Add Artemis stress test for detecting cache aliasing bugs. Include statistical hypothesis evaluations, fix validation data, and run reports for validation across architectures.

Signed-off-by: Robert Allan James <robert.allan.james@gmail.com>
This commit is contained in:
Robert Allan James
2026-08-02 14:49:52 -04:00
parent 148c4aa12c
commit 1cb68502fb
44 changed files with 5610173 additions and 2680 deletions
+29
View File
@@ -177,6 +177,35 @@ Acceptance criteria status, verified 2026-08-02:
Not done: everything under "FUTURE MATERIAL" below (zones, USB hot-plug, Not done: everything under "FUTURE MATERIAL" below (zones, USB hot-plug,
ACL records, PKI) — that line remains accurate and unchanged. ACL records, PKI) — that line remains accurate and unchanged.
### Surface Stress Test — permanent fixture, found a real bug (2026-08-02)
`ART-STRESS-TEST ( seed reps -- )` (`capsules/artemis/init.4th`, blocks 4160–4173) is a permanent, on-demand exercise of
the block-storage path across the whole data pool: `reps` distinct random blocks (coprime-stride walk, collision-free,
never touches anything already allocated by someone else), written, then verified after all writes complete so real
cache-eviction traffic gets forced through both caching layers. Every trial emits a `[ARTSTRESS]`-tagged CSV row to the
serial log (`HEADER`/`SUMMARY` markers make a truncated run unmistakable). Invoke as
`ART-STRESS` (fixed seed, 50 reps) or `<seed> <reps> ART-STRESS-TEST`. Not wired into the boot sequence — block 4170
carries the entry point commented out (`\ ART-STRESS`), same disable-by-default convention as
`ACL.4th`'s self-activation toggle, so ordinary boots pay nothing for it.
Its first completed run found a real, pre-existing data-corruption bug on amd64: 25 of 50 trials (50%) returned the
wrong byte pattern. Root cause, confirmed by source review and fixed in `src/word_source/block_words.c`
(not Artemis-specific — this is shared block-subsystem plumbing every VM using `BLOCK`/`UPDATE`/`FLUSH` goes through):
the VM-level "window" cache (`vm->blk_vm_cbuf[]`) stores a raw pointer directly into one of the block-subsystem's eight
`cache_slot_t` devblock-cache entries (`block_subsystem.c`). That cache's own FIFO eviction relocates entries via
struct-copy (`slot->cache[i] = slot->cache[i+1]`) once more than eight distinct devblocks are touched in one VM
session — the *addresses*
of `cache[0..7]` never move, but what they hold does, silently, aliasing any window pointer captured before the shift.
No prior test had ever touched enough distinct devblocks in one session to trigger it. Fixed by re-resolving the buffer
pointer by logical block number (`blk_get_buffer`)
immediately before every write-through-cache-pointer operation in
`block_word_update`, `blk_vm_evict`, and `blk_vm_flush_all`, instead of trusting a pointer captured earlier — confined
to the consumer side, no change to the block subsystem's cache architecture. Independent of, and predates, the
`BLK-CONFIRM-FORMAT` fix above; neither touches the other's code path. Re-verified 50/50 on amd64, aarch64, and riscv64
after the fix. Full write-up with hypothesis-tested analysis (a naive "consecutive writes fail together" reading did not
survive a formal runs test — see the report for what did and didn't hold up):
`experiments/artemis_stress/analysis/report/artemis_stress_report.pdf`.
--- ---
## What Artemis Owns ## What Artemis Owns
+221
View File
@@ -0,0 +1,221 @@
# CONSOLE.md — Console VM Architecture (DRAFT)
# StarshipOS / StarForth — Captain Bob (Robert Allan James)
# This document constrains Claude Code behavior once Console work begins. Read it
# completely before touching any Console code.
#
# STATUS: design-stage draft, 2026-08-02. Nothing described here is implemented yet.
# This is the output of a planning conversation, captured so it can keep being
# refined instead of living only in chat history. Do not treat anything below as
# "already built" — there is no Console code in the tree yet.
---
## The Pivot This Document Reflects
Console was first framed as "wrap the existing bitmap/VT100 framebuffer driver in a 4th Tripod VM." That framing is
superseded. The framebuffer is now treated as a **drawing fabric**: full-color, Cartesian-addressed, rasterized-vector —
not a character-cell bitmap terminal. Text is one thing drawn on the fabric, not a special case. Rationale (Captain Bob,
verbatim intent): everything in the eventual GUI is going to be drawn anyway, so the console should be drawn the same
way from the start rather than built on a bitmap terminal that gets thrown away later.
The existing `src/starkernel/hal/framebuffer.c` (raw UEFI GOP pixel plotting) is still the right low-level substrate —
that part is reused. `font_8x16.c` and
`vt100.c` (the bitmap-cell VT100 emulator built on top of it) are what's being superseded; see Open Decisions below for
their fate.
---
## Language Constraint — Non-Negotiable
All Console implementation is in StarForth dialect ONLY, with the same narrow exception every other Tripod VM has: raw
hardware access (the actual pixel write into GOP framebuffer memory) is C, registered directly as Forth primitives —
exactly how `BLOCK`/`UPDATE`/`FLUSH`/`RANDOM`/`SEED` already work. Everything above that raw pixel-write boundary —
glyph composition, cursor logic, dirty-cell tracking, VT100 semantics, fonts — is StarForth.
C99 beyond that narrow hardware boundary is forbidden unless explicitly blocked in StarForth AND explicit written
permission is given by Captain Bob for that specific construct. "I could not figure out how to do this in StarForth" is
not permission. Ask first. Wait for the answer.
---
## When Stuck — Non-Negotiable
If you are stuck, STOP.
Do not confabulate. Do not invent dictionary words that do not exist. Do not fabricate primitives. Do not guess at
behavior. Do not work around the problem silently.
STOP. Report exactly what is blocking you. Wait for instruction.
---
## What Console Is
Console is the fleet's fourth leg — Quadrupod, not Tripod. A real StarForth VM, spawned by Hera, participating in fleet
K≡1.0 like Hermes and Artemis. Its contract:
Console owns the drawing fabric (the framebuffer) and, in a later phase, keyboard input. It renders. It does not route
messages (Hermes), does not own persistent storage (Artemis), and does not make VM lifecycle decisions (Hera).
Mythological name: **not yet chosen.** Candidates raised (Argus, Iris, Hestia) were not settled on. Do not pick one
unilaterally when implementation starts — ask Captain Bob. This document uses "Console" as a working label until named.
---
## Coordinate System
True Cartesian: **origin bottom-left, Y increases upward.** This is a firm decision, not a placeholder. The
framebuffer's native memory layout is top-left origin, Y-down (standard raster convention) — every drawing primitive
that touches the framebuffer must apply the Y-flip transform. Get this at the lowest primitive layer so nothing above it
ever has to think about it again.
---
## Primitive Layer — C, registered directly as Forth words
Raw, hardware-touching, absolute-coordinate. Not turtle/relative — see Font & Pen Layer below for why the split falls
here.
- `PLOT ( x y color -- )` — set one pixel (post Y-flip transform)
- `LINE ( x1 y1 x2 y2 color -- )` — Bresenham line between two absolute points
Absolute coordinates were chosen deliberately: the long-term direction (wireframes — triangles/quadrangles, eventually
textured/shaded) is fed by projected vertex data, which is naturally absolute per-vertex, not relative turtle deltas.
Building the hardware boundary around absolute coordinates means the eventual triangle-fill / texture-sample primitives
extend this layer instead of fighting it.
**Performance principle — non-negotiable:** the actual per-pixel work (Bresenham walk, eventually triangle
rasterization, texture sampling) must live in this C layer, compiled. Forth orchestrates — decides what to draw and
when — but never walks pixels one at a time through the interpreter. This is the same C-does- hardware /
Forth-does-policy split every other Tripod VM already uses; it is not a new rule invented for Console, just applied
here.
Known unknown: actual achievable frame rate. Feasible in principle (this is how every pre-GPU software-rasterized engine
worked), but unmeasured, and QEMU TCG (the only acceptance environment per `.claude/CLAUDE.md`) will be meaningfully
slower than real hardware for CPU-emulated per-pixel work. Do not assume a frame-rate target without measuring on the
actual target environment first.
---
## Font & Pen Layer — StarForth, composable
Fonts are not a C data table. Each glyph is a StarForth word, e.g.
`GLYPH-A ( x y -- )`, expressed as a short sequence of pen strokes relative to the glyph's origin — a small pen DSL
(`STROKE dx dy`, pen up/down, within a unit em-box) that resolves down to calls into the absolute `LINE` primitive. This
is deliberately hackable: a font is just another `.4th` capsule, content-addressed and hash-verified exactly like every
other capsule in this system. Swap the capsule, get a different typeface. No new loading mechanism — the existing
capsule system already does this.
This does not fight the performance principle above: glyph composition is a handful of word dispatches (5–15 strokes per
character), not per-pixel work. The expensive part — actually writing pixels — stays in the C `LINE` primitive
regardless of how the glyph was authored. Combined with heat-driven redraw (below), most glyphs aren't recomposed most
ticks anyway.
A thin turtle/pen convenience layer (position + heading state, `FORWARD`/`RIGHT`/
`PENUP`/`PENDOWN`) may be built on top of the same absolute `LINE` primitive for generative drawing use cases. It is
StarForth-only, costs nothing at the C layer, and does not change the hardware-boundary decision to stay absolute.
Phase 1 requires the **full printable-ASCII stroke-glyph table**, not a subset — a VT100-equivalent terminal needs to
render arbitrary text from day one. A narrower subset only made sense as a pipeline smoke test in isolation, which is no
longer the goal.
---
## Heat-Driven Redraw Model
Screen redraw is dirty-cell tracking, expressed in the same compudynamic vocabulary as everything else in this system —
not decorative, it's the actual mechanism.
- Each screen cell carries a heat value, structurally the same idea as `BLK-HEAT`
for Artemis's data blocks.
- A cell goes hot (`Q.1`) the moment its content changes.
- Console's own tick (driven off Hera's fleet heartbeat, same pattern as
`ART-TICK`) scans for hot cells only.
- For each hot cell, Console redraws it via the glyph/`LINE` primitives — and that redraw **is** the reap event. Heat
goes to zero because the work got done, not because of a time-based decay curve like Artemis blocks or Hermes
messages.
- Cells at K=0 are skipped entirely this tick — untouched pixels, no wasted work.
This cell-heat contributes to Console's share of fleet K, same as every other compudynamic object in the system. Reap
(redraw-to-zero) must rebalance K correctly, same invariant every other VM already honors.
---
## Immediate Goal
Scope for the first real implementation pass. Nothing past this section is authorized to build yet.
**The target:** a VT100-equivalent interactive terminal, rendered entirely through the vector fabric described above
(not the old bitmap driver), with:
1. Console spawns as the fleet's 4th leg under Hera's birth protocol, joins fleet K≡1.0 like Hermes and Artemis.
2. Full printable-ASCII stroke font, composable/StarForth-authored, loaded as a capsule.
3. Cursor positioning, basic scrolling, and enough VT100 semantics to be a usable terminal — the same functional bar the
current bitmap `vt100.c` already clears, just achieved through the new fabric instead.
4. Heat-driven redraw — only changed cells get rasterized per tick.
5. Text arrives at Console the same way everything else moves between Tripod VMs:
via Hermes. Console does not invent a second message-passing mechanism.
**Explicitly out of scope for this phase:** keyboard input (see Future Material — there is no keyboard driver anywhere
in this codebase yet, this is a separate, substantial piece of work), double buffering (open decision, see below), the
wireframe/textured/shaded GUI vision (that's the reason the fabric is built this way, but building it is a different,
later phase).
---
## Open Decisions — must be settled before or during implementation
1. **Mythological name.** Not chosen. Ask Captain Bob before implementation starts.
2. **Fate of `font_8x16.c` / `vt100.c`.** Kept only for pre-Console boot diagnostics (before Console VM is alive to take
over), or retired outright once the fabric is proven? Not decided.
3. **Double buffering.** Direct framebuffer writes will tear once Console is doing partial redraws every tick instead of
static boot text. Cheaper to decide now than to retrofit through every draw call later. Not decided.
4. **Actual frame-rate/performance numbers.** Unmeasured. Do not commit to a real-time target without benchmarking on
real hardware and under QEMU TCG separately — they will differ meaningfully.
---
## FUTURE MATERIAL — Not In Scope
Everything below is real direction, preserved intentionally, and is **not** part of the Immediate Goal above. Do not
build any of this without Captain Bob explicitly reopening it — same rule ARTEMIS.md and TRIPOD.md already hold their
own future chapters to.
### Full wireframe GUI
Triangles/quadrangles from projected vertex data, applied textures and shading, anti-aliased/high-resolution appearance
("not blocky") even at small polygon sizes. Visually in the spirit of vector-display arcade games (Asteroids),
rasterized in software onto the linear framebuffer — no GPU, no hardware acceleration available on this platform. The
absolute-coordinate `PLOT`/`LINE` primitive layer is designed so triangle-fill/texture-sample primitives extend it later
rather than replacing it.
### Keyboard input
No driver exists anywhere in this codebase today. PS/2 8042 controller is the pragmatic first target — QEMU supports it
cleanly. USB HID through the existing virtio/PCI stack is a materially bigger lift and should come later, if at all.
`M8: REPL keyboard input` in the top-level roadmap has been "planned, not started"
for the life of this project; this is the same gap.
### Interactive REPL on the framebuffer
Once keyboard input exists, point a real interactive REPL loop at Console instead of (or alongside) the serial `ok>`.
This is the piece that actually starts to make the system feel like an operating system rather than a kernel with a
serial console. Depends entirely on keyboard input landing first.
### Multi-window / multi-session
Not discussed in depth. Flagged here only so it isn't forgotten — if the fabric approach pans out, multiple independent
drawing regions/sessions is a natural next question once single-console VT100-equivalent behavior is solid.
---
*This document is a living draft, not yet authoritative in the sense TRIPOD.md/ ARTEMIS.md/HERMES.md are — those
describe implemented, verified systems. This one describes a plan. Update it as decisions get made; do not let it drift
out of sync with what actually gets built, the way the other three documents do not drift.*
+216 -201
View File
File diff suppressed because it is too large Load Diff
+153
View File
@@ -249,6 +249,159 @@ Block 4133
ART-INIT ART-INIT
ART-BOOT-ENTRY ART-BOOT-ENTRY
WELCOME WELCOME
Block 4160
( Surface stress test storage. Manual-invoke only -- )
( never called from the entry sequence above. )
2000 CONSTANT ART-STRESS-CAP
CREATE ART-STRESS-LBN ART-STRESS-CAP CELLS ALLOT
CREATE ART-STRESS-PAT ART-STRESS-CAP CELLS ALLOT
VARIABLE ART-STRESS-N
VARIABLE ART-STRESS-PASS
VARIABLE ART-STRESS-FAIL
VARIABLE S-I
VARIABLE S-IDX
VARIABLE S-STRIDE
VARIABLE S-CUR
VARIABLE S-ACTUAL
Block 4161
( Trial record accessors )
: SLBN@ ( i -- lbn ) CELLS ART-STRESS-LBN + @ ;
: SLBN! ( lbn i -- ) CELLS ART-STRESS-LBN + ! ;
: SPAT@ ( i -- pat ) CELLS ART-STRESS-PAT + @ ;
: SPAT! ( pat i -- ) CELLS ART-STRESS-PAT + ! ;
Block 4172
( CSV emission part 1 -- one row per trial, tagged )
( so it can be grepped like the DoE CSVs. Every )
( trial emits a row, pass or fail -- no silent skips. )
: SCSV-HEADER ( n -- )
." [ARTSTRESS] HEADER,n=" DUP . CR
." [ARTSTRESS] trial,lbn,pat,result" CR ;
: SCSV-ROW ( -- )
." [ARTSTRESS] "
S-I @ . ." ,"
S-I @ SLBN@ . ." ,"
S-I @ SPAT@ . ." ,"
S-ACTUAL @ S-I @ SPAT@ = IF 1 ELSE 0 THEN . CR ;
Block 4173
( CSV emission part 2 -- SUMMARY marker. A run that )
( got killed/timed out mid-way never prints this -- )
( its absence means the data is incomplete, full )
( stop, not "probably fine." )
: SCSV-SUMMARY ( -- )
." [ARTSTRESS] SUMMARY,"
ART-STRESS-N @ . ." ,"
ART-STRESS-PASS @ . ." ,"
ART-STRESS-FAIL @ . CR ;
Block 4162
( Coprime-stride walk over the data index space -- )
( visits every index at most once before repeating, )
( so no O(n) scan against past picks is needed. )
( ART-DATA-BLKS = 22998 = 2 * 3 * 3833. )
: S-COPRIME? ( n -- flag )
DUP 3 MOD 0= IF DROP 0 EXIT THEN
3833 MOD 0<> ;
: S-INIT-WALK ( -- )
0 ART-DATA-BLKS 1 - RANDOM S-CUR !
BEGIN
1 ART-DATA-BLKS 1 - RANDOM 1 OR
DUP S-COPRIME?
UNTIL
S-STRIDE ! ;
Block 4171
( Step the walk; skip anything already allocated by )
( someone else -- never clobbers pre-existing data. )
: SPICK ( -- idx )
BEGIN
S-CUR @ S-STRIDE @ + ART-DATA-BLKS MOD S-CUR !
S-CUR @ FM-TEST 0=
UNTIL
S-CUR @ ;
Block 4163
( Write one trial: alloc a fresh random free block, )
( tag it with a trial-derived pattern, and persist )
( it through the real BLK-ALLOC-style path. )
: SWRITE1 ( -- )
SPICK S-IDX !
S-IDX @ FM-SET
Q.1 S-IDX @ BLK-HEAT!
S-IDX @ ART-DATA-LBN + S-I @ SLBN!
S-I @ 255 AND S-I @ SPAT!
S-I @ SLBN@ BLK-FETCH
1024 S-I @ SPAT@ FILL
S-I @ SLBN@ BLK-PERSIST ;
Block 4164
( Verify one trial: by now the 8-slot devblock cache )
( has likely evicted and reloaded this block many )
( times over -- a real round trip, not a cache hit. )
: SVERIFY1 ( -- )
S-I @ SLBN@ BLK-FETCH C@ S-ACTUAL !
S-ACTUAL @ S-I @ SPAT@ = IF
1 ART-STRESS-PASS +!
ELSE
1 ART-STRESS-FAIL +!
." STRESS MISMATCH trial=" S-I @ .
." lbn=" S-I @ SLBN@ . CR
THEN
SCSV-ROW ;
Block 4166
( Clamp reps into range, kept in its own block so )
( SVERIFY1 above has room for CSV emission. )
: ART-STRESS-CLAMP ( reps -- reps' )
DUP ART-STRESS-CAP > IF DROP ART-STRESS-CAP THEN
DUP 1 < IF DROP 1 THEN ;
Block 4165
( Driver part 1: banner, then the write+verify loops. )
( Split across several blocks -- a colon definition )
( must fit in a single 1024-byte/16-line block here. )
( This is a data-collection fixture, not a fast/live )
( check -- same posture as the DoE campaign. K is not )
( gated on live, it's recorded for offline analysis. )
: ART-STRESS-BANNER ( n -- )
DUP SCSV-HEADER
." Artemis: surface stress test, " . ." trials" CR
ART-K-TOTAL Q.PRINT ." K before" CR ;
: ART-STRESS-RUN ( -- )
ART-STRESS-N @ 0 DO I S-I ! SWRITE1 LOOP
ART-STRESS-N @ 0 DO I S-I ! SVERIFY1 LOOP ;
Block 4167
( Driver part 2: pass/fail summary line )
: ART-STRESS-REPORT ( -- )
." Artemis stress: "
ART-STRESS-PASS @ . ." passed, "
ART-STRESS-FAIL @ . ." failed" CR ;
Block 4168
( Driver part 3: free everything this run touched, )
( print K after for the record. Only write/verify )
( mismatches gate PASS/FAIL -- a background heartbeat )
( decay tick during a long run is expected and is not )
( itself a failure. )
: ART-STRESS-CLEANUP ( -- )
ART-STRESS-N @ 0 DO
I S-I ! S-I @ SLBN@ BLK-FREE
LOOP
ART-K-TOTAL Q.PRINT ." K after (should match)" CR
SCSV-SUMMARY
ART-STRESS-FAIL @ 0 = IF
." PASS: surface stress test" CR
ELSE
." FAIL: surface stress test" CR
THEN ;
Block 4169
( Top-level entry points )
: ART-STRESS-TEST ( seed reps -- )
ART-STRESS-CLAMP ART-STRESS-N ! SEED
0 ART-STRESS-PASS ! 0 ART-STRESS-FAIL !
S-INIT-WALK
ART-STRESS-N @ ART-STRESS-BANNER
ART-STRESS-RUN ART-STRESS-REPORT
ART-STRESS-CLEANUP ;
: ART-STRESS ( -- ) 424242 50 ART-STRESS-TEST ;
Block 4170
( Disabled by default -- uncomment to run the surface )
( stress test automatically at every boot. Normally )
( invoked manually instead: ART-STRESS or )
( <seed> <reps> ART-STRESS-TEST from the console. )
\ ART-STRESS
Block 4851 Block 4851
( Cheap touch target for Phase 3 fleet DoE workload -- O(1), ) ( Cheap touch target for Phase 3 fleet DoE workload -- O(1), )
( unlike ART-TICK/ART-STATUS which scan all ART-DATA-BLKS. ) ( unlike ART-TICK/ART-STATUS which scan all ART-DATA-BLKS. )
BIN
View File
Binary file not shown.
@@ -0,0 +1,269 @@
#!/usr/bin/env Rscript
# analyse_artemis_stress.R
#
# Analysis for the Artemis surface stress test bug hunt (2026-08-02):
# a stale-pointer aliasing bug between the block-subsystem's devblock
# cache and the VM-level window cache in src/word_source/block_words.c,
# found by the first completed run of ART-STRESS-TEST, fixed, and
# re-verified on all three architectures.
#
# Reads experiments/artemis_stress/runs/*.csv, writes SVG charts to
# analysis/charts/ and PNG copies to analysis/img/.
suppressMessages({
library(ggplot2)
library(dplyr)
library(tidyr)
library(scales)
library(svglite)
})
run_dir <- file.path("experiments", "artemis_stress", "runs")
chart_dir <- file.path("experiments", "artemis_stress", "analysis", "charts")
img_dir <- file.path("experiments", "artemis_stress", "analysis", "img")
dir.create(chart_dir, recursive = TRUE, showWarnings = FALSE)
dir.create(img_dir, recursive = TRUE, showWarnings = FALSE)
## ---- palette (matches experiments/bare_metal house style) ----
sfblue <- "#1F77B4"
sforange <- "#FF7F0E"
sfgreen <- "#2CA02C"
sfred <- "#D62728"
sfgray <- "#505050"
theme_sf <- function() {
theme_minimal(base_size = 12) +
theme(
panel.grid.minor = element_blank(),
plot.title = element_text(face = "bold"),
plot.subtitle = element_text(color = sfgray),
legend.position = "bottom"
)
}
save_chart <- function(plot, name, width = 7, height = 4.2) {
svg_path <- file.path(chart_dir, paste0(name, ".svg"))
png_path <- file.path(img_dir, paste0(name, ".png"))
ggsave(svg_path, plot, width = width, height = height, device = svglite)
ggsave(png_path, plot, width = width, height = height, dpi = 200)
cat(" wrote", svg_path, "and", png_path, "\n")
}
## ---- load runs ----
runs <- list(
amd64_buggy = "amd64 (pre-fix)",
amd64_fixed = "amd64 (post-fix)",
aarch64_fixed = "aarch64 (post-fix)",
riscv64_fixed = "riscv64 (post-fix)"
)
load_run <- function(key, label) {
df <- read.csv(file.path(run_dir, paste0(key, ".csv")))
df$run <- key
df$label <- label
df$status <- factor(ifelse(df$result == 1, "PASS", "FAIL"), levels = c("PASS", "FAIL"))
df
}
all_runs <- bind_rows(Map(load_run, names(runs), runs))
## ---- Chart 1: pass rate before/after, all runs ----
summary_tbl <- all_runs %>%
group_by(run, label) %>%
summarise(n = n(), passed = sum(result), .groups = "drop") %>%
mutate(pass_rate = passed / n,
phase = ifelse(grepl("pre-fix", label), "pre-fix", "post-fix"))
p1 <- ggplot(summary_tbl, aes(x = reorder(label, pass_rate), y = pass_rate, fill = phase)) +
geom_col(width = 0.6) +
geom_text(aes(label = sprintf("%d/%d", passed, n)), vjust = -0.5, size = 4) +
scale_y_continuous(labels = percent_format(), limits = c(0, 1.08)) +
scale_fill_manual(values = c("pre-fix" = sfred, "post-fix" = sfgreen)) +
coord_flip() +
labs(
title = "Artemis surface stress test: pass rate before/after fix",
subtitle = "50 trials per run, unique random blocks across the 22998-block data pool",
x = NULL, y = "Pass rate", fill = NULL
) +
theme_sf()
save_chart(p1, "pass_rate_comparison", width = 8)
## ---- Chart 2: pass/fail sequence in write/verify order, buggy run ----
## A visual scan of this sequence first suggested serial clustering; the
## Wald-Wolfowitz runs test (H1, below) does not support that reading --
## observed runs (26) exactly match the count expected under a random
## ordering. The chart is kept because it is still the raw sequence the
## tests are run against, not because it shows a confirmed pattern.
buggy <- all_runs %>% filter(run == "amd64_buggy") %>% arrange(trial)
p2 <- ggplot(buggy, aes(x = trial, y = 1, fill = status)) +
geom_tile(color = "white", linewidth = 0.6, height = 1) +
scale_fill_manual(values = c(PASS = sfgreen, FAIL = sfred)) +
scale_x_continuous(breaks = seq(0, 49, 5)) +
labs(
title = "Pass/fail sequence in write order, amd64 pre-fix run",
subtitle = "25 of 50 trials failed; a formal runs test (H1) finds no\nsignificant serial clustering in this sequence -- see Section 4",
x = "Trial index (= write/verify order)", y = NULL, fill = NULL
) +
theme_sf() +
theme(axis.text.y = element_blank(), panel.grid = element_blank())
save_chart(p2, "failure_clustering", height = 2.6)
## ---- Chart 3: surface coverage map, buggy vs one fixed run ----
cov <- all_runs %>% filter(run %in% c("amd64_buggy", "amd64_fixed"))
p3 <- ggplot(cov, aes(x = lbn, y = trial, color = status)) +
geom_point(size = 2.2, alpha = 0.85) +
scale_color_manual(values = c(PASS = sfgreen, FAIL = sfred)) +
facet_wrap(~label, ncol = 1) +
labs(
title = "Surface coverage: scattered across the full data pool",
subtitle = "Each point is one trial's block. A Wilcoxon test (H2) finds LBN and\noutcome are not independent, but no sub-region is visually implicated",
x = "Logical block number (data pool: 3076-26073)", y = "Trial index", color = NULL
) +
theme_sf()
save_chart(p3, "surface_coverage_map", height = 5.5)
## ---- summary table for the report ----
write.csv(summary_tbl, file.path("experiments", "artemis_stress", "analysis", "summary.csv"),
row.names = FALSE)
cat("\nDone. Summary:\n")
print(as.data.frame(summary_tbl))
## ═══════════════════════════════════════════════════════════════════════
## Hypothesis tests
##
## Three questions, each with an explicit null hypothesis, tested against
## the actual recorded trial data rather than asserted from eyeballing
## charts. Results are written to tables/ for the LaTeX report to quote
## verbatim -- no numbers in the report are hand-typed.
## ═══════════════════════════════════════════════════════════════════════
table_dir <- file.path("experiments", "artemis_stress", "analysis", "tables")
dir.create(table_dir, recursive = TRUE, showWarnings = FALSE)
sink_lines <- c()
say <- function(...) {
line <- paste0(...)
sink_lines <<- c(sink_lines, line)
cat(line, "\n")
}
say("Artemis surface stress test -- hypothesis tests")
say("=================================================")
say("")
## ---- H1: are pre-fix failures independently distributed across trials, ----
## ---- or do they show serial clustering (a runs test)? ----
## H0: the pass/fail sequence is a random ordering (no serial correlation).
## Manual Wald-Wolfowitz runs test (base R only, no extra package needed).
runs_test <- function(seq01) {
n1 <- sum(seq01 == 1); n2 <- sum(seq01 == 0); n <- n1 + n2
r <- 1 + sum(diff(seq01) != 0)
mu <- 1 + 2 * n1 * n2 / n
var_r <- (2 * n1 * n2 * (2 * n1 * n2 - n)) / (n^2 * (n - 1))
z <- (r - mu) / sqrt(var_r)
p <- 2 * pnorm(-abs(z))
list(n1 = n1, n2 = n2, runs = r, expected = mu, z = z, p = p)
}
buggy_seq <- buggy$result[order(buggy$trial)]
rt <- runs_test(buggy_seq)
say("H1 -- pre-fix failure clustering (Wald-Wolfowitz runs test)")
say(" H0: pass/fail outcomes are independently ordered (no clustering).")
say(sprintf(" Observed runs = %d (expected under H0 = %.2f, sd = %.2f)",
rt$runs, rt$expected, sqrt((2*rt$n1*rt$n2*(2*rt$n1*rt$n2-(rt$n1+rt$n2))) /
((rt$n1+rt$n2)^2*(rt$n1+rt$n2-1)))))
say(sprintf(" z = %.3f, p = %.2e", rt$z, rt$p))
say(sprintf(" -> %s H0 at alpha=0.05: failures %s serially clustered.",
ifelse(rt$p < 0.05, "REJECT", "fail to reject"),
ifelse(rt$p < 0.05, "ARE", "are not")))
say("")
## ---- H1b: is outcome associated with trial/write order directly ----
## ---- (not via the runs test, which only detects consecutive-value ----
## ---- clustering and can miss a more general order-dependent trend)? ----
## H0: trial index is drawn from the same distribution for PASS and FAIL.
wt_order <- wilcox.test(trial ~ status, data = buggy)
say("H1b -- pre-fix outcome vs. trial/write order directly (Wilcoxon)")
say(" H0: trial index is drawn from the same distribution for PASS/FAIL")
say(" (complements H1: the runs test only catches consecutive-value")
say(" clustering, not a general rank-order trend).")
say(sprintf(" W = %.1f, p = %.3f", wt_order$statistic, wt_order$p.value))
say(sprintf(" -> %s H0 at alpha=0.05: outcome %s associated with write order.",
ifelse(wt_order$p.value < 0.05, "REJECT", "fail to reject"),
ifelse(wt_order$p.value < 0.05, "IS", "is not")))
say("")
## ---- H2: is failure status independent of disk location (logical ----
## ---- block number)? ----
## H0: LBN is drawn from the same distribution for PASS and FAIL trials.
## Wilcoxon rank-sum test (non-parametric, no normality assumption on LBN).
wt <- wilcox.test(lbn ~ status, data = buggy)
say("H2 -- pre-fix failure vs. disk location (Wilcoxon rank-sum test)")
say(" H0: logical block number is drawn from the same distribution")
say(" for PASS and FAIL trials (failure does not depend on location).")
say(sprintf(" W = %.1f, p = %.3f", wt$statistic, wt$p.value))
say(sprintf(" -> %s H0 at alpha=0.05: failure %s associated with LBN.",
ifelse(wt$p.value < 0.05, "REJECT", "fail to reject"),
ifelse(wt$p.value < 0.05, "IS", "is not")))
say("")
## Secondary check: sibling position within devblock (0/1/2).
buggy$sib_pos <- (buggy$lbn - 3076) %% 3
pos_tbl <- table(buggy$sib_pos, buggy$status)
ct <- suppressWarnings(chisq.test(pos_tbl))
say(" Secondary check -- sibling position (0/1/2) vs. outcome (chi-squared)")
say(sprintf(" chi-sq = %.3f, df = %d, p = %.3f",
ct$statistic, ct$parameter, ct$p.value))
say(sprintf(" -> %s H0: outcome %s associated with sibling position.",
ifelse(ct$p.value < 0.05, "REJECT", "fail to reject"),
ifelse(ct$p.value < 0.05, "IS", "is not")))
say("")
## ---- H3: did the fix change the pass rate, or is 25/50 vs 150/150 ----
## ---- explainable by chance? ----
## H0: pre-fix and post-fix runs share the same true pass probability.
post_passed <- sum(all_runs$run != "amd64_buggy" & all_runs$result == 1)
post_n <- sum(all_runs$run != "amd64_buggy")
pre_passed <- sum(all_runs$run == "amd64_buggy" & all_runs$result == 1)
pre_n <- sum(all_runs$run == "amd64_buggy")
pt <- prop.test(c(pre_passed, post_passed), c(pre_n, post_n), correct = TRUE)
say("H3 -- did the fix change the pass rate (two-proportion test)")
say(" H0: pre-fix and post-fix trials share the same true pass rate.")
say(sprintf(" Pre-fix: %d/%d = %.1f%% Post-fix (pooled, 3 arches): %d/%d = %.1f%%",
pre_passed, pre_n, 100*pre_passed/pre_n,
post_passed, post_n, 100*post_passed/post_n))
say(sprintf(" chi-sq = %.2f, df = %d, p = %.2e", pt$statistic, 1, pt$p.value))
say(sprintf(" 95%% CI on the difference in pass rate: [%.3f, %.3f]",
pt$conf.int[1], pt$conf.int[2]))
say(sprintf(" -> %s H0 at alpha=0.05: the fix %s the pass rate.",
ifelse(pt$p.value < 0.05, "REJECT", "fail to reject"),
ifelse(pt$p.value < 0.05, "DID CHANGE", "did not measurably change")))
writeLines(sink_lines, file.path(table_dir, "hypothesis_tests.txt"))
## Machine-readable version for \newcommand substitution in the report.
hyp_df <- data.frame(
key = c("h1_runs_obs","h1_runs_exp","h1_z","h1_p",
"h1b_W","h1b_p",
"h2_W","h2_p","h2b_chisq","h2b_df","h2b_p",
"h3_pre_n","h3_pre_pass","h3_post_n","h3_post_pass",
"h3_chisq","h3_p","h3_ci_lo","h3_ci_hi"),
value = c(rt$runs, rt$expected, rt$z, rt$p,
unname(wt_order$statistic), wt_order$p.value,
unname(wt$statistic), wt$p.value, unname(ct$statistic), unname(ct$parameter), ct$p.value,
pre_n, pre_passed, post_n, post_passed,
unname(pt$statistic), pt$p.value, pt$conf.int[1], pt$conf.int[2])
)
write.csv(hyp_df, file.path(table_dir, "hypothesis_tests.csv"), row.names = FALSE)
cat("\nWrote", file.path(table_dir, "hypothesis_tests.txt"), "and .csv\n")
@@ -0,0 +1,208 @@
<?xml version='1.0' encoding='UTF-8' ?>
<svg xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink' width='504.00pt' height='187.20pt'
viewBox='0 0 504.00 187.20'>
<g class='svglite'>
<defs>
<style type='text/css'><![CDATA[
.svglite line, .svglite polyline, .svglite polygon, .svglite path, .svglite rect, .svglite circle {
fill: none;
stroke: #000000;
stroke-linecap: round;
stroke-linejoin: round;
stroke-miterlimit: 10.00;
}
.svglite text {
white-space: pre;
}
.svglite g.glyphgroup path {
fill: inherit;
stroke: none;
}
]]></style>
</defs>
<rect width='100%' height='100%' style='stroke: none; fill: #FFFFFF;'/>
<defs>
<clipPath id='cpMC4wMHw1MDQuMDB8MC4wMHwxODcuMjA='>
<rect x='0.00' y='0.00' width='504.00' height='187.20'/>
</clipPath>
</defs>
<g clip-path='url(#cpMC4wMHw1MDQuMDB8MC4wMHwxODcuMjA=)'>
<rect x='0.00' y='0.00' width='504.00' height='187.20'
style='stroke-width: 1.16; stroke: none; fill: #FFFFFF;'/>
</g>
<defs>
<clipPath id='cpNS45OHw0OTguMDJ8NTQuNTN8MTEyLjMz'>
<rect x='5.98' y='54.53' width='492.04' height='57.80'/>
</clipPath>
</defs>
<g clip-path='url(#cpNS45OHw0OTguMDJ8NTQuNTN8MTEyLjMz)'>
<rect x='28.34' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='37.29' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='46.24' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='55.18' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='64.13' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='73.07' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='82.02' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='90.97' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='99.91' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='108.86' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='117.81' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='126.75' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='135.70' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='144.64' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='153.59' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='162.54' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='171.48' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='180.43' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='189.38' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='198.32' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='207.27' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='216.21' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='225.16' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='234.11' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='243.05' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='252.00' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='260.95' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='269.89' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='278.84' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='287.79' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='296.73' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='305.68' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='314.62' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='323.57' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='332.52' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='341.46' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='350.41' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='359.36' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='368.30' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='377.25' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='386.19' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='395.14' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='404.09' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='413.03' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='421.98' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='430.93' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='439.87' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='448.82' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='457.76' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='466.71' y='57.16' width='8.95' height='52.55'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
</g>
<g clip-path='url(#cpMC4wMHw1MDQuMDB8MC4wMHwxODcuMjA=)'>
<text x='32.82' y='124.31' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='5.33px'
lengthAdjust='spacingAndGlyphs'>0
</text>
<text x='77.55' y='124.31' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='5.33px'
lengthAdjust='spacingAndGlyphs'>5
</text>
<text x='122.28' y='124.31' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='10.66px'
lengthAdjust='spacingAndGlyphs'>10
</text>
<text x='167.01' y='124.31' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='10.66px'
lengthAdjust='spacingAndGlyphs'>15
</text>
<text x='211.74' y='124.31' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='10.66px'
lengthAdjust='spacingAndGlyphs'>20
</text>
<text x='256.47' y='124.31' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='10.66px'
lengthAdjust='spacingAndGlyphs'>25
</text>
<text x='301.20' y='124.31' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='10.66px'
lengthAdjust='spacingAndGlyphs'>30
</text>
<text x='345.94' y='124.31' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='10.66px'
lengthAdjust='spacingAndGlyphs'>35
</text>
<text x='390.67' y='124.31' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='10.66px'
lengthAdjust='spacingAndGlyphs'>40
</text>
<text x='435.40' y='124.31' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='10.66px'
lengthAdjust='spacingAndGlyphs'>45
</text>
<text x='252.00' y='137.54' text-anchor='middle' style='font-size: 12.00px; font-family: "Liberation Sans";'
textLength='165.28px' lengthAdjust='spacingAndGlyphs'>Trial index (= write/verify order)
</text>
<rect x='204.28' y='158.82' width='15.58' height='15.58'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='258.43' y='158.82' width='15.58' height='15.58'
style='stroke-width: 1.28; stroke: #FFFFFF; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<text x='226.69' y='169.90' style='font-size: 9.60px; font-family: "Liberation Sans";' textLength='24.91px'
lengthAdjust='spacingAndGlyphs'>PASS
</text>
<text x='280.83' y='169.90' style='font-size: 9.60px; font-family: "Liberation Sans";' textLength='19.73px'
lengthAdjust='spacingAndGlyphs'>FAIL
</text>
<text x='5.98' y='33.10' style='font-size: 12.00px;fill: #505050; font-family: "Liberation Sans";'
textLength='275.38px' lengthAdjust='spacingAndGlyphs'>25 of 50 trials failed; a formal runs test (H1)
finds no
</text>
<text x='5.98' y='46.06' style='font-size: 12.00px;fill: #505050; font-family: "Liberation Sans";'
textLength='317.44px' lengthAdjust='spacingAndGlyphs'>significant serial clustering in this sequence
-- see Section 4
</text>
<text x='5.98' y='15.88' style='font-size: 14.40px; font-weight: bold; font-family: "Liberation Sans";'
textLength='351.91px' lengthAdjust='spacingAndGlyphs'>Pass/fail sequence in write order, amd64 pre-fix
run
</text>
</g>
</g>
</svg>

After

Width:  |  Height:  |  Size: 15 KiB

@@ -0,0 +1,132 @@
<?xml version='1.0' encoding='UTF-8' ?>
<svg xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink' width='576.00pt' height='302.40pt'
viewBox='0 0 576.00 302.40'>
<g class='svglite'>
<defs>
<style type='text/css'><![CDATA[
.svglite line, .svglite polyline, .svglite polygon, .svglite path, .svglite rect, .svglite circle {
fill: none;
stroke: #000000;
stroke-linecap: round;
stroke-linejoin: round;
stroke-miterlimit: 10.00;
}
.svglite text {
white-space: pre;
}
.svglite g.glyphgroup path {
fill: inherit;
stroke: none;
}
]]></style>
</defs>
<rect width='100%' height='100%' style='stroke: none; fill: #FFFFFF;'/>
<defs>
<clipPath id='cpMC4wMHw1NzYuMDB8MC4wMHwzMDIuNDA='>
<rect x='0.00' y='0.00' width='576.00' height='302.40'/>
</clipPath>
</defs>
<g clip-path='url(#cpMC4wMHw1NzYuMDB8MC4wMHwzMDIuNDA=)'>
<rect x='0.00' y='0.000000000000057' width='576.00' height='302.40'
style='stroke-width: 1.16; stroke: none; fill: #FFFFFF;'/>
</g>
<defs>
<clipPath id='cpODUuOTR8NTcwLjAyfDQxLjU3fDIyNy41Mw=='>
<rect x='85.94' y='41.57' width='484.09' height='185.96'/>
</clipPath>
</defs>
<g clip-path='url(#cpODUuOTR8NTcwLjAyfDQxLjU3fDIyNy41Mw==)'>
<polyline points='85.94,200.97 570.02,200.97 '
style='stroke-width: 1.16; stroke: #EBEBEB; stroke-linecap: butt;'/>
<polyline points='85.94,156.69 570.02,156.69 '
style='stroke-width: 1.16; stroke: #EBEBEB; stroke-linecap: butt;'/>
<polyline points='85.94,112.41 570.02,112.41 '
style='stroke-width: 1.16; stroke: #EBEBEB; stroke-linecap: butt;'/>
<polyline points='85.94,68.14 570.02,68.14 '
style='stroke-width: 1.16; stroke: #EBEBEB; stroke-linecap: butt;'/>
<polyline points='107.94,227.53 107.94,41.57 '
style='stroke-width: 1.16; stroke: #EBEBEB; stroke-linecap: butt;'/>
<polyline points='230.18,227.53 230.18,41.57 '
style='stroke-width: 1.16; stroke: #EBEBEB; stroke-linecap: butt;'/>
<polyline points='352.43,227.53 352.43,41.57 '
style='stroke-width: 1.16; stroke: #EBEBEB; stroke-linecap: butt;'/>
<polyline points='474.67,227.53 474.67,41.57 '
style='stroke-width: 1.16; stroke: #EBEBEB; stroke-linecap: butt;'/>
<rect x='107.94' y='143.41' width='407.48' height='26.57'
style='stroke-width: 1.16; stroke: none; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='107.94' y='187.68' width='203.74' height='26.57'
style='stroke-width: 1.16; stroke: none; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<rect x='107.94' y='99.13' width='407.48' height='26.57'
style='stroke-width: 1.16; stroke: none; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='107.94' y='54.85' width='407.48' height='26.57'
style='stroke-width: 1.16; stroke: none; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<text x='515.42' y='152.78' text-anchor='middle' style='font-size: 11.38px; font-family: "Liberation Sans";'
textLength='28.47px' lengthAdjust='spacingAndGlyphs'>50/50
</text>
<text x='311.68' y='197.05' text-anchor='middle' style='font-size: 11.38px; font-family: "Liberation Sans";'
textLength='28.47px' lengthAdjust='spacingAndGlyphs'>25/50
</text>
<text x='515.42' y='108.50' text-anchor='middle' style='font-size: 11.38px; font-family: "Liberation Sans";'
textLength='28.47px' lengthAdjust='spacingAndGlyphs'>50/50
</text>
<text x='515.42' y='64.22' text-anchor='middle' style='font-size: 11.38px; font-family: "Liberation Sans";'
textLength='28.47px' lengthAdjust='spacingAndGlyphs'>50/50
</text>
</g>
<g clip-path='url(#cpMC4wMHw1NzYuMDB8MC4wMHwzMDIuNDA=)'>
<text x='80.56' y='204.26' text-anchor='end'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='64.97px'
lengthAdjust='spacingAndGlyphs'>amd64 (pre-fix)
</text>
<text x='80.56' y='159.99' text-anchor='end'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='74.58px'
lengthAdjust='spacingAndGlyphs'>aarch64 (post-fix)
</text>
<text x='80.56' y='115.71' text-anchor='end'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='69.25px'
lengthAdjust='spacingAndGlyphs'>amd64 (post-fix)
</text>
<text x='80.56' y='71.43' text-anchor='end'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='70.31px'
lengthAdjust='spacingAndGlyphs'>riscv64 (post-fix)
</text>
<text x='107.94' y='239.51' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='13.86px'
lengthAdjust='spacingAndGlyphs'>0%
</text>
<text x='230.18' y='239.51' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='19.19px'
lengthAdjust='spacingAndGlyphs'>30%
</text>
<text x='352.43' y='239.51' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='19.19px'
lengthAdjust='spacingAndGlyphs'>60%
</text>
<text x='474.67' y='239.51' text-anchor='middle'
style='font-size: 9.60px;fill: #4D4D4D; font-family: "Liberation Sans";' textLength='19.19px'
lengthAdjust='spacingAndGlyphs'>90%
</text>
<text x='327.98' y='252.74' text-anchor='middle' style='font-size: 12.00px; font-family: "Liberation Sans";'
textLength='50.67px' lengthAdjust='spacingAndGlyphs'>Pass rate
</text>
<rect x='273.74' y='273.94' width='15.73' height='15.73'
style='stroke-width: 1.16; stroke: none; stroke-linecap: butt; stroke-linejoin: miter; fill: #2CA02C;'/>
<rect x='333.88' y='273.94' width='15.73' height='15.73'
style='stroke-width: 1.16; stroke: none; stroke-linecap: butt; stroke-linejoin: miter; fill: #D62728;'/>
<text x='296.22' y='285.10' style='font-size: 9.60px; font-family: "Liberation Sans";' textLength='30.91px'
lengthAdjust='spacingAndGlyphs'>post-fix
</text>
<text x='356.37' y='285.10' style='font-size: 9.60px; font-family: "Liberation Sans";' textLength='26.62px'
lengthAdjust='spacingAndGlyphs'>pre-fix
</text>
<text x='85.94' y='33.10' style='font-size: 12.00px;fill: #505050; font-family: "Liberation Sans";'
textLength='387.47px' lengthAdjust='spacingAndGlyphs'>50 trials per run, unique random blocks across
the 22998-block data pool
</text>
<text x='85.94' y='15.88' style='font-size: 14.40px; font-weight: bold; font-family: "Liberation Sans";'
textLength='360.70px' lengthAdjust='spacingAndGlyphs'>Artemis surface stress test: pass rate
before/after fix
</text>
</g>
</g>
</svg>

After

Width:  |  Height:  |  Size: 7.8 KiB

File diff suppressed because it is too large Load Diff

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 61 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 133 KiB

@@ -0,0 +1,7 @@
*.aux
*.log
*.out
*.toc
*.synctex.gz
*.fls
*.fdb_latexmk
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
"run","label","n","passed","pass_rate","phase"
"aarch64_fixed","aarch64 (post-fix)",50,50,1,"post-fix"
"amd64_buggy","amd64 (pre-fix)",50,25,0.5,"pre-fix"
"amd64_fixed","amd64 (post-fix)",50,50,1,"post-fix"
"riscv64_fixed","riscv64 (post-fix)",50,50,1,"post-fix"
1 run label n passed pass_rate phase
2 aarch64_fixed aarch64 (post-fix) 50 50 1 post-fix
3 amd64_buggy amd64 (pre-fix) 50 25 0.5 pre-fix
4 amd64_fixed amd64 (post-fix) 50 50 1 post-fix
5 riscv64_fixed riscv64 (post-fix) 50 50 1 post-fix
@@ -0,0 +1,35 @@
#!/usr/bin/env python3
"""Convert all SVGs in charts/ to PDF for LaTeX \\includegraphics.
Mirrors experiments/bare_metal/analysis/svg_to_pdf.py, scoped to this
experiment's own charts/report/figures directories.
"""
import subprocess
import sys
from pathlib import Path
CHARTS = Path(__file__).parent / "charts"
PDFS = Path(__file__).parent / "report" / "figures"
PDFS.mkdir(parents=True, exist_ok=True)
svgs = sorted(CHARTS.glob("*.svg"))
if not svgs:
print("No SVGs found in", CHARTS)
sys.exit(1)
ok = fail = 0
for svg in svgs:
pdf = PDFS / svg.with_suffix(".pdf").name
result = subprocess.run(
["rsvg-convert", "--format=pdf", "--output", str(pdf), str(svg)],
capture_output=True, text=True
)
if result.returncode == 0:
print(f" OK {svg.name} -> {pdf.name}")
ok += 1
else:
print(f" ERR {svg.name}: {result.stderr.strip()}")
fail += 1
print(f"\n{ok} converted, {fail} failed.")
sys.exit(0 if fail == 0 else 1)
@@ -0,0 +1,20 @@
"key","value"
"h1_runs_obs",26
"h1_runs_exp",26
"h1_z",0
"h1_p",1
"h1b_W",332
"h1b_p",0.714773975516392
"h2_W",505
"h2_p",0.000107733285139378
"h2b_chisq",0.790350877192982
"h2b_df",2
"h2b_p",0.67356186013519
"h3_pre_n",50
"h3_pre_pass",25
"h3_post_n",150
"h3_post_pass",150
"h3_chisq",81.2038095238095
"h3_p",2.03599124928093e-19
"h3_ci_lo",-0.651923715768301
"h3_ci_hi",-0.348076284231699
1 key value
2 h1_runs_obs 26
3 h1_runs_exp 26
4 h1_z 0
5 h1_p 1
6 h1b_W 332
7 h1b_p 0.714773975516392
8 h2_W 505
9 h2_p 0.000107733285139378
10 h2b_chisq 0.790350877192982
11 h2b_df 2
12 h2b_p 0.67356186013519
13 h3_pre_n 50
14 h3_pre_pass 25
15 h3_post_n 150
16 h3_post_pass 150
17 h3_chisq 81.2038095238095
18 h3_p 2.03599124928093e-19
19 h3_ci_lo -0.651923715768301
20 h3_ci_hi -0.348076284231699
@@ -0,0 +1,32 @@
Artemis surface stress test -- hypothesis tests
=================================================
H1 -- pre-fix failure clustering (Wald-Wolfowitz runs test)
H0: pass/fail outcomes are independently ordered (no clustering).
Observed runs = 26 (expected under H0 = 26.00, sd = 3.50)
z = 0.000, p = 1.00e+00
-> fail to reject H0 at alpha=0.05: failures are not serially clustered.
H1b -- pre-fix outcome vs. trial/write order directly (Wilcoxon)
H0: trial index is drawn from the same distribution for PASS/FAIL
(complements H1: the runs test only catches consecutive-value
clustering, not a general rank-order trend).
W = 332.0, p = 0.715
-> fail to reject H0 at alpha=0.05: outcome is not associated with write order.
H2 -- pre-fix failure vs. disk location (Wilcoxon rank-sum test)
H0: logical block number is drawn from the same distribution
for PASS and FAIL trials (failure does not depend on location).
W = 505.0, p = 0.000
-> REJECT H0 at alpha=0.05: failure IS associated with LBN.
Secondary check -- sibling position (0/1/2) vs. outcome (chi-squared)
chi-sq = 0.790, df = 2, p = 0.674
-> fail to reject H0: outcome is not associated with sibling position.
H3 -- did the fix change the pass rate (two-proportion test)
H0: pre-fix and post-fix trials share the same true pass rate.
Pre-fix: 25/50 = 50.0% Post-fix (pooled, 3 arches): 150/150 = 100.0%
chi-sq = 81.20, df = 1, p = 2.04e-19
95% CI on the difference in pass rate: [-0.652, -0.348]
-> REJECT H0 at alpha=0.05: the fix DID CHANGE the pass rate.
@@ -0,0 +1,51 @@
trial,lbn,pat,result
0,11306,0,1
1,8587,1,1
2,3149,2,1
3,23428,3,1
4,20709,4,1
5,15271,5,1
6,12552,6,1
7,7114,7,1
8,4395,8,1
9,24674,9,1
10,21955,10,1
11,16517,11,1
12,25920,12,1
13,23201,13,1
14,20482,14,1
15,17763,15,1
16,12325,16,1
17,24447,17,1
18,21728,18,1
19,16290,19,1
20,13571,20,1
21,25693,21,1
22,22974,22,1
23,20255,23,1
24,17536,24,1
25,12098,25,1
26,3941,26,1
27,24220,27,1
28,21501,28,1
29,16063,29,1
30,13344,30,1
31,5187,31,1
32,25466,32,1
33,22747,33,1
34,20028,34,1
35,11871,35,1
36,23993,36,1
37,21274,37,1
38,13117,38,1
39,25239,39,1
40,22520,40,1
41,19801,41,1
42,17082,42,1
43,14363,43,1
44,23766,44,1
45,21047,45,1
46,18328,46,1
47,12890,47,1
48,4733,48,1
49,25012,49,1
1 trial lbn pat result
2 0 11306 0 1
3 1 8587 1 1
4 2 3149 2 1
5 3 23428 3 1
6 4 20709 4 1
7 5 15271 5 1
8 6 12552 6 1
9 7 7114 7 1
10 8 4395 8 1
11 9 24674 9 1
12 10 21955 10 1
13 11 16517 11 1
14 12 25920 12 1
15 13 23201 13 1
16 14 20482 14 1
17 15 17763 15 1
18 16 12325 16 1
19 17 24447 17 1
20 18 21728 18 1
21 19 16290 19 1
22 20 13571 20 1
23 21 25693 21 1
24 22 22974 22 1
25 23 20255 23 1
26 24 17536 24 1
27 25 12098 25 1
28 26 3941 26 1
29 27 24220 27 1
30 28 21501 28 1
31 29 16063 29 1
32 30 13344 30 1
33 31 5187 31 1
34 32 25466 32 1
35 33 22747 33 1
36 34 20028 34 1
37 35 11871 35 1
38 36 23993 36 1
39 37 21274 37 1
40 38 13117 38 1
41 39 25239 39 1
42 40 22520 40 1
43 41 19801 41 1
44 42 17082 42 1
45 43 14363 43 1
46 44 23766 44 1
47 45 21047 45 1
48 46 18328 46 1
49 47 12890 47 1
50 48 4733 48 1
51 49 25012 49 1
@@ -0,0 +1,51 @@
trial,lbn,pat,result
0,11306,0,0
1,8587,1,0
2,3149,2,0
3,20709,3,1
4,15271,4,1
5,12552,5,0
6,9833,6,1
7,7114,7,1
8,4395,8,0
9,24674,9,0
10,21955,10,1
11,16517,11,0
12,5641,12,1
13,25920,13,1
14,23201,14,1
15,20482,15,1
16,17763,16,1
17,12325,17,0
18,21728,18,0
19,16290,19,1
20,13571,20,0
21,8133,21,0
22,25693,22,1
23,17536,23,1
24,12098,24,0
25,9379,25,0
26,3941,26,0
27,21501,27,1
28,16063,28,1
29,13344,29,0
30,7906,30,0
31,5187,31,0
32,25466,32,1
33,22747,33,0
34,11871,34,0
35,3714,35,0
36,23993,36,1
37,21274,37,1
38,13117,38,0
39,22520,39,1
40,19801,40,0
41,17082,41,1
42,14363,42,1
43,8925,43,0
44,18328,44,1
45,12890,45,1
46,10171,46,0
47,4733,47,0
48,22293,48,1
49,16855,49,1
1 trial lbn pat result
2 0 11306 0 0
3 1 8587 1 0
4 2 3149 2 0
5 3 20709 3 1
6 4 15271 4 1
7 5 12552 5 0
8 6 9833 6 1
9 7 7114 7 1
10 8 4395 8 0
11 9 24674 9 0
12 10 21955 10 1
13 11 16517 11 0
14 12 5641 12 1
15 13 25920 13 1
16 14 23201 14 1
17 15 20482 15 1
18 16 17763 16 1
19 17 12325 17 0
20 18 21728 18 0
21 19 16290 19 1
22 20 13571 20 0
23 21 8133 21 0
24 22 25693 22 1
25 23 17536 23 1
26 24 12098 24 0
27 25 9379 25 0
28 26 3941 26 0
29 27 21501 27 1
30 28 16063 28 1
31 29 13344 29 0
32 30 7906 30 0
33 31 5187 31 0
34 32 25466 32 1
35 33 22747 33 0
36 34 11871 34 0
37 35 3714 35 0
38 36 23993 36 1
39 37 21274 37 1
40 38 13117 38 0
41 39 22520 39 1
42 40 19801 40 0
43 41 17082 41 1
44 42 14363 42 1
45 43 8925 43 0
46 44 18328 44 1
47 45 12890 45 1
48 46 10171 46 0
49 47 4733 47 0
50 48 22293 48 1
51 49 16855 49 1
@@ -0,0 +1,51 @@
trial,lbn,pat,result
0,11306,0,1
1,8587,1,1
2,3149,2,1
3,23428,3,1
4,20709,4,1
5,15271,5,1
6,12552,6,1
7,7114,7,1
8,4395,8,1
9,24674,9,1
10,21955,10,1
11,16517,11,1
12,25920,12,1
13,23201,13,1
14,20482,14,1
15,17763,15,1
16,12325,16,1
17,24447,17,1
18,21728,18,1
19,16290,19,1
20,13571,20,1
21,25693,21,1
22,22974,22,1
23,20255,23,1
24,17536,24,1
25,12098,25,1
26,3941,26,1
27,24220,27,1
28,21501,28,1
29,16063,29,1
30,13344,30,1
31,5187,31,1
32,25466,32,1
33,22747,33,1
34,20028,34,1
35,11871,35,1
36,23993,36,1
37,21274,37,1
38,13117,38,1
39,25239,39,1
40,22520,40,1
41,19801,41,1
42,17082,42,1
43,14363,43,1
44,23766,44,1
45,21047,45,1
46,18328,46,1
47,12890,47,1
48,4733,48,1
49,25012,49,1
1 trial lbn pat result
2 0 11306 0 1
3 1 8587 1 1
4 2 3149 2 1
5 3 23428 3 1
6 4 20709 4 1
7 5 15271 5 1
8 6 12552 6 1
9 7 7114 7 1
10 8 4395 8 1
11 9 24674 9 1
12 10 21955 10 1
13 11 16517 11 1
14 12 25920 12 1
15 13 23201 13 1
16 14 20482 14 1
17 15 17763 15 1
18 16 12325 16 1
19 17 24447 17 1
20 18 21728 18 1
21 19 16290 19 1
22 20 13571 20 1
23 21 25693 21 1
24 22 22974 22 1
25 23 20255 23 1
26 24 17536 24 1
27 25 12098 25 1
28 26 3941 26 1
29 27 24220 27 1
30 28 21501 28 1
31 29 16063 29 1
32 30 13344 30 1
33 31 5187 31 1
34 32 25466 32 1
35 33 22747 33 1
36 34 20028 34 1
37 35 11871 35 1
38 36 23993 36 1
39 37 21274 37 1
40 38 13117 38 1
41 39 25239 39 1
42 40 22520 40 1
43 41 19801 41 1
44 42 17082 42 1
45 43 14363 43 1
46 44 23766 44 1
47 45 21047 45 1
48 46 18328 46 1
49 47 12890 47 1
50 48 4733 48 1
51 49 25012 49 1
@@ -0,0 +1,51 @@
trial,lbn,pat,result
0,11306,0,1
1,8587,1,1
2,3149,2,1
3,23428,3,1
4,20709,4,1
5,15271,5,1
6,12552,6,1
7,7114,7,1
8,4395,8,1
9,24674,9,1
10,21955,10,1
11,16517,11,1
12,25920,12,1
13,23201,13,1
14,20482,14,1
15,17763,15,1
16,12325,16,1
17,24447,17,1
18,21728,18,1
19,16290,19,1
20,13571,20,1
21,25693,21,1
22,22974,22,1
23,20255,23,1
24,17536,24,1
25,12098,25,1
26,3941,26,1
27,24220,27,1
28,21501,28,1
29,16063,29,1
30,13344,30,1
31,5187,31,1
32,25466,32,1
33,22747,33,1
34,20028,34,1
35,11871,35,1
36,23993,36,1
37,21274,37,1
38,13117,38,1
39,25239,39,1
40,22520,40,1
41,19801,41,1
42,17082,42,1
43,14363,43,1
44,23766,44,1
45,21047,45,1
46,18328,46,1
47,12890,47,1
48,4733,48,1
49,25012,49,1
1 trial lbn pat result
2 0 11306 0 1
3 1 8587 1 1
4 2 3149 2 1
5 3 23428 3 1
6 4 20709 4 1
7 5 15271 5 1
8 6 12552 6 1
9 7 7114 7 1
10 8 4395 8 1
11 9 24674 9 1
12 10 21955 10 1
13 11 16517 11 1
14 12 25920 12 1
15 13 23201 13 1
16 14 20482 14 1
17 15 17763 15 1
18 16 12325 16 1
19 17 24447 17 1
20 18 21728 18 1
21 19 16290 19 1
22 20 13571 20 1
23 21 25693 21 1
24 22 22974 22 1
25 23 20255 23 1
26 24 17536 24 1
27 25 12098 25 1
28 26 3941 26 1
29 27 24220 27 1
30 28 21501 28 1
31 29 16063 29 1
32 30 13344 30 1
33 31 5187 31 1
34 32 25466 32 1
35 33 22747 33 1
36 34 20028 34 1
37 35 11871 35 1
38 36 23993 36 1
39 37 21274 37 1
40 38 13117 38 1
41 39 25239 39 1
42 40 22520 40 1
43 41 19801 41 1
44 42 17082 42 1
45 43 14363 43 1
46 44 23766 44 1
47 45 21047 45 1
48 46 18328 46 1
49 47 12890 47 1
50 48 4733 48 1
51 49 25012 49 1
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More