Fix two catastrophically heavy workloads found running the real campaign
Build / build-amd64-iso (push) Canceled after 0s
Build / build-aarch64-iso (push) Canceled after 0s
Build / build-riscv64-img (push) Canceled after 0s

The full 360-trial campaign was launched, then killed after 4h39m of
CPU time with zero trials completed -- still stuck on the very first
touch of the very first trial.

Root cause, quantified from the workload's own source, not estimated:
RUN-CHAOS5 (workload-5.4th) is 1000000 0 DO CHAOS-FIELD CHAOS-RIPPLE
LOOP, multiplying out to ~2.4 trillion word executions for one call --
~495 days at the observed rate. Never designed to be called to
completion as a single touch in a repeatedly-birthed worker.

A second problem found computing the fix rather than discovering it
mid-run again: workload-1.4th's own birth-time self-execution (20
SQUARE-WAVE + 1500 SQUARE-BURST + 100000x MICRO-BURST, all three run
automatically at capsule load) sums to ~875 million words -- ~4.3
hours just to birth one worker -- and worker index 1 always maps to
it in heterogeneous mode, landing in half the campaign's cells
regardless of concurrency level.

Fix: two new capsules, workload-1-lite.4th and workload-5-lite.4th,
carrying the same word bodies verbatim but a bounded self-execution
tail, comparable in scale to the campaign's other workloads. The
originals are untouched; only multiuser-doe.4th's own WL-CAPSULE/
WL-ENTRY index 1 and 5 mappings were repointed.

Verified live before relaunching a third time: the actual worst case
in isolation (5 0 998 MU-RUN-TRIAL, concurrency=8 heterogeneous,
includes both fixed workloads plus RUN-OMNI) completed in ~20 minutes
wall-clock, Hera stayed healthy throughout. Clean 3-arch qemu boot.

Reps reduced 60 -> 30/cell (Bob's call after seeing the real per-trial
cost) -- still matches the project's "rule of 3's" DoE convention
(same as ACL-RWT's own 30 reps). 6 cfgs x 30 reps = 180 trials.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BWpNjdwPtFLuVLaAq44L9K
This commit is contained in:
Robert Allan James
2026-09-16 15:47:30 -04:00
co-authored by Claude Sonnet 5
parent 29f91f9531
commit 94215e8b47
16 changed files with 2048842 additions and 18 deletions
+42
View File
@@ -5090,3 +5090,45 @@ workloads like `RUN-CHAOS5`'s 1,000,000 iterations) -- a genuinely long-running
TCG, deliberately not kicked off without explicit confirmation first. The WIREBIND-automation
fixed arm also remains unbuilt.
### XXXIII.7 -- Two workloads too heavy to touch as-is, found live at real cost
**Approved and launched, then killed after finding a real problem the design review never
caught:** the full campaign ran for 4h39m of CPU time without completing even the first
touch of the first trial. Root cause quantified directly from the workload's own source
(`workload-5.4th`), not estimated: `RUN-CHAOS5` is `1000000 0 DO CHAOS-FIELD CHAOS-RIPPLE
LOOP`, and `CHAOS-FIELD` alone is `24 0 DO CHAOS-BLOCK LOOP` with `CHAOS-BLOCK` itself ~100K
word executions -- multiplying out to **~2.4 trillion word executions for one call**. At the
observed rate (~56,300 words/sec), that's roughly 495 days for a single touch. This workload
was never designed to be called to completion as one touch in a repeatedly-birthed-and-killed
worker; whatever its role in the original single-VM L8 DoE, it doesn't fit this campaign's
shape at all.
**A second, related problem found computing the fix, not by running it first:** `workload-
1.4th`'s own birth-time self-execution (`20 SQUARE-WAVE` + `1500 SQUARE-BURST` + `100000x
MICRO-BURST`, all three run automatically at capsule load) sums to roughly **875 million word
executions -- ~4.3 hours just to birth one worker**, and worker index 1 always maps to this
capsule in heterogeneous mode, so the cost lands in half the campaign's cells regardless of
concurrency level. Caught by computing the real per-workload word counts from every capsule's
own source before relaunching a third time, not by discovering it mid-run again.
**Fix, matching the pattern Captain Bob had already authorized ("workloads are not carved in
stone, create new extended workloads where needed"):** two new capsules,
`workload-5-lite.4th` and `workload-1-lite.4th`, carrying the *same* word bodies verbatim
(`CHAOS-BLOCK`/`CHAOS-FIELD`; `COMPUTE`/`BURST`/`MICRO-BURST`) but a bounded self-execution
tail (`CHAOS-FIELD` once; `1000x MICRO-BURST`) -- comparable in scale to the campaign's other
workloads (~2-30M words), not the original files' unbounded repeat counts. The originals
(`workload-1.4th`, `workload-5.4th`) are untouched, still real, still available for whatever
their original context needs -- only `multiuser-doe.4th`'s own `WL-CAPSULE`/`WL-ENTRY` index
1 and 5 mappings were repointed to the lite variants.
**Verified live before relaunching the full campaign a third time:** ran the actual worst
case in isolation -- `5 0 998 MU-RUN-TRIAL` (cfg=5: concurrency=8, heterogeneous, so it
includes both previously-catastrophic workloads plus `RUN-OMNI`'s own ~30M words) -- and it
completed in ~20 minutes wall-clock (`MU-TRIAL run=998 cfg=5 rep=0 nw=8 mode=1 fail=0`), Hera
stayed healthy throughout. Clean 3-arch qemu boot on the real committed capsule changes.
**Reps reduced from 60 to 30 per cell**, Captain Bob's own call after seeing the real per-
trial cost -- still matches the project's "rule of 3's" DoE convention exactly (the same 30
reps as the ACL-RWT campaign). 6 cfgs x 30 reps = 180 main-block trials, roughly half the
prior 360-trial estimate.