diff --git a/FABRIC-3.5.md b/FABRIC-3.5.md index 09f7b4b2..16f17468 100644 --- a/FABRIC-3.5.md +++ b/FABRIC-3.5.md @@ -3583,3 +3583,171 @@ Stated as tripwires rather than hopes, since the point of this pass is confidenc - Item 28 (prove the allocator) becomes **Stage B** and keeps its priority. - Item 27 (channels: negotiate or one membership) **gains urgency** — §XXXIV.1 shows retiring channels directly relieves the transition's budget pressure. + +--- + +## XXXV. Pre-mortem: it is six months on, the reshuffle failed — why? (2026-09-19) + +Run at Captain Bob's request before pulling the build trigger. **Method: assume failure, work +backwards, and ground every cause in this project's own recorded failure record rather than in +generic risk.** Two of the six are new findings that change the design; one is a hypothesis that +weakened when checked, recorded as such. + +### XXXV.0 — This codebase's dominant failure mode, evidenced + +Before the specific causes, the pattern they mostly instantiate. **Things here fail without +saying anything:** + +- §XXVIII.1: "Stage 3 emits nothing per switch by default, so **a switch storm and a healthy + idle REPL produce an identical serial log.** The corruption was already live in every one of + those 'clean' Stage 3 boots." +- §XXXII.2 (`FABRIC-3`): a console-name mismatch "**silently falls back to direct + interpretation with no error** — just quietly never relays." +- §XIV/§XXXI: near-simultaneous attach — **6 of 9 identities silently never triggered** their + attach line. +- §XX: "Referencing an undefined word during FORTH compilation doesn't raise a hard error, it + **silently drops the definition being compiled**." + +**Four independent subsystems, one signature.** Any pre-mortem for this project that does not +put silent failure first is not about this project. + +### XXXV.1 — Cause 1, most likely: the sinking latch never fired, because nothing could raise it + +**This is the pre-mortem's principal finding, and it is a design gap, not a risk.** + +§VIII.1 has Hermes raise a latch when it detects itself sinking, holding it until it goes down, +to give the fleet a window. That was conceived when Hermes was **a VM on the Stadium floor** — +a patron with heat, a TTL and a reservoir, capable of degrading. + +**Kernel-Hermes is none of those things.** `stadium_admit()` is called only from +`capsule_birth.c:808`, on birth. Kernel-Hermes is not born, so **it is not a Stadium patron: no +heat, no TTL, no eviction, no compudynamic death.** Which raises the question §VIII never had to +answer: + +> **What does it mean for a kernel-resident C subsystem to *sink*?** + +Its real failure modes are: +- **A bug → `sk_hal_panic()` → `while(1){arch_halt();}`.** Immediate and total. **No window at + all** — the fleet never gets to shut down cleanly, which is precisely what §VIII.1 exists to + provide. +- **Resource exhaustion → allocation refusals.** Degraded, but not dying, and recoverable. + Raising a monotonic "I am sinking" latch here would be wrong (§XXIII.3 made it irreversible). + +**And §VII's ladder does not apply to it either.** Rung 1 is warm restart, rung 2 is a +from-scratch `BIRTH` — **but `BIRTH` births VMs.** There is no birthing a kernel subsystem. +§XXI.3's table gives every rung an actor *for VMs*; kernel-Hermes sits outside that table +entirely. + +**So the failure story for the one component that cannot use `SOS` is undesigned.** Six months +on, the most likely shape of "it failed" is: the arbiter hit a bug, panicked, halted the +machine instantly, and every VM died mid-write with no window — with the latch never set, +because there was no state between healthy and gone. + +**Punch item 31, and it gates §VIII/§XXIII rather than following them.** Candidate directions, +none chosen: give kernel-Hermes an explicit degraded state it *can* detect and announce (queue +exhaustion, corrupted routing table) and raise the latch there; or accept that its only failure +is a panic, and make the panic path itself raise the latch and yield briefly before halting; or +conclude §VIII was scoped to a component that no longer exists and retire it. **The third is a +real possibility and should not be dismissed for tidiness.** + +### XXXV.2 — Cause 2: the fleet hit a ceiling of 16 + +`SK_SWITCH_MAX_SLOTS` is **16** (`capsule_vm_switch_signal.c:42`), with the header noting +compaction "after `SK_SWITCH_MAX_SLOTS` attach/detach cycles." + +That was a reasonable bound when the switcher was one of *two* ways a VM got control. **§XXXII +made it the only one.** So the moment that ruling lands, **16 becomes a hard ceiling on how many +VMs can receive control at all** — and §XXII's own comment says "the real fleet is headed toward +hundreds of VMs." + +**This is the §XXII pump story repeating one layer over**: a bound that is invisible at Tripod +scale, fine at 9, and a wall later. §XXII found its wall live, at 9 VMs, on the slowest +architecture, mid-campaign. **This one is findable now, on paper, before it is built on.** + +**Punch item 32: size the switch-slot table against the intended fleet before §XXXII's ruling +is implemented.** Not necessarily hard — but it must be a decision, not a default carried +forward from when it did not matter. + +### XXXV.3 — Cause 3, weakened on inspection and recorded honestly + +**The hypothesis:** the reshuffle's tripwires (§XXXIV.6) all depend on `fleet_conserved`, while +§XXV.1 has the DoE's invocative paths being rewritten in the same period — so the instrument +that verifies K would be under reconstruction exactly when it is most needed. + +**Checked, and it is weaker than it looked.** `vm_physics_conserved()` lives in +`capsule_vm_physics.c:492` — a kernel function, independent of the DoE — and is exposed as a +FORTH word at `mama_forth_words.c:2099` (`VM-CONSERVED?`). `doe_log.c` only *reports* it, in +columns 19/20. **The capability survives a DoE rewrite.** + +**The residual is real but smaller:** what the DoE provides is the *continuous, per-tick record* +of K across a long run. A word you have to call gives you a point sample. **Stage B's proof +(§XXXIV.3) should therefore not lean on the DoE** — it needs its own deterministic +alloc/free/check loop calling `VM-CONSERVED?` directly. Recorded so Stage B is designed for an +instrument that exists rather than one mid-rewrite. + +### XXXV.4 — Cause 4: Hestia turned out to be ceremonial + +§XVIII.2 established that the HAL must never know a VM exists, so **Hestia's ownership of the +fabric is by vocabulary and registration — not enforcement.** §XVIII.2 called that "weaker than +it sounds and exactly right." + +The six-months-on failure: **nothing changed.** Other VMs still reach the framebuffer, Hestia +owns the fabric only in the documentation, and the reorganization was cosmetic — a rename plus a +capsule. + +**The test is narrow and checkable:** after the reshuffle, is `PLOT`/`FB-WIDTH`/`FB-HEIGHT` +registered *anywhere* except Hestia's word table? In particular, `register_child_vm_words()` +must **not** hand them out the way it hands out the eight `STADIUM-*` primitives. If it does, +ownership is a fiction on day one. + +**Punch item 33: verify the framebuffer primitives are registered to Hestia alone**, and treat +any other registration site as a defect rather than a convenience. + +### XXXV.5 — Cause 5: the strip took something live, and the usual net was down + +§XXII.2's near-miss (the `init-l8-*` family reading as zero-reference) plus §XXII.3's acceptance +asymmetry — a bad DoE-capsule strip **passes all three architectures cleanly** and surfaces +months later during a campaign. + +**What makes this worse than §XXII assumed:** §XXV.1 has the DoE's invocative paths being +rewritten in the same window. **The campaign that would eventually catch a wrong strip is itself +in flux**, so "we would have noticed at the next campaign" is a weaker assurance than it was +when §XXII was written. + +**Mitigation, no new mechanism needed:** do the Category A strip **before** the DoE rewrite +begins, while the existing campaign apparatus still runs, and run one campaign against the +stripped tree as the strip's real acceptance — not just a boot. + +### XXXV.6 — Cause 6: it was declared done, and it wasn't + +The project's most repeated failure, by its own record: §XXVIII's "Stage 3 CLOSED" corrected by +§XXVIII.1; §XXV corrected by §XXVI ("wrong VM tested"); `FABRIC-2`'s close header overclaiming +(§XXXI.3); the `v2.0.1` bump rolled back (§I.2). + +**Already partly defended** — §XXIX refused the LTS label and wrote criteria instead, §XXXIV.6 +states tripwires as stop conditions. **The remaining exposure is the acceptance definition +itself:** three green boots prove the boot path, and §XXXV.0 says this codebase's failures do +not announce themselves on the boot path. + +**Punch item 34: for each of Stages A–E, name in advance the observation that would prove it +worked** — not "it booted," but the specific evidence, chosen before the stage runs. §XXVIII.1 +is the cautionary case: a stage that emits nothing cannot be verified by reading a log that +looks identical either way. + +### XXXV.7 — What the pre-mortem changes + +**Two new design gaps** (§XXXV.1's undesigned kernel-arbiter failure story, §XXXV.2's slot +ceiling), **two verification corrections** (§XXXV.3's Stage B instrument, §XXXV.6's +name-the-evidence rule), **one sequencing change** (§XXXV.5: strip before the DoE rewrite), and +**one checkable invariant** (§XXXV.4). + +- ⬜ **31 — design kernel-Hermes's own failure story**, or retire §VIII. **Gates §VIII/§XXIII.** +- ⬜ **32 — size `SK_SWITCH_MAX_SLOTS` against the intended fleet** before §XXXII lands. +- ⬜ **33 — framebuffer primitives registered to Hestia alone**, verified not assumed. +- ⬜ **34 — per-stage success evidence named in advance**, per §XXVIII.1's lesson. +- ⬜ **35 — Category A strip before the DoE rewrite**, with a campaign as its acceptance. + +**The honest summary: the most likely cause of failure is not the messaging rewrite.** That is +now sized (§XXXIII), staged (§XXXIV) and instrumented. **It is that the one component which +cannot call for help has no defined way to fail** — and that this codebase's failures are +characteristically quiet.