Fix O(N) idle-loop messaging pump; full 3x9x3 turn-attractor campaign clean on all 3 architectures (FABRIC-3.md §XXII)
Build / build-amd64-iso (push) Canceled after 0s
Build / build-aarch64-iso (push) Canceled after 0s
Build / build-riscv64-img (push) Canceled after 0s

sk_repl_idle()'s messaging pump (repl.c) walked the entire live-VM
registry every idle beat (~1Hz) and dispatched a full VM-EXEC
"MSG-TICK" -- a dictionary lookup plus a 32-slot arena scan -- into
every live VM, every tick, unconditionally, forever. Fine at Tripod's
original 3-VM scale; a full 9-identity turn-attractor campaign
exposed it as a genuine wall on riscv64 specifically (its TCG makes
each dispatch cost more): the same campaign that completed in 194s/
339s on amd64/aarch64 never finished on riscv64 at 9 VMs across three
attempts, while 8 VMs there was fine in 159s.

Ruled out capacity explanations before touching anything: bumping
riscv64's QEMU RAM 1024->4096 changed nothing (reverted), and a live
STADIUM-RES@/MSG-STATUS probe with all 9 VMs attached showed no
depletion. Host memory pressure was also ruled out directly (one
background task did get OOM-killed once during the investigation,
but the identical stall reproduced again with 9.2GB free). The real
mistake was three premature kills under 4 minutes with no way to
tell "slow" from "stuck" from outside the guest -- fixed by having
run_doe_batch.sh sample the qemu process's own /proc/<pid>/stat utime
every 60s; with that signal, riscv64 at 9 VMs was unambiguously alive
(climbing utime, no hang), just disproportionately slow going from 8
VMs (159s) to 9 (600s+ and climbing).

This was never really a riscv64-only bug: an O(N) per-second walk
over the full VM population doesn't scale to the hundreds of VMs this
fleet is headed toward, on any architecture -- riscv64 just made it
visible first, at N=9, because its per-dispatch cost is highest.

Fixed by round-robin batching: the pump now dispatches to at most
SK_MSG_PUMP_BATCH (4) live VMs per idle beat via a persistent cursor
that resumes where the previous beat left off, instead of all of them
every time. Bounds both the scan and dispatch cost to O(K) regardless
of total VM count; any single VM's queue now drains roughly every
ceil(N/K) beats instead of every beat, still bounded and still
matching the pump's own existing best-effort contract. No new C
primitives, no messaging/Stadium changes.

Verified: clean build on all 3 architectures, then the full 3x9x3
campaign re-run on all 3 (not just riscv64) per the standing rule
that a defect repair requires a clean re-run everywhere before
anything counts as closed:

  amd64   198s  0 faults  99/99 tokens x8  656/656 K-conserved
  aarch64 339s  0 faults  99/99 tokens x8  659/659 K-conserved
  riscv64 178s  0 faults  99/99 tokens x8  659/659 K-conserved

riscv64 went from "never completes" to faster than aarch64, same
campaign, same seed, same identity set. No regression on amd64/
aarch64.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EXieurDfDSsDFdnSyusuWo
This commit is contained in:
Robert Allan James
2026-09-12 19:29:14 -04:00
co-authored by Claude Sonnet 5
parent 19916717b2
commit 71bcb72a59
38 changed files with 191163 additions and 650 deletions
+64 -33
View File
@@ -293,6 +293,29 @@ static void sk_cursor_show(void)
static int g_mama_interpreting; /* zero-initialized (BSS) */
static int g_idle_pump_active; /* zero-initialized (BSS) */
/* FABRIC-3.md SXXII (2026-09-12): the messaging pump below used to walk
* the ENTIRE live-VM registry every idle beat and dispatch a full
* VM-EXEC "MSG-TICK" into every one of them -- O(N) per tick, forever,
* with no cap. Fine at Tripod's original 3-VM scale; live-caught as a
* real bottleneck at just 9 VMs on riscv64 (a std79 turn-attractor
* campaign that completed in ~200-340s on amd64/aarch64 never finished
* on riscv64 at the same VM count, while 8 VMs was fine there too --
* the qemu-instance progress check (utime climbing steadily, ~66-75%
* CPU) ruled out a hang; it was genuinely just this loop, at riscv64's
* slower per-dispatch cost, dominating wall time). The real fleet is
* headed toward hundreds of VMs, at which point this isn't a riscv64
* quirk -- unconditional O(N) per second is a wall on every
* architecture. Fixed by round-robin batching: SK_MSG_PUMP_BATCH live
* VMs get dispatched per beat, not all of them, with a persistent
* cursor picking up where the previous beat left off (wrapping around
* the registry). Bounds both the scan and the dispatch cost to O(K)
* regardless of total VM count -- the tradeoff is that any single VM's
* own queue now drains roughly every ceil(N/K) beats instead of every
* beat, which is still bounded and still matches the pump's own
* existing "best-effort, resumes next beat" contract. */
#define SK_MSG_PUMP_BATCH 4u /* max VM-EXEC MSG-TICK dispatches per idle beat */
static uint32_t g_msg_pump_cursor; /* zero-initialized (BSS) */
static void sk_repl_idle(VM *active_vm)
{
/* Close any dangling output line before this bottom half emits its own
@@ -582,44 +605,52 @@ static void sk_repl_idle(VM *active_vm)
g_idle_pump_active = 1;
{
uint32_t count = capsule_vm_registry_count();
uint32_t i;
for (i = 0; i < count; i++) {
uint32_t count = capsule_vm_registry_count();
uint32_t dispatched = 0;
uint32_t scanned = 0;
uint32_t i = (count > 0) ? (g_msg_pump_cursor % count) : 0;
while (count > 0 && dispatched < SK_MSG_PUMP_BATCH && scanned < count) {
VMRegistryEntry ent;
if (capsule_vm_registry_get_by_index(i, &ent) != 0) continue;
if (ent.state != VM_STATE_LIVE) continue;
if (ent.vm_ptr == (void *)mama) continue;
/* FABRIC-3.md, 2026-09-09: a per-VM flag (e.g. "this one is
* std79-locked") doesn't generalize -- any VM whose own
* dictionary lacks MSG-TICK, for whatever reason (never
* loaded common:messaging.4th, a future personality that
* drops it, ACL denial), hits the identical failure. So
* check the actual target VM's own dictionary fresh every
* tick, the same source of truth the interpreter's own ACL
* enforcement uses (vm_core.c), rather than pre-flag specific
* personalities. Skip silently rather than spam "VM-EXEC:
* ERROR in <name>" every idle tick forever for a VM that
* plain doesn't have -- or isn't allowed -- MSG-TICK
* (live-caught 2026-09-07/09). */
{
scanned++;
if (capsule_vm_registry_get_by_index(i, &ent) == 0 &&
ent.state == VM_STATE_LIVE &&
ent.vm_ptr != (void *)mama) {
/* FABRIC-3.md, 2026-09-09: a per-VM flag (e.g. "this one is
* std79-locked") doesn't generalize -- any VM whose own
* dictionary lacks MSG-TICK, for whatever reason (never
* loaded common:messaging.4th, a future personality that
* drops it, ACL denial), hits the identical failure. So
* check the actual target VM's own dictionary fresh every
* tick, the same source of truth the interpreter's own ACL
* enforcement uses (vm_core.c), rather than pre-flag specific
* personalities. Skip silently rather than spam "VM-EXEC:
* ERROR in <name>" every idle tick forever for a VM that
* plain doesn't have -- or isn't allowed -- MSG-TICK
* (live-caught 2026-09-07/09). */
DictEntry *msgtick = vm_find_word((VM *)ent.vm_ptr, "MSG-TICK", 8);
if (!msgtick || !msgtick->acl_allow) continue;
if (msgtick && msgtick->acl_allow) {
static const char PFX[] = "S\" MSG-TICK\" S\" ";
static const char SFX[] = "\" VM-EXEC";
char cmd[128];
size_t p = 0;
size_t name_len = strlen(ent.name);
if (name_len > VM_NAME_MAX - 1u) name_len = VM_NAME_MAX - 1u;
memcpy(cmd + p, PFX, sizeof(PFX) - 1u); p += sizeof(PFX) - 1u;
memcpy(cmd + p, ent.name, name_len); p += name_len;
memcpy(cmd + p, SFX, sizeof(SFX) - 1u); p += sizeof(SFX) - 1u;
cmd[p] = '\0';
vm_interpret(mama, cmd);
dispatched++;
}
}
static const char PFX[] = "S\" MSG-TICK\" S\" ";
static const char SFX[] = "\" VM-EXEC";
char cmd[128];
size_t p = 0;
size_t name_len = strlen(ent.name);
if (name_len > VM_NAME_MAX - 1u) name_len = VM_NAME_MAX - 1u;
memcpy(cmd + p, PFX, sizeof(PFX) - 1u); p += sizeof(PFX) - 1u;
memcpy(cmd + p, ent.name, name_len); p += name_len;
memcpy(cmd + p, SFX, sizeof(SFX) - 1u); p += sizeof(SFX) - 1u;
cmd[p] = '\0';
vm_interpret(mama, cmd);
i = (i + 1u) % count;
}
g_msg_pump_cursor = i;
}
/* FABRIC-3.md SXX: Hera now genuinely has her own MSG-ARENA/CH-ARENA
* (the messaging-load root cause above is fixed -- see kernel_main.c's