Fix O(N) idle-loop messaging pump; full 3x9x3 turn-attractor campaign clean on all 3 architectures (FABRIC-3.md §XXII)
sk_repl_idle()'s messaging pump (repl.c) walked the entire live-VM registry every idle beat (~1Hz) and dispatched a full VM-EXEC "MSG-TICK" -- a dictionary lookup plus a 32-slot arena scan -- into every live VM, every tick, unconditionally, forever. Fine at Tripod's original 3-VM scale; a full 9-identity turn-attractor campaign exposed it as a genuine wall on riscv64 specifically (its TCG makes each dispatch cost more): the same campaign that completed in 194s/ 339s on amd64/aarch64 never finished on riscv64 at 9 VMs across three attempts, while 8 VMs there was fine in 159s. Ruled out capacity explanations before touching anything: bumping riscv64's QEMU RAM 1024->4096 changed nothing (reverted), and a live STADIUM-RES@/MSG-STATUS probe with all 9 VMs attached showed no depletion. Host memory pressure was also ruled out directly (one background task did get OOM-killed once during the investigation, but the identical stall reproduced again with 9.2GB free). The real mistake was three premature kills under 4 minutes with no way to tell "slow" from "stuck" from outside the guest -- fixed by having run_doe_batch.sh sample the qemu process's own /proc/<pid>/stat utime every 60s; with that signal, riscv64 at 9 VMs was unambiguously alive (climbing utime, no hang), just disproportionately slow going from 8 VMs (159s) to 9 (600s+ and climbing). This was never really a riscv64-only bug: an O(N) per-second walk over the full VM population doesn't scale to the hundreds of VMs this fleet is headed toward, on any architecture -- riscv64 just made it visible first, at N=9, because its per-dispatch cost is highest. Fixed by round-robin batching: the pump now dispatches to at most SK_MSG_PUMP_BATCH (4) live VMs per idle beat via a persistent cursor that resumes where the previous beat left off, instead of all of them every time. Bounds both the scan and dispatch cost to O(K) regardless of total VM count; any single VM's queue now drains roughly every ceil(N/K) beats instead of every beat, still bounded and still matching the pump's own existing best-effort contract. No new C primitives, no messaging/Stadium changes. Verified: clean build on all 3 architectures, then the full 3x9x3 campaign re-run on all 3 (not just riscv64) per the standing rule that a defect repair requires a clean re-run everywhere before anything counts as closed: amd64 198s 0 faults 99/99 tokens x8 656/656 K-conserved aarch64 339s 0 faults 99/99 tokens x8 659/659 K-conserved riscv64 178s 0 faults 99/99 tokens x8 659/659 K-conserved riscv64 went from "never completes" to faster than aarch64, same campaign, same seed, same identity set. No regression on amd64/ aarch64. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXieurDfDSsDFdnSyusuWo
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
19916717b2
commit
71bcb72a59
+64
-33
@@ -293,6 +293,29 @@ static void sk_cursor_show(void)
|
||||
static int g_mama_interpreting; /* zero-initialized (BSS) */
|
||||
static int g_idle_pump_active; /* zero-initialized (BSS) */
|
||||
|
||||
/* FABRIC-3.md SXXII (2026-09-12): the messaging pump below used to walk
|
||||
* the ENTIRE live-VM registry every idle beat and dispatch a full
|
||||
* VM-EXEC "MSG-TICK" into every one of them -- O(N) per tick, forever,
|
||||
* with no cap. Fine at Tripod's original 3-VM scale; live-caught as a
|
||||
* real bottleneck at just 9 VMs on riscv64 (a std79 turn-attractor
|
||||
* campaign that completed in ~200-340s on amd64/aarch64 never finished
|
||||
* on riscv64 at the same VM count, while 8 VMs was fine there too --
|
||||
* the qemu-instance progress check (utime climbing steadily, ~66-75%
|
||||
* CPU) ruled out a hang; it was genuinely just this loop, at riscv64's
|
||||
* slower per-dispatch cost, dominating wall time). The real fleet is
|
||||
* headed toward hundreds of VMs, at which point this isn't a riscv64
|
||||
* quirk -- unconditional O(N) per second is a wall on every
|
||||
* architecture. Fixed by round-robin batching: SK_MSG_PUMP_BATCH live
|
||||
* VMs get dispatched per beat, not all of them, with a persistent
|
||||
* cursor picking up where the previous beat left off (wrapping around
|
||||
* the registry). Bounds both the scan and the dispatch cost to O(K)
|
||||
* regardless of total VM count -- the tradeoff is that any single VM's
|
||||
* own queue now drains roughly every ceil(N/K) beats instead of every
|
||||
* beat, which is still bounded and still matches the pump's own
|
||||
* existing "best-effort, resumes next beat" contract. */
|
||||
#define SK_MSG_PUMP_BATCH 4u /* max VM-EXEC MSG-TICK dispatches per idle beat */
|
||||
static uint32_t g_msg_pump_cursor; /* zero-initialized (BSS) */
|
||||
|
||||
static void sk_repl_idle(VM *active_vm)
|
||||
{
|
||||
/* Close any dangling output line before this bottom half emits its own
|
||||
@@ -582,44 +605,52 @@ static void sk_repl_idle(VM *active_vm)
|
||||
|
||||
g_idle_pump_active = 1;
|
||||
{
|
||||
uint32_t count = capsule_vm_registry_count();
|
||||
uint32_t i;
|
||||
for (i = 0; i < count; i++) {
|
||||
uint32_t count = capsule_vm_registry_count();
|
||||
uint32_t dispatched = 0;
|
||||
uint32_t scanned = 0;
|
||||
uint32_t i = (count > 0) ? (g_msg_pump_cursor % count) : 0;
|
||||
|
||||
while (count > 0 && dispatched < SK_MSG_PUMP_BATCH && scanned < count) {
|
||||
VMRegistryEntry ent;
|
||||
if (capsule_vm_registry_get_by_index(i, &ent) != 0) continue;
|
||||
if (ent.state != VM_STATE_LIVE) continue;
|
||||
if (ent.vm_ptr == (void *)mama) continue;
|
||||
/* FABRIC-3.md, 2026-09-09: a per-VM flag (e.g. "this one is
|
||||
* std79-locked") doesn't generalize -- any VM whose own
|
||||
* dictionary lacks MSG-TICK, for whatever reason (never
|
||||
* loaded common:messaging.4th, a future personality that
|
||||
* drops it, ACL denial), hits the identical failure. So
|
||||
* check the actual target VM's own dictionary fresh every
|
||||
* tick, the same source of truth the interpreter's own ACL
|
||||
* enforcement uses (vm_core.c), rather than pre-flag specific
|
||||
* personalities. Skip silently rather than spam "VM-EXEC:
|
||||
* ERROR in <name>" every idle tick forever for a VM that
|
||||
* plain doesn't have -- or isn't allowed -- MSG-TICK
|
||||
* (live-caught 2026-09-07/09). */
|
||||
{
|
||||
scanned++;
|
||||
|
||||
if (capsule_vm_registry_get_by_index(i, &ent) == 0 &&
|
||||
ent.state == VM_STATE_LIVE &&
|
||||
ent.vm_ptr != (void *)mama) {
|
||||
/* FABRIC-3.md, 2026-09-09: a per-VM flag (e.g. "this one is
|
||||
* std79-locked") doesn't generalize -- any VM whose own
|
||||
* dictionary lacks MSG-TICK, for whatever reason (never
|
||||
* loaded common:messaging.4th, a future personality that
|
||||
* drops it, ACL denial), hits the identical failure. So
|
||||
* check the actual target VM's own dictionary fresh every
|
||||
* tick, the same source of truth the interpreter's own ACL
|
||||
* enforcement uses (vm_core.c), rather than pre-flag specific
|
||||
* personalities. Skip silently rather than spam "VM-EXEC:
|
||||
* ERROR in <name>" every idle tick forever for a VM that
|
||||
* plain doesn't have -- or isn't allowed -- MSG-TICK
|
||||
* (live-caught 2026-09-07/09). */
|
||||
DictEntry *msgtick = vm_find_word((VM *)ent.vm_ptr, "MSG-TICK", 8);
|
||||
if (!msgtick || !msgtick->acl_allow) continue;
|
||||
if (msgtick && msgtick->acl_allow) {
|
||||
static const char PFX[] = "S\" MSG-TICK\" S\" ";
|
||||
static const char SFX[] = "\" VM-EXEC";
|
||||
char cmd[128];
|
||||
size_t p = 0;
|
||||
size_t name_len = strlen(ent.name);
|
||||
if (name_len > VM_NAME_MAX - 1u) name_len = VM_NAME_MAX - 1u;
|
||||
|
||||
memcpy(cmd + p, PFX, sizeof(PFX) - 1u); p += sizeof(PFX) - 1u;
|
||||
memcpy(cmd + p, ent.name, name_len); p += name_len;
|
||||
memcpy(cmd + p, SFX, sizeof(SFX) - 1u); p += sizeof(SFX) - 1u;
|
||||
cmd[p] = '\0';
|
||||
|
||||
vm_interpret(mama, cmd);
|
||||
dispatched++;
|
||||
}
|
||||
}
|
||||
|
||||
static const char PFX[] = "S\" MSG-TICK\" S\" ";
|
||||
static const char SFX[] = "\" VM-EXEC";
|
||||
char cmd[128];
|
||||
size_t p = 0;
|
||||
size_t name_len = strlen(ent.name);
|
||||
if (name_len > VM_NAME_MAX - 1u) name_len = VM_NAME_MAX - 1u;
|
||||
|
||||
memcpy(cmd + p, PFX, sizeof(PFX) - 1u); p += sizeof(PFX) - 1u;
|
||||
memcpy(cmd + p, ent.name, name_len); p += name_len;
|
||||
memcpy(cmd + p, SFX, sizeof(SFX) - 1u); p += sizeof(SFX) - 1u;
|
||||
cmd[p] = '\0';
|
||||
|
||||
vm_interpret(mama, cmd);
|
||||
i = (i + 1u) % count;
|
||||
}
|
||||
g_msg_pump_cursor = i;
|
||||
}
|
||||
/* FABRIC-3.md SXX: Hera now genuinely has her own MSG-ARENA/CH-ARENA
|
||||
* (the messaging-load root cause above is fixed -- see kernel_main.c's
|
||||
|
||||
Reference in New Issue
Block a user