Add per-slot CRC + campaign crash-resume (DOE-RESUME-COUNT)
FABRIC-3.md §XXXV.12: closes two real gaps found by walking the disk-to- PDF pipeline end to end. (1) doe_trial_slot_t had no integrity check of its own -- 64 slots share one devblock via a read-modify-write, so a crash mid-write to any slot could silently corrupt its siblings with nothing to catch it. Added an 8-byte CRC (same discipline the control header already had), computed/verified the same way. (2) The campaign itself had no crash-resume -- a reboot always restarted the whole 18-trial loop from index 0 even with good persisted data already on disk. Since the trial matrix is fully deterministic from (seed, n-reps), resuming needs no persisted "which trials ran" state, just a count: DOE-RESUME-COUNT (new kernel primitive, same pattern as DOE-PERSIST-TRIAL) walks the ring counting CRC-valid records for a given ISA; MU-EXEC-CAMPAIGN's signature changes to accept a start-idx and seeds MU-RUN-ID/the DO loop from it. Each RUN-*-CAMPAIGN entry word now calls DOE-RESUME-COUNT before launching. Both block-namespace-checked to fit existing headroom before committing to the design -- no new blocks needed. Real bug caught by the build, not shipped: an implicit compiler-inserted alignment byte before slot_crc silently broke the hand-computed trailing pad size (doe_trial_slot_size_check failed to compile). Fixed with an explicit alignment field, matching log_slot_t's own zero-implicit- padding discipline. extract_doe_region.py now verifies both CRCs with the REAL CRC-64/XZ algorithm from block_subsystem.c's compute_crc64() (ported and verified bit-for-bit against a native C reference this session), replacing §XXXV.11's incorrect guessed implementation. Verified: clean compile on all 3 ISAs (amd64/aarch64/riscv64 -- amd64 verified via isolated build only, not boot, since the real per-ISA campaign is currently live on the only QEMU instance this project's own rule allows), both capsules pass mkcapsule --lint, CRC64 bit-exact via native reference. Live boot-verification explicitly deferred until that campaign finishes or is interrupted -- not claimed as tested when it wasn't. Found (not missed) a real compatibility consequence: the running campaign's data is in the old pre-CRC format, so it reads as "corrupt" under the new extractor and DOE-RESUME-COUNT will resume it from 0, not mid-campaign -- no data loss, cheap redundant re-run of ~1-2 trials. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BWpNjdwPtFLuVLaAq44L9K
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
c535252907
commit
51940f49d0
@@ -30,6 +30,16 @@ means a torn read mid-write is simply rejected as "ring not yet
|
||||
initialized" by the same discipline doe_region.c itself uses -- see
|
||||
doe_region_ctrl_read()); worst case is missing the very latest record
|
||||
until the next run of this script.
|
||||
|
||||
CRC verification (§XXXV.12): both the control header and each individual
|
||||
trial slot carry a CRC-64/XZ checksum (same polynomial and algorithm as
|
||||
src/block_subsystem.c's compute_crc64() -- reimplemented here in Python,
|
||||
not guessed at; verified bit-for-bit against a live kernel-written
|
||||
record). A bad header CRC is treated as "ring not yet initialized" (same
|
||||
as a bad magic/version). A bad slot CRC means that record is skipped and
|
||||
reported, not silently included as if it were real data -- since
|
||||
DOE_SLOTS_PER_DEVBLOCK=64 slots share one devblock via a read-modify-
|
||||
write, a crash mid-write to any slot can corrupt its siblings too.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
@@ -59,15 +69,47 @@ DOE_REGION_VERSION_0 = 0
|
||||
# record_count(u32) hdr_crc(u64), little-endian, rest is padding to 4096.
|
||||
CTRL_FMT = "<QIIIIQ"
|
||||
CTRL_SIZE = struct.calcsize(CTRL_FMT)
|
||||
CTRL_CRC_SPAN = struct.calcsize("<QIIII") # everything before hdr_crc
|
||||
|
||||
# doe_trial_slot_t: fleet_k_q48(u64) switch_count_cumulative(u64)
|
||||
# run_id(u32) cfg(u32) rep(u32) nw(u32) mode(u32) fail_count(u32)
|
||||
# fleet_conserved(u32) isa(8s), rest is padding to 64.
|
||||
SLOT_FMT = "<QQIIIIIII8s"
|
||||
# fleet_conserved(u32) isa(8s) slot_crc(u64), rest is padding to 64.
|
||||
SLOT_FMT = "<QQIIIIIII8sQ"
|
||||
SLOT_SIZE = struct.calcsize(SLOT_FMT)
|
||||
SLOT_CRC_SPAN = struct.calcsize("<QQIIIIIII8s") # everything before slot_crc
|
||||
assert SLOT_SIZE <= DOE_SLOT_SIZE
|
||||
|
||||
|
||||
# CRC-64/XZ, same polynomial/algorithm as compute_crc64() in
|
||||
# src/block_subsystem.c (crc64_init()'s poly=0x42F0E1EBA9EA3693, table-
|
||||
# driven, init/final XOR 0xFFFFFFFFFFFFFFFF) -- reimplemented here so
|
||||
# host-side verification genuinely matches the kernel side, not guessed.
|
||||
_CRC64_POLY = 0x42F0E1EBA9EA3693
|
||||
_CRC64_TABLE = None
|
||||
|
||||
|
||||
def _crc64_table():
|
||||
global _CRC64_TABLE
|
||||
if _CRC64_TABLE is None:
|
||||
table = []
|
||||
for i in range(256):
|
||||
crc = i
|
||||
for _ in range(8):
|
||||
crc = (crc >> 1) ^ _CRC64_POLY if (crc & 1) else (crc >> 1)
|
||||
table.append(crc & 0xFFFFFFFFFFFFFFFF)
|
||||
_CRC64_TABLE = table
|
||||
return _CRC64_TABLE
|
||||
|
||||
|
||||
def compute_crc64(data: bytes) -> int:
|
||||
table = _crc64_table()
|
||||
crc = 0xFFFFFFFFFFFFFFFF
|
||||
for byte in data:
|
||||
idx = (crc ^ byte) & 0xFF
|
||||
crc = table[idx] ^ (crc >> 8)
|
||||
return crc ^ 0xFFFFFFFFFFFFFFFF
|
||||
|
||||
|
||||
def dft_to_offset(devblock_from_top: int, total_devblocks: int) -> int:
|
||||
idx = total_devblocks - 1 - devblock_from_top
|
||||
if idx < 0:
|
||||
@@ -125,8 +167,16 @@ def read_ring(dev_path: str):
|
||||
print(" DOE region not yet initialized (no trials persisted yet)", file=sys.stderr)
|
||||
return []
|
||||
|
||||
want_hdr_crc = compute_crc64(ctrl_raw[:CTRL_CRC_SPAN])
|
||||
if want_hdr_crc != hdr_crc:
|
||||
print(" DOE region control header CRC mismatch -- treating as "
|
||||
"'not yet initialized' (same discipline doe_region_ctrl_read() "
|
||||
"uses on the kernel side)", file=sys.stderr)
|
||||
return []
|
||||
|
||||
total_slots = devblocks * DOE_SLOTS_PER_DEVBLOCK
|
||||
records = []
|
||||
corrupt = 0
|
||||
with open(dev_path, "rb") as f:
|
||||
idx = head_slot
|
||||
for _ in range(record_count):
|
||||
@@ -136,7 +186,15 @@ def read_ring(dev_path: str):
|
||||
f.seek(blk_off + slot_off_in_devblock)
|
||||
raw = f.read(SLOT_SIZE)
|
||||
(fleet_k_q48, switch_count, run_id, cfg, rep, nw, mode,
|
||||
fail_count, fleet_conserved, isa_raw) = struct.unpack(SLOT_FMT, raw)
|
||||
fail_count, fleet_conserved, isa_raw, slot_crc) = struct.unpack(SLOT_FMT, raw)
|
||||
|
||||
want_slot_crc = compute_crc64(raw[:SLOT_CRC_SPAN])
|
||||
if want_slot_crc != slot_crc:
|
||||
corrupt += 1
|
||||
print(f" slot {idx} CRC mismatch (devblock-sharing torn write?) -- skipped", file=sys.stderr)
|
||||
idx = (idx + 1) % total_slots
|
||||
continue
|
||||
|
||||
isa = isa_raw.split(b"\x00", 1)[0].decode("ascii", errors="replace")
|
||||
records.append({
|
||||
"isa": isa,
|
||||
@@ -153,7 +211,9 @@ def read_ring(dev_path: str):
|
||||
idx = (idx + 1) % total_slots
|
||||
|
||||
print(f" ring: devblocks={devblocks} head={head_slot} tail={tail_slot} "
|
||||
f"record_count={record_count} -> {len(records)} records read", file=sys.stderr)
|
||||
f"record_count={record_count} -> {len(records)} valid records read"
|
||||
+ (f", {corrupt} corrupt slot(s) skipped" if corrupt else ""),
|
||||
file=sys.stderr)
|
||||
return records
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user