feat(v4, ci): the fault dump on a node's stop, and the EMERGENCY diagnostic build after a failed stage
Captain Bob's CI directive (2026-11-17) and the design he ruled point by point (docs/v4.0.0/plans/2026-11-17-emergency-diagnostic-build.md). - v4/system/dump.c, v4/include/v4/dump.h: what a node was doing when it stopped -- the fault's kind, address and count; P A B; the data stack, T first; the return stack, R first, each entry named by the word it falls in (the backtrace; ROM and below-every-word said so); the running line. Bounded dictionary walk; no C library. Ruled: every build, on a stop only (a handled fault keeps its line). - kernel/src/v4/sk_v4.c: the dump before each of the three halts (not started; POST's blocks not let go; a node stopped on a fault) -- Hera and any stopped node. kernel/Makefile links dump.o. - scripts/nightly.sh diagnose [<stage>]: after a failed stage, that configuration built again with EMERGENCY_CONSOLE_ENABLED=1 -- build only, nothing booted (ruled) -- its artifacts kept in logs/<TS>/diagnose/ only from a build that passed, with a README saying how to launch it with --log-level=debug. Always exits 0: evidence, never a result. The protocol runs it after a failed stage and still exits with that stage's code; acceptance.yml, nightly.yml and release.yml run it `if: failure()`, before the logs are kept. Any failed stage triggers it (ruled). - capsules/BLOCK_MAP.md: as the build writes it -- the v4:physics.4th row, missing since28ffed8f(ruled: commit the regenerated map). - Test first: test_dump.c failed with no dump.c (logs/20261117-dump), then 15 checks at both widths and under ASan+UBSan. The diagnose stage: a real EMERGENCY amd64 build; a failing build exits 0 and keeps no artifacts (a defect found and fixed this way); the protocol path keeps the stage's exit. - Also the write-up of the regression at046ce0be, for ruling. Not yet: acceptance on a clean tree (the next run); the kernel's stop wiring is compile-tested only -- no boot reaches a stop on purpose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RzZgrf7V7Zk2iczBXDRbm
This commit is contained in:
@@ -42,6 +42,16 @@
|
||||
# run's serial logs and DoE CSVs -- the evidence for the rest; and the proof log, when the
|
||||
# proofs ran before it, as a release runs them).
|
||||
#
|
||||
# scripts/nightly.sh diagnose [<stage that failed>]
|
||||
# After a failed stage: the EMERGENCY diagnostic build (docs/v4.0.0/plans/
|
||||
# 2026-11-17-emergency-diagnostic-build.md; Captain Bob, 2026-11-17). The failed configuration
|
||||
# is built again with EMERGENCY_CONSOLE_ENABLED=1 -- the architecture of a failed accept, or all
|
||||
# three for any other stage -- and its artifacts kept in logs/<TS>/diagnose/: the loader, the
|
||||
# kernel ELF with its symbols, the capsule map, each build's log and result, and a README that
|
||||
# says how to launch it with diagnostic logging (--log-level=debug, as always). BUILD ONLY:
|
||||
# nothing is booted (ruled). It is evidence, never a result: it always exits 0, whatever its own
|
||||
# build did, and the stage that failed stays the run's failure.
|
||||
#
|
||||
# The documents (a separate pipeline):
|
||||
#
|
||||
# scripts/nightly.sh docs
|
||||
@@ -83,6 +93,7 @@ usage() {
|
||||
echo "Usage: scripts/nightly.sh accept <amd64|aarch64|riscv64|stick-high> | test-32 | v4-checks | proofs | images" >&2
|
||||
echo " | publish <URL> | docs | publish-docs <URL> | gitea-release <tag>" >&2
|
||||
echo " | protocol [<stage>...] | protocol-report | dict-hash | post-count" >&2
|
||||
echo " | diagnose [<stage that failed>]" >&2
|
||||
echo " protocol stages: $PROTOCOL_STAGES" >&2
|
||||
exit 2
|
||||
}
|
||||
@@ -308,6 +319,62 @@ publish-docs)
|
||||
"$DOCS/vol1.pdf" pdf vol1 "$DOCS/vol2.pdf" pdf vol2 "$DOCS/vol3.pdf" pdf vol3 \
|
||||
"$DOCS/api.pdf" pdf api
|
||||
;;
|
||||
diagnose)
|
||||
# The EMERGENCY diagnostic build: see the header. Evidence only -- it exits 0 whatever happens.
|
||||
failed=${2:-}
|
||||
if [ -z "$failed" ]; then # the workflows: the first accept that did not pass
|
||||
for b in amd64 aarch64 riscv64 stick-high; do
|
||||
grep -q '^RESULT: PASS' "$OUT/accept-$b.txt" 2>/dev/null || { failed=accept-$b; break; }
|
||||
done
|
||||
failed=${failed:-unknown}
|
||||
fi
|
||||
case $failed in
|
||||
accept-stick-high|accept-amd64) archs=amd64 ;;
|
||||
accept-aarch64) archs=aarch64 ;;
|
||||
accept-riscv64) archs=riscv64 ;;
|
||||
*) archs="amd64 aarch64 riscv64" ;;
|
||||
esac
|
||||
ts=$(date -u +%Y%m%d-%H%M%S); d=logs/$ts/diagnose; mkdir -p "$d"
|
||||
echo "=== nightly: diagnose after $failed: the EMERGENCY build of $archs (build only)"
|
||||
{
|
||||
echo "# EMERGENCY diagnostic build -- evidence, not a result"
|
||||
echo
|
||||
echo "After: \`$failed\` failed (that failure is the run's result). Commit: $(git rev-parse HEAD)."
|
||||
echo "Built with EMERGENCY_CONSOLE_ENABLED=1; nothing was booted (build only, ruled 2026-11-17)."
|
||||
echo
|
||||
echo "To launch it by hand, with diagnostic logging, one QEMU at a time, in the foreground:"
|
||||
echo
|
||||
for a in $archs; do
|
||||
echo " timeout 570 make -f kernel/Makefile ARCH=$a EMERGENCY_CONSOLE_ENABLED=1 KERNEL_ARGS=--log-level=debug qemu"
|
||||
done
|
||||
echo
|
||||
echo "| Architecture | Build | Log |"
|
||||
echo "|---|---|---|"
|
||||
} > "$d/README.md"
|
||||
for a in $archs; do
|
||||
set +e
|
||||
make --no-print-directory -f kernel/Makefile ARCH=$a EMERGENCY_CONSOLE_ENABLED=1 clean all > "$d/build-$a.log" 2>&1
|
||||
rc=$?
|
||||
set -e
|
||||
if [ $rc = 0 ]; then st=PASS; else st="FAIL (exit $rc)"; fi
|
||||
# the artifacts only of a build that passed: what is in build/$a after a failed one is an older
|
||||
# build's, and must not pass for this one
|
||||
if [ $rc = 0 ]; then
|
||||
mkdir -p "$d/$a"
|
||||
for f in starkernel_loader.efi starkernel_kernel.elf; do
|
||||
[ -f "build/$a/kernel/$f" ] && cp "build/$a/kernel/$f" "$d/$a/" || true
|
||||
done
|
||||
[ -f "build/$a/.config" ] && cp "build/$a/.config" "$d/$a/config" || true
|
||||
cp capsules/BLOCK_MAP.md "$d/$a/" 2>/dev/null || true
|
||||
else
|
||||
st="$st -- no artifacts (the build did not complete)"
|
||||
fi
|
||||
echo "$rc" > "$d/build-$a.rc"
|
||||
echo "| $a | $st | \`$d/build-$a.log\` |" >> "$d/README.md"
|
||||
echo "=== nightly: diagnose $a: $st"
|
||||
done
|
||||
exit 0
|
||||
;;
|
||||
protocol)
|
||||
# The developer protocol (#135): the stages a PR, the nightly and a release run, never publish.
|
||||
shift
|
||||
@@ -337,6 +404,7 @@ protocol)
|
||||
if [ "$rc" != 0 ]; then
|
||||
echo "nightly: protocol stage $s failed (exit $rc)" >&2
|
||||
[ $all = 0 ] || "$self" protocol-report || true
|
||||
"$self" diagnose "$s" || true # evidence only: the stage's failure stays the result
|
||||
exit "$rc"
|
||||
fi
|
||||
done
|
||||
|
||||
Reference in New Issue
Block a user