CI: a passing acceptance is turned red by its log upload, which sends every log in the tree #184

New Issue

2026-11-12T00:47:07Z

rajames commented

2026-11-12 00:47:07 +00:00

Found

Acceptance run 250 (PR 164, 4d42d25c) passed every stage -- the four boots and test-32 (39 programs, 0 failures) -- and was marked failed by its last step, Keep the logs and DoE CSVs (actions/upload-artifact@v3, .gitea/workflows/acceptance.yml:70). Gitea's artifact service timed out finalizing the upload, then answered 500 on four retries: Finalize artifact upload failed: Artifact service responded with 500. Run 252 uploaded the same paths and succeeded, so it is intermittent.

The step uploads all of logs/ and experiments/bare_metal/runs/ -- 2,620 files, every run in the tree's history, not this run's -- which is the likely reason the finalize request timed out. The nightly (nightly.yml:91) and the release do the same. Those logs are committed to git already; the artifact is a convenience copy, yet a failed upload turns a passing gate red.

Done when

The uploads keep only what this run wrote (as scripts/nightly.sh publish does, -newer build/nightly/.started; or logs/<TS>/ of this run)
Decided: a failed artifact upload fails the gate, or is a warning that cannot turn a passing acceptance red -- and the decision written in the workflows' comments
Seen: an acceptance run whose upload is a handful of files, not thousands

🤖 Filed with Claude Code -- found checking PR 164's acceptance runs (Captain Bob, 2026-11-12: "file the artifact issue").

### Found Acceptance run 250 (PR 164, `4d42d25c`) passed every stage -- the four boots and test-32 (39 programs, 0 failures) -- and was marked failed by its last step, **Keep the logs and DoE CSVs** (`actions/upload-artifact@v3`, `.gitea/workflows/acceptance.yml:70`). Gitea's artifact service timed out finalizing the upload, then answered 500 on four retries: `Finalize artifact upload failed: Artifact service responded with 500`. Run 252 uploaded the same paths and succeeded, so it is intermittent. The step uploads all of `logs/` and `experiments/bare_metal/runs/` -- 2,620 files, every run in the tree's history, not this run's -- which is the likely reason the finalize request timed out. The nightly (`nightly.yml:91`) and the release do the same. Those logs are committed to git already; the artifact is a convenience copy, yet a failed upload turns a passing gate red. ### Done when - [ ] The uploads keep only what this run wrote (as `scripts/nightly.sh publish` does, `-newer build/nightly/.started`; or `logs/<TS>/` of this run) - [ ] Decided: a failed artifact upload fails the gate, or is a warning that cannot turn a passing acceptance red -- and the decision written in the workflows' comments - [ ] Seen: an acceptance run whose upload is a handful of files, not thousands 🤖 Filed with [Claude Code](https://claude.com/claude-code) -- found checking PR 164's acceptance runs (Captain Bob, 2026-11-12: "file the artifact issue").

rajames added the

labels 2026-11-12 00:47:08 +00:00

rajames added this to the v4.0.0-alpha.5: SDK milestone 2026-11-12 11:43:39 +00:00

Sign in to join this conversation.