ci: nightly images, nightly documents and tagged releases, in the tree
scripts/nightly.sh, one stage per call, a step each in the workflows:
accept <amd64|aarch64|riscv64|stick-high> the acceptance protocol --
qemu_session.py zuse, and stick_zuse --target qemu-stick-high;
after the last, dict_hash must be equal on the three
test-32 make -C v4 test-32
proofs isabelle build -c -D proof/ (StarForth checked afresh each run)
images the Pi 5 and Mars images, xz for Raspberry Pi Imager
publish org.starship:kernelImage: pc (iso), raspi5, milkv (img.xz),
logs (tar.gz: this run's serial logs, proof log, DoE CSVs)
docs, publish-docs book, vol1-3, api -> org.starship:documentation
.gitea/workflows/nightly.yml (03:00) and nightly-docs.yml (04:00) publish
<LITHOS_VERSION>-SNAPSHOT to starshipos-snapshots; release.yml, on a tag
v*, builds and publishes both to starshipos-releases at the tag's version,
refusing a tag that is not v<LITHOS_VERSION>. Until this branch is merged
to master, master carries copies of the two nightlies naming it.
make init: INIT_DOCS=0 leaves out TeX and Doxygen, INIT_PROOFS=0 Isabelle.
Checked here, stage by stage, in order: the four acceptance boots PASS,
dict_hash 0xfec8c47cfc6f437f on the three, test-32 0 failures, StarForth
proved (0:00:41), images built; publish to a local file:// repository,
and the images' publish to Nexus starshipos-snapshots read back byte for
byte. The docs stage and publish-docs: not yet checked when committed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GE3SDvQn5EoxdGufk84zz2
This commit is contained in:
rajames
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
deb71faeb2
commit
a5a29cc857
+129
-50
@@ -1,49 +1,127 @@
|
||||
#!/bin/bash
|
||||
# nightly.sh -- the nightly snapshot: boot-check each ISA, build the board images, publish them.
|
||||
# Three stages, run in this order (.gitea/workflows/nightly.yml runs each as a step):
|
||||
# nightly.sh -- the nightly and release builds: the acceptance protocol, the proofs, the board
|
||||
# images, the documents, and Nexus. One stage per call; the workflows run each as a step
|
||||
# (.gitea/workflows/nightly.yml, release.yml, nightly-docs.yml).
|
||||
#
|
||||
# scripts/nightly.sh boot <amd64|aarch64|riscv64>
|
||||
# Boots that ISA headless in QEMU (scripts/qemu_session.py headless: POST: PASSED, and no
|
||||
# prompt with no Zuse drive), in the foreground; one QEMU at a time, the acceptance order
|
||||
# amd64, aarch64, riscv64. A failure ends the nightly. amd64 also keeps the ISO it booted,
|
||||
# byte for byte, as build/nightly/StarshipOS.iso.
|
||||
# The images, in this order (any failure ends the run; nothing is published after it):
|
||||
#
|
||||
# scripts/nightly.sh accept <amd64|aarch64|riscv64|stick-high>
|
||||
# One boot of the acceptance protocol (.claude/CLAUDE.md), in the foreground, one QEMU at a
|
||||
# time, in that order: scripts/qemu_session.py zuse <arch> -- [zuse@Hera] ok>, `3 4 + .`
|
||||
# -> 7 ok, POST passed -- or stick_zuse amd64 --target qemu-stick-high, the PC stick loaded
|
||||
# above 2 GiB. amd64 keeps the ISO it booted, byte for byte, as build/nightly/StarshipOS.iso.
|
||||
# The last of the four (stick-high) also checks PARITY:V4_SYSTEM's dict_hash is the same
|
||||
# on amd64, aarch64 and riscv64.
|
||||
# scripts/nightly.sh test-32
|
||||
# make -C v4 test-32: v4 with 32-bit cells, every test (the acceptance's last part).
|
||||
# scripts/nightly.sh proofs
|
||||
# isabelle build -c -D proof/: the StarForth session checked afresh every run, whatever is
|
||||
# cached (HOL-Library is reused); its log is logs/<TS>/isabelle/, as a boot's is.
|
||||
# scripts/nightly.sh images
|
||||
# The board images, into build/nightly/:
|
||||
# raspi5.img.xz Raspberry Pi 5 (make boot_image TARGET=RASPI), for Raspberry Pi Imager
|
||||
# MilkV.img.xz Milk-V Mars (make boot_image TARGET=MILKV)
|
||||
# SHA256SUMS of StarshipOS.iso and both images, before and after compression
|
||||
# (Imager's extract_sha256 is the uncompressed one)
|
||||
# QEMU boots the aarch64 and riscv64 kernels, not these images: they are built from the
|
||||
# same sources, for hardware QEMU does not model.
|
||||
# scripts/nightly.sh publish <URL>
|
||||
# Deploys the three to a Maven repository (Nexus's starshipos-snapshots) as
|
||||
# org.starship:kernelImage:<LITHOS_VERSION>-SNAPSHOT, one classifier per board --
|
||||
# pc (iso), raspi5 (img.xz), milkv (img.xz) -- in one deploy, so they share a timestamp.
|
||||
# NEXUS_USER (default ci-publisher) and NEXUS_PASSWORD. Needs mvn and a JRE.
|
||||
# org.starship:kernelImage:<VERSION> to that Maven repository, in one deploy (one
|
||||
# timestamp): classifier pc (iso), raspi5 and milkv (img.xz), and logs (tar.gz: this
|
||||
# run's serial logs, proof log and DoE CSVs -- the evidence for the rest).
|
||||
#
|
||||
# NIGHTLY_VERSION overrides the version. The serial logs are logs/<TS>/<arch>/ and the DoE CSVs
|
||||
# experiments/bare_metal/runs/, as for any boot (.claude/CLAUDE.md). Needs what
|
||||
# make ARCH=amd64,aarch64,riscv64 init INIT_DOCS=0 installs.
|
||||
# The documents (a separate pipeline):
|
||||
#
|
||||
# scripts/nightly.sh docs
|
||||
# The book (make docs), the formal volumes (make -C docs/formal vols) and the API reference
|
||||
# (make -C docs/formal doxygen), into build/nightly-docs/.
|
||||
# scripts/nightly.sh publish-docs <URL>
|
||||
# org.starship:documentation:<VERSION>: classifier book, vol1, vol2, vol3, api (pdf).
|
||||
#
|
||||
# VERSION is <LITHOS_VERSION>-SNAPSHOT, or NIGHTLY_VERSION when set (a release sets it from its
|
||||
# tag). NEXUS_USER (default ci-publisher) and NEXUS_PASSWORD; publishing needs mvn and a JRE.
|
||||
# Needs what make ARCH=amd64,aarch64,riscv64 init installs (INIT_DOCS=0 for the images,
|
||||
# INIT_PROOFS=0 for the documents).
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
OUT=build/nightly
|
||||
DOCS=build/nightly-docs
|
||||
LITHOS=$(sed -n 's/^LITHOS_VERSION *?= *//p' kernel/Makefile)
|
||||
VERSION=${NIGHTLY_VERSION:-$LITHOS-SNAPSHOT}
|
||||
usage() { echo "Usage: scripts/nightly.sh boot <amd64|aarch64|riscv64> | images | publish <URL>" >&2; exit 2; }
|
||||
usage() {
|
||||
echo "Usage: scripts/nightly.sh accept <amd64|aarch64|riscv64|stick-high> | test-32 | proofs | images" >&2
|
||||
echo " | publish <URL> | docs | publish-docs <URL>" >&2
|
||||
exit 2
|
||||
}
|
||||
need() { for f in "$@"; do [ -f "$f" ] || { echo "nightly: no $f -- run the stages before this one" >&2; exit 1; }; done; }
|
||||
|
||||
# deploy <URL> <artifactId> <main file> <main ext> <main classifier> [file ext classifier]...
|
||||
deploy() {
|
||||
local url=$1 artifact=$2 file=$3 ext=$4 cls=$5 files= exts= clss= settings
|
||||
shift 5
|
||||
while [ $# -gt 0 ]; do files+=${files:+,}$1; exts+=${exts:+,}$2; clss+=${clss:+,}$3; shift 3; done
|
||||
: "${NEXUS_PASSWORD:?publishing needs NEXUS_PASSWORD}"
|
||||
command -v mvn >/dev/null || { echo "nightly: publishing needs mvn (apt-get install maven)" >&2; exit 1; }
|
||||
settings=$(mktemp)
|
||||
cat > "$settings" <<'EOF'
|
||||
<settings><servers><server>
|
||||
<id>starship-nexus</id>
|
||||
<username>${env.NEXUS_USER}</username>
|
||||
<password>${env.NEXUS_PASSWORD}</password>
|
||||
</server></servers></settings>
|
||||
EOF
|
||||
export NEXUS_USER=${NEXUS_USER:-ci-publisher}
|
||||
echo "=== nightly: publish org.starship:$artifact:$VERSION -> $url"
|
||||
mvn -B -q -s "$settings" org.apache.maven.plugins:maven-deploy-plugin:3.1.2:deploy-file \
|
||||
-Durl="$url" -DrepositoryId=starship-nexus \
|
||||
-DgroupId=org.starship -DartifactId="$artifact" -Dversion="$VERSION" \
|
||||
-Dfile="$file" -Dpackaging="$ext" -Dclassifier="$cls" \
|
||||
${files:+-Dfiles="$files" -Dtypes="$exts" -Dclassifiers="$clss"} -DgeneratePom=true \
|
||||
|| { rm -f "$settings"; exit 1; }
|
||||
rm -f "$settings"
|
||||
echo "=== nightly: published"
|
||||
}
|
||||
|
||||
case ${1:-} in
|
||||
boot)
|
||||
arch=${2:-}
|
||||
case $arch in amd64|aarch64|riscv64) ;; *) usage ;; esac
|
||||
[ "$arch" != amd64 ] || { rm -rf "$OUT"; mkdir -p "$OUT"; } # amd64 is the first stage
|
||||
echo "=== nightly: org.starship:kernelImage:$VERSION from $(git rev-parse --short HEAD); boot check $arch"
|
||||
git checkout -q -- disk/zuse-scratch # the committed Zuse pair, as before each acceptance boot
|
||||
python3 scripts/qemu_session.py headless "$arch" --label "nightly-$arch" --make "QEMU_DISPLAY=none"
|
||||
[ "$arch" != amd64 ] || cp build/amd64/kernel/starkernel.iso "$OUT/StarshipOS.iso"
|
||||
accept)
|
||||
what=${2:-}
|
||||
case $what in
|
||||
amd64) rm -rf "$OUT"; mkdir -p "$OUT"; touch "$OUT/.started" # the first stage
|
||||
mode=zuse; arch=amd64; extra= ;;
|
||||
aarch64|riscv64) mode=zuse; arch=$what; extra= ;;
|
||||
stick-high) mode=stick_zuse; arch=amd64; extra="--target qemu-stick-high" ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
need "$OUT/.started"
|
||||
echo "=== nightly: org.starship:kernelImage:$VERSION from $(git rev-parse --short HEAD); accept $what"
|
||||
git checkout -q -- disk/zuse-scratch # the committed Zuse pair, before each architecture
|
||||
python3 scripts/qemu_session.py $mode $arch $extra --label "nightly-$what" --make "QEMU_DISPLAY=none" \
|
||||
| tee "$OUT/accept-$what.txt"
|
||||
[ "$what" != amd64 ] || cp build/amd64/kernel/starkernel.iso "$OUT/StarshipOS.iso"
|
||||
if [ "$what" = stick-high ]; then
|
||||
# the three architectures' boots of this run: one dict_hash
|
||||
hashes=$(for a in amd64 aarch64 riscv64; do
|
||||
log=$(sed -n 's/^LOG: //p' "$OUT/accept-$a.txt")
|
||||
grep -aoE "PARITY:V4_SYSTEM [^[:cntrl:]]*" "$log" | sed -n 's/.*dict_hash=//p'
|
||||
done | sort -u)
|
||||
echo "=== nightly: dict_hash on the three: $hashes"
|
||||
[ "$(echo "$hashes" | wc -l)" = 1 ] && [ -n "$hashes" ] \
|
||||
|| { echo "nightly: the three architectures do not agree on dict_hash" >&2; exit 1; }
|
||||
fi
|
||||
;;
|
||||
test-32)
|
||||
need "$OUT/.started"
|
||||
make --no-print-directory -C v4 test-32 2>&1 | tee "$OUT/test-32.txt"
|
||||
echo "=== nightly: test-32: $(grep -cE '[0-9]+ checks, 0 failures' "$OUT/test-32.txt") programs, 0 failures"
|
||||
;;
|
||||
proofs)
|
||||
need "$OUT/.started"
|
||||
ts=$(date -u +%Y%m%d-%H%M%S)
|
||||
mkdir -p "logs/$ts/isabelle"
|
||||
isabelle build -c -v -D proof/ 2>&1 | tee "logs/$ts/isabelle/isabelle-$ts.log"
|
||||
;;
|
||||
images)
|
||||
[ -f "$OUT/StarshipOS.iso" ] || { echo "nightly: no $OUT/StarshipOS.iso -- run boot amd64 first" >&2; exit 1; }
|
||||
need "$OUT/StarshipOS.iso"
|
||||
for b in "RASPI raspi/lithos-raspi.img raspi5.img" "MILKV milkv/lithos-milkv.img MilkV.img"; do
|
||||
set -- $b
|
||||
echo "=== nightly: image $1"
|
||||
@@ -58,31 +136,32 @@ images)
|
||||
(cd "$OUT" && sha256sum raspi5.img.xz MilkV.img.xz >> SHA256SUMS && cat SHA256SUMS)
|
||||
;;
|
||||
publish)
|
||||
url=${2:-}
|
||||
[ -n "$url" ] || usage
|
||||
: "${NEXUS_PASSWORD:?publish needs NEXUS_PASSWORD}"
|
||||
command -v mvn >/dev/null || { echo "nightly: publish needs mvn (apt-get install maven)" >&2; exit 1; }
|
||||
for f in StarshipOS.iso raspi5.img.xz MilkV.img.xz; do
|
||||
[ -f "$OUT/$f" ] || { echo "nightly: no $OUT/$f -- run boot and images first" >&2; exit 1; }
|
||||
done
|
||||
settings=$(mktemp)
|
||||
trap 'rm -f "$settings"' EXIT
|
||||
cat > "$settings" <<'EOF'
|
||||
<settings><servers><server>
|
||||
<id>starship-nexus</id>
|
||||
<username>${env.NEXUS_USER}</username>
|
||||
<password>${env.NEXUS_PASSWORD}</password>
|
||||
</server></servers></settings>
|
||||
EOF
|
||||
export NEXUS_USER=${NEXUS_USER:-ci-publisher}
|
||||
echo "=== nightly: publish org.starship:kernelImage:$VERSION -> $url"
|
||||
mvn -B -q -s "$settings" org.apache.maven.plugins:maven-deploy-plugin:3.1.2:deploy-file \
|
||||
-Durl="$url" -DrepositoryId=starship-nexus \
|
||||
-DgroupId=org.starship -DartifactId=kernelImage -Dversion="$VERSION" \
|
||||
-Dfile="$OUT/StarshipOS.iso" -Dpackaging=iso -Dclassifier=pc \
|
||||
-Dfiles="$OUT/raspi5.img.xz,$OUT/MilkV.img.xz" -Dclassifiers=raspi5,milkv -Dtypes=img.xz,img.xz \
|
||||
-DgeneratePom=true
|
||||
echo "=== nightly: published"
|
||||
url=${2:-}; [ -n "$url" ] || usage
|
||||
need "$OUT/StarshipOS.iso" "$OUT/raspi5.img.xz" "$OUT/MilkV.img.xz"
|
||||
# the evidence: what this run wrote under logs/ and experiments/bare_metal/runs/
|
||||
find logs experiments/bare_metal/runs -type f -newer "$OUT/.started" -print0 \
|
||||
| tar --null -czf "$OUT/logs.tar.gz" -T -
|
||||
echo "=== nightly: logs.tar.gz: $(tar -tzf "$OUT/logs.tar.gz" | wc -l) files"
|
||||
deploy "$url" kernelImage "$OUT/StarshipOS.iso" iso pc \
|
||||
"$OUT/raspi5.img.xz" img.xz raspi5 \
|
||||
"$OUT/MilkV.img.xz" img.xz milkv \
|
||||
"$OUT/logs.tar.gz" tar.gz logs
|
||||
;;
|
||||
docs)
|
||||
rm -rf "$DOCS"; mkdir -p "$DOCS"
|
||||
make --no-print-directory docs
|
||||
make --no-print-directory -C docs/formal vols doxygen
|
||||
cp build/docs/LithosAnanke.pdf "$DOCS/book.pdf"
|
||||
for v in 1 2 3; do cp "docs/formal/build/vol$v/main.pdf" "$DOCS/vol$v.pdf"; done
|
||||
cp docs/formal/build/doxygen/StarForth-API-Reference.pdf "$DOCS/api.pdf"
|
||||
(cd "$DOCS" && sha256sum ./*.pdf | tee SHA256SUMS)
|
||||
;;
|
||||
publish-docs)
|
||||
url=${2:-}; [ -n "$url" ] || usage
|
||||
need "$DOCS/book.pdf" "$DOCS/vol1.pdf" "$DOCS/vol2.pdf" "$DOCS/vol3.pdf" "$DOCS/api.pdf"
|
||||
deploy "$url" documentation "$DOCS/book.pdf" pdf book \
|
||||
"$DOCS/vol1.pdf" pdf vol1 "$DOCS/vol2.pdf" pdf vol2 "$DOCS/vol3.pdf" pdf vol3 \
|
||||
"$DOCS/api.pdf" pdf api
|
||||
;;
|
||||
*) usage ;;
|
||||
esac
|
||||
|
||||
Reference in New Issue
Block a user