LithosAnanake/docs/v4.0.0/NUCLEUS.md

StarForth v4.0.0 — Nucleus, FORTH-79 Capsule and POST

Design, 2026-10-05. Ruled by Captain Bob in conversation the same day; this file records those rulings. It sits beside JUSTIFICATION.md (why v4) and DECOMPOSITION.md (every v3 word's fate), and changes how the vocabulary DECOMPOSITION.md describes is delivered, not what the words do.

1. Goal

v4 comes up on amd64, aarch64 and riscv64, both as the bare-metal kernel under QEMU and as the hosted Linux binary, in this order:

1. start the assembled nucleus;
2. load FORTH-79 from a capsule;
3. run POST on FORTH-79;
4. reach the ok> prompt.

That is all this work delivers. Both the kernel and the hosted binary are products.

2. What is wrong today

• All 292 named words are written in F18 assembler text (v4/capsule/*.v4). Only the editor and SEE are FORTH source.
• v4/tools/mkimage.c assembles and compiles everything on the build machine. The kernel links the finished memory image. Nothing is loaded from a capsule at boot.
• There is no v4 POST.
• No QEMU log of a v4 boot exists, so bare-metal v4 has never been accepted.

3. Scope

In scope: the FORTH-79 Required Word Set, as colon definitions in a capsule, and a POST for it.

Out of scope until this is accepted: the Double Number extension set, Q48 arithmetic, logging, access control, the editor, SEE, DEFER, and every other StarForth extension. Their .v4 and .fth sources stay in the tree but are not built into the nucleus image or loaded at boot.

4. The nucleus

The nucleus is the part of the vocabulary that stays in assembler. A word is in the nucleus only if one of these holds.

1. It is an opcode or a register access. v4 compiles colon definitions to native instruction words, so FORTH source cannot write an opcode. DUP DROP OVER + AND XOR 2* 2/ @ ! >R R> R@, the stack-depth registers, and the console registers behind KEY and EMIT.
2. The node needs it to read a capsule. WORD FIND NUMBER INTERPRET QUIT, the error trap, HERE ALLOT ,, CREATE, :, ;, IMMEDIATE, LITERAL, [, ], ( and the code generator.
3. It lays down code for a colon definition in the capsule. The control words become colon definitions, and they need the code generator's words by name: (OP,), (LIT,), (LABEL), (BRANCH>), (RESOLVE), (JUMP,), (CALL,), and the control-flow stack words >CF, CF> and (PAIR). These get headers. They are not FORTH-79 words.

Everything else in the Required Word Set is a colon definition in the capsule. That includes words that are assembler today only for speed: SWAP ROT - * /MOD 0= < C@ C! CMOVE FILL . <# # #> TYPE COUNT VARIABLE CONSTANT, the control words, the vocabulary words and the block words.

The capsule's definitions will run slower than the assembled ones they replace. That is accepted: the goal is the minimum in assembler.

The list in rule 2 is the starting boundary, not the final one. A word leaves the nucleus whenever a colon definition of it passes POST (section 8, step 3). The nucleus is at its minimum when no remaining word can be moved.

5. Capsules

5.1 Files

File	Capsule name	Contents
capsules/v4/forth79.4th	v4:forth79.4th	The Required Word Set as colon definitions
capsules/v4/post79.4th		Gone 2026-10-07: POST is the kernel's (6.3); its cases are v4/system/post_cases.c

Both are ordinary .4th capsules: Block N headers, lines of at most 64 characters, at most 16 lines to a block. tools/mkcapsule.c bakes them into capsule_generated.c with every other file under capsules/, hashed, and signed when the signing key is present. v3 does not load them: it runs init.4th and only what that file EXECs.

Named blobs (any non-.4th file under capsules/) are baked the same way and stay available to v4 by name. This work does not need one.

5.2 The block-number rule

validate_forth_blocks (tools/mkcapsule.c:409) accepts block numbers in [2048, 5120). The floor is real: blocks 0–2047 are the VM's fast RAM. The ceiling matches no device.

Change: the check becomes "block number is at least 2048". There is no upper bound. Blocks 0–2047 are the only forbidden ones. The collision checks against other capsules and tools/capsule-reserved.txt are unchanged. experiments/bare_metal/README.md and .claude/CLAUDE.md state the old range and are corrected with it.

The v4 capsules take a free range found from capsules/BLOCK_MAP.md and tools/capsule-reserved.txt.

5.3 Loading

A capsule reaches the node as source lines through its console input, the way v3's capsule_exec_payload interprets a payload line by line. LOAD is not used: the block words are themselves in the capsule.

For each capsule, in order:

1. find it by name (capsule_find_by_name);
2. recompute its hash (capsule_validate); a mismatch stops the boot;
3. check its signature (capsule_verify_signature); invalid stops the boot, missing warns — v3's rule at capsule_birth.c:585;
4. split the payload on Block N headers and feed each line to the node, running the node until it waits for input again;
5. require the node to answer ok to every line.

A line that is not accepted stops the boot and prints the capsule name, block number, line number and what the node said. v3's retry, which skips a failing line and runs the block again, is not carried over: a skipped line in the standard's own capsule must not pass unnoticed.

What the node prints while loading is not shown, except on failure.

In v4 the block numbers are labels for now. They become storage addresses when the block words exist and a capsule is copied to block storage.

6. POST

6.1 Source of the cases

POST is v3's own test cases, ported. The cases are in v3/src/test_runner/modules/*.c; each is a name, a line of FORTH and a flag saying whether an error is expected. Only cases for Required Word Set words are ported, chosen by word name against the standard's list.

6.2 Judging

v3 judges a case only by whether it raised an error (v3/src/test_runner/test_common.c:258-285); the "expected" text is a comment. A wrong result that raises no error passes. That is not enough for POST's purpose here, which is to prove each colon definition against the assembled word it replaces.

Ruling: every ported case compares its result. The expected data stack and printed output are taken from running the same line on the hosted v3 binary, so v3 remains the reference. A case v3 expects to raise an error must raise one on v4.

Where v3 departs from FORTH-79 for a standard word, FORTH-79 wins (standing ruling); the case's expected value is then the standard's, and the departure is reported.

6.3 Form

Ruled 2026-10-07 (MESH.md step 6b): POST is the kernel's, as in v3. The kernel holds the cases and feeds them to the node; nothing of POST's harness is loaded into the node's dictionary. What this section said before — a harness in FORTH in post79.4th, and two variables in the nucleus for it, (CATCH) and (EMIT-HOOK) — is withdrawn; 6.3a keeps it for the record.

v3, as built. The cases are C tables compiled into the kernel (v3/src/test_runner/modules/*.c). For each, the kernel hands the line to the interpreter, reads the VM's error flag, and puts the VM's stack pointers, error and mode back (test_common.c, run_single_test). It runs once at boot on the first VM, against block RAM of POST's own, and the parity line is taken afterwards. What the cases define does not stay: run_test_suite saves the dictionary before each word's cases and puts it back after them, "to remove test-created words" (test_common.c:333, :365).

v4.

• The cases are a table in C, v4/system/post_cases.c, generated by v4/tools/mkpost.py (6.4) and linked into the hosted program and the kernel. Each is a name, its lines of FORTH, whether an error is expected, and what it must leave: the data stack, or only how many values for an address-dependent case (6.5), and the text it prints.
• The runner, v4/system/post.c, is the kernel's and is shared by the hosted program, the bare-metal kernel and the tests. For each case it empties the node's data stack from outside; sends DECIMAL FORTH DEFINITIONS, the state every expected value was generated from (6.4); sends the case's lines one at a time, as the boot sends any line, keeping what the node prints and showing none of it; and judges. A case passes when it ended in an error exactly if one was expected and, if none was, its stack and its printed text are what the table has. An error ends the line it is on, so a case is several short lines; it has ended in an error if any of its lines did.
• What it prints. For a failing case, POST FAIL: name out<...> stack<...>. Then PARITY:V4_POST tests=N pass=N fail=N.
• What is left: nothing, as in v3. The system is sealed before POST, and after the last case the kernel has the node do COLD, which comes back to the system as sealed; what COLD prints is not shown. The dictionary is put back once, after all the cases, and not after each word's as v3 does: the expected results were taken from one v3 session in which nothing was put back between cases (6.4), and the capsule did the same with its one FORGET at the end. How this came to be ruled twice. On 2026-10-07 Captain Bob was told that v3 leaves the cases' definitions in the dictionary, and ruled "as v3: they stay". That was false: it was read from run_single_test, which puts back only the stacks, without reading run_test_suite. It was built so, and the review of the step found it; shown what v3 does, he ruled that POST leaves nothing.
• A case the node does not come back from — it has stopped, or runs for ever — fails, and POST ends there: the runner names it and says how many cases were not run.
• What use still changes. After POST and COLD, HERE and LATEST are where they were, but the dictionary hash is not what it was before POST: each word POST had interpreted has a different TTL in its entry, the count of uses left before its access is checked again (v4/capsule/compile.v4, (ACL?)). 138 cells, measured 2026-10-07. It is the same on every system, so the six still agree.
• Gone: capsules/v4/post79.4th and its blocks 7000 up; the harness words; (CATCH) and (EMIT-HOOK), with what EMIT and the prompt loop did for them.

6.3a As it was until step 6b

post79.4th defined a small harness in FORTH and a tally, and needed two variables from the nucleus: (CATCH), so that a line ending in an error ended ok and POST could go on; and (EMIT-HOOK), the xt of a word given each character in place of the console, so that POST could compare what a case printed. Its last block was T-REPORT and FORGET T#: the harness and everything the cases had defined were forgotten.

6.4 Generating the expected values

v4/tools/mkpost.py reads the v3 test modules, keeps the cases for Required Word Set words, runs them in one session of the hosted v3 binary, and writes v4/system/post_cases.c (until step 6b, capsules/v4/post79.4th). It is a development tool (make -C v4 post79), run when the cases or the rules change; its output is committed and reviewed like any source. Every case starts from the same state on both machines: empty stack, DECIMAL, FORTH DEFINITIONS.

6.5 Where POST does not take v3's word (ruled 2026-10-05)

v4/tools/post79_rules.py holds the exceptions, each with its reason, and mkpost.py writes them out as POST79.md, which lists every one.

1. Address-dependent cases. A case that leaves or prints a memory address is checked for the number of values it leaves, and for what it prints unless an address is printed. The values are not compared.
2. Where v3 departs from FORTH-79. The expected result is the standard's. Where v3's line is not valid FORTH-79 (>R at the prompt, a single number given to <#), the line is rewritten so that it is.
3. Cases written for v3's machine. A v4 address unit is a cell (D-1); cases that assume v3's byte addresses are rewritten in v4's terms.
4. Words v3 has no case for get cases written by hand, with the standard's results. U* and U/MOD, which neither v3 nor the v4 nucleus had, are colon definitions in forth79.4th.

Not tested: KEY, EXPECT and QUERY, which wait for the keyboard, and QUIT, which returns to the terminal without ok.

7. Hashing, signing and parity

Hashing and signing are the existing capsule mechanisms, used unchanged (section 5.3).

Parity needs one new function. sk_dict_canonical_hash walks v3's DictEntry list and cannot hash a v4 node. The v4 dictionary hash is FNV-1a (fnv1a_64, already in kernel/src/vm/parity.c) over the node's memory from the start of code to HERE, and LATEST. Stacks, input buffers, block buffers and heat are left out.

The boot prints, in order:

PARITY:V4_NUCLEUS words=N image_hash=0x...
PARITY:V4_CAPSULE name=v4:forth79.4th capsule_id=0x... capsule_hash=0x... dict_hash=0x...
PARITY:V4_POST tests=N pass=N fail=N
PARITY:V4_SYSTEM word_count=N dict_hash=0x...
PARITY:OK
POST: PASSED
ok>

PARITY:V4_SYSTEM (step 6b) is the parity of the system as it is sealed, after POST, as v3's PARITY:M7.1a is taken after POST: how many words FORTH holds and the dictionary hash. Until step 6b the POST capsule had a PARITY:V4_CAPSULE line of its own in that place.

On any failure it prints PARITY:FAIL and POST: FAILED in place of the last three lines and does not give a prompt.

The tags are PARITY:V4_* so tooling that looks for v3's PARITY:M7.1a is not misled.

Every build uses 64-bit cells on the same engine, so every line above is identical on all six builds of one commit (section 9). The dictionary hash changes whenever a word moves between the nucleus and the capsule, so there is no fixed golden hash until decomposition is finished.

The nucleus is data linked into the binary and is trusted as the binary's own code is. It is hashed for the parity line and not signed.

8. Order of work

1. Capsule loading at boot, hosted and bare metal, with the block-number change. The capsule may be nearly empty at this step.
2. POST against today's assembled words. POST must pass here, on words make -C v4 test already covers. This proves POST before it judges anything new.
3. Decompose group by group. Move one group from .v4 to forth79.4th and run POST hosted: stack, arithmetic, comparison, memory, strings, number output, control, defining, vocabulary, blocks. A failure points at the group just moved.
4. Acceptance (section 9).

9. Products and acceptance

The hosted binary and the kernel run the same engine (v4/src), the same nucleus image, the same baked capsule directory and the same boot sequence (section 5.3 and section 7). Only the console differs: stdin and stdout hosted, the serial console on bare metal. The hosted binary links capsule_generated.c and reads nothing from the source tree.

Acceptance is six boots of one commit:

	amd64	aarch64	riscv64
Hosted Linux	native	user-mode QEMU	user-mode QEMU
Bare metal	clean qemu	clean qemu	clean qemu

Each must print identical PARITY:V4_* lines, pass POST and reach ok>. Bare-metal runs follow the repository's QEMU rules: one instance at a time, in the foreground, clean before qemu, logs kept under logs/.

make -C v4 test also runs at 32-bit cells. That stays as a development check for the FPGA gateway and is not part of this acceptance.

10. What is reused and what is new

Need	Existing	Change
Bake capsules	tools/mkcapsule.c, kernel/Makefile capsule rule	Block-number check only (5.2)
Find, hash-check, verify	capsule_find.c, capsule_validate.c, capsule_sig.c	None; also compiled into the hosted binary
Split a payload into blocks and lines	is_block_header and the walk in capsule_loader.c	Moved to a file both loaders call
Start a node from an image	v4/src/image.c, v4_image_boot	None
Build the nucleus image	v4/tools/mkimage.c	File list shrinks as groups move; the step that types editor.fth and tools.fth is removed
Kernel entry	kernel/src/v4/sk_v4.c, Kconfig STARFORTH_V4	Calls the shared boot sequence before reading the keyboard
Hosted entry	v4/tools/hosted.c	Same
Parity lines and FNV-1a	kernel/src/vm/parity.c	v4 dictionary hash added
v3 cases	v3/src/test_runner/modules/*.c	Read only
Expected values	hosted v3 binary	Read only
Hosted builds per ISA	v4/Makefile	Cross-compiler targets for aarch64 and riscv64

New: the shared v4 boot sequence (one file), the v4 dictionary hash, the case-extraction script, forth79.4th and post79.4th (the last gone since step 6b, 6.3).

The v3 boot is untouched: STARFORTH_V4 defaults to off.

11. Not decided here

• Where the v4 hosted binaries are installed. They are built under v4/build/; lfs/ holds v3's and is left alone.
• Signing the nucleus. It would have to become a named blob under capsules/, which puts a generated file in a source directory.