LithosAnanake/docs/v4.0.0/NUCLEUS.md

# StarForth v4.0.0 — Nucleus, FORTH-79 Capsule and POST

Design, 2026-10-05. Ruled by Captain Bob in conversation the same day; this
file records those rulings. It sits beside `JUSTIFICATION.md` (why v4) and
`DECOMPOSITION.md` (every v3 word's fate), and changes how the vocabulary
`DECOMPOSITION.md` describes is delivered, not what the words do.

## 1. Goal

v4 comes up on amd64, aarch64 and riscv64, both as the bare-metal kernel
under QEMU and as the hosted Linux binary, in this order:

1. start the assembled nucleus;
2. load FORTH-79 from a capsule;
3. run POST on FORTH-79;
4. reach the `ok>` prompt.

That is all this work delivers. Both the kernel and the hosted binary are
products.

## 2. What is wrong today

- All 292 named words are written in F18 assembler text (`v4/capsule/*.v4`).
  Only the editor and `SEE` are FORTH source.
- `v4/tools/mkimage.c` assembles and compiles everything on the build
  machine. The kernel links the finished memory image. Nothing is loaded
  from a capsule at boot.
- There is no v4 POST.
- No QEMU log of a v4 boot exists, so bare-metal v4 has never been accepted.

## 3. Scope

In scope: the FORTH-79 Required Word Set, as colon definitions in a capsule,
and a POST for it.

Out of scope until this is accepted: the Double Number extension set, Q48
arithmetic, logging, access control, the editor, `SEE`, `DEFER`, and every
other StarForth extension. Their `.v4` and `.fth` sources stay in the tree
but are not built into the nucleus image or loaded at boot.

## 4. The nucleus

The nucleus is the part of the vocabulary that stays in assembler. A word is
in the nucleus only if one of these holds.

1. **It is an opcode or a register access.** v4 compiles colon definitions
   to native instruction words, so FORTH source cannot write an opcode.
   `DUP DROP OVER + AND XOR 2* 2/ @ ! >R R> R@`, the stack-depth registers,
   and the console registers behind `KEY` and `EMIT`.
2. **The node needs it to read a capsule.** `WORD FIND NUMBER INTERPRET
   QUIT`, the error trap, `HERE ALLOT ,`, `CREATE`, `:`, `;`, `IMMEDIATE`,
   `LITERAL`, `[`, `]`, `(` and the code generator.
3. **It lays down code for a colon definition in the capsule.** The control
   words become colon definitions, and they need the code generator's
   words by name: `(OP,)`, `(LIT,)`, `(LABEL)`, `(BRANCH>)`, `(RESOLVE)`,
   `(JUMP,)`, `(CALL,)`, and the control-flow stack words `>CF`, `CF>` and
   `(PAIR)`. These get headers. They are not FORTH-79 words.

Everything else in the Required Word Set is a colon definition in the
capsule. That includes words that are assembler today only for speed:
`SWAP ROT - * /MOD 0= < C@ C! CMOVE FILL . <# # #> TYPE COUNT VARIABLE
CONSTANT`, the control words, the vocabulary words and the block words.

The capsule's definitions will run slower than the assembled ones they
replace. That is accepted: the goal is the minimum in assembler.

The list in rule 2 is the starting boundary, not the final one. A word
leaves the nucleus whenever a colon definition of it passes POST (section
8, step 3). The nucleus is at its minimum when no remaining word can be
moved.

## 5. Capsules

### 5.1 Files

| File | Capsule name | Contents |
|---|---|---|
| `capsules/v4/forth79.4th` | `v4:forth79.4th` | The Required Word Set as colon definitions |
| `capsules/v4/post79.4th` | `v4:post79.4th` | The POST harness and its cases |

Both are ordinary `.4th` capsules: `Block N` headers, lines of at most 64
characters, at most 16 lines to a block. `tools/mkcapsule.c` bakes them into
`capsule_generated.c` with every other file under `capsules/`, hashed, and
signed when the signing key is present. v3 does not load them: it runs
`init.4th` and only what that file `EXEC`s.

Named blobs (any non-`.4th` file under `capsules/`) are baked the same way
and stay available to v4 by name. This work does not need one.

### 5.2 The block-number rule

`validate_forth_blocks` (`tools/mkcapsule.c:409`) accepts block numbers in
`[2048, 5120)`. The floor is real: blocks 0–2047 are the VM's fast RAM. The
ceiling matches no device.

**Change:** the check becomes "block number is at least 2048". There is no
upper bound. Blocks 0–2047 are the only forbidden ones. The collision checks
against other capsules and `tools/capsule-reserved.txt` are unchanged.
`experiments/bare_metal/README.md` and `.claude/CLAUDE.md` state the old
range and are corrected with it.

The v4 capsules take a free range found from `capsules/BLOCK_MAP.md` and
`tools/capsule-reserved.txt`.

### 5.3 Loading

A capsule reaches the node as source lines through its console input, the
way v3's `capsule_exec_payload` interprets a payload line by line. `LOAD`
is not used: the block words are themselves in the capsule.

For each capsule, in order:

1. find it by name (`capsule_find_by_name`);
2. recompute its hash (`capsule_validate`); a mismatch stops the boot;
3. check its signature (`capsule_verify_signature`); invalid stops the
   boot, missing warns — v3's rule at `capsule_birth.c:585`;
4. split the payload on `Block N` headers and feed each line to the node,
   running the node until it waits for input again;
5. require the node to answer ` ok` to every line.

A line that is not accepted stops the boot and prints the capsule name,
block number, line number and what the node said. v3's retry, which skips a
failing line and runs the block again, is not carried over: a skipped line
in the standard's own capsule must not pass unnoticed.

What the node prints while loading is not shown, except on failure.

In v4 the block numbers are labels for now. They become storage addresses
when the block words exist and a capsule is copied to block storage.

## 6. POST

### 6.1 Source of the cases

POST is v3's own test cases, ported. The cases are in
`v3/src/test_runner/modules/*.c`; each is a name, a line of FORTH and a
flag saying whether an error is expected. Only cases for Required Word Set
words are ported, chosen by word name against the standard's list.

### 6.2 Judging

v3 judges a case only by whether it raised an error
(`v3/src/test_runner/test_common.c:258-285`); the "expected" text is a
comment. A wrong result that raises no error passes. That is not enough for
POST's purpose here, which is to prove each colon definition against the
assembled word it replaces.

**Ruling:** every ported case compares its result. The expected data stack
and printed output are taken from running the same line on the hosted v3
binary, so v3 remains the reference. A case v3 expects to raise an error
must raise one on v4.

Where v3 departs from FORTH-79 for a standard word, FORTH-79 wins (standing
ruling); the case's expected value is then the standard's, and the
departure is reported.

### 6.3 Form

**Ruled 2026-10-07 (`MESH.md` step 6b): POST is the kernel's, as in v3.**
The kernel holds the cases and feeds them to the node; nothing of POST's
harness is loaded into the node's dictionary. What this section said
before — a harness in FORTH in `post79.4th`, and two variables in the
nucleus for it, `(CATCH)` and `(EMIT-HOOK)` — is withdrawn; 6.3a keeps it
for the record.

**v3, as built.** The cases are C tables compiled into the kernel
(`v3/src/test_runner/modules/*.c`). For each, the kernel hands the line to
the interpreter, reads the VM's error flag, and puts the VM's stack
pointers, error and mode back (`test_common.c`, `run_single_test`). It
runs once at boot on the first VM, against block RAM of POST's own, and
the parity line is taken afterwards. Whatever the cases define stays in
the dictionary.

**v4.**

- *The cases* are a table in C, `v4/system/post_cases.c`, generated by
  `v4/tools/mkpost.py` (6.4) and linked into the hosted program and the
  kernel. Each is a name, its lines of FORTH, whether an error is
  expected, and what it must leave: the data stack, or only how many
  values for an address-dependent case (6.5), and the text it prints.
- *The runner*, `v4/system/post.c`, is the kernel's and is shared by the
  hosted program, the bare-metal kernel and the tests. For each case it
  empties the node's data stack from outside; sends `DECIMAL FORTH
  DEFINITIONS`, the state every expected value was generated from (6.4);
  sends the case's lines one at a time, as the boot sends any line,
  keeping what the node prints and showing none of it; and judges. A case
  passes when it ended in an error exactly if one was expected and, if
  none was, its stack and its printed text are what the table has. An
  error ends the line it is on, so a case is several short lines; it has
  ended in an error if any of its lines did.
- *What it prints.* For a failing case, `POST FAIL: name out<...>
  stack<...>`. Then `PARITY:V4_POST tests=N pass=N fail=N`.
- *What is left.* What the cases define stays in the dictionary, as in
  v3 (ruled 2026-10-07, on being told that the capsule had ended with
  `FORGET` and left nothing). The system is sealed after POST, so `COLD`
  comes back to those words and `FORGET` will not go below them. Measured
  2026-10-07: they take about 1,290 cells of the 5,632 in the dictionary
  space.
- *Gone:* `capsules/v4/post79.4th` and its blocks 7000 up; the harness
  words; `(CATCH)` and `(EMIT-HOOK)`, with what `EMIT` and the prompt
  loop did for them.

### 6.3a As it was until step 6b

`post79.4th` defined a small harness in FORTH and a tally, and needed two
variables from the nucleus: `(CATCH)`, so that a line ending in an error
ended ` ok` and POST could go on; and `(EMIT-HOOK)`, the xt of a word given
each character in place of the console, so that POST could compare what a
case printed. Its last block was `T-REPORT` and `FORGET T#`: the harness
and everything the cases had defined were forgotten.

### 6.4 Generating the expected values

`v4/tools/mkpost.py` reads the v3 test modules, keeps the cases for
Required Word Set words, runs them in one session of the hosted v3 binary,
and writes `v4/system/post_cases.c` (until step 6b, `capsules/v4/post79.4th`). It is a development tool
(`make -C v4 post79`), run when the cases or the rules change; its output
is committed and reviewed like any source. Every case starts from the same
state on both machines: empty stack, `DECIMAL`, `FORTH DEFINITIONS`.

### 6.5 Where POST does not take v3's word (ruled 2026-10-05)

`v4/tools/post79_rules.py` holds the exceptions, each with its reason, and
`mkpost.py` writes them out as `POST79.md`, which lists every one.

1. **Address-dependent cases.** A case that leaves or prints a memory
   address is checked for the number of values it leaves, and for what it
   prints unless an address is printed. The values are not compared.
2. **Where v3 departs from FORTH-79.** The expected result is the
   standard's. Where v3's line is not valid FORTH-79 (`>R` at the prompt, a
   single number given to `<#`), the line is rewritten so that it is.
3. **Cases written for v3's machine.** A v4 address unit is a cell (D-1);
   cases that assume v3's byte addresses are rewritten in v4's terms.
4. **Words v3 has no case for** get cases written by hand, with the
   standard's results. `U*` and `U/MOD`, which neither v3 nor the v4
   nucleus had, are colon definitions in `forth79.4th`.

Not tested: `KEY`, `EXPECT` and `QUERY`, which wait for the keyboard, and
`QUIT`, which returns to the terminal without ` ok`.

## 7. Hashing, signing and parity

Hashing and signing are the existing capsule mechanisms, used unchanged
(section 5.3).

Parity needs one new function. `sk_dict_canonical_hash` walks v3's
`DictEntry` list and cannot hash a v4 node. The v4 dictionary hash is
FNV-1a (`fnv1a_64`, already in `kernel/src/vm/parity.c`) over the node's
memory from the start of code to `HERE`, and `LATEST`. Stacks, input
buffers, block buffers and heat are left out.

The boot prints, in order:

```
PARITY:V4_NUCLEUS words=N image_hash=0x...
PARITY:V4_CAPSULE name=v4:forth79.4th capsule_id=0x... capsule_hash=0x... dict_hash=0x...
PARITY:V4_POST tests=N pass=N fail=N
PARITY:V4_SYSTEM word_count=N dict_hash=0x...
PARITY:OK
POST: PASSED
ok>
```

`PARITY:V4_SYSTEM` (step 6b) is the parity of the system as it is sealed,
after POST, as v3's `PARITY:M7.1a` is taken after POST: how many words
FORTH holds and the dictionary hash. Until step 6b the POST capsule had a
`PARITY:V4_CAPSULE` line of its own in that place.

On any failure it prints `PARITY:FAIL` and `POST: FAILED` in place of the
last three lines and does not give a prompt.

The tags are `PARITY:V4_*` so tooling that looks for v3's `PARITY:M7.1a`
is not misled.

Every build uses 64-bit cells on the same engine, so every line above is
identical on all six builds of one commit (section 9). The dictionary hash
changes whenever a word moves between the nucleus and the capsule, so
there is no fixed golden hash until decomposition is finished.

The nucleus is data linked into the binary and is trusted as the binary's
own code is. It is hashed for the parity line and not signed.

## 8. Order of work

1. **Capsule loading at boot**, hosted and bare metal, with the block-number
   change. The capsule may be nearly empty at this step.
2. **POST against today's assembled words.** POST must pass here, on words
   `make -C v4 test` already covers. This proves POST before it judges
   anything new.
3. **Decompose group by group.** Move one group from `.v4` to
   `forth79.4th` and run POST hosted: stack, arithmetic, comparison,
   memory, strings, number output, control, defining, vocabulary, blocks.
   A failure points at the group just moved.
4. **Acceptance** (section 9).

## 9. Products and acceptance

The hosted binary and the kernel run the same engine (`v4/src`), the same
nucleus image, the same baked capsule directory and the same boot sequence
(section 5.3 and section 7). Only the console differs: stdin and stdout
hosted, the serial console on bare metal. The hosted binary links
`capsule_generated.c` and reads nothing from the source tree.

Acceptance is six boots of one commit:

| | amd64 | aarch64 | riscv64 |
|---|---|---|---|
| Hosted Linux | native | user-mode QEMU | user-mode QEMU |
| Bare metal | `clean qemu` | `clean qemu` | `clean qemu` |

Each must print identical `PARITY:V4_*` lines, pass POST and reach `ok>`.
Bare-metal runs follow the repository's QEMU rules: one instance at a time,
in the foreground, `clean` before `qemu`, logs kept under `logs/`.

`make -C v4 test` also runs at 32-bit cells. That stays as a development
check for the FPGA gateway and is not part of this acceptance.

## 10. What is reused and what is new

| Need | Existing | Change |
|---|---|---|
| Bake capsules | `tools/mkcapsule.c`, `kernel/Makefile` capsule rule | Block-number check only (5.2) |
| Find, hash-check, verify | `capsule_find.c`, `capsule_validate.c`, `capsule_sig.c` | None; also compiled into the hosted binary |
| Split a payload into blocks and lines | `is_block_header` and the walk in `capsule_loader.c` | Moved to a file both loaders call |
| Start a node from an image | `v4/src/image.c`, `v4_image_boot` | None |
| Build the nucleus image | `v4/tools/mkimage.c` | File list shrinks as groups move; the step that types `editor.fth` and `tools.fth` is removed |
| Kernel entry | `kernel/src/v4/sk_v4.c`, Kconfig `STARFORTH_V4` | Calls the shared boot sequence before reading the keyboard |
| Hosted entry | `v4/tools/hosted.c` | Same |
| Parity lines and FNV-1a | `kernel/src/vm/parity.c` | v4 dictionary hash added |
| v3 cases | `v3/src/test_runner/modules/*.c` | Read only |
| Expected values | hosted v3 binary | Read only |
| Hosted builds per ISA | `v4/Makefile` | Cross-compiler targets for aarch64 and riscv64 |

New: the shared v4 boot sequence (one file), the v4 dictionary hash, the
case-extraction script, `forth79.4th` and `post79.4th`.

The v3 boot is untouched: `STARFORTH_V4` defaults to off.

## 11. Not decided here

- Where the v4 hosted binaries are installed. They are built under
  `v4/build/`; `lfs/` holds v3's and is left alone.
- Signing the nucleus. It would have to become a named blob under
  `capsules/`, which puts a generated file in a source directory.