LithosAnanake/docs/v4.0.0/NUCLEUS.md

# StarForth v4.0.0 — Nucleus, FORTH-79 Capsule and POST

Design, 2026-10-05. Ruled by Captain Bob in conversation the same day; this
file records those rulings. It sits beside `JUSTIFICATION.md` (why v4) and
`DECOMPOSITION.md` (every v3 word's fate), and changes how the vocabulary
`DECOMPOSITION.md` describes is delivered, not what the words do.

## 1. Goal

v4 comes up on amd64, aarch64 and riscv64, both as the bare-metal kernel
under QEMU and as the hosted Linux binary, in this order:

1. start the assembled nucleus;
2. load FORTH-79 from a capsule;
3. run POST on FORTH-79;
4. reach the `ok>` prompt.

That is all this work delivers. Both the kernel and the hosted binary are
products.

## 2. What is wrong today

- All 292 named words are written in F18 assembler text (`v4/capsule/*.v4`).
  Only the editor and `SEE` are FORTH source.
- `v4/tools/mkimage.c` assembles and compiles everything on the build
  machine. The kernel links the finished memory image. Nothing is loaded
  from a capsule at boot.
- There is no v4 POST.
- No QEMU log of a v4 boot exists, so bare-metal v4 has never been accepted.

## 3. Scope

In scope: the FORTH-79 Required Word Set, as colon definitions in a capsule,
and a POST for it.

Out of scope until this is accepted: the Double Number extension set, Q48
arithmetic, logging, access control, the editor, `SEE`, `DEFER`, and every
other StarForth extension. Their `.v4` and `.fth` sources stay in the tree
but are not built into the nucleus image or loaded at boot.

## 4. The nucleus

The nucleus is the part of the vocabulary that stays in assembler. A word is
in the nucleus only if one of these holds.

1. **It is an opcode or a register access.** v4 compiles colon definitions
   to native instruction words, so FORTH source cannot write an opcode.
   `DUP DROP OVER + AND XOR 2* 2/ @ ! >R R> R@`, the stack-depth registers,
   and the console registers behind `KEY` and `EMIT`.
2. **The node needs it to read a capsule.** `WORD FIND NUMBER INTERPRET
   QUIT`, the error trap, `HERE ALLOT ,`, `CREATE`, `:`, `;`, `IMMEDIATE`,
   `LITERAL`, `[`, `]`, `(` and the code generator.
3. **It lays down code for a colon definition in the capsule.** The control
   words become colon definitions, and they need the code generator's
   words by name: `(OP,)`, `(LIT,)`, `(LABEL)`, `(BRANCH>)`, `(RESOLVE)`,
   `(JUMP,)`, `(CALL,)`, and the control-flow stack words `>CF`, `CF>` and
   `(PAIR)`. These get headers. They are not FORTH-79 words.

Everything else in the Required Word Set is a colon definition in the
capsule. That includes words that are assembler today only for speed:
`SWAP ROT - * /MOD 0= < C@ C! CMOVE FILL . <# # #> TYPE COUNT VARIABLE
CONSTANT`, the control words, the vocabulary words and the block words.

The capsule's definitions will run slower than the assembled ones they
replace. That is accepted: the goal is the minimum in assembler.

The list in rule 2 is the starting boundary, not the final one. A word
leaves the nucleus whenever a colon definition of it passes POST (section
8, step 3). The nucleus is at its minimum when no remaining word can be
moved.

## 5. Capsules

### 5.1 Files

| File | Capsule name | Contents |
|---|---|---|
| `capsules/v4/forth79.4th` | `v4:forth79.4th` | The Required Word Set as colon definitions |
| `capsules/v4/post79.4th` | `v4:post79.4th` | The POST harness and its cases |

Both are ordinary `.4th` capsules: `Block N` headers, lines of at most 64
characters, at most 16 lines to a block. `tools/mkcapsule.c` bakes them into
`capsule_generated.c` with every other file under `capsules/`, hashed, and
signed when the signing key is present. v3 does not load them: it runs
`init.4th` and only what that file `EXEC`s.

Named blobs (any non-`.4th` file under `capsules/`) are baked the same way
and stay available to v4 by name. This work does not need one.

### 5.2 The block-number rule

`validate_forth_blocks` (`tools/mkcapsule.c:409`) accepts block numbers in
`[2048, 5120)`. The floor is real: blocks 0–2047 are the VM's fast RAM. The
ceiling matches no device.

**Change:** the check becomes "block number is at least 2048". There is no
upper bound. Blocks 0–2047 are the only forbidden ones. The collision checks
against other capsules and `tools/capsule-reserved.txt` are unchanged.
`experiments/bare_metal/README.md` and `.claude/CLAUDE.md` state the old
range and are corrected with it.

The v4 capsules take a free range found from `capsules/BLOCK_MAP.md` and
`tools/capsule-reserved.txt`.

### 5.3 Loading

A capsule reaches the node as source lines through its console input, the
way v3's `capsule_exec_payload` interprets a payload line by line. `LOAD`
is not used: the block words are themselves in the capsule.

For each capsule, in order:

1. find it by name (`capsule_find_by_name`);
2. recompute its hash (`capsule_validate`); a mismatch stops the boot;
3. check its signature (`capsule_verify_signature`); invalid stops the
   boot, missing warns — v3's rule at `capsule_birth.c:585`;
4. split the payload on `Block N` headers and feed each line to the node,
   running the node until it waits for input again;
5. require the node to answer ` ok` to every line.

A line that is not accepted stops the boot and prints the capsule name,
block number, line number and what the node said. v3's retry, which skips a
failing line and runs the block again, is not carried over: a skipped line
in the standard's own capsule must not pass unnoticed.

What the node prints while loading is not shown, except on failure.

In v4 the block numbers are labels for now. They become storage addresses
when the block words exist and a capsule is copied to block storage.

## 6. POST

### 6.1 Source of the cases

POST is v3's own test cases, ported. The cases are in
`v3/src/test_runner/modules/*.c`; each is a name, a line of FORTH and a
flag saying whether an error is expected. Only cases for Required Word Set
words are ported, chosen by word name against the standard's list.

### 6.2 Judging

v3 judges a case only by whether it raised an error
(`v3/src/test_runner/test_common.c:258-285`); the "expected" text is a
comment. A wrong result that raises no error passes. That is not enough for
POST's purpose here, which is to prove each colon definition against the
assembled word it replaces.

**Ruling:** every ported case compares its result. The expected data stack
and printed output are taken from running the same line on the hosted v3
binary, so v3 remains the reference. A case v3 expects to raise an error
must raise one on v4.

Where v3 departs from FORTH-79 for a standard word, FORTH-79 wins (standing
ruling); the case's expected value is then the standard's, and the
departure is reported.

### 6.3 Form

`post79.4th` defines a small harness in FORTH and a tally. The harness
needs two things from the nucleus that FORTH-79 does not provide. Both are
variables an assembled word consults, and both have FORTH names:

| Variable | Consulted by | Effect while not zero |
|---|---|---|
| `(CATCH)` | the prompt loop (`quit.v4`) | A line that ends in an error does not say `ERROR`: `(CATCH)` is set to −1 and the line ends ` ok`. POST can then go on to its next case, and can check a case that must fail. |
| `(EMIT-HOOK)` | `EMIT` (`core.v4`) | It holds the xt of a word ( c -- ) that is given each character instead of the console. POST uses it to compare what a case prints, and it keeps the messages of expected errors out of the boot log. |

The prompt loop sets `(EMIT-HOOK)` to 0 at the end of every line, so the
console cannot be left silent. (This section first said one nucleus word
would do; reading `quit.v4` and `core.v4` showed it takes these two.
Amended 2026-10-05.)

An error always ends the line it is on, so a case is several short lines
and the harness words work across them.

After the last case POST prints the number of cases, passes and failures,
and names each failing case.

### 6.4 Generating the expected values

`v4/tools/mkpost.py` reads the v3 test modules, keeps the cases for
Required Word Set words, runs them in one session of the hosted v3 binary,
and writes `capsules/v4/post79.4th`. It is a development tool
(`make -C v4 post79`), run when the cases or the rules change; its output
is committed and reviewed like any source. Every case starts from the same
state on both machines: empty stack, `DECIMAL`, `FORTH DEFINITIONS`.

### 6.5 Where POST does not take v3's word (ruled 2026-10-05)

`v4/tools/post79_rules.py` holds the exceptions, each with its reason, and
`mkpost.py` writes them out as `POST79.md`, which lists every one.

1. **Address-dependent cases.** A case that leaves or prints a memory
   address is checked for the number of values it leaves, and for what it
   prints unless an address is printed. The values are not compared.
2. **Where v3 departs from FORTH-79.** The expected result is the
   standard's. Where v3's line is not valid FORTH-79 (`>R` at the prompt, a
   single number given to `<#`), the line is rewritten so that it is.
3. **Cases written for v3's machine.** A v4 address unit is a cell (D-1);
   cases that assume v3's byte addresses are rewritten in v4's terms.
4. **Words v3 has no case for** get cases written by hand, with the
   standard's results. `U*` and `U/MOD`, which neither v3 nor the v4
   nucleus had, are colon definitions in `forth79.4th`.

Not tested: `KEY`, `EXPECT` and `QUERY`, which wait for the keyboard, and
`QUIT`, which returns to the terminal without ` ok`.

## 7. Hashing, signing and parity

Hashing and signing are the existing capsule mechanisms, used unchanged
(section 5.3).

Parity needs one new function. `sk_dict_canonical_hash` walks v3's
`DictEntry` list and cannot hash a v4 node. The v4 dictionary hash is
FNV-1a (`fnv1a_64`, already in `kernel/src/vm/parity.c`) over the node's
memory from the start of code to `HERE`, and `LATEST`. Stacks, input
buffers, block buffers and heat are left out.

The boot prints, in order:

```
PARITY:V4_NUCLEUS words=N image_hash=0x...
PARITY:V4_CAPSULE name=v4:forth79.4th capsule_id=0x... capsule_hash=0x... dict_hash=0x...
PARITY:V4_CAPSULE name=v4:post79.4th capsule_id=0x... capsule_hash=0x... dict_hash=0x...
PARITY:V4_POST tests=N pass=N fail=N
PARITY:OK
POST: PASSED
ok>
```

On any failure it prints `PARITY:FAIL` and `POST: FAILED` in place of the
last three lines and does not give a prompt.

The tags are `PARITY:V4_*` so tooling that looks for v3's `PARITY:M7.1a`
is not misled.

Every build uses 64-bit cells on the same engine, so every line above is
identical on all six builds of one commit (section 9). The dictionary hash
changes whenever a word moves between the nucleus and the capsule, so
there is no fixed golden hash until decomposition is finished.

The nucleus is data linked into the binary and is trusted as the binary's
own code is. It is hashed for the parity line and not signed.

## 8. Order of work

1. **Capsule loading at boot**, hosted and bare metal, with the block-number
   change. The capsule may be nearly empty at this step.
2. **POST against today's assembled words.** POST must pass here, on words
   `make -C v4 test` already covers. This proves POST before it judges
   anything new.
3. **Decompose group by group.** Move one group from `.v4` to
   `forth79.4th` and run POST hosted: stack, arithmetic, comparison,
   memory, strings, number output, control, defining, vocabulary, blocks.
   A failure points at the group just moved.
4. **Acceptance** (section 9).

## 9. Products and acceptance

The hosted binary and the kernel run the same engine (`v4/src`), the same
nucleus image, the same baked capsule directory and the same boot sequence
(section 5.3 and section 7). Only the console differs: stdin and stdout
hosted, the serial console on bare metal. The hosted binary links
`capsule_generated.c` and reads nothing from the source tree.

Acceptance is six boots of one commit:

| | amd64 | aarch64 | riscv64 |
|---|---|---|---|
| Hosted Linux | native | user-mode QEMU | user-mode QEMU |
| Bare metal | `clean qemu` | `clean qemu` | `clean qemu` |

Each must print identical `PARITY:V4_*` lines, pass POST and reach `ok>`.
Bare-metal runs follow the repository's QEMU rules: one instance at a time,
in the foreground, `clean` before `qemu`, logs kept under `logs/`.

`make -C v4 test` also runs at 32-bit cells. That stays as a development
check for the FPGA gateway and is not part of this acceptance.

## 10. What is reused and what is new

| Need | Existing | Change |
|---|---|---|
| Bake capsules | `tools/mkcapsule.c`, `kernel/Makefile` capsule rule | Block-number check only (5.2) |
| Find, hash-check, verify | `capsule_find.c`, `capsule_validate.c`, `capsule_sig.c` | None; also compiled into the hosted binary |
| Split a payload into blocks and lines | `is_block_header` and the walk in `capsule_loader.c` | Moved to a file both loaders call |
| Start a node from an image | `v4/src/image.c`, `v4_image_boot` | None |
| Build the nucleus image | `v4/tools/mkimage.c` | File list shrinks as groups move; the step that types `editor.fth` and `tools.fth` is removed |
| Kernel entry | `kernel/src/v4/sk_v4.c`, Kconfig `STARFORTH_V4` | Calls the shared boot sequence before reading the keyboard |
| Hosted entry | `v4/tools/hosted.c` | Same |
| Parity lines and FNV-1a | `kernel/src/vm/parity.c` | v4 dictionary hash added |
| v3 cases | `v3/src/test_runner/modules/*.c` | Read only |
| Expected values | hosted v3 binary | Read only |
| Hosted builds per ISA | `v4/Makefile` | Cross-compiler targets for aarch64 and riscv64 |

New: the shared v4 boot sequence (one file), the v4 dictionary hash, the
case-extraction script, `forth79.4th` and `post79.4th`.

The v3 boot is untouched: `STARFORTH_V4` defaults to off.

## 11. Not decided here

- Where the v4 hosted binaries are installed. They are built under
  `v4/build/`; `lfs/` holds v3's and is left alone.
- Signing the nucleus. It would have to become a named blob under
  `capsules/`, which puts a generated file in a source directory.