LithosAnanake/docs/v4.0.0/MESH.md

# StarForth v4.0.0 — Talking nodes

Design, 2026-10-06. Ruled by Captain Bob by question and answer on
2026-10-05 and -06; each ruling is recorded with his words in `ENGINE.md`
section 3d. This file is the design that follows from them. Where it goes
beyond a ruling it says so, and those parts are proposals.

## 1. What this step is

> The entire point is that ultimately we have F18 engines digesting
> capsules alone, and in a sense can be anything written in F18 assembler
> for our fabric — 144 someday as a 12x12 grid, but I want a 12^3 FPGA
> ultimately, where using a capsule digester like StarForth, it's more
> than an operating system.

> The next step is talking nodes sharing the common SSD, and [they] may or
> may not have block storage available.

More than one node, each born empty and made into something by the capsule
it takes in, talking to each other through ports, sharing a disk. It comes
before making v4 equal v3 on bare metal (`ENGINE.md` step 4), which waits.

## 2. The rulings

1. **The ports are the transport; the message is what is transported.**
   Node to node, a write blocks until the neighbour reads and a read blocks
   until the neighbour writes (`DECOMPOSITION.md` section 6). What travels
   is v3's Hermes message with what it carries, so v3's messaging rules are
   not dropped: they go with the message.
2. **Some nodes have storage of their own**, in addition to or instead of
   the common SSD. Some have none.
3. **The first set of nodes is "2x2 + 1 central"**: five.
4. **The geometry is data, not design.** A node has a number of ports, and
   that number is a parameter. Which port connects to what is a table that
   can change while the system runs. "Not constrained by a 3D world. Other
   geometries might be better."
5. **The first geometry: "Only the central node can connect to only another
   central node."** Four outer nodes and their centre are a unit. An outer
   node is wired only inside its unit. Units are joined centre to centre.
6. **It must scale at run time, adaptively.**
7. **Hera decides which nodes exist and are awake, not whose turn it is.**
   Every node that is not blocked runs. Cooperative is a node blocking
   itself at a port; preemptive is Hera putting a node to sleep or killing
   it between any two instruction words. An idle node does not spin: it is
   blocked reading its ports.
8. **A node is born empty.** It has nothing but the ability to listen. The
   first thing a neighbour sends it is a capsule of F18 code. StarForth's
   nucleus is such a capsule.
9. **Built in the shared engine, proven hosted on three ISAs first, then on
   bare metal.**

## 3. Acceptance (approved 2026-10-06)

1. **Five nodes come up from nothing.** Hera is born empty, takes in the
   nucleus capsule and the FORTH-79 capsule, and passes POST. She births
   the four outer nodes while running; each is born empty, takes in the
   same capsules through its port from Hera, and passes POST. Each prints a
   parity line; the four outer nodes' dictionary hashes are identical.
2. **They talk.** A line typed at the console reaches Hera as a message. A
   message from Hera runs on an outer node and its output comes back. A
   message between two outer nodes that are not wired to each other is
   forwarded by a node in between.
3. **They share the SSD.** A block written by one node is read by another.
   One outer node has a drive of its own and uses it. One has no storage,
   and `BLOCK` on it is an error with a message.
4. **It scales while running.** A second unit of five is born and joined
   centre to centre. A message crosses from one unit to the other. The
   second unit is removed and the first carries on.
5. **Hera manages.** An idle node executes nothing while it waits, which
   the anti-clock shows. Hera puts a node to sleep and wakes it. Hera kills
   a node that is stuck in an endless loop, and everything else keeps
   running.
6. **On every build.** The three hosted ISAs, then the three bare-metal
   ISAs under QEMU with the real console and disk.

**Left for the step after, by agreement:**

- Hera sleeping, waking and killing nodes by herself from each node's
  heat. Here she does it by command. The rule for it has not been given.
- v3's messaging rules checked at every hop. From this step a message
  carries its heat, TTL and ACL tag; checking them is the router's, and
  the checks await rulings.

## 4. The engine: ports

Everything in this section is the engine's (`v4/src`) and knows nothing of
StarForth or of any kernel.

### 4.1 A node's ports

A node has `V4_PORTS` ports. The number is a build parameter, as
`V4_CELL_BITS` and `V4_NODE_WORDS` are. (Proposal: 8 for now, which is what
the first geometry needs of a centre — four outer nodes, two devices, two
other centres — with nothing depending on the number.)

Ports are addresses, as `DECOMPOSITION.md` section 6 has them. Attached at
word address `base`:

| Address | What |
|---|---|
| `base` … `base + V4_PORTS − 1` | Port 0 … port `V4_PORTS − 1` |
| `base + V4_PORTS` | Any port: a read here takes from whichever port has a neighbour writing |
| `base + V4_PORTS + 1` | Which port the last read from "any" came from. Read only. |

- **A write to a port blocks the node until the neighbour has read it.**
  Built 2026-10-05 for one port; the opcode after the write runs when the
  write has been taken.
- **A read from a port blocks the node until the neighbour writes.** The
  fetch does not happen until there is something to fetch. This is new.
- **A read is any fetch**: `@`, `@b`, `@+`, the literal fetch `@p`, and the
  fetch of an instruction word when `P` is a port address. When `P` is a
  port, it is not advanced: the node goes on executing what arrives there.
  That is how an F18 node runs code from a port, and it is what makes
  ruling 8 need no code in a newborn node.
- A port with nothing on the other end blocks for ever, as on the fabric.

### 4.2 A node at reset

`P` is the "any port" address, both stacks are empty, memory is zero. The
node is blocked reading its ports. Nothing else is in it.

### 4.3 The fabric

`v4/src/fabric.c`: the nodes there are, how their ports are wired, and
time passing for all of them at once.

- **The nodes.** A set that grows and shrinks while the system runs
  (ruling 6). A node is added empty (4.2) and removed whole.
- **The wiring.** For each port of each node: nothing, or a port of
  another node, or a device. It is a table, changed at run time
  (ruling 4). The fabric does not know what shape it makes.
- **A device** is what is on the other end of a port that is not a node:
  two functions, one that takes a word the node writes and one that gives
  a word when the node reads, each able to say "not yet". The console, a
  disk, and the kernel that serves a node's requests (`ENGINE.md` 3.3) are
  devices.
- **A step.** Every node that is awake and not blocked executes one
  instruction word. Then every write that has a reader waiting on the
  other end of its wire is handed over, and both nodes are unblocked. That
  is all: there is no choice of whose turn it is (ruling 7).
- **Asleep.** A node that is asleep executes nothing and nothing is handed
  to it or taken from it. It is put to sleep and woken from outside, at
  any instruction word.

### 4.4 What is not in the engine

Messages, routing, capsules, the unit of five, Hera. Those are sections 5
to 8 and are made of capsule code and of the host that owns the devices.

## 5. A capsule of F18 code, and how an empty node takes it in

**As built.** A capsule of F18 code is not a format a node has to
understand. It is the words a neighbour writes to the node's port, in the
order they are written: for each stretch of memory that is not zero the
two instruction words below with their address and count and then the
words themselves, and at the end a jump to where to start. The file is
those words, each in a cell's bytes, low byte first, and its hash is the
hash of exactly what is sent. The nucleus is one, in the capsule directory
with a name, a hash and a signature like any other. `mkcapsule` was not
changed: the nucleus capsule is a built file kept under `capsules/v4/`.

A neighbour puts it into an empty node by writing to the port between
them, and the node executes what arrives (4.1). For each run it sends

```
@p a! @p push       \ then the address, then count − 1: executed from the port
@p !+ unext         \ then the words: each is fetched from the port and stored
```

and at the end `jump` to the start address. That is the F18's own way,
and it needs nothing in the node beforehand.

## 6. A message

**Proposal**, from `DECOMPOSITION.md` section 6.1 and v3's `SkHermesMessage`
(`kernel/include/starkernel/vm/kernel_hermes.h`), which it must be able to
carry whole:

| Word | Contents |
|---|---|
| 0 | To: the node it is for |
| 1 | From: the node that sent it |
| 2 | Type, and the channel |
| 3 | Heat and TTL |
| 4 | ACL tag: the sender's identity fingerprint |
| 5 | Sequence |
| 6 | Payload length in characters, 0 to 1024 |
| 7 … | Payload: FORTH text, four characters to a word |

It is written to a port a word at a time and read a word at a time. Words
3 and 4 are carried from this step on and are not yet checked
(section 3).

A node that is a StarForth digester, when it has nothing to do, reads a
message from "any port". If it is for this node, the payload is
interpreted, as a line is today (`ENGINE.md` 3.1). If it is for another,
it is written to the port that leads there (section 7).

What a node prints goes to its console, and its console is a place like
any other: for the node wired to the console device, that port; for any
other node, a message to the node that is. So what an outer node prints
comes back through its centre.

## 7. Finding the way

**Proposal.** Each node has a small table: for a destination, the port
that leads toward it; and one port for everything not in the table.
Whoever wires a node writes its table, and changes it when the wiring
changes. Under the first geometry an outer node's table is its two grid
neighbours and "everything else to my centre"; a centre's is its four
outer nodes, and for each other unit the port toward that unit's centre.

A different geometry is a different way of filling in the wiring table
and these tables. Nothing else changes.

## 7a. Two neighbours writing to each other (open; awaits a ruling)

The fault is in section 10, step 4. Three ways out, none built:

**A. A node writes only to a neighbour that is already reading.** The
engine gives a node two more things to fetch, as the F18's `io` register
does: which ports have a neighbour writing, and which have a neighbour
reading. A node with a message to send looks first. If the neighbour is
reading, it writes, and the write does not wait. If the neighbour is
writing to it instead, it takes that message in first, and deals with it,
and looks again. A node therefore holds messages waiting to go, in its own
memory, and what it does when that room is full has to be decided (the
message is lost and counted, as one with nowhere to go is now). Writes
still block, as ruled; a node simply does not start one it cannot finish.
This is how it is done on the chip. It is all in the nucleus but for the
two fetches.

**B. Passing on is not the node's programme's work.** `DECOMPOSITION.md`
6.1 says "the router in each node forwards packets". Read as a thing apart
from the programme -- logic beside the F18 on the FPGA, a part of the
fabric in the model -- it takes in and sends on whatever the programme is
doing, and holds what is for the programme until it reads. The programme's
own writes then never wait on a neighbour's programme. This is more
hardware per node and takes passing-on out of F18 code, and out of what a
capsule can change.

**C. No answer to a `SEND`.** It removes the case found and leaves the
fault: any two messages crossing still stop both nodes. Not offered as a
way out; listed so that it is seen to have been weighed.

Recommended: A. It keeps every ruling as it stands, keeps a node an F18
and nothing more, and what it adds to the engine the chip already has.

## 8. Storage

**Proposal.** A disk is a device on a port. Asking for a block is a
message to it — read block *n*, or write block *n* with 1024 characters —
and the answer is a message back; a block is exactly the most a message
carries. Under v3's one block-number space, each device has its own range
of numbers.

- A node wired to a disk asks it directly.
- A node not wired to one asks through the node that is: the message is
  forwarded like any other. So the common SSD is shared by every node that
  has a way to the node that holds it.
- A node with its own drive has that device on one of its own ports.
- A node with no way to any disk has no storage, and `BLOCK` says so.

Who may have which block is the block card's, decided where the disk is,
under the identity in the message's ACL tag (`V3-PARITY.md` 1d). Not
checked in this step.

## 9. Birth, and Hera

**Proposal.** Hera asks, through the port where her requests go
(`ENGINE.md` 3.3), for a node to be added and wired; she then sends the
newborn its capsules through the port that joins them. Putting a node to
sleep, waking it, and removing it are requests of the same kind. Only
Hera's requests are honoured.

The unit rule (ruling 5) is Hera's, in capsule code: it is what she does
with those requests. The engine and the host do not know it.

## 10. Steps

Each is tested, committed and pushed before the next.

1. **Ports and the fabric (section 4).** Reads; "any port"; a node at
   reset; execution from a port; nodes added and removed; wiring changed;
   asleep and awake. Tests at the level of opcodes: two nodes exchange
   words; an empty node is filled through its port and runs what it was
   sent; a word is passed on by a node in between; a node waiting executes
   nothing; a node is put to sleep, woken, and removed while looping.
   **Done 2026-10-06.** `v4/src/node.c` (the ports, `v4_node_born`),
   `v4/src/exec.c` (a fetch from a port waits; execution from a port; the
   slot a blocked node goes on from), `v4/src/fabric.c`. `V4_PORTS` is 8.
   `v4/tests/test_fabric.c`, 53 checks at both cell widths and under ASan
   and UBSan: all of the above, with the empty node filled once by a device
   and once by another node holding the capsule in its own memory; the
   fabric given more room while nodes run; what cannot be wired refused.
   The single-node products are unchanged by it: `hosted-check` on three
   ISAs, and the three bare-metal boots with lines typed at each prompt
   (`logs/20261006-074907`, `-075150`, `-075532`).
2. **The nucleus as a capsule, and an empty node made a StarForth node
   through its port (section 5).**
   **Done 2026-10-06.** `v4/src/capsule.c`, `v4_capsule_write`: any
   node's memory as the words to send an empty node. `mkimage` writes the
   nucleus so, to `capsules/v4/nucleus-64.f18`, a built file kept in the
   tree as `BLOCK_MAP.md` is; `mkcapsule` is unchanged and bakes, hashes
   and signs it with the rest. The boot's node is born empty
   (`v4_image_born`) and is given the nucleus a word at a time as it reads
   its port, after the capsule's hash and signature are checked
   (`v4/system/boot.c`). Nothing of the nucleus is linked into either
   product any more. `test_fabric.c`: a memory with a programme and words
   here and there arrives word for word and runs. Both products boot so:
   `hosted-check` on three ISAs; `logs/20261006-102421`, `-102706`,
   `-103048`.
   **What a newborn node still has from its host:** the registers the
   nucleus expects — the console's three, the two stack registers, the
   error register, the fault table's address, the storage registers — are
   attached by the host when the node is born, from the description
   `mkimage` writes. They are the node's hardware as the golden model has
   it, at addresses the memory map (D-4, still open) will fix. The console
   and storage ones go when those become devices on ports (steps 3 and
   6).
3. **Messages (section 6):** a StarForth node that reads messages when
   idle; the console as a device; a line typed is a message.
   **Done 2026-10-06, but for the keyboard.** A node with nothing to do is
   blocked reading "any port" (`(IDLE)`, `v4/capsule/quit.v4`). What
   arrives is a message; text for this node is interpreted; what it prints
   is kept and sent back as messages to the sender, on the port the
   message came on, and then a message saying how the text ended (`EMIT`,
   `(FLUSH-OUT)`, `(HDR)` in `core.v4`; `(FINISH)` in `quit.v4`).
   `v4/src/message.c` is the same format for whatever is on the other end
   of a port and is not a node. The boot is that, on the node's port 1, as
   the console (`v4_boot_line`); the prompt tests are too. Handing a node a
   line by writing its input buffer and setting its `P` is gone
   (`v4_line_begin` and the rest). Nothing reads the `CONSOLE-TX` register
   any more.
   `EMIT` still needs one free cell of the data stack and no more, as
   before; it keeps its working values on the return stack. The full-stack
   tests hold at the same figures as before.
   All v4 tests pass at both widths and under ASan and UBSan;
   `hosted-check` on three ISAs; bare metal `logs/20261006-110551` (amd64),
   `-111621` (aarch64), `-111341` (riscv64). `-110837` is an aarch64 run
   that was ended by the test wrapper's limit while still in UEFI firmware,
   before the kernel had started; it shows nothing about v4.
   **Not done:** `KEY`, `EXPECT` and `QUERY` still read the console's two
   input registers, not a message. A message not for this node, or not
   text, is let go: passing it on is step 4.
4. **Finding the way (section 7).**
   **Built 2026-10-06; NOT done: one fault stands, below, and it needs a
   ruling.** Each node has a table of up to 16 destinations and the port
   toward each, and one port for everything else (`ROUTE ( node port -- )`,
   `DEFAULT-ROUTE ( port -- )`, `NO-ROUTES`; `(PORT-FOR)` in
   `v4/capsule/core.v4`). A message not for this node is written, whole,
   to the port its destination's entry names (`(PASS-ON)`,
   `v4/capsule/quit.v4`); with no entry and no port for everything else it
   is dropped and counted (`(LOST)`). What text prints, and how it ended,
   go back to the node the text came from by the same table, so an answer
   crosses as many nodes as the text did. `SEND ( baddr u node -- )` sends
   text to another node; `(ME)` is a node's own number; `(CONSOLE)`, when
   set, is where a node's printing goes instead of to the sender.
   `v4/tests/test_host_mesh.c`, 28 checks at both widths and under ASan
   and UBSan: three StarForth nodes in a row behind a console; text for
   the far one passes through the other two and its answer comes back;
   each node keeps its own dictionary; output longer than one message; a
   1024-character message passed on whole; a message with nowhere to go
   counted; a table changed while running. The single-node products are
   unchanged: `hosted-check` on three ISAs; bare metal
   `logs/20261006-115225` (amd64), `-115501` (aarch64), `-115849`
   (riscv64).
   **THE FAULT: two neighbours that write to each other at once wait for
   ever.** A write blocks until the neighbour reads (ruling, section 4.1),
   and a node that is blocked writing is not reading. Passing a message on
   is done here by the node's own programme, so a node cannot pass
   anything on, or take anything in, while it waits to write. Found so: the
   far node `SEND`s text to the middle one and then goes to send word of
   how its own text ended, which goes by the middle one; the middle one has
   done the text and goes to answer the far one. Each is writing the first
   word of a message to the other. It is not peculiar to `SEND`: any two
   messages going opposite ways on one wire at the same moment do it, and
   so would a ring of nodes each writing to the next. The last check in
   `test_host_mesh.c` shows it and is named KNOWN FAULT; it is there so the
   fault is on record and is seen to go.
   It is not a slip in the code. It is what blocking writes and a
   programme that does one thing at a time give. Section 7 said nothing
   about it and should have. What is to be done about it is not decided:
   the choices are in section 7a.
5. **Birth and Hera's requests (section 9); the unit of five.**
6. **Storage (section 8).**
7. **A second unit; scaling while running; sleep, wake and kill by
   command.**
8. **The hosted product is the five nodes**, on three ISAs: acceptance 1
   to 5.
9. **The same on bare metal:** acceptance 6.

## 11. What becomes of what is there

- `v4/system/boot.c` and `v4/tools/hosted.c` hand one node a line and run
  it to the end. They stay until step 8, where the hosted product becomes
  the five nodes.
- `(LINE)` and `(IDLE)` (`v4/capsule/quit.v4`): `(LINE)` stays, as what
  interprets a message's payload. `(IDLE)` becomes the read of a message
  at step 3.
- The one port of 2026-10-05, where a node's requests go, is port 0 of the
  ports of 4.1.
- The nucleus linked into the binary goes at step 2.
- `DECOMPOSITION.md` section 6 says four ports; it is corrected at step 1.