admin/LithosAnanake
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: reorganise -- current reference gathered, finished work archived, superseded and dead ideas marked as such

Browse Source 
docs/CLAUDE.md Phase 2, 2026-10-14.  All moves by git mv; every archived
file carries a header saying what it is, why, and where to look instead.

- docs/reference/ (new): roadmap and versioning policy, M7.1, word-ACL
  design, kernel args, hosted acceptance test, dictionary, primitives,
  ontology, L8 table -- previously spread over docs/, docs/lithosananke/
  and docs/03-architecture/.
- docs/working/archive/historical/: finished work (session plan, test-gap
  audit, birthing plan, fix write-ups, the January architecture draft).
- docs/working/archive/superseded/: the Hermes-VM Tripod docs and July
  Hermes designs (-> FABRIC-3.5/3.6), the StarForth roadmap, every working
  doc already flagged superseded or obsolete.
- docs/working/archive/dead/: the hal_* target contract (never adopted),
  L4Re, metrics-and-knobs.
- Papers and proposals into docs/working/papers/.
- References rewritten across the repo (not logs/, not docs/patent/);
  CLAUDE.md's documentation section rewritten.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
rajames
2026-10-14 16:03:56 -04:00
co-authored by Claude Opus 5.5
parent 1a4b163861
commit 7aed588f78
116 changed files with 305 additions and 175 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/working/archive/DOCUMENTATION_AUDIT.md
+4 -4
View File
@@ -148,7 +148,7 @@ This directory contains experimental designs, execution guides, and results for
- Results for publication: `06-research/results-for-publication.md`
```
#### `docs/03-architecture/README.md`
#### `docs/working/archive/superseded/2026-08-tripod/03-architecture-README.md`
```markdown
# Architecture
@@ -427,7 +427,7 @@ This document provides **kernel-specific implementation details**...
#### Incomplete Documents
**`docs/ONTOLOGY.md`:**
**`docs/reference/ONTOLOGY.md`:**
- Appears to be high-level conceptual framework
- Could benefit from linking to specific implementation docs
- Should reference HAL as part of platform layer
@@ -510,7 +510,7 @@ The `archive/` directory preserves historical context without cluttering active
- Mention StarKernel/StarshipOS vision
- **Effort:** 15 minutes
4. **Create `docs/03-architecture/README.md`**
4. **Create `docs/working/archive/superseded/2026-08-tripod/03-architecture-README.md`**
- Big picture of how subsystems fit together
- Roadmap (StarForth → StarKernel → StarshipOS)
- **Effort:** 1 hour
@@ -581,7 +581,7 @@ The `archive/` directory preserves historical context without cluttering active
1. Create 6 missing README files for numbered directories
2. Update `docs/CLAUDE.md` with HAL section
3. Update `docs/README.md` with current work
4. Create `docs/03-architecture/README.md`
4. Create `docs/working/archive/superseded/2026-08-tripod/03-architecture-README.md`
### Short-term Actions (1-2 weeks)
docs/working/archive/dead/hal-target-contract/README.md
+246
View File
@@ -0,0 +1,246 @@
> **DEAD (docs triage 2026-10-14).** The proposed `hal_*` interface contract and its migration plan were never adopted; the kernel grew its own HAL (kernel/src/hal/, kernel/include/starkernel/hal/).
> Instead: kernel/src/hal/README.md. Moved here from `docs/working/architecture/03-architecture/hal/README.md`.
<!-- Moved from docs/03-architecture/hal/README.md to docs/working/archive/dead/hal-target-contract/README.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ TARGET HAL — NOT THE BUILT INTERFACE (flagged 2026-10-09).** This document family
> describes a proposed `hal_*`-named interface and a `src/platform/<platform>/` layout that
> was never adopted as written. The implemented kernel HAL lives in `src/starkernel/hal/`
> with different (time-backend-style) naming. Treat all five documents here as design intent,
> not the current API. For current architecture see `.claude/CLAUDE.md`.
# Hardware Abstraction Layer (HAL) Documentation
This directory contains comprehensive documentation for StarForth's Hardware Abstraction Layer (HAL), which enables the evolution from hosted VM to StarKernel to StarshipOS.
> **HISTORICAL — L4Re/Fiasco.OC**: L4Re references throughout this directory's docs describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build.
---
## Document Organization
Read in this order for full understanding:
### 1. **[overview.md](overview.md)** - Start Here
**What:** High-level HAL architecture, design principles, and rationale
**Who:** Everyone (developers, architects, contributors)
**Why read:** Understand the problem HAL solves and how it fits into StarForth → StarKernel → StarshipOS
**Key sections:**
- The Problem (why HAL is necessary)
- Architecture Overview (3-layer model)
- Design Principles (VM purity, contract-first, testability)
- HAL Subsystems (time, interrupts, memory, console, CPU)
- HAL and Physics Subsystems
- Success Criteria
---
### 2. **[interfaces.md](interfaces.md)** - The Contract
**What:** Detailed HAL interface specifications (function signatures, semantics, guarantees)
**Who:** VM developers, platform implementers
**Why read:** Understand exactly what each HAL function does and requires
**Key sections:**
- Time & Timers interface (`hal_time.h`)
- Interrupts interface (`hal_interrupt.h`)
- Memory interface (`hal_memory.h`)
- Console interface (`hal_console.h`)
- CPU interface (`hal_cpu.h`)
- Panic/Error interface (`hal_panic.h`)
- Interface safety summary (ISR-safe, thread-safe)
**Critical details:**
- Function semantics (what it does, guaranteed behavior)
- Error handling (when it can fail, how to handle)
- Performance expectations (latency, overhead)
- Concurrency model (thread-safe? ISR-safe?)
---
### 3. **[platform-implementations.md](platform-implementations.md)** - How to Implement
**What:** Platform-specific implementation examples and guidance
**Who:** Platform developers adding HAL support for new targets
**Why read:** Learn how to implement HAL for a specific platform
**Key sections:**
- Platform directory structure
- Linux HAL implementation (reference platform)
- Kernel HAL implementation (StarKernel)
- Platform testing strategy
- Common implementation pitfalls
**Includes complete code examples for:**
- Linux (POSIX, hosted)
- Kernel (freestanding, bare metal)
---
### 4. **[migration-plan.md](migration-plan.md)** - Refactoring Guide
**What:** Step-by-step plan for refactoring existing StarForth to use HAL
**Who:** Core developers doing the migration work
**Why read:** Execute the HAL migration safely with zero regressions
**Key sections:**
- 7-phase migration strategy
- Incremental refactoring approach
- Test validation after each phase
- Rollback strategy
- Timeline estimates (2-4 weeks)
**Phases:**
1. Define HAL interfaces (headers only)
2. Implement HAL for Linux
3. Migrate VM core to use HAL
4. Migrate physics subsystems to use HAL
5. Migrate REPL and word implementations
6. Implement HAL for L4Re (optional validation)
7. Validate deterministic behavior
---
### 5. **[starkernel-integration.md](starkernel-integration.md)** - Kernel Details
**What:** StarKernel-specific HAL implementation and boot sequence
**Who:** Kernel developers building StarKernel
**Why read:** Understand UEFI boot, freestanding C, hardware initialization
**Key sections:**
- StarKernel architecture (boot sequence)
- Directory structure (`src/platform/kernel/`)
- Boot sequence detail (UEFI → StarKernel → VM)
- HAL implementation details (time, interrupts, memory, console, CPU)
- Build system (freestanding toolchain, linker script)
- Testing on QEMU/OVMF
- Debugging techniques
- Roadmap to `ok` prompt
**Includes:**
- Complete UEFI loader implementation
- Kernel entry point
- Hardware initialization sequence
- QEMU testing guide
- Debugging strategies
---
## Quick Reference
### For Different Roles
**I'm a VM developer:**
- Read: `overview.md`, `interfaces.md`
- Use HAL functions in VM code
- Never call platform-specific APIs directly
**I'm adding a new platform:**
- Read: `overview.md`, `interfaces.md`, `platform-implementations.md`
- Implement all HAL interfaces for your platform
- Test against VM test suite (936+ tests must pass)
**I'm migrating existing code to HAL:**
- Read: `migration-plan.md`
- Follow 7-phase incremental approach
- Test after each phase
**I'm building StarKernel:**
- Read: All documents, especially `starkernel-integration.md`
- Start with UEFI boot + serial output
- Build incrementally to `ok` prompt
---
## Key Concepts
### HAL Principles
1. **VM Purity:** VM code is platform-agnostic, never knows which platform it's on
2. **Contract-First:** HAL interfaces are contracts with precise semantics
3. **Testability:** Develop/test on Linux, deploy on kernel
4. **Zero Overhead:** HAL inlines to direct hardware access when optimized
5. **Fail-Fast:** Validate platform assumptions at init, not during execution
### Success Criteria
The HAL is successful if:
- ✅ VM core has zero platform-specific code
- ✅ All 936+ tests pass on all platforms
- ✅ 0% algorithmic variance maintained
- ✅ No measurable performance regression
- ✅ StarKernel boots to `ok` prompt
- ✅ Physics subsystems work identically on all platforms
---
## Document Status
| Document | Status | Last Updated |
|----------|--------|--------------|
| overview.md | ✅ Complete | 2025-12-14 |
| interfaces.md | ✅ Complete | 2025-12-14 |
| platform-implementations.md | ✅ Complete | 2025-12-14 |
| migration-plan.md | ✅ Complete | 2025-12-14 |
| starkernel-integration.md | ✅ Complete | 2025-12-14 |
---
## Next Steps
### Immediate (HAL Migration)
1. Review and approve HAL interface specifications
2. Execute migration plan Phase 1 (define interfaces)
3. Execute migration plan Phase 2 (implement Linux HAL)
4. Continue through Phase 7 (validation)
### Near-Term (StarKernel)
1. Implement UEFI boot loader
2. Implement kernel HAL subsystems
3. Boot to `ok` prompt on QEMU
4. Validate physics subsystems on bare metal
### Long-Term (StarshipOS)
1. Add storage drivers (AHCI, NVMe)
2. Add networking stack (TCP/IP)
3. Implement process model (Forth tasks)
4. Build device model (block/net/char)
---
## References
### StarForth Documentation
- `docs/CLAUDE.md` - Project overview and build instructions
- `docs/03-architecture/heartbeat-system/` - Heartbeat subsystem details
- `docs/FEEDBACK_LOOPS.md` - Physics feedback loops
- `README.md` - Project quick start
### External Resources
- [UEFI Specification](https://uefi.org/specifications)
- [GNU-EFI](https://sourceforge.net/projects/gnu-efi/)
- [OSDev Wiki](https://wiki.osdev.org/)
- [Intel Software Developer Manual](https://www.intel.com/sdm)
- [ACPI Specification](https://uefi.org/acpi)
---
## Contributing
When updating HAL documentation:
1. **Keep documents synchronized** - Changes to interfaces require updates to implementation guides
2. **Maintain code examples** - Examples must compile and work
3. **Update this README** - Reflect new sections or document reorganization
4. **Version documentation** - Note "Last Updated" dates
5. **Test instructions** - Verify all commands and code snippets work
---
## Questions?
For questions about HAL architecture or StarKernel implementation, see:
- GitHub Issues: https://github.com/anthropics/starforth/issues (if public)
- Project maintainer: (contact info)
- Documentation feedback: (preferred method)
---
*The HAL is not just a kernel bootstrapping tool—it's the architectural foundation for StarForth → StarKernel → StarshipOS.*
docs/working/archive/dead/hal-target-contract/interfaces.md
+713
View File
@@ -0,0 +1,713 @@
> **DEAD (docs triage 2026-10-14).** The proposed `hal_*` interface contract and its migration plan were never adopted; the kernel grew its own HAL (kernel/src/hal/, kernel/include/starkernel/hal/).
> Instead: kernel/src/hal/README.md. Moved here from `docs/working/architecture/03-architecture/hal/interfaces.md`.
<!-- Moved from docs/03-architecture/hal/interfaces.md to docs/working/archive/dead/hal-target-contract/interfaces.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ TARGET HAL — NOT THE BUILT INTERFACE (flagged 2026-10-09).** This document defines the
> proposed `hal_*` interface contract; it was never adopted as written. The implemented kernel
> HAL is `src/starkernel/hal/` with different naming. See `README.md` in this directory and
> `.claude/CLAUDE.md` for what is actually built.
# HAL Interface Specifications
## Overview
This document defines the **contract** for each HAL subsystem. These interfaces are platform-agnostic and must be implemented identically (semantically) on all platforms.
> **HISTORICAL — L4Re/Fiasco.OC**: L4Re references below describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build.
**Critical:** These are not optional "convenience wrappers"—they are the **only** way VM and physics code access platform resources.
---
## 1. Time & Timers (`include/hal/hal_time.h`)
### Purpose
Provide monotonic time, periodic/oneshot timers, and calibration for physics subsystems.
### Interface
```c
#ifndef HAL_TIME_H
#define HAL_TIME_H
#include <stdint.h>
/**
* Initialize time subsystem.
* Must be called before any other hal_time_* functions.
*
* Responsibilities:
* - Calibrate high-resolution timer (TSC, HPET, etc.)
* - Validate monotonicity
* - Set up timer interrupt sources (if needed)
*
* Error handling:
* - hal_panic() if no suitable time source available
* - hal_panic() if calibration fails
*/
void hal_time_init(void);
/**
* Get current monotonic time in nanoseconds.
*
* Guarantees:
* - MUST be monotonic (never goes backward)
* - SHOULD have sub-microsecond resolution
* - MUST be callable from interrupt context (ISR-safe)
* - MUST be callable concurrently (thread-safe)
*
* Performance:
* - Target: < 50 cycles on x86_64
* - Acceptable: < 200 cycles
*
* Returns:
* - Nanoseconds since arbitrary epoch (typically boot time)
*/
uint64_t hal_time_now_ns(void);
/**
* Busy-wait delay for specified nanoseconds.
* Used for calibration and short delays.
*
* Guarantees:
* - MUST delay for at least `ns` nanoseconds
* - MAY delay longer (but not by more than 10%)
* - MUST be callable from interrupt context
*
* Parameters:
* - ns: Nanoseconds to delay (0 = no delay)
*
* Notes:
* - This is a busy-wait, not a sleep
* - For delays > 1ms, prefer hal_timer_oneshot()
*/
void hal_time_delay_ns(uint64_t ns);
/**
* Timer callback function type.
*
* Parameters:
* - ctx: User-provided context pointer
*
* Context:
* - MAY be called from interrupt context (ISR)
* - MUST NOT block or sleep
* - MUST complete quickly (< 10µs target)
*/
typedef void (*hal_timer_callback_t)(void *ctx);
/**
* Schedule a oneshot timer to fire after specified nanoseconds.
*
* Guarantees:
* - Callback will fire at least `ns` nanoseconds in the future
* - MAY fire up to 10% late (jitter allowed)
* - Callback runs in interrupt context (ISR)
*
* Parameters:
* - ns: Nanoseconds until callback fires (0 = ASAP)
* - callback: Function to call when timer fires
* - ctx: User context passed to callback
*
* Error handling:
* - Returns 0 on success
* - Returns -1 if timer resources exhausted
*
* Concurrency:
* - Thread-safe (can be called from multiple threads)
* - NOT reentrant from same timer callback
*/
int hal_timer_oneshot(uint64_t ns, hal_timer_callback_t callback, void *ctx);
/**
* Schedule a periodic timer to fire every `period_ns` nanoseconds.
*
* Guarantees:
* - Callback will fire repeatedly at ~period_ns intervals
* - Jitter: < 10% of period (target: < 1%)
* - Callback runs in interrupt context (ISR)
* - Timer continues until hal_timer_cancel() called
*
* Parameters:
* - period_ns: Period in nanoseconds (minimum: 1000ns = 1µs)
* - callback: Function to call on each period
* - ctx: User context passed to callback
*
* Returns:
* - Timer ID (> 0) on success
* - -1 if timer resources exhausted
* - -2 if period_ns < minimum
*
* Notes:
* - This is the PRIMARY mechanism for heartbeat ISR
* - Platform MUST minimize jitter for experimental validity
*/
int hal_timer_periodic(uint64_t period_ns, hal_timer_callback_t callback, void *ctx);
/**
* Cancel a periodic timer.
*
* Parameters:
* - timer_id: ID returned from hal_timer_periodic()
*
* Guarantees:
* - Callback will not fire after this call returns
* - Safe to call even if timer already fired or was cancelled
*
* Returns:
* - 0 on success
* - -1 if timer_id invalid
*/
int hal_timer_cancel(int timer_id);
/**
* Get timer frequency in Hz.
* Used for diagnostics and experiment metadata.
*
* Returns:
* - Frequency of underlying time source (e.g., TSC frequency)
* - 0 if frequency is variable or unknown
*/
uint64_t hal_time_frequency_hz(void);
#endif /* HAL_TIME_H */
```
### Platform Implementation Notes
**Linux (`platform/linux/hal_time.c`):**
- `hal_time_now_ns()` → `clock_gettime(CLOCK_MONOTONIC)`
- `hal_timer_periodic()` → `timer_create()` with `SIGRTMIN` + signal handler
- Challenge: Signal-based timers have ~1-10µs latency jitter
**L4Re (`platform/l4re/hal_time.c`):**
- `hal_time_now_ns()` → `l4re_kip_clock()`
- `hal_timer_periodic()` → L4Re IPC-based timer + IRQ object
- Challenge: IPC overhead, but lower jitter than Linux
**Kernel (`platform/kernel/hal_time.c`):**
- `hal_time_now_ns()` → `rdtsc()` calibrated against HPET
- `hal_timer_periodic()` → APIC timer interrupt
- Challenge: TSC drift on old hardware, calibration complexity
---
## 2. Interrupts (`include/hal/hal_interrupt.h`)
### Purpose
Enable/disable interrupts, register ISRs, query interrupt context.
### Interface
```c
#ifndef HAL_INTERRUPT_H
#define HAL_INTERRUPT_H
#include <stdint.h>
/**
* Initialize interrupt subsystem.
* Must be called before any other hal_interrupt_* functions.
*
* Responsibilities:
* - Set up interrupt controller (PIC/APIC on x86_64)
* - Initialize IDT (on kernel platforms)
* - Prepare for ISR registration
*
* Error handling:
* - hal_panic() if interrupt controller unavailable
*/
void hal_interrupt_init(void);
/**
* Enable interrupts globally.
*
* Guarantees:
* - Interrupts will be delivered after this call
* - Safe to call multiple times (idempotent)
* - NOT callable from interrupt context
*
* Notes:
* - On x86_64 kernel: sti
* - On Linux: no-op (interrupts always enabled)
*/
void hal_irq_enable(void);
/**
* Disable interrupts globally.
*
* Guarantees:
* - Interrupts will NOT be delivered after this call
* - Safe to call multiple times (idempotent)
* - Callable from interrupt context
*
* Returns:
* - Previous interrupt state (for hal_irq_restore)
*
* Notes:
* - On x86_64 kernel: cli
* - On Linux: no-op (interrupts always enabled)
*/
unsigned long hal_irq_disable(void);
/**
* Restore previous interrupt state.
*
* Parameters:
* - state: Value returned from hal_irq_disable()
*
* Use case:
* unsigned long flags = hal_irq_disable();
* // critical section
* hal_irq_restore(flags);
*/
void hal_irq_restore(unsigned long state);
/**
* Interrupt service routine function type.
*
* Parameters:
* - irq: IRQ number that triggered
* - ctx: User-provided context pointer
*
* Context:
* - ALWAYS called from interrupt context
* - MUST NOT block or sleep
* - MUST complete quickly (< 10µs target, < 100µs acceptable)
* - MUST NOT call non-ISR-safe HAL functions
*/
typedef void (*hal_isr_t)(unsigned int irq, void *ctx);
/**
* Register an interrupt service routine.
*
* Parameters:
* - irq: IRQ number to register (platform-specific numbering)
* - isr: Function to call when IRQ fires
* - ctx: User context passed to ISR
*
* Guarantees:
* - ISR will be called when IRQ fires
* - Only one ISR per IRQ (replaces previous if registered)
* - ISR runs with interrupts disabled (not reentrant)
*
* Returns:
* - 0 on success
* - -1 if IRQ number invalid
* - -2 if ISR registration failed
*
* Notes:
* - Used by hal_timer_periodic() internally
* - Used by device drivers for hardware IRQs
*/
int hal_irq_register(unsigned int irq, hal_isr_t isr, void *ctx);
/**
* Unregister an interrupt service routine.
*
* Parameters:
* - irq: IRQ number to unregister
*
* Returns:
* - 0 on success
* - -1 if IRQ was not registered
*/
int hal_irq_unregister(unsigned int irq);
/**
* Check if currently executing in interrupt context.
*
* Returns:
* - 1 if in interrupt context (ISR)
* - 0 if in normal execution context
*
* Use case:
* - Validate ISR-only functions (e.g., heartbeat sampling)
* - Conditional behavior (ISR vs. normal path)
*
* Performance:
* - MUST be fast (< 10 cycles)
* - Often implemented as checking a per-CPU flag
*/
int hal_in_interrupt_context(void);
#endif /* HAL_INTERRUPT_H */
```
### Platform Implementation Notes
**Linux:**
- `hal_irq_enable/disable()` → no-op (can't disable signals)
- `hal_irq_register()` → `sigaction()` + signal handler
- `hal_in_interrupt_context()` → thread-local flag set by signal handler
**Kernel:**
- `hal_irq_enable/disable()` → `sti`/`cli` (x86_64)
- `hal_irq_register()` → IDT entry + APIC configuration
- `hal_in_interrupt_context()` → check per-CPU interrupt nesting count
---
## 3. Memory (`include/hal/hal_memory.h`)
### Purpose
Allocate/free memory, map pages, manage heap.
### Interface
```c
#ifndef HAL_MEMORY_H
#define HAL_MEMORY_H
#include <stddef.h>
#include <stdint.h>
/**
* Initialize memory subsystem.
* Must be called before any other hal_mem_* functions.
*
* Responsibilities:
* - Set up heap allocator
* - Initialize page tables (kernel platforms)
* - Parse memory map (kernel platforms)
*
* Error handling:
* - hal_panic() if insufficient memory
*/
void hal_mem_init(void);
/**
* Allocate memory from heap.
*
* Parameters:
* - size: Bytes to allocate (0 = returns NULL)
*
* Guarantees:
* - Returns aligned pointer (at least 8-byte aligned)
* - Memory is zero-initialized
* - Thread-safe (can be called concurrently)
* - NOT callable from interrupt context
*
* Returns:
* - Pointer to allocated memory
* - NULL if allocation failed
*
* Notes:
* - On Linux: wrapper around malloc()
* - On kernel: custom heap allocator (kmalloc)
*/
void *hal_mem_alloc(size_t size);
/**
* Free memory allocated by hal_mem_alloc().
*
* Parameters:
* - ptr: Pointer returned from hal_mem_alloc() (NULL = no-op)
*
* Guarantees:
* - Thread-safe
* - NOT callable from interrupt context
* - Safe to call with NULL
*/
void hal_mem_free(void *ptr);
/**
* Allocate contiguous physical pages.
* Used for DMA buffers, page tables, large allocations.
*
* Parameters:
* - count: Number of pages to allocate (page size is platform-specific)
*
* Returns:
* - Physical address of first page
* - 0 if allocation failed
*
* Notes:
* - Pages are NOT mapped into virtual address space
* - Caller must call hal_mem_map() to access
* - On Linux: may use mmap() with MAP_ANONYMOUS
* - On kernel: physical memory manager (PMM)
*/
uint64_t hal_mem_alloc_pages(size_t count);
/**
* Free physical pages allocated by hal_mem_alloc_pages().
*
* Parameters:
* - paddr: Physical address returned from hal_mem_alloc_pages()
* - count: Number of pages to free
*/
void hal_mem_free_pages(uint64_t paddr, size_t count);
/**
* Map physical memory into virtual address space.
*
* Parameters:
* - vaddr: Virtual address to map (must be page-aligned)
* - paddr: Physical address to map (must be page-aligned)
* - size: Size in bytes (must be page-multiple)
* - flags: Mapping flags (HAL_MEM_*)
*
* Returns:
* - 0 on success
* - -1 if mapping failed
*
* Flags:
* - HAL_MEM_READ: Readable
* - HAL_MEM_WRITE: Writable
* - HAL_MEM_EXEC: Executable
* - HAL_MEM_NOCACHE: Uncached (for MMIO)
*
* Notes:
* - On Linux: no-op (identity mapping assumed)
* - On kernel: update page tables, flush TLB
*/
int hal_mem_map(uint64_t vaddr, uint64_t paddr, size_t size, unsigned int flags);
/* Mapping flags */
#define HAL_MEM_READ (1 << 0)
#define HAL_MEM_WRITE (1 << 1)
#define HAL_MEM_EXEC (1 << 2)
#define HAL_MEM_NOCACHE (1 << 3)
/**
* Get page size in bytes.
*
* Returns:
* - Page size (typically 4096 on x86_64)
*/
size_t hal_mem_page_size(void);
#endif /* HAL_MEMORY_H */
```
---
## 4. Console (`include/hal/hal_console.h`)
### Purpose
Character I/O for REPL and diagnostics.
### Interface
```c
#ifndef HAL_CONSOLE_H
#define HAL_CONSOLE_H
/**
* Initialize console subsystem.
* Must be called before any other hal_console_* functions.
*
* Responsibilities:
* - Initialize UART (kernel platforms)
* - Set up framebuffer (if available)
* - Configure terminal settings (Linux)
*
* Error handling:
* - hal_panic() if no console available
*/
void hal_console_init(void);
/**
* Write a single character to console.
*
* Parameters:
* - c: Character to write
*
* Guarantees:
* - Character will be visible on console
* - Blocking (waits if output buffer full)
* - Callable from interrupt context (for panic messages)
*
* Performance:
* - May be slow (UART is ~1µs per char)
* - Buffering recommended for bulk output
*/
void hal_console_putc(char c);
/**
* Write a null-terminated string to console.
*
* Parameters:
* - s: String to write (NULL = no-op)
*
* Notes:
* - Convenience wrapper around hal_console_putc()
*/
void hal_console_puts(const char *s);
/**
* Read a single character from console.
*
* Guarantees:
* - Blocking (waits until character available)
* - NOT callable from interrupt context
*
* Returns:
* - Character read (0-255)
* - -1 on error or EOF
*/
int hal_console_getc(void);
/**
* Check if input is available (non-blocking).
*
* Returns:
* - 1 if character available (hal_console_getc() will not block)
* - 0 if no character available
*/
int hal_console_has_input(void);
#endif /* HAL_CONSOLE_H */
```
---
## 5. CPU (`include/hal/hal_cpu.h`)
### Purpose
CPU identification, relax/halt, SMP coordination.
### Interface
```c
#ifndef HAL_CPU_H
#define HAL_CPU_H
#include <stdint.h>
/**
* Initialize CPU subsystem.
* Must be called before any other hal_cpu_* functions.
*
* Responsibilities:
* - Detect CPU features (SSE, AVX, etc.)
* - Set up per-CPU storage
* - Bring up secondary CPUs (SMP)
*
* Error handling:
* - hal_panic() if CPU unsupported
*/
void hal_cpu_init(void);
/**
* Get current CPU ID.
*
* Returns:
* - CPU ID (0 = boot processor, 1+ = application processors)
*
* Guarantees:
* - Callable from interrupt context
* - Fast (< 10 cycles)
*
* Notes:
* - On x86_64 kernel: APIC ID
* - On Linux: thread ID (not true CPU ID)
*/
unsigned int hal_cpu_id(void);
/**
* Hint to CPU to relax (pause in spin-wait loop).
*
* Guarantees:
* - Callable from interrupt context
* - No functional effect (pure performance hint)
*
* Notes:
* - On x86_64: pause instruction
* - On ARM: yield instruction
* - On Linux: sched_yield()
*/
void hal_cpu_relax(void);
/**
* Halt CPU until next interrupt.
*
* Guarantees:
* - CPU enters low-power state
* - Wakes on next interrupt
* - NOT callable from interrupt context
*
* Notes:
* - On x86_64 kernel: hlt instruction
* - On Linux: select() or nanosleep()
*/
void hal_cpu_halt(void);
/**
* Get number of CPUs in system.
*
* Returns:
* - Number of CPUs (1 = single-core, 2+ = multi-core)
*/
unsigned int hal_cpu_count(void);
#endif /* HAL_CPU_H */
```
---
## 6. Panic/Error Handling (`include/hal/hal_panic.h`)
### Purpose
Fatal error handling (system halt or abort).
### Interface
```c
#ifndef HAL_PANIC_H
#define HAL_PANIC_H
/**
* Fatal error: print message and halt system.
*
* Parameters:
* - msg: Error message (NULL = "panic")
*
* Guarantees:
* - NEVER returns
* - Message printed to console (if available)
* - System halted (kernel) or abort() (hosted)
* - Callable from interrupt context
*
* Notes:
* - Use for unrecoverable errors only
* - On kernel: disable interrupts + hlt loop
* - On Linux: fprintf(stderr) + abort()
*/
void hal_panic(const char *msg) __attribute__((noreturn));
#endif /* HAL_PANIC_H */
```
---
## Interface Summary
| Subsystem | Key Functions | ISR-Safe | Thread-Safe |
|-----------|---------------|----------|-------------|
| **Time** | `hal_time_now_ns()` | ✅ | ✅ |
| | `hal_timer_periodic()` | ❌ | ✅ |
| **Interrupt** | `hal_in_interrupt_context()` | ✅ | ✅ |
| | `hal_irq_disable()` | ✅ | ✅ |
| **Memory** | `hal_mem_alloc()` | ❌ | ✅ |
| | `hal_mem_map()` | ❌ | ⚠️ |
| **Console** | `hal_console_putc()` | ✅ | ⚠️ |
| | `hal_console_getc()` | ❌ | ❌ |
| **CPU** | `hal_cpu_id()` | ✅ | ✅ |
| | `hal_cpu_relax()` | ✅ | ✅ |
**Legend:**
- ✅ = Safe
- ❌ = Unsafe
- ⚠️ = Platform-dependent
---
## Next Steps
See companion documentation:
- `platform-implementations.md` - How to implement HAL for a platform
- `migration-plan.md` - Refactoring existing code to use HAL
- `starkernel-integration.md` - StarKernel-specific HAL implementation
docs/working/archive/dead/hal-target-contract/migration-plan.md
+660
View File
@@ -0,0 +1,660 @@
> **DEAD (docs triage 2026-10-14).** The proposed `hal_*` interface contract and its migration plan were never adopted; the kernel grew its own HAL (kernel/src/hal/, kernel/include/starkernel/hal/).
> Instead: kernel/src/hal/README.md. Moved here from `docs/working/architecture/03-architecture/hal/migration-plan.md`.
<!-- Moved from docs/03-architecture/hal/migration-plan.md to docs/working/archive/dead/hal-target-contract/migration-plan.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ TARGET HAL — NOT THE BUILT INTERFACE (flagged 2026-10-09).** This migration plan targets
> the proposed `hal_*` interface, which was never adopted as written. The kernel grew its own
> HAL directly in `src/starkernel/hal/`. Kept for design rationale; see `.claude/CLAUDE.md`
> for the built architecture.
# HAL Migration Plan
## Overview
This document provides a **step-by-step plan** for refactoring StarForth to use the Hardware Abstraction Layer (HAL). The migration preserves all existing functionality while preparing the codebase for StarKernel.
> **HISTORICAL — L4Re/Fiasco.OC**: L4Re references below describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build.
**Goal:** Zero functional regressions, all 936+ tests pass, 0% algorithmic variance maintained.
---
## Migration Strategy
### Principles
1. **Incremental refactoring** - One subsystem at a time
2. **Test after each step** - Build + test suite must pass
3. **Platform parity** - Linux and L4Re work throughout migration
4. **No big bang** - HAL interfaces defined first, then adopted gradually
### Phases
```
Phase 1: Define HAL Interfaces (headers only)
↓
Phase 2: Implement HAL for Linux (refactor existing code)
↓
Phase 3: Migrate VM Core to use HAL
↓
Phase 4: Migrate Physics Subsystems to use HAL
↓
Phase 5: Migrate REPL and Word Implementations
↓
Phase 6: Implement HAL for L4Re (optional validation)
↓
Phase 7: Validate Deterministic Behavior
```
---
## Phase 1: Define HAL Interfaces
**Duration:** 1-2 days
**Risk:** Low (no code changes, only headers)
### Tasks
1. **Create HAL header directory**
```bash
mkdir -p include/hal
```
2. **Write HAL interface headers**
- `include/hal/hal_time.h`
- `include/hal/hal_interrupt.h`
- `include/hal/hal_memory.h`
- `include/hal/hal_console.h`
- `include/hal/hal_cpu.h`
- `include/hal/hal_panic.h`
See `interfaces.md` for full specifications.
3. **Update Makefile to include HAL headers**
```makefile
INCLUDES += -Iinclude/hal
```
4. **Compile check (headers only)**
```bash
make clean && make PLATFORM=linux
```
Should compile without errors (HAL functions not yet called).
### Success Criteria
✅ All HAL headers compile without errors
✅ No changes to VM code yet
✅ Documentation reviewed and approved
---
## Phase 2: Implement HAL for Linux
**Duration:** 3-5 days
**Risk:** Medium (refactoring existing platform code)
### Current Platform Code Locations
Audit existing platform-specific code:
```bash
# Find POSIX-specific calls
grep -r "clock_gettime\|pthread\|signal\|malloc\|printf" src/
```
**Expected locations:**
- Timing: `src/platform/linux/time.c` (already exists)
- Memory: Direct `malloc()` calls in VM code (needs refactoring)
- Console: Direct `printf()` calls in REPL (needs refactoring)
### Tasks
1. **Create Linux HAL directory**
```bash
mkdir -p src/platform/linux/hal
```
2. **Implement Linux HAL subsystems**
**2a. `hal_time.c`** - Refactor `src/platform/linux/time.c`
```c
/* Before: */
uint64_t platform_time_ns(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return ...;
}
/* After: */
uint64_t hal_time_now_ns(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (uint64_t)ts.tv_sec * 1000000000ULL + ts.tv_nsec;
}
```
**2b. `hal_memory.c`** - Wrap malloc/free
```c
void *hal_mem_alloc(size_t size) {
if (size == 0) return NULL;
void *ptr = malloc(size);
if (ptr) memset(ptr, 0, size);
return ptr;
}
void hal_mem_free(void *ptr) {
free(ptr);
}
```
**2c. `hal_console.c`** - Wrap stdio
```c
void hal_console_putc(char c) {
putchar(c);
fflush(stdout);
}
int hal_console_getc(void) {
return getchar();
}
```
**2d. `hal_interrupt.c`** - Signal-based interrupts
See `platform-implementations.md` for full implementation.
**2e. `hal_cpu.c`** - CPU info
```c
unsigned int hal_cpu_id(void) {
return 0; /* Single-threaded */
}
void hal_cpu_relax(void) {
sched_yield();
}
```
**2f. `hal_panic.c`** - Error handling
```c
void hal_panic(const char *msg) {
fprintf(stderr, "PANIC: %s\n", msg ? msg : "unknown");
abort();
}
```
3. **Update Makefile**
```makefile
ifeq ($(PLATFORM),linux)
PLATFORM_SOURCES = \
src/platform/linux/hal_time.c \
src/platform/linux/hal_interrupt.c \
src/platform/linux/hal_memory.c \
src/platform/linux/hal_console.c \
src/platform/linux/hal_cpu.c \
src/platform/linux/hal_panic.c
endif
```
4. **Build test (HAL code compiles)**
```bash
make clean && make PLATFORM=linux
```
### Success Criteria
✅ Linux HAL compiles without errors
✅ Existing VM code still builds (not yet using HAL)
✅ No functional changes yet
---
## Phase 3: Migrate VM Core to use HAL
**Duration:** 2-3 days
**Risk:** High (core VM changes)
### Files to Modify
- `src/vm.c` - Core interpreter loop
- `src/vm_api.c` - External API
- `src/memory_management.c` - Dictionary allocator
- `include/vm.h` - VM struct
### Tasks
1. **Replace direct malloc/free in VM core**
**Before:**
```c
/* src/memory_management.c */
void *mem = malloc(size);
```
**After:**
```c
#include "hal/hal_memory.h"
void *mem = hal_mem_alloc(size);
```
2. **Replace timing calls in heartbeat**
**Before:**
```c
/* src/vm.c */
#include <time.h>
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
uint64_t now = ts.tv_sec * 1000000000ULL + ts.tv_nsec;
```
**After:**
```c
#include "hal/hal_time.h"
uint64_t now = hal_time_now_ns();
```
3. **Add HAL initialization to VM startup**
**In `src/main.c`:**
```c
int main(int argc, char **argv) {
/* Initialize HAL subsystems */
hal_time_init();
hal_interrupt_init();
hal_mem_init();
hal_console_init();
hal_cpu_init();
/* Continue with existing VM initialization */
VM *vm = vm_create();
...
}
```
4. **Remove platform-specific #ifdefs**
**Before:**
```c
#ifdef PLATFORM_LINUX
clock_gettime(CLOCK_MONOTONIC, &ts);
#elif PLATFORM_L4RE
l4re_kip_clock(kip);
#endif
```
**After:**
```c
uint64_t now = hal_time_now_ns(); /* Works on all platforms */
```
5. **Build and test**
```bash
make clean && make PLATFORM=linux
make test
```
### Expected Failures
- **Compilation errors:** Missing `#include "hal/hal_*.h"`
- **Link errors:** HAL functions not implemented
- **Runtime errors:** HAL not initialized before VM
**Fix incrementally:** Add includes, implement missing HAL functions, ensure init order.
### Success Criteria
✅ VM builds using HAL interfaces
✅ All 936+ tests pass
✅ No platform-specific code in `src/vm.c`
---
## Phase 4: Migrate Physics Subsystems to use HAL
**Duration:** 2-3 days
**Risk:** Medium (determinism must be preserved)
### Files to Modify
- `src/dictionary_heat_optimization.c`
- `src/rolling_window_of_truth.c`
- `src/physics_hotwords_cache.c`
- `src/physics_pipelining_metrics.c`
- `src/inference_engine.c`
- `src/heartbeat.c`
### Tasks
1. **Audit timing calls in physics subsystems**
```bash
grep -n "clock_gettime\|time\.h" src/*physics*.c src/heartbeat.c src/inference*.c
```
2. **Replace timing calls**
**Example: `src/rolling_window_of_truth.c`**
```c
/* Before: */
#include <time.h>
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
uint64_t timestamp = ...;
/* After: */
#include "hal/hal_time.h"
uint64_t timestamp = hal_time_now_ns();
```
3. **Migrate heartbeat to HAL timer**
**Before:** `src/heartbeat.c` uses pthread + sleep
```c
void *heartbeat_thread(void *arg) {
while (running) {
usleep(period_us);
vm_tick(vm);
}
}
```
**After:** Use HAL periodic timer
```c
static VM *heartbeat_vm = NULL;
static void heartbeat_isr(void *ctx) {
(void)ctx;
if (heartbeat_vm) {
vm_tick_isr(heartbeat_vm); /* ISR-safe VM tick */
}
}
void heartbeat_start(VM *vm, uint64_t rate_hz) {
heartbeat_vm = vm;
uint64_t period_ns = 1000000000ULL / rate_hz;
hal_timer_periodic(period_ns, heartbeat_isr, NULL);
}
```
**Critical:** `vm_tick_isr()` must be ISR-safe:
- No malloc/free
- No blocking I/O
- Only lock-free ring buffer updates
4. **Test deterministic behavior**
```bash
make fastest PLATFORM=linux
./build/amd64/fastest/starforth --doe > results_after_hal.csv
```
Compare with baseline (before HAL migration):
```bash
diff results_before_hal.csv results_after_hal.csv
```
**Expected:** Identical results (0% algorithmic variance).
### Success Criteria
✅ Physics subsystems use HAL for all timing
✅ Heartbeat runs via HAL periodic timer
✅ 0% algorithmic variance maintained
✅ All tests pass
---
## Phase 5: Migrate REPL and Word Implementations
**Duration:** 1-2 days
**Risk:** Low (mostly console I/O)
### Files to Modify
- `src/repl.c` - Read-eval-print loop
- `src/word_source/*_words.c` - Words that do I/O (emit, key, etc.)
### Tasks
1. **Replace stdio in REPL**
**Before:** `src/repl.c`
```c
#include <stdio.h>
char c = getchar();
printf("ok\n");
```
**After:**
```c
#include "hal/hal_console.h"
char c = hal_console_getc();
hal_console_puts("ok\n");
```
2. **Update I/O words**
**`src/word_source/io_words.c`:**
```c
/* EMIT ( c -- ) */
static void word_emit(VM *vm) {
int c = vm_pop(vm);
hal_console_putc((char)c);
}
/* KEY ( -- c ) */
static void word_key(VM *vm) {
int c = hal_console_getc();
vm_push(vm, c);
}
```
3. **Test REPL interactively**
```bash
./build/amd64/standard/starforth
> 1 2 + .
3 ok
> BYE
```
### Success Criteria
✅ REPL works identically to before
✅ All I/O words use HAL
✅ No direct stdio calls in VM code
---
## Phase 6: Implement HAL for L4Re (Optional)
**Duration:** 3-5 days
**Risk:** Low (validates HAL portability)
This phase is **optional** but highly recommended to validate that the HAL abstraction actually works across platforms.
### Tasks
1. **Create L4Re HAL directory**
```bash
mkdir -p src/platform/l4re/hal
```
2. **Implement L4Re HAL subsystems**
- `hal_time.c` → L4Re clock API
- `hal_memory.c` → L4Re dataspaces
- `hal_console.c` → L4Re console service
- `hal_interrupt.c` → L4Re IRQ objects
See L4Re documentation for API details.
3. **Build for L4Re**
```bash
make PLATFORM=l4re
```
4. **Run tests on L4Re**
```bash
make PLATFORM=l4re test
```
### Success Criteria
✅ L4Re HAL compiles
✅ VM runs on L4Re without source changes
✅ Tests pass on L4Re
---
## Phase 7: Validate Deterministic Behavior
**Duration:** 1 day
**Risk:** High (final validation)
### Tasks
1. **Run DoE on Linux (HAL)**
```bash
make fastest PLATFORM=linux
for i in {1..10}; do
./build/amd64/fastest/starforth --doe > results_linux_$i.csv
done
```
2. **Verify 0% variance across runs**
```bash
# All CSV files should be identical
md5sum results_linux_*.csv
```
3. **Compare with pre-HAL baseline**
```bash
diff results_before_hal.csv results_linux_1.csv
```
**Expected:** Identical (HAL added zero overhead).
4. **Run full test suite**
```bash
make test PLATFORM=linux
```
**Expected:** All 936+ tests pass.
5. **Benchmark performance**
```bash
make bench PLATFORM=linux
```
**Expected:** < 5% performance delta from pre-HAL baseline.
### Success Criteria
✅ 0% algorithmic variance maintained
✅ All tests pass
✅ < 5% performance regression
✅ HAL abstraction validated
---
## Rollback Strategy
If migration fails at any phase:
1. **Git branch strategy**
```bash
git checkout -b hal-migration
# Work on branch, test each phase
git commit -m "Phase N complete"
# If phase fails, revert:
git reset --hard HEAD~1
```
2. **Incremental commits**
- Commit after each phase passes tests
- Never commit broken code
- Each commit should build and pass tests
3. **Feature flag (if needed)**
```c
#ifdef ENABLE_HAL
uint64_t now = hal_time_now_ns();
#else
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
uint64_t now = ...;
#endif
```
Enable HAL incrementally, fall back if issues arise.
---
## Post-Migration Cleanup
After successful migration:
1. **Remove old platform code**
```bash
# If src/platform/linux/time.c was fully replaced by hal_time.c
git rm src/platform/linux/time.c
```
2. **Update documentation**
- `docs/CLAUDE.md` - Mention HAL architecture
- `README.md` - Update build instructions
- `docs/DEVELOPER.md` - Add HAL section
3. **Update .gitignore**
```
# Build artifacts for all platforms
build/linux/
build/l4re/
build/kernel/
```
---
## Timeline Estimate
| Phase | Duration | Cumulative |
|-------|----------|------------|
| 1. Define HAL Interfaces | 1-2 days | 1-2 days |
| 2. Implement Linux HAL | 3-5 days | 4-7 days |
| 3. Migrate VM Core | 2-3 days | 6-10 days |
| 4. Migrate Physics | 2-3 days | 8-13 days |
| 5. Migrate REPL | 1-2 days | 9-15 days |
| 6. L4Re HAL (optional) | 3-5 days | 12-20 days |
| 7. Validation | 1 day | 13-21 days |
**Total:** 2-4 weeks (depending on L4Re inclusion)
---
## Success Metrics
The HAL migration is successful if:
1. ✅ **All tests pass** - 936+ tests on Linux (and L4Re if implemented)
2. ✅ **0% variance** - DoE results identical pre/post migration
3. ✅ **No regressions** - Performance < 5% slower
4. ✅ **Zero platform code in VM** - No #ifdef PLATFORM_X in core
5. ✅ **Clean abstractions** - HAL interfaces well-documented
6. ✅ **Ready for StarKernel** - Platform layer can be replaced
---
## Next Steps
After successful HAL migration:
1. **Implement StarKernel platform** (`src/platform/kernel/`)
2. **UEFI boot loader** (`src/platform/kernel/boot/uefi_loader.c`)
3. **Boot to `ok` prompt** on QEMU/OVMF
4. **Validate physics on bare metal** (determinism on real hardware)
See `starkernel-integration.md` for StarKernel-specific implementation details.
docs/working/archive/dead/hal-target-contract/overview.md
+349
View File
@@ -0,0 +1,349 @@
> **DEAD (docs triage 2026-10-14).** The proposed `hal_*` interface contract and its migration plan were never adopted; the kernel grew its own HAL (kernel/src/hal/, kernel/include/starkernel/hal/).
> Instead: kernel/src/hal/README.md. Moved here from `docs/working/architecture/03-architecture/hal/overview.md`.
<!-- Moved from docs/03-architecture/hal/overview.md to docs/working/archive/dead/hal-target-contract/overview.md on 2026-06-16 (docs reorg Phase 2) -->
# Hardware Abstraction Layer (HAL) Architecture
> **⚠️ Important Note**: This document describes the **target HAL architecture** with standardized `hal_*` function naming. The current codebase uses `sf_time_backend_t` and similar abstractions with different naming conventions. See [migration-plan.md](migration-plan.md) for the refactoring steps to migrate existing platform code to the HAL interfaces described here.
>
> **HISTORICAL — L4Re/Fiasco.OC**: this document's L4Re references (platform comparison table, microkernel discussion, etc.) describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build. Treat every L4Re mention below as historical design context, not a currently buildable platform.
## Executive Summary
The Hardware Abstraction Layer (HAL) is the architectural foundation that enables StarForth to evolve from a hosted VM into StarKernel while preserving the physics-driven adaptive runtime's deterministic behavior across all platforms.
**Critical insight:** The HAL is not an afterthought—it's the linchpin that makes StarKernel → StarshipOS possible without compromising StarForth's experimental integrity.
---
## The Problem
StarForth currently runs on multiple platforms:
- **Linux** (POSIX, hosted)
- **L4Re/Fiasco.OC** (microkernel)
- **Bare metal** (limited, experimental)
Each platform has its own:
- Timing mechanisms (POSIX timers vs. hardware timers)
- Interrupt handling (signals vs. IDT/APIC)
- Memory allocation (malloc vs. physical page frames)
- I/O mechanisms (stdin/stdout vs. UART/framebuffer)
**Without a HAL:** Platform-specific code bleeds into the VM core, physics subsystems, and word implementations. This creates:
- ❌ Fragile #ifdef PLATFORM_X conditionals throughout codebase
- ❌ Platform-specific bugs in supposedly portable code
- ❌ Inability to test kernel code on hosted platforms
- ❌ Risk to deterministic behavior guarantees (0% algorithmic variance)
**With a HAL:** Clean separation between VM logic and platform implementation:
- ✅ VM and physics subsystems are platform-agnostic
- ✅ Platform code is isolated and testable
- ✅ New platforms (StarKernel) can be added without touching VM core
- ✅ Deterministic behavior guaranteed by HAL contract, not platform quirks
---
## Architecture Overview
```
┌────────────────────────────────────────────────────────────┐
│ StarForth VM Core + Physics Subsystems │
│ • Interpreter loop (vm.c) │
│ • Execution heat tracking │
│ • Rolling window of truth │
│ • Hot-words cache │
│ • Pipelining metrics │
│ • Inference engine │
│ • Heartbeat coordination │
│ │
│ ↓ Calls HAL interfaces (platform-agnostic) │
├────────────────────────────────────────────────────────────┤
│ Hardware Abstraction Layer (HAL) │
│ • hal_time.h - Monotonic time, timers, calibration │
│ • hal_interrupt.h - IRQ management, ISR registration │
│ • hal_memory.h - Allocation, page mapping, heap │
│ • hal_console.h - Character I/O (serial, framebuffer) │
│ • hal_cpu.h - CPU ID, relax/halt, SMP coordination │
│ │
│ ↓ Platform-specific implementations │
├────────────────────────────────────────────────────────────┤
│ Platform Implementations │
│ ┌──────────────┬──────────────┬──────────────────────┐ │
│ │ Linux │ L4Re │ Kernel (StarKernel) │ │
│ │ (POSIX) │ (microkernel)│ (freestanding) │ │
│ ├──────────────┼──────────────┼──────────────────────┤ │
│ │ clock_gettime│ L4Re::Clock │ TSC + HPET + APIC │ │
│ │ timerfd │ L4Re::IrqEoi │ IDT + APIC IRQ │ │
│ │ malloc/free │ dataspaces │ PMM + VMM + kmalloc │ │
│ │ stdin/stdout │ L4Re::Console│ UART + framebuffer │ │
│ │ pthread │ L4Re::Thread │ SMP bring-up │ │
│ └──────────────┴──────────────┴──────────────────────┘ │
└────────────────────────────────────────────────────────────┘
```
---
## Design Principles
### 1. **VM Purity**
The VM core must never know which platform it's running on. All platform awareness lives in HAL implementations.
**Anti-pattern (current):**
```c
#ifdef PLATFORM_LINUX
clock_gettime(CLOCK_MONOTONIC, &ts);
#elif PLATFORM_L4RE
l4re_kip_clock(kip);
#elif PLATFORM_KERNEL
rdtsc();
#endif
```
**Correct pattern (HAL):**
```c
// In VM code (platform-agnostic)
uint64_t now = hal_time_now_ns();
// In platform/linux/hal_time.c
uint64_t hal_time_now_ns(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (uint64_t)ts.tv_sec * 1000000000ULL + ts.tv_nsec;
}
// In platform/kernel/hal_time.c
uint64_t hal_time_now_ns(void) {
return tsc_to_ns(rdtsc());
}
```
### 2. **Contract-First Design**
HAL interfaces are **contracts**, not convenience wrappers. Each HAL function has:
- **Precise semantics** - What it does, guaranteed across all platforms
- **Error handling** - When it can fail and how
- **Performance expectations** - Allowed latency/overhead
- **Concurrency model** - Thread-safe? ISR-safe?
### 3. **Testability on Hosted Platforms**
StarKernel code must be developable and testable on Linux/L4Re before deploying to bare metal.
**Example:** Heartbeat ISR development
- **Kernel implementation:** APIC timer interrupt → ISR → ring buffer
- **Linux test implementation:** timerfd + signal handler → ISR → ring buffer
- **Same HAL interface, same VM code, different platform layer**
### 4. **Zero Overhead When Possible**
HAL calls should compile to direct hardware access on kernel platforms, not add abstraction tax.
**Good:** `hal_time_now_ns()` inlines to `rdtsc()` with `-O2`
**Bad:** Function pointer indirection adds 5-10 cycles per call
### 5. **Fail-Fast Validation**
HAL implementations validate platform assumptions at init time, not during execution.
**Example:**
```c
void hal_time_init(void) {
// Calibrate TSC frequency at boot
tsc_calibrate_hpet();
// Validate monotonicity
uint64_t t1 = hal_time_now_ns();
hal_cpu_relax();
uint64_t t2 = hal_time_now_ns();
if (t2 < t1) {
hal_panic("hal_time: TSC not monotonic!");
}
}
```
---
## HAL Subsystems
### 1. **Time & Timers** (`hal_time.h`)
**Purpose:** Monotonic time, periodic/oneshot timers, calibration
**Critical for StarForth:** The heartbeat subsystem and physics feedback loops depend on precise, jitter-free timing. HAL must guarantee:
- Monotonic time (never goes backward)
- Sub-microsecond resolution
- Calibrated frequency (for TSC-based timing)
**Platform challenges:**
- **Linux:** `clock_gettime()` is good, but signal-based timers have latency
- **Kernel:** TSC drift, HPET/PIT fallback, per-core calibration
### 2. **Interrupts** (`hal_interrupt.h`)
**Purpose:** Enable/disable IRQs, register ISRs, query interrupt context
**Critical for StarForth:** Heartbeat ISR must run at precise intervals without VM involvement.
**Platform challenges:**
- **Linux:** Signals are "interrupt-like" but not true IRQs
- **Kernel:** IDT setup, APIC configuration, spurious interrupt handling
### 3. **Memory** (`hal_memory.h`)
**Purpose:** Allocate/free memory, page mapping, heap management
**Critical for StarForth:** Dictionary allocation, stack allocation, heap allocator must work identically across platforms.
**Platform challenges:**
- **Linux:** malloc/free are simple
- **Kernel:** Physical memory manager, virtual memory manager, heap allocator—all from scratch
### 4. **Console** (`hal_console.h`)
**Purpose:** Character I/O for REPL and diagnostics
**Critical for StarForth:** The REPL must work on all platforms for interactive experimentation.
**Platform challenges:**
- **Linux:** stdin/stdout are perfect
- **Kernel:** UART 16550 is well-documented but framebuffer is tricky
### 5. **CPU** (`hal_cpu.h`)
**Purpose:** CPU ID, relax/halt, SMP coordination
**Critical for StarForth:** Per-core execution heat tracking (future), SMP scalability (future).
**Platform challenges:**
- **Linux:** pthread local storage
- **Kernel:** Local APIC, per-core stacks, CPU-local storage
---
## HAL and the Physics Subsystems
The physics-driven adaptive runtime is the **primary beneficiary** of the HAL:
| Physics Subsystem | HAL Dependency | Why |
|-------------------|----------------|-----|
| **Execution Heat** | None (pure VM state) | Tracks word execution frequency |
| **Rolling Window** | `hal_time_now_ns()` | Timestamps for window entries |
| **Hot-Words Cache** | None (pure dictionary state) | Frequency-based reordering |
| **Pipelining Metrics** | None (pure transition tracking) | Word-to-word prediction |
| **Inference Engine** | `hal_time_now_ns()` | Calibration timing, ANOVA |
| **Heartbeat** | `hal_timer_periodic()`, `hal_interrupt.*` | ISR-based sampling |
**Key insight:** Only 2 of 6 subsystems touch the HAL directly, and only via clean interfaces. This preserves deterministic behavior while enabling kernel deployment.
---
## HAL and StarKernel
StarKernel is a **new platform implementation** of the HAL:
```
src/platform/kernel/
├── boot/
│ └── uefi_loader.c # UEFI entry point → BootInfo handoff
├── hal_time.c # TSC + HPET + APIC timer
├── hal_interrupt.c # IDT + Local APIC + IOAPIC
├── hal_memory.c # PMM + VMM + kmalloc
├── hal_console.c # UART 16550 + framebuffer
├── hal_cpu.c # SMP + CPU-local storage
└── drivers/ # PCI, AHCI, NVMe, VirtIO, etc.
```
**StarKernel does NOT modify:**
- VM core (`src/vm.c`)
- Physics subsystems (`src/dictionary_heat_optimization.c`, etc.)
- Word implementations (`src/word_source/*.c`)
**StarKernel ONLY implements:**
- HAL interfaces (`hal_*.c`)
- UEFI boot loader (`boot/uefi_loader.c`)
- Device drivers (`drivers/*.c`)
This is the **proof** that the HAL abstraction works: StarKernel is a pure platform layer, not a VM fork.
---
## HAL and StarshipOS
StarshipOS builds on StarKernel by adding:
- **Process model** - Forth tasks vs. traditional processes
- **Filesystem** - FAT32, ext2, or log-structured
- **Networking** - TCP/IP stack, DHCP, DNS
- **Device model** - Unified block/net/char device interfaces
- **Security model** - Capabilities, ACL, Forth-based access control
All of these **still use the HAL** for low-level access:
- Filesystem → `hal_memory` for caching, `hal_interrupt` for async I/O
- Networking → `hal_interrupt` for packet RX, `hal_time` for timeouts
- Device model → HAL as the common substrate
**The HAL is not just a kernel bootstrapping tool—it's the foundation for the entire OS.**
---
## Migration Strategy
Current StarForth codebase must be refactored to introduce the HAL:
### Phase 1: Define HAL Interfaces
- Write `include/hal/*.h` headers with contracts
- Document semantics, error handling, performance expectations
- **No implementation yet**
### Phase 2: Refactor Existing Platforms
- Create `src/platform/linux/hal_*.c` implementing HAL
- Create `src/platform/l4re/hal_*.c` implementing HAL
- Update VM code to call HAL instead of platform-specific APIs
- **VM must still build and pass all 936+ tests**
### Phase 3: Validate HAL
- Build and test on Linux
- Build and test on L4Re (if available)
- Verify deterministic behavior (0% algorithmic variance) still holds
- Benchmark: HAL must not add measurable overhead
### Phase 4: Implement StarKernel Platform
- Create `src/platform/kernel/` with HAL implementations
- UEFI boot loader
- Basic MM, console, timing
- **Goal:** Boot to `ok` prompt on QEMU/OVMF
### Phase 5: Full StarKernel
- Complete HAL implementations (interrupts, SMP, drivers)
- Heartbeat ISR running at kernel level
- Physics subsystems operational
- **Goal:** Reproduce DoE results on bare metal
### Phase 6: StarshipOS
- Process model, filesystem, networking
- Forth as native control plane
- **Goal:** Self-hosting OS
---
## Success Criteria
The HAL is successful if:
1. ✅ **VM core has zero platform-specific code**
2. ✅ **All 936+ tests pass on Linux, L4Re, and StarKernel**
3. ✅ **0% algorithmic variance maintained across platforms**
4. ✅ **No measurable performance regression from HAL abstraction**
5. ✅ **StarKernel boots to `ok` prompt and runs REPL**
6. ✅ **Heartbeat subsystem works identically on all platforms**
7. ✅ **New platforms can be added without touching VM code**
---
## Next Steps
See companion documentation:
- `interfaces.md` - Detailed HAL interface specifications
- `platform-implementations.md` - How to implement HAL for new platforms
- `migration-plan.md` - Step-by-step refactoring guide
- `starkernel-integration.md` - StarKernel-specific implementation details
---
## References
- StarForth VM core: `src/vm.c`
- Heartbeat system: `docs/03-architecture/heartbeat-system/`
- Physics feedback loops: `docs/FEEDBACK_LOOPS.md`
- Platform abstraction (current): `src/platform/`
docs/working/archive/dead/hal-target-contract/platform-implementations.md
+741
View File
@@ -0,0 +1,741 @@
> **DEAD (docs triage 2026-10-14).** The proposed `hal_*` interface contract and its migration plan were never adopted; the kernel grew its own HAL (kernel/src/hal/, kernel/include/starkernel/hal/).
> Instead: kernel/src/hal/README.md. Moved here from `docs/working/architecture/03-architecture/hal/platform-implementations.md`.
<!-- Moved from docs/03-architecture/hal/platform-implementations.md to docs/working/archive/dead/hal-target-contract/platform-implementations.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ OBSOLETE (flagged 2026-10-09).** Assumes a `src/platform/<platform>/hal_*.c` layout
> whose platform names ("Linux", "L4Re", "StarKernel") and interface never matched the built
> tree: the actual kernel HAL is `src/starkernel/hal/`, L4Re is no longer an active target,
> and the kernel is well past a hypothetical "Boot + Serial Output" starting point. Do not use
> as a platform porting guide. See the TARGET-HAL caveat in `README.md` and
> `.claude/CLAUDE.md`.
# HAL Platform Implementation Guide
## Overview
This document explains how to implement the HAL for a new platform. It provides detailed examples for Linux, L4Re, and StarKernel (freestanding) implementations.
> **HISTORICAL — L4Re/Fiasco.OC**: the L4Re examples below describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build.
**Target audience:** Platform developers adding HAL support for a new target.
---
## Platform Directory Structure
```
src/platform/<platform>/
├── hal_time.c # Time & timer HAL implementation
├── hal_interrupt.c # Interrupt HAL implementation
├── hal_memory.c # Memory HAL implementation
├── hal_console.c # Console HAL implementation
├── hal_cpu.c # CPU HAL implementation
├── hal_panic.c # Panic/error HAL implementation
└── platform_init.c # Platform-specific initialization (optional)
```
### Build Integration
Each platform is compiled independently via Makefile:
```makefile
# Platform selection
PLATFORM ?= linux # linux | l4re | kernel
# Platform-specific source files
PLATFORM_SOURCES = \
src/platform/$(PLATFORM)/hal_time.c \
src/platform/$(PLATFORM)/hal_interrupt.c \
src/platform/$(PLATFORM)/hal_memory.c \
src/platform/$(PLATFORM)/hal_console.c \
src/platform/$(PLATFORM)/hal_cpu.c \
src/platform/$(PLATFORM)/hal_panic.c
# Platform-specific flags
ifeq ($(PLATFORM),kernel)
CFLAGS += -ffreestanding -nostdlib -mno-red-zone
LDFLAGS += -nostdlib -static
endif
```
---
## Implementation Examples
### 1. Linux Platform (`src/platform/linux/`)
Linux is the **reference platform** for development and testing. It provides the richest debugging environment.
#### `hal_time.c` (Linux)
```c
#include "hal/hal_time.h"
#include <time.h>
#include <signal.h>
#include <string.h>
#include <stdlib.h>
/* Timer state */
typedef struct {
timer_t timer_id;
hal_timer_callback_t callback;
void *ctx;
} linux_timer_t;
static linux_timer_t *periodic_timers = NULL;
static int timer_count = 0;
void hal_time_init(void) {
/* Linux: clock_gettime is always available */
struct timespec ts;
if (clock_gettime(CLOCK_MONOTONIC, &ts) != 0) {
hal_panic("hal_time_init: CLOCK_MONOTONIC unavailable");
}
}
uint64_t hal_time_now_ns(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (uint64_t)ts.tv_sec * 1000000000ULL + ts.tv_nsec;
}
void hal_time_delay_ns(uint64_t ns) {
struct timespec req;
req.tv_sec = ns / 1000000000ULL;
req.tv_nsec = ns % 1000000000ULL;
nanosleep(&req, NULL);
}
/* Signal handler for periodic timer */
static void timer_signal_handler(int signo, siginfo_t *si, void *uc) {
(void)signo;
(void)uc;
linux_timer_t *timer = (linux_timer_t *)si->si_value.sival_ptr;
if (timer && timer->callback) {
timer->callback(timer->ctx);
}
}
int hal_timer_periodic(uint64_t period_ns, hal_timer_callback_t callback, void *ctx) {
/* Allocate timer structure */
linux_timer_t *timer = malloc(sizeof(linux_timer_t));
if (!timer) return -1;
timer->callback = callback;
timer->ctx = ctx;
/* Set up signal handler */
struct sigaction sa;
memset(&sa, 0, sizeof(sa));
sa.sa_flags = SA_SIGINFO;
sa.sa_sigaction = timer_signal_handler;
sigemptyset(&sa.sa_mask);
sigaction(SIGRTMIN, &sa, NULL);
/* Create POSIX timer */
struct sigevent sev;
memset(&sev, 0, sizeof(sev));
sev.sigev_notify = SIGEV_SIGNAL;
sev.sigev_signo = SIGRTMIN;
sev.sigev_value.sival_ptr = timer;
if (timer_create(CLOCK_MONOTONIC, &sev, &timer->timer_id) != 0) {
free(timer);
return -1;
}
/* Arm the timer */
struct itimerspec its;
its.it_value.tv_sec = period_ns / 1000000000ULL;
its.it_value.tv_nsec = period_ns % 1000000000ULL;
its.it_interval = its.it_value; /* Periodic */
if (timer_settime(timer->timer_id, 0, &its, NULL) != 0) {
timer_delete(timer->timer_id);
free(timer);
return -1;
}
return timer_count++; /* Return timer ID */
}
int hal_timer_cancel(int timer_id) {
/* TODO: Implement timer cancellation */
(void)timer_id;
return 0;
}
uint64_t hal_time_frequency_hz(void) {
/* Linux: CLOCK_MONOTONIC is variable frequency, return 0 */
return 0;
}
```
#### `hal_interrupt.c` (Linux)
```c
#include "hal/hal_interrupt.h"
#include <signal.h>
/* Linux uses signals as "interrupts" */
static __thread int in_signal_handler = 0;
void hal_interrupt_init(void) {
/* No initialization needed on Linux */
}
void hal_irq_enable(void) {
/* Linux: interrupts (signals) always enabled */
}
unsigned long hal_irq_disable(void) {
/* Linux: can't disable signals, return dummy state */
return 0;
}
void hal_irq_restore(unsigned long state) {
(void)state;
}
int hal_irq_register(unsigned int irq, hal_isr_t isr, void *ctx) {
/* Linux: IRQs are handled via signal mechanism in hal_timer_periodic() */
(void)irq;
(void)isr;
(void)ctx;
return 0;
}
int hal_irq_unregister(unsigned int irq) {
(void)irq;
return 0;
}
int hal_in_interrupt_context(void) {
return in_signal_handler;
}
```
#### `hal_memory.c` (Linux)
```c
#include "hal/hal_memory.h"
#include <stdlib.h>
#include <string.h>
void hal_mem_init(void) {
/* malloc is always available on Linux */
}
void *hal_mem_alloc(size_t size) {
if (size == 0) return NULL;
void *ptr = malloc(size);
if (ptr) {
memset(ptr, 0, size); /* Zero-initialize */
}
return ptr;
}
void hal_mem_free(void *ptr) {
free(ptr);
}
uint64_t hal_mem_alloc_pages(size_t count) {
/* Linux: just allocate from heap (no separate PMM) */
size_t size = count * hal_mem_page_size();
return (uint64_t)hal_mem_alloc(size);
}
void hal_mem_free_pages(uint64_t paddr, size_t count) {
(void)count;
hal_mem_free((void *)paddr);
}
int hal_mem_map(uint64_t vaddr, uint64_t paddr, size_t size, unsigned int flags) {
/* Linux: identity mapping, no-op */
(void)vaddr;
(void)paddr;
(void)size;
(void)flags;
return 0;
}
size_t hal_mem_page_size(void) {
return 4096; /* Standard page size */
}
```
#### `hal_console.c` (Linux)
```c
#include "hal/hal_console.h"
#include <stdio.h>
#include <unistd.h>
#include <poll.h>
void hal_console_init(void) {
/* stdin/stdout always available */
}
void hal_console_putc(char c) {
putchar(c);
fflush(stdout); /* Ensure character is visible */
}
void hal_console_puts(const char *s) {
if (s) {
fputs(s, stdout);
fflush(stdout);
}
}
int hal_console_getc(void) {
return getchar();
}
int hal_console_has_input(void) {
struct pollfd pfd;
pfd.fd = STDIN_FILENO;
pfd.events = POLLIN;
return poll(&pfd, 1, 0) > 0;
}
```
#### `hal_cpu.c` (Linux)
```c
#include "hal/hal_cpu.h"
#include <unistd.h>
#include <sched.h>
void hal_cpu_init(void) {
/* No initialization needed */
}
unsigned int hal_cpu_id(void) {
/* Linux: return thread ID (not true CPU ID) */
return 0; /* Single-threaded for now */
}
void hal_cpu_relax(void) {
sched_yield();
}
void hal_cpu_halt(void) {
pause(); /* Wait for signal */
}
unsigned int hal_cpu_count(void) {
return sysconf(_SC_NPROCESSORS_ONLN);
}
```
#### `hal_panic.c` (Linux)
```c
#include "hal/hal_panic.h"
#include <stdio.h>
#include <stdlib.h>
void hal_panic(const char *msg) {
fprintf(stderr, "PANIC: %s\n", msg ? msg : "unknown error");
fflush(stderr);
abort();
}
```
---
### 2. Kernel Platform (`src/platform/kernel/`)
The kernel platform is **freestanding**—no libc, no OS, direct hardware access.
#### `hal_time.c` (Kernel)
```c
#include "hal/hal_time.h"
#include "hal/hal_interrupt.h"
#include "kernel/apic.h"
#include "kernel/hpet.h"
#include "kernel/tsc.h"
/* TSC calibration state */
static uint64_t tsc_frequency_hz = 0;
static uint64_t tsc_offset_ns = 0;
void hal_time_init(void) {
/* Calibrate TSC using HPET */
tsc_frequency_hz = tsc_calibrate_hpet();
if (tsc_frequency_hz == 0) {
hal_panic("hal_time_init: TSC calibration failed");
}
/* Validate monotonicity */
uint64_t t1 = hal_time_now_ns();
for (volatile int i = 0; i < 1000; i++);
uint64_t t2 = hal_time_now_ns();
if (t2 < t1) {
hal_panic("hal_time_init: TSC not monotonic");
}
}
uint64_t hal_time_now_ns(void) {
/* Read TSC and convert to nanoseconds */
uint64_t tsc = rdtsc();
return tsc_to_ns(tsc) - tsc_offset_ns;
}
void hal_time_delay_ns(uint64_t ns) {
uint64_t end = hal_time_now_ns() + ns;
while (hal_time_now_ns() < end) {
hal_cpu_relax();
}
}
/* APIC timer state */
static hal_timer_callback_t apic_timer_callback = NULL;
static void *apic_timer_ctx = NULL;
/* APIC timer ISR (called from IDT) */
void apic_timer_isr(void) {
if (apic_timer_callback) {
apic_timer_callback(apic_timer_ctx);
}
apic_eoi(); /* Send End-Of-Interrupt */
}
int hal_timer_periodic(uint64_t period_ns, hal_timer_callback_t callback, void *ctx) {
if (period_ns < 1000) return -2; /* Minimum 1µs */
apic_timer_callback = callback;
apic_timer_ctx = ctx;
/* Configure APIC timer */
uint64_t ticks = (period_ns * apic_timer_frequency_hz()) / 1000000000ULL;
apic_timer_periodic(ticks);
return 1; /* Timer ID */
}
int hal_timer_cancel(int timer_id) {
(void)timer_id;
apic_timer_stop();
apic_timer_callback = NULL;
return 0;
}
uint64_t hal_time_frequency_hz(void) {
return tsc_frequency_hz;
}
```
#### `hal_interrupt.c` (Kernel)
```c
#include "hal/hal_interrupt.h"
#include "kernel/idt.h"
#include "kernel/apic.h"
/* Per-CPU interrupt nesting count */
static __thread unsigned int interrupt_depth = 0;
void hal_interrupt_init(void) {
/* Initialize IDT */
idt_init();
/* Initialize Local APIC */
apic_init();
/* Disable legacy PIC (use APIC instead) */
pic_disable();
}
void hal_irq_enable(void) {
__asm__ volatile("sti");
}
unsigned long hal_irq_disable(void) {
unsigned long flags;
__asm__ volatile(
"pushfq\n"
"popq %0\n"
"cli\n"
: "=r"(flags)
);
return flags;
}
void hal_irq_restore(unsigned long state) {
__asm__ volatile(
"pushq %0\n"
"popfq\n"
:
: "r"(state)
);
}
/* ISR table (256 entries for x86_64 IDT) */
static hal_isr_t isr_table[256] = {0};
static void *isr_ctx_table[256] = {0};
/* Common ISR entry point (called from IDT stubs) */
void common_isr_handler(unsigned int vector) {
interrupt_depth++;
if (isr_table[vector]) {
isr_table[vector](vector, isr_ctx_table[vector]);
}
interrupt_depth--;
apic_eoi();
}
int hal_irq_register(unsigned int irq, hal_isr_t isr, void *ctx) {
if (irq >= 256) return -1;
isr_table[irq] = isr;
isr_ctx_table[irq] = ctx;
/* Configure IOAPIC to route IRQ to Local APIC */
ioapic_route_irq(irq, 0); /* Route to CPU 0 */
return 0;
}
int hal_irq_unregister(unsigned int irq) {
if (irq >= 256) return -1;
isr_table[irq] = NULL;
isr_ctx_table[irq] = NULL;
return 0;
}
int hal_in_interrupt_context(void) {
return interrupt_depth > 0;
}
```
#### `hal_memory.c` (Kernel)
```c
#include "hal/hal_memory.h"
#include "kernel/pmm.h" /* Physical Memory Manager */
#include "kernel/vmm.h" /* Virtual Memory Manager */
#include "kernel/kmalloc.h" /* Kernel heap allocator */
void hal_mem_init(void) {
/* Initialize physical memory manager (parse UEFI memory map) */
pmm_init();
/* Initialize virtual memory manager (set up page tables) */
vmm_init();
/* Initialize kernel heap allocator */
kmalloc_init();
}
void *hal_mem_alloc(size_t size) {
if (size == 0) return NULL;
return kmalloc(size); /* Kernel heap allocator */
}
void hal_mem_free(void *ptr) {
kfree(ptr);
}
uint64_t hal_mem_alloc_pages(size_t count) {
return pmm_alloc_pages(count);
}
void hal_mem_free_pages(uint64_t paddr, size_t count) {
pmm_free_pages(paddr, count);
}
int hal_mem_map(uint64_t vaddr, uint64_t paddr, size_t size, unsigned int flags) {
/* Translate HAL flags to page table flags */
unsigned int pt_flags = 0;
if (flags & HAL_MEM_WRITE) pt_flags |= PAGE_WRITE;
if (flags & HAL_MEM_EXEC) pt_flags |= PAGE_EXEC;
if (flags & HAL_MEM_NOCACHE) pt_flags |= PAGE_NOCACHE;
return vmm_map(vaddr, paddr, size, pt_flags);
}
size_t hal_mem_page_size(void) {
return 4096;
}
```
#### `hal_console.c` (Kernel)
```c
#include "hal/hal_console.h"
#include "kernel/uart.h"
#include "kernel/framebuffer.h"
void hal_console_init(void) {
/* Initialize UART 16550 (serial console) */
uart_init(0x3F8, 115200); /* COM1, 115200 baud */
/* Initialize framebuffer (if available from UEFI GOP) */
fb_init();
}
void hal_console_putc(char c) {
/* Output to both serial and framebuffer */
uart_putc(c);
fb_putc(c);
}
void hal_console_puts(const char *s) {
if (!s) return;
while (*s) {
hal_console_putc(*s++);
}
}
int hal_console_getc(void) {
/* Read from UART (blocking) */
return uart_getc();
}
int hal_console_has_input(void) {
return uart_has_data();
}
```
#### `hal_cpu.c` (Kernel)
```c
#include "hal/hal_cpu.h"
#include "kernel/apic.h"
static unsigned int cpu_count = 1;
void hal_cpu_init(void) {
/* Detect CPU features */
cpu_detect_features();
/* Bring up secondary CPUs (SMP) */
cpu_count = smp_init();
}
unsigned int hal_cpu_id(void) {
/* Get Local APIC ID */
return apic_get_id();
}
void hal_cpu_relax(void) {
__asm__ volatile("pause");
}
void hal_cpu_halt(void) {
__asm__ volatile("hlt");
}
unsigned int hal_cpu_count(void) {
return cpu_count;
}
```
---
## Platform Testing Strategy
### 1. **Unit Tests (Per-Platform)**
Each platform should have unit tests for HAL functions:
```c
/* test/platform/linux/hal_time_test.c */
void test_hal_time_monotonic(void) {
hal_time_init();
uint64_t t1 = hal_time_now_ns();
hal_time_delay_ns(1000000); /* 1ms */
uint64_t t2 = hal_time_now_ns();
assert(t2 > t1);
assert((t2 - t1) >= 1000000);
assert((t2 - t1) < 2000000); /* Allow 100% jitter for test */
}
```
### 2. **Cross-Platform VM Tests**
VM tests should run identically on all platforms:
```bash
# Linux
make PLATFORM=linux test
# L4Re
make PLATFORM=l4re test
# Kernel (QEMU)
make PLATFORM=kernel test-qemu
```
**Success criteria:** All 936+ tests pass on all platforms.
### 3. **Determinism Validation**
Physics subsystems must show 0% algorithmic variance:
```bash
# Run DoE on each platform
for platform in linux l4re kernel; do
make PLATFORM=$platform fastest
./build/$ARCH/fastest/starforth --doe > results_$platform.csv
done
# Compare results (should be identical)
diff results_linux.csv results_kernel.csv
```
---
## Common Implementation Pitfalls
### 1. **Non-Monotonic Time**
**Symptom:** Physics metrics show negative deltas, rolling window corrupted.
**Cause:** TSC goes backward (multi-core, frequency scaling).
**Fix:** Use `rdtscp` instead of `rdtsc`, synchronize TSC across cores, or fall back to HPET.
### 2. **Timer Jitter**
**Symptom:** Heartbeat intervals vary by 10-50%, metrics show high variance.
**Cause:** Signal latency (Linux), interrupt latency (kernel).
**Fix:** Minimize ISR work, use high-priority timer interrupt, disable frequency scaling.
### 3. **Memory Leaks**
**Symptom:** `hal_mem_alloc()` calls grow without matching `hal_mem_free()`.
**Cause:** Missing free in error paths, lost pointers.
**Fix:** Valgrind on Linux, manual leak tracking on kernel.
### 4. **Interrupt Context Violations**
**Symptom:** Crashes, hangs, or corruption when calling blocking HAL functions from ISR.
**Cause:** Calling `hal_mem_alloc()` or `hal_console_getc()` from timer ISR.
**Fix:** Assert `!hal_in_interrupt_context()` in blocking functions.
---
## Next Steps
- See `migration-plan.md` for refactoring existing code
- See `starkernel-integration.md` for kernel-specific details
docs/working/archive/dead/hal-target-contract/starkernel-integration.md
+773
View File
@@ -0,0 +1,773 @@
> **DEAD (docs triage 2026-10-14).** The proposed `hal_*` interface contract and its migration plan were never adopted; the kernel grew its own HAL (kernel/src/hal/, kernel/include/starkernel/hal/).
> Instead: kernel/src/hal/README.md. Moved here from `docs/working/architecture/03-architecture/hal/starkernel-integration.md`.
<!-- Moved from docs/03-architecture/hal/starkernel-integration.md to docs/working/archive/dead/hal-target-contract/starkernel-integration.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ OBSOLETE (flagged 2026-10-09).** Describes implementing `src/platform/kernel/` and a
> "Milestone 1: Boot + Serial Output" starting point. True architecture is far ahead: the
> kernel lives in `src/starkernel/` with M0–M7 complete and M7.1 live. Kept for historical
> context; for the real boot/kernel story see `.claude/CLAUDE.md` and `docs/fabric/`.
# StarKernel HAL Integration Guide
## Overview
This document provides **kernel-specific implementation details** for the HAL platform layer that becomes StarKernel. It covers UEFI boot, freestanding C environment, hardware initialization, and the path to a working `ok` prompt.
**Audience:** Kernel developers implementing `src/platform/kernel/`
---
## StarKernel Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ UEFI Firmware │
│ • Initializes hardware │
│ • Provides boot services (memory map, ACPI, GOP) │
│ • Loads BOOTX64.EFI │
└─────────────────────┬───────────────────────────────────────┘
│
↓ ExitBootServices()
┌─────────────────────────────────────────────────────────────┐
│ StarKernel Boot Loader (stage 0) │
│ • Collects BootInfo (memory map, ACPI tables, framebuffer) │
│ • Sets up initial page tables │
│ • Exits UEFI boot services │
│ • Jumps to StarKernel proper │
└─────────────────────┬───────────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ StarKernel HAL (stage 1) │
│ • CPU initialization (GDT, IDT, interrupts) │
│ • Memory subsystem (PMM, VMM, heap) │
│ • Time subsystem (TSC, HPET, APIC timer) │
│ • Console (UART + framebuffer) │
└─────────────────────┬───────────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ StarForth VM │
│ • vm_create() │
│ • Initialize physics subsystems │
│ • Start REPL │
│ • Print "ok" prompt │
└─────────────────────────────────────────────────────────────┘
```
---
## Directory Structure
```
src/platform/kernel/
├── boot/
│ ├── uefi_loader.c # UEFI entry point (BOOTX64.EFI)
│ ├── bootinfo.h # BootInfo struct definition
│ └── handoff.S # ASM trampoline (UEFI → StarKernel)
│
├── cpu/
│ ├── gdt.c # Global Descriptor Table
│ ├── idt.c # Interrupt Descriptor Table
│ ├── isr.S # Interrupt service routine stubs
│ └── smp.c # SMP bring-up (future)
│
├── mm/
│ ├── pmm.c # Physical Memory Manager
│ ├── vmm.c # Virtual Memory Manager
│ ├── kmalloc.c # Kernel heap allocator
│ └── paging.S # Page table manipulation ASM
│
├── time/
│ ├── tsc.c # Time Stamp Counter
│ ├── hpet.c # High Precision Event Timer
│ ├── apic_timer.c # Local APIC timer
│ └── pit.c # Programmable Interval Timer (fallback)
│
├── drivers/
│ ├── uart.c # UART 16550 serial console
│ ├── framebuffer.c # UEFI GOP framebuffer
│ ├── apic.c # Local APIC + IOAPIC
│ ├── acpi.c # ACPI table parsing
│ └── pci.c # PCI enumeration (future)
│
├── hal_time.c # HAL time implementation
├── hal_interrupt.c # HAL interrupt implementation
├── hal_memory.c # HAL memory implementation
├── hal_console.c # HAL console implementation
├── hal_cpu.c # HAL CPU implementation
├── hal_panic.c # HAL panic implementation
│
└── kernel_main.c # StarKernel entry point (after UEFI)
```
---
## Boot Sequence Detail
### 1. UEFI Loader (`boot/uefi_loader.c`)
**Responsibilities:**
- Collect system information from UEFI
- Allocate kernel memory
- Exit UEFI boot services
- Jump to StarKernel
**Implementation:**
```c
/* boot/uefi_loader.c */
#include <efi.h>
#include <efilib.h>
#include "bootinfo.h"
/* BootInfo passed to StarKernel */
typedef struct {
uint64_t memory_map_addr;
uint64_t memory_map_size;
uint64_t memory_map_descriptor_size;
uint64_t acpi_rsdp_addr;
uint64_t framebuffer_addr;
uint32_t framebuffer_width;
uint32_t framebuffer_height;
uint32_t framebuffer_pitch;
} BootInfo;
EFI_STATUS EFIAPI efi_main(EFI_HANDLE ImageHandle, EFI_SYSTEM_TABLE *SystemTable) {
InitializeLib(ImageHandle, SystemTable);
BootInfo *boot_info = (BootInfo *)AllocatePool(sizeof(BootInfo));
/* 1. Get memory map */
UINTN map_size = 0, map_key, descriptor_size;
UINT32 descriptor_version;
EFI_MEMORY_DESCRIPTOR *memory_map = NULL;
uefi_call_wrapper(BS->GetMemoryMap, 5, &map_size, memory_map, &map_key,
&descriptor_size, &descriptor_version);
map_size += 2 * descriptor_size; /* Extra space for ExitBootServices */
memory_map = (EFI_MEMORY_DESCRIPTOR *)AllocatePool(map_size);
uefi_call_wrapper(BS->GetMemoryMap, 5, &map_size, memory_map, &map_key,
&descriptor_size, &descriptor_version);
boot_info->memory_map_addr = (uint64_t)memory_map;
boot_info->memory_map_size = map_size;
boot_info->memory_map_descriptor_size = descriptor_size;
/* 2. Get ACPI RSDP */
EFI_GUID acpi_20_guid = ACPI_20_TABLE_GUID;
void *rsdp = NULL;
for (UINTN i = 0; i < ST->NumberOfTableEntries; i++) {
if (CompareGuid(&ST->ConfigurationTable[i].VendorGuid, &acpi_20_guid) == 0) {
rsdp = ST->ConfigurationTable[i].VendorTable;
break;
}
}
boot_info->acpi_rsdp_addr = (uint64_t)rsdp;
/* 3. Get framebuffer from GOP (Graphics Output Protocol) */
EFI_GUID gop_guid = EFI_GRAPHICS_OUTPUT_PROTOCOL_GUID;
EFI_GRAPHICS_OUTPUT_PROTOCOL *gop = NULL;
uefi_call_wrapper(BS->LocateProtocol, 3, &gop_guid, NULL, (void **)&gop);
if (gop) {
boot_info->framebuffer_addr = gop->Mode->FrameBufferBase;
boot_info->framebuffer_width = gop->Mode->Info->HorizontalResolution;
boot_info->framebuffer_height = gop->Mode->Info->VerticalResolution;
boot_info->framebuffer_pitch = gop->Mode->Info->PixelsPerScanLine * 4;
}
/* 4. Exit boot services */
uefi_call_wrapper(BS->ExitBootServices, 2, ImageHandle, map_key);
/* 5. Jump to StarKernel */
extern void kernel_main(BootInfo *boot_info);
kernel_main(boot_info);
/* Never returns */
while (1) __asm__ volatile("hlt");
return EFI_SUCCESS;
}
```
### 2. StarKernel Entry Point (`kernel_main.c`)
**Responsibilities:**
- Parse BootInfo
- Initialize CPU (GDT, IDT)
- Initialize memory subsystem
- Initialize HAL
- Call `main()` to start VM
**Implementation:**
```c
/* kernel_main.c */
#include "bootinfo.h"
#include "cpu/gdt.h"
#include "cpu/idt.h"
#include "mm/pmm.h"
#include "mm/vmm.h"
#include "drivers/uart.h"
#include "drivers/framebuffer.h"
/* Global BootInfo (available to all kernel code) */
BootInfo *g_boot_info = NULL;
void kernel_main(BootInfo *boot_info) {
g_boot_info = boot_info;
/* 1. Initialize serial console ASAP (for debug output) */
uart_early_init(0x3F8, 115200); /* COM1 */
uart_puts("StarKernel booting...\r\n");
/* 2. Initialize CPU structures */
gdt_init();
idt_init();
/* 3. Initialize physical memory manager */
pmm_init(boot_info);
uart_puts("PMM initialized\r\n");
/* 4. Initialize virtual memory manager */
vmm_init();
uart_puts("VMM initialized\r\n");
/* 5. Initialize kernel heap */
kmalloc_init();
uart_puts("Heap initialized\r\n");
/* 6. Initialize framebuffer */
fb_init(boot_info);
/* 7. Initialize HAL */
hal_time_init();
hal_interrupt_init();
hal_mem_init();
hal_console_init();
hal_cpu_init();
uart_puts("HAL initialized\r\n");
/* 8. Call standard main() to start VM */
extern int main(int argc, char **argv);
char *argv[] = {"starforth", NULL};
main(1, argv);
/* Never returns, but halt if it does */
hal_panic("main() returned unexpectedly");
}
```
---
## HAL Implementation Details
### 1. HAL Time (`hal_time.c`)
**Hardware:** TSC (Time Stamp Counter), HPET (High Precision Event Timer), APIC timer
**Key challenges:**
- TSC calibration against HPET
- TSC frequency varies on old CPUs (need invariant TSC check)
- APIC timer setup for periodic interrupts
**Implementation outline:**
```c
/* hal_time.c */
#include "hal/hal_time.h"
#include "time/tsc.h"
#include "time/hpet.h"
#include "time/apic_timer.h"
static uint64_t tsc_hz = 0;
static uint64_t boot_tsc = 0;
void hal_time_init(void) {
/* Check for invariant TSC */
if (!tsc_is_invariant()) {
hal_panic("hal_time: TSC not invariant, falling back to HPET");
/* TODO: Use HPET directly if TSC unreliable */
}
/* Calibrate TSC frequency using HPET */
tsc_hz = tsc_calibrate_hpet();
if (tsc_hz == 0) {
hal_panic("hal_time: TSC calibration failed");
}
boot_tsc = rdtsc();
/* Initialize APIC timer for periodic interrupts */
apic_timer_init();
}
uint64_t hal_time_now_ns(void) {
uint64_t tsc = rdtsc() - boot_tsc;
return (tsc * 1000000000ULL) / tsc_hz;
}
int hal_timer_periodic(uint64_t period_ns, hal_timer_callback_t callback, void *ctx) {
return apic_timer_periodic(period_ns, callback, ctx);
}
```
**See:** `time/tsc.c`, `time/hpet.c`, `time/apic_timer.c` for low-level implementations.
---
### 2. HAL Interrupt (`hal_interrupt.c`)
**Hardware:** IDT (Interrupt Descriptor Table), Local APIC, IOAPIC
**Key challenges:**
- Setting up 256 IDT entries
- Routing hardware IRQs through IOAPIC
- Tracking interrupt nesting depth (for `hal_in_interrupt_context()`)
**Implementation outline:**
```c
/* hal_interrupt.c */
#include "hal/hal_interrupt.h"
#include "cpu/idt.h"
#include "drivers/apic.h"
/* Per-CPU interrupt depth (thread-local on SMP) */
static __thread unsigned int irq_depth = 0;
void hal_interrupt_init(void) {
/* Set up IDT with 256 entries */
idt_init();
/* Disable legacy PIC (use APIC instead) */
outb(0x21, 0xFF); /* Mask all PIC1 IRQs */
outb(0xA1, 0xFF); /* Mask all PIC2 IRQs */
/* Initialize Local APIC */
apic_init();
/* Initialize IOAPIC */
ioapic_init();
}
void hal_irq_enable(void) {
__asm__ volatile("sti");
}
unsigned long hal_irq_disable(void) {
unsigned long flags;
__asm__ volatile("pushfq; popq %0; cli" : "=r"(flags));
return flags;
}
void hal_irq_restore(unsigned long flags) {
__asm__ volatile("pushq %0; popfq" :: "r"(flags));
}
int hal_in_interrupt_context(void) {
return irq_depth > 0;
}
/* Common ISR entry (called from IDT stubs) */
void isr_common_handler(uint64_t vector, uint64_t error_code) {
irq_depth++;
/* Dispatch to registered handler */
extern void isr_dispatch(uint64_t vector, uint64_t error_code);
isr_dispatch(vector, error_code);
irq_depth--;
/* Send EOI to APIC */
apic_eoi();
}
```
**See:** `cpu/idt.c`, `cpu/isr.S`, `drivers/apic.c` for low-level implementations.
---
### 3. HAL Memory (`hal_memory.c`)
**Components:**
- **PMM (Physical Memory Manager):** Tracks free/used physical page frames
- **VMM (Virtual Memory Manager):** Manages page tables (4-level paging on x86_64)
- **kmalloc:** Kernel heap allocator (slab or buddy allocator)
**Implementation outline:**
```c
/* hal_memory.c */
#include "hal/hal_memory.h"
#include "mm/pmm.h"
#include "mm/vmm.h"
#include "mm/kmalloc.h"
void hal_mem_init(void) {
/* PMM and VMM initialized in kernel_main() before HAL */
/* Just verify they're ready */
if (!pmm_is_initialized()) {
hal_panic("hal_mem_init: PMM not initialized");
}
}
void *hal_mem_alloc(size_t size) {
return kmalloc(size);
}
void hal_mem_free(void *ptr) {
kfree(ptr);
}
uint64_t hal_mem_alloc_pages(size_t count) {
return pmm_alloc_pages(count);
}
void hal_mem_free_pages(uint64_t paddr, size_t count) {
pmm_free_pages(paddr, count);
}
int hal_mem_map(uint64_t vaddr, uint64_t paddr, size_t size, unsigned int flags) {
return vmm_map_range(vaddr, paddr, size, flags);
}
size_t hal_mem_page_size(void) {
return 4096;
}
```
**PMM details** (`mm/pmm.c`):
- Bitmap allocator (1 bit per 4KB page)
- Parse UEFI memory map to mark usable vs. reserved regions
- Reserve kernel image, ACPI tables, framebuffer
**VMM details** (`mm/vmm.c`):
- 4-level page tables (PML4 → PDPT → PD → PT)
- Recursive mapping trick for page table access
- Identity-map kernel, higher-half kernel optional
**kmalloc details** (`mm/kmalloc.c`):
- Slab allocator for common sizes (16, 32, 64, ..., 4096 bytes)
- Buddy allocator for large allocations
- Zero-initialization required by HAL contract
---
### 4. HAL Console (`hal_console.c`)
**Hardware:** UART 16550 (serial), UEFI GOP framebuffer (video)
**Implementation outline:**
```c
/* hal_console.c */
#include "hal/hal_console.h"
#include "drivers/uart.h"
#include "drivers/framebuffer.h"
void hal_console_init(void) {
/* UART already initialized in kernel_main() for early debug */
/* Framebuffer initialized from BootInfo */
}
void hal_console_putc(char c) {
uart_putc(c); /* Always output to serial */
fb_putc(c); /* Also output to framebuffer if available */
}
void hal_console_puts(const char *s) {
while (*s) {
hal_console_putc(*s++);
}
}
int hal_console_getc(void) {
/* Block until UART has data */
return uart_getc();
}
int hal_console_has_input(void) {
return uart_has_data();
}
```
**UART details** (`drivers/uart.c`):
- 16550 compatible (COM1 = 0x3F8, COM2 = 0x2F8)
- 115200 baud default
- Polling mode (no interrupts for simplicity)
**Framebuffer details** (`drivers/framebuffer.c`):
- Linear framebuffer from UEFI GOP
- 32-bit RGB pixels
- Software text rendering (8x16 font)
- Scrolling, cursor, ANSI escape codes (optional)
---
### 5. HAL CPU (`hal_cpu.c`)
**Implementation outline:**
```c
/* hal_cpu.c */
#include "hal/hal_cpu.h"
#include "drivers/apic.h"
static unsigned int num_cpus = 1;
void hal_cpu_init(void) {
/* Detect CPU features (CPUID) */
/* TODO: Bring up secondary CPUs (SMP) */
}
unsigned int hal_cpu_id(void) {
return apic_get_id(); /* Local APIC ID */
}
void hal_cpu_relax(void) {
__asm__ volatile("pause");
}
void hal_cpu_halt(void) {
__asm__ volatile("hlt");
}
unsigned int hal_cpu_count(void) {
return num_cpus;
}
```
---
## Build System
### Toolchain Requirements
**Freestanding C environment:**
- `gcc` or `clang` with `-ffreestanding`
- `ld` with custom linker script
- `objcopy` to create PE32+ executable (for UEFI)
### Makefile Additions
```makefile
# Platform: kernel
ifeq ($(PLATFORM),kernel)
CC = gcc
LD = ld
OBJCOPY = objcopy
CFLAGS += -ffreestanding -nostdlib -mno-red-zone -mcmodel=large
CFLAGS += -mno-mmx -mno-sse -mno-sse2 # No FP in kernel
CFLAGS += -I/usr/include/efi -I/usr/include/efi/x86_64
LDFLAGS += -nostdlib -static -T src/platform/kernel/linker.ld
# Platform sources
PLATFORM_SOURCES = \
src/platform/kernel/boot/uefi_loader.c \
src/platform/kernel/kernel_main.c \
src/platform/kernel/cpu/gdt.c \
src/platform/kernel/cpu/idt.c \
src/platform/kernel/cpu/isr.S \
src/platform/kernel/mm/pmm.c \
src/platform/kernel/mm/vmm.c \
src/platform/kernel/mm/kmalloc.c \
src/platform/kernel/drivers/uart.c \
src/platform/kernel/drivers/framebuffer.c \
src/platform/kernel/drivers/apic.c \
src/platform/kernel/hal_time.c \
src/platform/kernel/hal_interrupt.c \
src/platform/kernel/hal_memory.c \
src/platform/kernel/hal_console.c \
src/platform/kernel/hal_cpu.c \
src/platform/kernel/hal_panic.c
# Build UEFI PE32+ executable
starforth.efi: $(OBJECTS)
$(LD) $(LDFLAGS) -o starforth.so $(OBJECTS)
$(OBJCOPY) -j .text -j .data -j .rodata -j .reloc \
--target=efi-app-x86_64 starforth.so $@
endif
```
### Linker Script (`src/platform/kernel/linker.ld`)
```ld
OUTPUT_FORMAT("elf64-x86-64")
ENTRY(efi_main)
SECTIONS {
. = 0x100000; /* Load at 1MB (standard kernel load address) */
.text : {
*(.text .text.*)
}
.rodata : {
*(.rodata .rodata.*)
}
.data : {
*(.data .data.*)
}
.bss : {
*(COMMON)
*(.bss .bss.*)
}
/DISCARD/ : {
*(.eh_frame)
}
}
```
---
## Testing on QEMU
### QEMU + OVMF Setup
```bash
# Install OVMF (UEFI firmware for QEMU)
sudo apt install ovmf
# Create ESP (EFI System Partition)
mkdir -p esp/EFI/BOOT
cp starforth.efi esp/EFI/BOOT/BOOTX64.EFI
# Run QEMU
qemu-system-x86_64 \
-bios /usr/share/ovmf/OVMF.fd \
-drive file=fat:rw:esp/,format=raw \
-serial stdio \
-m 512M \
-enable-kvm
```
### Expected Output
```
StarKernel booting...
PMM initialized
VMM initialized
Heap initialized
HAL initialized
StarForth VM v1.0
ok
```
---
## Debugging
### Serial Debugging
```c
/* Early debug output via UART */
void debug_puts(const char *s) {
while (*s) {
while (!(inb(0x3FD) & 0x20)); /* Wait for UART ready */
outb(0x3F8, *s++); /* Write character */
}
}
```
### GDB Remote Debugging
```bash
# Start QEMU with GDB server
qemu-system-x86_64 ... -s -S
# In another terminal
gdb starforth.elf
(gdb) target remote :1234
(gdb) break kernel_main
(gdb) continue
```
### Panic Handler
```c
/* hal_panic.c */
void hal_panic(const char *msg) {
hal_irq_disable();
hal_console_puts("\n*** KERNEL PANIC ***\n");
hal_console_puts(msg ? msg : "unknown error");
hal_console_puts("\nSystem halted.\n");
/* Halt all CPUs */
while (1) {
hal_cpu_halt();
}
}
```
---
## Roadmap to `ok` Prompt
### Milestone 1: Boot + Serial Output
- UEFI loader runs
- Kernel prints "StarKernel booting..." to serial
- System doesn't triple-fault
### Milestone 2: Memory Works
- PMM tracks physical pages
- VMM maps kernel
- kmalloc/kfree work
### Milestone 3: HAL Initialized
- All `hal_*_init()` functions complete
- HAL functions callable (even if not fully functional)
### Milestone 4: VM Starts
- `vm_create()` succeeds
- Dictionary allocated
- No crashes in VM init
### Milestone 5: REPL Runs
- REPL prints "`ok`" prompt
- Can type characters (echoed to serial)
- Can execute simple words (`1 2 + .` → `3`)
### Milestone 6: Physics Subsystems Work
- Execution heat tracking operational
- Heartbeat timer fires
- Rolling window captures execution history
### Milestone 7: Full Test Suite
- All 936+ tests pass on kernel
- DoE mode works
- 0% algorithmic variance on bare metal
---
## Future Work (StarshipOS)
After achieving `ok` prompt on StarKernel:
1. **Storage:** AHCI/NVMe drivers, FAT32 filesystem
2. **Networking:** VirtIO-net driver, TCP/IP stack
3. **Process model:** Forth tasks, scheduling, IPC
4. **Device model:** Unified block/net/char subsystem
5. **Security:** Capabilities, ACL, Forth-based access control
**The HAL remains the foundation throughout.**
---
## References
- UEFI Specification: https://uefi.org/specifications
- GNU-EFI: https://sourceforge.net/projects/gnu-efi/
- OSDev Wiki: https://wiki.osdev.org/
- Intel SDM: https://www.intel.com/sdm
- ACPI Specification: https://uefi.org/acpi
docs/working/archive/dead/l4re/L4RE_DICTIONARY_ALLOCATION.adoc
+203
View File
@@ -0,0 +1,203 @@
// DEAD (docs triage 2026-10-14). L4Re/Fiasco.OC was removed as a platform target in mid-2026. Instead: .claude/CLAUDE.md (Project Overview). Moved from docs/working/hardware/platform-integration/L4RE_DICTIONARY_ALLOCATION.adoc.
// Moved from docs/src/platform-integration/L4RE_DICTIONARY_ALLOCATION.adoc to docs/working/scratch/src/platform-integration/L4RE_DICTIONARY_ALLOCATION.adoc on 2026-06-16 (docs reorg Phase 2)
== Dictionary Allocation Strategy for L4Re
:toc: left
:toc-title: Contents
:toclevels: 3
xref:../README.adoc[← Back to Documentation Index]
=== Current Implementation
StarForth currently uses `+malloc()+` and `+free()+` from the host C
library for dictionary entry allocation:
* *Location*: `+src/dictionary_management.c:228+`
* *Function*: `+vm_create_word()+` uses `+malloc(total)+` to allocate
DictEntry structures
* *Cleanup*: `+FORGET+` uses `+free(e)+` to deallocate entries (line 554
in defining_words.c)
This works fine for hosted environments but creates a dependency on libc
heap allocator.
=== Why This Matters for L4Re/StarshipOS
==== Issues with Current Approach:
[arabic]
. *Heap Dependency*: Requires full malloc implementation in L4Re
environment
. *Non-Deterministic*: Heap fragmentation and unpredictable allocation
behavior
. *Separate Address Space*: Dictionary entries live outside VM memory
model
. *Hard to Introspect*: Can’t dump entire VM state as single memory
region
==== Benefits of VM-Based Allocation:
[arabic]
. *Zero External Dependencies*: No libc malloc needed
. *Deterministic*: Simple bump allocator, predictable behavior
. *Unified Address Space*: Everything in one 5MB VM memory region
. *Easy Snapshots*: Can serialize entire VM state
. *Better for Microkernel*: Aligns with L4Re dataspace model
=== Recommended Approach for L4Re
==== Strategy: Conditional Compilation
Add a compile-time flag to switch between malloc and VM-based
allocation:
[source,c]
----
// In dictionary_management.c
#ifdef STARFORTH_VM_DICT_ALLOC
/* VM-based allocation (for L4Re/embedded) */
DictEntry *entry = dict_alloc_from_vm(vm, total);
#else
/* Standard malloc (for hosted environments) */
DictEntry *entry = (DictEntry *) malloc(total);
#endif
----
==== Memory Layout (VM-Based Mode):
....
vm->memory[0...VM_MEMORY_SIZE] (5MB):
┌──────────────────────────────────────────┐
│ Dictionary Headers (256KB) │ ← dict_headers_used grows up
│ - DictEntry structures │
│ - Allocated via dict_alloc() │
├──────────────────────────────────────────┤
│ User Data & Code (variable) │ ← HERE grows up
│ - Compiled definitions │
│ - Variables, constants │
│ - String literals │
├──────────────────────────────────────────┤
│ Block Storage (remainder) │
│ - 1KB blocks for Forth block system │
└──────────────────────────────────────────┘
....
==== Implementation Steps:
[arabic]
. *Add Dictionary Allocator* (256KB bump allocator at `+memory[0]+`):
[source,c]
----
static DictEntry *dict_alloc_from_vm(VM *vm, size_t total_bytes) {
/* Align to cell_t boundary */
size_t align = sizeof(cell_t);
total_bytes = (total_bytes + align - 1) & ~(align - 1);
/* Check space */
if (vm->dict_headers_used + total_bytes > DICT_HEADERS_MAX) {
vm->error = 1;
return NULL;
}
/* Allocate from VM memory */
DictEntry *entry = (DictEntry *)(vm->memory + vm->dict_headers_used);
vm->dict_headers_used += total_bytes;
return entry;
}
----
[arabic, start=2]
. *Update FORGET* (no individual frees, just rewind):
[source,c]
----
/* Instead of free(e) loop, rewind allocator */
size_t target_offset = (uint8_t*)target - vm->memory;
vm->dict_headers_used = target_offset;
----
[arabic, start=3]
. *Adjust VM Initialization*:
[source,c]
----
vm->dict_headers_used = 0; /* Headers start at memory[0] */
vm->here = DICT_HEADERS_MAX; /* User data starts at 256KB */
----
[arabic, start=4]
. *No Changes to Fast-Lookup Cache*: Cache buckets still use malloc for
the pointer arrays, but entries themselves are in VM memory.
==== Testing Strategy:
[source,bash]
----
# Test with standard malloc (current):
make clean && make test
# Test with VM-based allocation:
make clean
CFLAGS="-DSTARFORTH_VM_DICT_ALLOC" make test
# Test L4Re minimal build:
make clean
CFLAGS="-DSTARFORTH_MINIMAL -DSTARFORTH_VM_DICT_ALLOC" make rpi4-cross
----
=== Current Status
*Decision*: Keep using `+malloc()+` for now, document the abstraction
point clearly.
*Rationale*:
* Current implementation is stable and well-tested
* L4Re port can implement the switch when needed
* Abstraction layer is simple and well-defined
* No performance difference between approaches
*Action Items for L4Re Port*:
[arabic]
. Define `+STARFORTH_VM_DICT_ALLOC+` in L4Re build
. Implement `+dict_alloc_from_vm()+` function
. Update `+FORGET+` to use rewind instead of free loop
. Test with full test suite
. Verify no memory leaks in microkernel environment
=== Minimal Code Changes Needed
The switch to VM-based allocation requires changes to only 3 functions:
[arabic]
. *`+vm_create_word()+`* - Use `+dict_alloc_from_vm()+` instead of
`+malloc()+`
. *`+dictionary_word_forget()+`* - Rewind `+dict_headers_used+` instead
of `+free()+` loop
. *`+vm_init()+`* - Initialize `+dict_headers_used+` and offset `+HERE+`
Add to `+VM+` struct in `+include/vm.h+`:
[source,c]
----
size_t dict_headers_used; /* Bytes used for dictionary headers (VM mode only) */
----
*Estimated effort*: 4 hours to implement + 2 hours testing = *6 hours
total*
=== References
* Current malloc usage: `+src/dictionary_management.c:228+`
* Current free usage: `+src/word_source/defining_words.c:554+`
* Fast-lookup cache: `+src/dictionary_management.c:43-67+` (unchanged)
* L4Re dataspace docs: `+docs/L4RE_INTEGRATION.md+`
'''''
*Author*: R. A. James *Date*: 2025-10-01 *Status*: Documented,
implementation deferred to L4Re port
docs/working/archive/dead/l4re/L4RE_INTEGRATION.adoc
+1215
View File
File diff suppressed because it is too large Load Diff
docs/working/archive/dead/l4re/L4RE_INTEGRATION_TODO.adoc
+349
View File
@@ -0,0 +1,349 @@
// DEAD (docs triage 2026-10-14). L4Re/Fiasco.OC was removed as a platform target in mid-2026. Instead: .claude/CLAUDE.md (Project Overview). Moved from docs/working/scratch/src/L4RE_INTEGRATION_TODO.adoc.
// Moved from docs/src/L4RE_INTEGRATION_TODO.adoc to docs/working/archive/dead/l4re/L4RE_INTEGRATION_TODO.adoc on 2026-06-16 (docs reorg Phase 2)
== L4Re Integration TODO
:toc: left
:toc-title: Contents
:toclevels: 3
xref:./README.adoc[← Back to Documentation Index]
=== Overview
The platform abstraction layer is complete and working for standalone
POSIX builds. This document outlines the specific steps needed to
integrate into the StarshipOS L4Re BID build system.
'''''
=== Pre-Integration Checklist
* [x] Platform abstraction layer implemented
* [x] POSIX backend tested and working
* [x] L4Re backend code written
* [x] Standalone builds work with `+make L4RE=1+`
* [ ] Platform files copied to StarshipOS tree
* [ ] BID Makefile updated
* [ ] L4Re build tested in StarshipOS tree
'''''
=== Integration Steps
==== 1. Copy Platform Files to StarshipOS
*Source Location:*
`+/home/rajames/CLionProjects/StarForth/src/platform/+`
*Destination:*
`+/home/rajames/CLionProjects/StarshipOS/l4/pkg/starforth/server/src/platform/+`
*Files to Copy:*
[source,bash]
----
cd /home/rajames/CLionProjects/StarForth
mkdir -p /home/rajames/CLionProjects/StarshipOS/l4/pkg/starforth/server/src/platform
cp src/platform/platform_init.c \
/home/rajames/CLionProjects/StarshipOS/l4/pkg/starforth/server/src/platform/
cp src/platform/time_l4re.c \
/home/rajames/CLionProjects/StarshipOS/l4/pkg/starforth/server/src/platform/
# Note: Do NOT copy time_posix.c - not needed in L4Re build
----
*Header File:*
[source,bash]
----
cp include/platform_time.h \
/home/rajames/CLionProjects/StarshipOS/l4/pkg/starforth/server/include/
----
'''''
==== 2. Update BID Makefile
*File:*
`+/home/rajames/CLionProjects/StarshipOS/l4/pkg/starforth/server/src/Makefile+`
*Changes Needed:*
[source,makefile]
----
# Add platform sources to SRC_C list
SRC_C = \
main.c \
log.c \
io.c \
vm.c \
vm_debug.c \
word_registry.c \
device_map.c \
stack_management.c \
panic.c \
repl_io.c \
memory_management.c \
dictionary_management.c \
vm_api.c \
bam.c \
platform/platform_init.c \ # ← ADD THIS
platform/time_l4re.c \ # ← ADD THIS
word_source/arithmetic_words.c \
# ... rest of existing files ...
# Add librtc to REQUIRES_LIBS
REQUIRES_LIBS = libc_be_l4re libc l4re l4re_c l4sys l4util libsigma0 libblock-device librtc
# ^^^^^^ ADD THIS
----
*Note:* BID already sets `+-D__l4__+` at line 77-78, so platform
selection happens automatically.
'''''
==== 3. Update Core Source Files in StarshipOS
These files need the same changes made to standalone StarForth:
*File:* `+server/src/log.c+`
* [ ] Add `+#include "../include/platform_time.h"+` to includes
* [ ] Replace `+get_timestamp()+` implementation with platform API calls
* [ ] Change: `+time(NULL)+` → `+sf_realtime_ns()+`
* [ ] Change: `+localtime()+` + `+strftime()+` →
`+sf_format_timestamp()+`
*File:* `+server/src/profiler.c+` (if it exists in L4Re version)
* [ ] Add `+#include "../include/platform_time.h"+` to includes
* [ ] Replace `+get_time_ns()+` implementation
* [ ] Change: `+clock_gettime(CLOCK_MONOTONIC)+` → `+sf_monotonic_ns()+`
*File:* `+server/src/main.c+`
* [ ] Add `+#include "platform_time.h"+` to includes
* [ ] Add `+sf_time_init();+` call at start of `+main()+` (before any
logging)
'''''
==== 4. Verify RTC Library Availability
*Check RTC package exists:*
[source,bash]
----
cd /home/rajames/CLionProjects/StarshipOS
ls -la l4/pkg/rtc/
ls -la l4/build-x86_64/lib/*/librtc.a
----
*Expected files:*
* `+l4/pkg/rtc/lib/client/librtc.cc+` - RTC client library
* `+l4/pkg/rtc/include/rtc+` - C++ header
* `+l4/pkg/rtc/include/rtc.h+` - C header
* `+l4/build-x86_64/lib/amd64_gen/l4f/librtc.a+` - Compiled library
*If missing:*
[source,bash]
----
cd l4/pkg/rtc
make
----
'''''
==== 5. Update Loader Configuration
*File:* Choose your loader script (e.g., `+l4/conf/modules.list+` or a
`+.lua+` file)
*Add RTC server startup:*
[source,lua]
----
-- Start RTC server (needed for wall-clock time)
ld:start({
caps = {
vbus = vbus_l4re,
icu = L4.default_loader:new_channel(),
},
log = L4.Env.log:m("rws", "magenta"),
}, "rom/rtc");
-- StarForth with RTC capability
ld:start({
caps = {
rtc = ld:wait("rtc", 1000), -- Wait for RTC server, timeout 1s
-- ... other caps ...
},
log = L4.Env.log:m("rws", "cyan"),
}, "rom/starforth");
----
*Note:* StarForth will work without RTC capability, but timestamps will
show epoch (1970).
'''''
==== 6. Build and Test
*Clean build:*
[source,bash]
----
cd /home/rajames/CLionProjects/StarshipOS
make clean-l4
make l4
----
*Check for platform files in build:*
[source,bash]
----
ls -la l4/build-x86_64/pkg/starforth/server/OBJ-amd64_gen/platform/
# Should see: platform_init.o, time_l4re.o
----
*Launch in QEMU:*
[source,bash]
----
scripts/runos.sh
----
*Verify in StarForth REPL:*
* Logging should show timestamps (may be 1970-01-01 if no RTC server)
* Profiler should work (uses KIP clock, doesn’t need RTC)
* No crashes or missing symbol errors
'''''
=== Verification Tests
==== Test 1: Basic Functionality
[source,forth]
----
\ In StarForth REPL:
1 2 + . \ Should print 3
: TEST 42 . ; \ Should work
TEST \ Should print 42
----
==== Test 2: Profiler (Monotonic Time)
[source,bash]
----
# From shell:
./build-x86_64/bin/amd64_gen/l4f/starforth --profile 2 --run-tests --profile-report
----
Should show timing information without errors.
==== Test 3: Logging (Real Time)
[source,bash]
----
./build-x86_64/bin/amd64_gen/l4f/starforth --log-debug
----
Should show timestamps in log output (may be 1970 epoch if no RTC).
==== Test 4: RTC Server Integration
[source,bash]
----
# In QEMU, after starting with RTC server
# Check for RTC capability
grep "RTC" /proc/l4re/caps
----
If RTC server is running and capability provided, timestamps should be
correct.
'''''
=== Troubleshooting
==== Build Errors
*Error:* `+platform_init.c: No such file or directory+` *Fix:* Check
that platform directory was created and files copied correctly
*Error:* `+undefined reference to l4_kip_clock_ns+` *Fix:* Make sure
linking against `+l4re+` library (should be in REQUIRES_LIBS)
*Error:* `+undefined reference to L4rtc::Rtc::get_timer+` *Fix:* Add
`+librtc+` to REQUIRES_LIBS
*Error:* `+fatal error: platform_time.h: No such file or directory+`
*Fix:* Copy header to `+server/include/+` directory
==== Runtime Errors
*Warning:* "`RTC server not found, assuming 1.1.1970`" *Fix:* Add RTC
server to loader configuration and provide `+rtc+` capability
*Issue:* Timestamps show 1970-01-01 *Fix:* Either RTC server not
running, or capability not provided to StarForth
*Issue:* Profiler shows 0 elapsed time *Fix:* Check that
`+sf_monotonic_ns()+` is being called (shouldn’t need RTC)
'''''
=== Rollback Plan
If integration fails, revert by:
[arabic]
. Remove platform files from `+SRC_C+` in Makefile
. Remove `+librtc+` from `+REQUIRES_LIBS+`
. Revert changes to `+log.c+`, `+profiler.c+`, `+main.c+`
. Delete `+platform/+` directory
. Rebuild: `+make clean-l4 && make l4+`
The L4Re build will work as before (using direct POSIX calls via libc).
'''''
=== Notes
* *BID build system* automatically sets `+-D__l4__+` flag (line 77-78 in
Makefile)
* *Platform selection* happens at compile-time based on `+__l4__+`
define
* *No conditional Makefiles needed* - BID handles everything
* *RTC is optional* - StarForth works without it (uses epoch time)
* *Profiler always works* - uses KIP clock, independent of RTC
'''''
=== Success Criteria
* [ ] StarForth builds cleanly in L4Re tree
* [ ] No linker errors for platform functions
* [ ] REPL starts without crashes
* [ ] Logging shows timestamps (even if 1970)
* [ ] Profiler works with `+--profile+` flag
* [ ] Tests pass with `+--run-tests+`
* [ ] If RTC server running: correct wall-clock time
* [ ] If no RTC: graceful fallback to epoch time
'''''
=== Future Enhancements
* [ ] Add RTC FORTH words (GET-TIME, SET-TIME)
* [ ] Expose monotonic timer to FORTH (for benchmarking)
* [ ] Add timezone support (currently UTC only)
* [ ] Support for RTC alarm/wakeup features
* [ ] Performance comparison: POSIX vs L4Re backend
docs/working/archive/dead/l4re/l_4_re_blkio_endpoints.adoc
+193
View File
@@ -0,0 +1,193 @@
// DEAD (docs triage 2026-10-14). L4Re/Fiasco.OC was removed as a platform target in mid-2026. Instead: .claude/CLAUDE.md (Project Overview). Moved from docs/working/hardware/platform-integration/l_4_re_blkio_endpoints.adoc.
// Moved from docs/src/platform-integration/l_4_re_blkio_endpoints.adoc to docs/working/scratch/src/platform-integration/l_4_re_blkio_endpoints.adoc on 2026-06-16 (docs reorg Phase 2)
== StarForth Block I/O — L4Re Porting Endpoints
:toc: left
:toc-title: Contents
:toclevels: 3
xref:../README.adoc[← Back to Documentation Index]
This document explains how the existing `+blkio+` API maps cleanly onto
L4Re, what you need to implement, and where your VM and Forth block
words connect. The idea is: the *northbound API never changes*; you only
swap in different backends (dataspace or IPC service) when porting to
L4Re.
'''''
=== 1. Northbound API (unchanged)
Your VM and block words always call:
[source,c]
----
int blkio_open (blkio_dev_t *dev, const blkio_vtable_t *vt, const blkio_params_t *p);
int blkio_close(blkio_dev_t *dev);
int blkio_read (blkio_dev_t *dev, uint32_t fblock, void *dst);
int blkio_write(blkio_dev_t *dev, uint32_t fblock, const void *src);
int blkio_flush(blkio_dev_t *dev);
int blkio_info (blkio_dev_t *dev, blkio_info_t *out);
----
No matter what backend you use, these remain stable. That’s your ABI.
'''''
=== 2. In-process L4Re Dataspace Backend
Use when your VM can directly attach a dataspace. This gives you the
fastest path: no IPC.
*Symbols you implement:*
[source,c]
----
size_t blkio_l4ds_state_size(void);
int blkio_l4ds_init_state(void *state_mem, size_t state_len,
l4_cap_idx_t ds_cap,
l4_addr_t attach_addr,
l4_size_t attach_size,
uint32_t total_blocks,
uint32_t fbs,
uint8_t read_only,
void **out_opaque);
const blkio_vtable_t *blkio_l4ds_vtable(void);
----
*State struct:*
[source,c]
----
typedef struct {
l4_cap_idx_t ds; /* dataspace cap */
void *base; /* attached VA */
l4_size_t size; /* mapped length */
uint32_t fbs; /* block size */
uint32_t blocks; /* #blocks */
uint8_t ro;
} blkio_l4ds_state_t;
----
*Ops:*
* `+open+`: `+l4re_rm_attach()+` dataspace, derive block count.
* `+read/write+`: `+memcpy+` from attached region.
* `+flush+`: NOP (or `+l4re_ds_op+` if needed).
* `+close+`: `+l4re_rm_detach()+`.
'''''
=== 3. Out-of-process Block Server (IPC Backend)
Use when storage is owned by a different task (e.g., AHCI, NVMe, eMMC
server).
*Client-side symbols:*
[source,c]
----
size_t blkio_l4svc_state_size(void);
int blkio_l4svc_init_state(void *state_mem, size_t state_len,
l4_cap_idx_t server_cap,
l4_cap_idx_t bounce_ds,
uint32_t fbs,
uint8_t read_only,
void **out_opaque);
const blkio_vtable_t *blkio_l4svc_vtable(void);
----
*IPC protocol (example):*
[source,c]
----
enum bsvc_op {
BSVC_INFO = 1,
BSVC_READ = 2,
BSVC_WRITE = 3,
BSVC_FLUSH = 4,
BSVC_PING = 5
};
----
Transport choices:
* Shared dataspace (preferred): server + client map the same page(s).
* Inline IPC (simpler, less efficient): block payload in IPC message.
*Ops mapping:*
* `+read+`: send op + fblock, server fills shared buffer.
* `+write+`: copy into shared buffer, send op + fblock.
* `+info+`: query block size, count, flags.
'''''
=== 4. Factory Helpers on L4Re
You can add overloads to `+blkio_factory+` for clarity:
[source,c]
----
int blkio_factory_open_l4ds (blkio_dev_t *dev, l4_cap_idx_t ds_cap, uint32_t fbs, uint8_t ro);
int blkio_factory_open_file (blkio_dev_t *dev, const char *path, uint32_t fbs, uint8_t ro);
int blkio_factory_open_l4svc(blkio_dev_t *dev, l4_cap_idx_t server_cap, l4_cap_idx_t bounce_ds, uint32_t fbs, uint8_t ro);
----
This way `+main.c+` can pick a backend cleanly depending on CLI args and
environment.
'''''
=== 5. CLI Semantics on L4Re
* `+--disk-img=<path>+` → Either:
** VFS+file server that returns a dataspace → *l4ds backend*
** Or connect to a block server → *l4svc backend*
* `+--ram-disk=<MB>+` → Allocate RAM dataspace, use *l4ds backend*.
Fallback: if no `+--disk-img+`, create RAM disk from allocator.
'''''
=== 6. Composite Backends (RAM + Disk, 4 KiB packing)
Your design needs:
* Blocks 0–1023 → RAM backend.
* Blocks ≥1024 → Disk backend.
* All wrapped in a composite vtable that dispatches based on `+fblock+`.
Later: add an outer wrapper that packs 3×1 KiB + 1 KiB metadata into 4
KiB pages for device alignment. Still the same five ops.
'''''
== TL;DR — Endpoints to Implement for L4Re
*New backends:*
[source,c]
----
/* In-proc dataspace */
blkio_l4ds_state_size();
blkio_l4ds_init_state(...);
blkio_l4ds_vtable();
/* IPC client/server */
blkio_l4svc_state_size();
blkio_l4svc_init_state(...);
blkio_l4svc_vtable();
----
*Optional factory helpers:*
[source,c]
----
blkio_factory_open_l4ds(...);
blkio_factory_open_l4svc(...);
----
*Everything else stays the same*: VM calls only `+blkio_read+`,
`+blkio_write+`, etc. You swap backends, not call sites.
docs/working/archive/dead/metrics-and-knobs.md
+148
View File
@@ -0,0 +1,148 @@
> **DEAD (docs triage 2026-10-14).** Already marked dead in its own header.
> Instead: its own header. Moved here from `docs/working/architecture/03-architecture/physics-engine/metrics-and-knobs.md`.
<!-- Moved from docs/03-architecture/physics-engine/metrics-and-knobs.md to docs/working/archive/dead/metrics-and-knobs.md on 2026-06-16 (docs reorg Phase 2) -->
# StarForth Metrics & Knobs: GAPS ONLY
## What's NOT Wired (Things to Add)
### DEAD CODE / NEVER CALLED
| What | Location | Status | Comment |
|------|----------|--------|---------|
| Bucket reordering | `physics_hotwords_cache.c:201-215` | Defined but NEVER CALLED | Bubble sort exists but no automatic trigger. Is this dead code or forgotten? |
---
### STACK METRICS (ZERO COLLECTION)
| Metric | File | Missing Since | Impact | Knob Needed |
|--------|------|---|---|---|
| Data stack peak depth | `stack_management.c` | Day 1 | Can't tell if workload needs larger stack | STACK_SIZE tuning blind |
| Return stack peak depth | `stack_management.c` | Day 1 | Can't measure colon definition nesting | Tail-call optimization impossible |
| Underflow attempts | `arithmetic_words.c:59-62` | Day 1 | Can't detect fragile code paths | Could optimize checks if zero underflows observed |
| Overflow attempts | `arithmetic_words.c` | Day 1 | Can't detect stack exhaustion | Early warning system missing |
---
### DICTIONARY METRICS (ZERO VISIBILITY)
| Metric | File | Missing Since | Impact | Why Matters |
|--------|------|---|---|---|
| Bucket load histogram | `vm.c: dict lookup` | Day 1 | First-char bucketing (26 buckets) may be unbalanced | Affects lookup latency, collision clustering |
| Dictionary growth rate | `memory_management.c` | Day 1 | No visibility into `here` pointer velocity | Can't predict "dictionary full" events |
| FORGET usage frequency | `dictionary_words.c` | Day 1 | No trace of word deletion patterns | Fragmentation unknown |
| Lookup failure rate | `vm.c` | Day 1 | How often is a word not found? | Error path frequency unknown |
| Word name length distribution | `vm.c` | Day 1 | Average name length unknown | Affects memory efficiency |
---
### BLOCK I/O METRICS (COMPLETELY DARK)
| Metric | File | Missing Since | Impact | Phase |
|--------|------|---|---|---|
| Block read frequency | `block_subsystem.c` | Day 1 | How often is BLOCK called? | DoE phase 1 |
| Block write frequency | `block_subsystem.c` | Day 1 | How often is UPDATE/FLUSH called? | DoE phase 1 |
| Writeback frequency | `block_subsystem.c:198-200` | Day 1 | When are dirty blocks flushed? | DoE phase 1 |
| Cache hit rate (disk) | `block_subsystem.c` | Day 1 | Blocks served from RAM cache % | DoE phase 1 |
| Dirty block ratio | `block_subsystem.c` | Day 1 | Blocks pending flush at time T | DoE phase 1 |
| I/O latency | `block_subsystem.c` | Day 1 | Time per read/write operation | DoE phase 2 |
| Fragmentation ratio | `block_subsystem.c` | Day 1 | Gaps in allocated blocks | DoE phase 2 |
---
### PER-WORD EXECUTION METRICS
| Metric | File | Missing Since | Impact | Use Case |
|--------|------|---|---|---|
| Per-word execution frequency | `vm.c: inner loop` | Day 1 | Which words are actually hot? | Profiling, hotspot detection |
| Per-word latency distribution | `vm.c` | Day 1 | Median/P99 time per word | Performance regression detection |
| Word category breakdown | `word_source/*.c` | Day 1 | Arithmetic vs. Stack vs. Control dominance | Workload characterization |
| Colon vs. native ratio | `vm.c` | Day 1 | User-defined word execution frequency | Code complexity analysis |
---
### HEAT DYNAMICS (PHASE 2 NOT ACTIVE)
| Metric | File | Status | Comment |
|--------|------|--------|---------|
| Heat decay trace | `physics_metadata.c: TODO` | Phase 2 placeholder | Need to measure if decay causes pattern loss |
| Heat percentile distribution | `physics_metadata.c` | Not updating dynamic | Stored but never dynamically recalculated |
| Heat concentration ratio | `physics_metadata.c` | Not tracked | (Top 10 words heat) / (total heat) |
| Heat velocity | `physics_metadata.c` | Not tracked | Rate of heat change per unit time |
| Heat plateau detection | `physics_metadata.c` | Not tracked | When does word heat level off? |
---
### PIPELINING METRICS (PHASE 1 INSTRUMENTATION, NOT USED)
| Metric | File | Status | Phase 2 Use |
|--------|------|--------|---|
| Prediction accuracy by depth | `physics_pipelining_metrics.c` | Not measured | Binary chop TRANSITION_WINDOW_SIZE |
| Context transition frequency | `physics_pipelining_metrics.c` | Counted but not reported | Validate context patterns |
| Speculation ROI histogram | `physics_pipelining_metrics.c` | Not aggregated | Tune SPECULATION_THRESHOLD |
| Misprediction cost (actual) | `physics_pipelining_metrics.c` | Measured but not validated | Calibrate cost model |
| Pipeline stall reduction | `physics_pipelining_metrics.c` | Not measured | Throughput impact unknown |
---
### CACHE POLLUTION & CYCLES
| Metric | File | Missing Since | Impact |
|--------|------|---|---|
| Cache eviction/re-promotion cycles | `physics_hotwords_cache.c` | Day 1 | Are we thrashing? |
| False negatives (words evicted before promotion) | `physics_hotwords_cache.c` | Day 1 | Should promotion threshold be lower? |
| Cache eviction candidate value | `physics_hotwords_cache.c` | Day 1 | Was LRU the right choice? |
| Bucket scan depth (position of hit) | `physics_hotwords_cache.c` | Day 1 | Is bucket order actually helping? |
| Promotion wait time | `physics_hotwords_cache.c` | Day 1 | How long does word stay in bucket before cache? |
---
## Priority for "Twisting the Dragon's Tail"
### 🔴 **CRITICAL (Do First)**
1. **Stack peak depth** - 5 minutes to add, massive insight
2. **Block I/O metrics** - Writeback behavior unknown (could be performance killer)
3. **Dictionary bucket load** - Unbalanced bucketing = latency tail
### 🟠 **HIGH (Phase 1 DoE)**
4. **Per-word execution frequency** - Validation for cache/window decisions
5. **Heat decay trace** - Validate Phase 2 feature before enabling
6. **Block I/O hit rate** - Complete the I/O picture
### 🟡 **MEDIUM (Phase 2 DoE)**
7. **Pipelining accuracy by depth** - Input to window size binary chop
8. **Colon definition nesting depth** - Tail-call optimization opportunities
9. **Dictionary growth velocity** - Predict "full" events
### 🟢 **NICE-TO-HAVE (Phase 3)**
10. Heat concentration ratio
11. Word category breakdown
12. Speculation ROI histogram
---
## Implementation Effort Estimate
| Category | Files | LOC | Time | Priority |
|----------|-------|-----|------|----------|
| Stack metrics | stack_management.c, vm.h | ~30 | 30 min | 🔴 |
| Dictionary metrics | vm.c, memory_management.c, vm.h | ~50 | 45 min | 🔴 |
| Block I/O metrics | block_subsystem.c, vm.h | ~80 | 1 hr | 🔴 |
| Per-word metrics | vm.c, physics_metadata.c, vm.h | ~100 | 1.5 hr | 🟠 |
| Heat dynamics | physics_metadata.c, vm.h | ~60 | 1 hr | 🟠 |
| Pipelining accuracy | physics_pipelining_metrics.c, vm.h | ~40 | 45 min | 🟡 |
---
## Bucket Reordering Mystery
**File**: `physics_hotwords_cache.c:201-215`
**Code**: `hotwords_bucket_reorder()` - bubble sort by execution_heat
**Status**: Function defined but **NEVER CALLED** from lookup path
**Question**: Is this dead code or forgot to hook it?
**If enabled, metrics needed:**
- Reorder frequency
- Cost (bubble sort latency)
- Benefit (does reordering actually improve hit latency?)
docs/working/archive/historical/2026-01-27-system-architecture-draft.md
+981
View File
@@ -0,0 +1,981 @@
> **HISTORICAL (docs triage 2026-10-14).** Draft 0.1 of the system vision ("everything is a VM, everything is a capsule"), 2026-01-27; the design that was built is recorded in the FABRIC documents.
> Instead: docs/fabric/ (FABRIC-3.5.md for the Tripod and kernel-Hermes, FABRIC-3.md for bare metal). Moved here from `docs/lithosananke/SYSTEM_ARCHITECTURE.md`.
# LithosAnanke System Architecture
**Version**: 0.1 (Draft)
**Date**: 2026-01-27
**Status**: Architectural Design
---
## 1. EXECUTIVE SUMMARY
LithosAnanke is a microkernel-style operating system where all system services run as isolated Virtual Machines (VMs) communicating exclusively through message passing. User identity is tied to physical media (thumbdrives), with cryptographically signed capability grants enforced at VM birth.
**Core Principles:**
- Everything is a VM, everything is a capsule
- All communication through messaging backbone (no exceptions)
- All storage through block I/O VM (no exceptions)
- Physical possession = authentication
- Cryptographic signing = capability enforcement
- One admin key, no recovery, no backdoors
---
## 2. VM HIERARCHY
### 2.1 Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ Mama VM │
│ (births and deaths ONLY) │
└─────────────────────────────────────────────────────────────────┘
│
┌─────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────────┐ ┌───────────────┐ ┌───────────────────┐
│ Messaging VM │ │ Block I/O VM │ │ User Session │
│ (all comms) │ │ (all storage)│ │ VMs... │
└───────────────────┘ └───────────────┘ └───────────────────┘
```
### 2.2 Mama VM
**Role:** Root VM, births and deaths other VMs. Nothing else.
**Responsibilities:**
- Execute Mama init capsule at boot
- Birth system VMs (Messaging, Block I/O)
- Birth user session VMs on thumbdrive insertion
- Death user session VMs on thumbdrive removal
- Maintain VM registry
**Does NOT do:**
- Message routing (Messaging VM does this)
- Block I/O (Block I/O VM does this)
- User interaction (User VMs do this)
**Capsule:** `mama-init.4th` with flag `CAPSULE_FLAG_MAMA_INIT`
### 2.3 Messaging Backbone VM
**Role:** All inter-VM communication routes through this VM. No exceptions.
**Responsibilities:**
- Route messages between VMs
- Enforce message-level ACLs
- Queue messages for sleeping/busy VMs
- Handle hardware interrupts (keyboard, USB events)
- Notify Mama of hardware events (thumbdrive insertion)
**Capabilities Required:**
- `CAP_ROUTE_MESSAGES` - Can route between any VMs
- `CAP_HARDWARE_IRQ` - Can receive hardware interrupts
- `CAP_VM_ENUMERATE` - Can query VM registry
**Capsule:** `messaging.4th` with flag `CAPSULE_FLAG_PRODUCTION`
### 2.4 Block I/O VM
**Role:** All storage access routes through this VM. No exceptions.
**Responsibilities:**
- Physical block I/O to all storage devices
- Map user virtual blocks to physical blocks
- Enforce block-level ACLs
- Manage Block Allocation Maps (BAM)
- Handle storage device hotplug
**Capabilities Required:**
- `CAP_BLOCK_PHYSICAL` - Direct hardware access to storage
- `CAP_BLOCK_MAP` - Can create/modify block mappings
- `CAP_ACL_ENFORCE` - Can check ACLs on block operations
**Capsule:** `block-io.4th` with flag `CAPSULE_FLAG_PRODUCTION`
### 2.5 User Session VMs
**Role:** Isolated user environment with REPL.
**Created when:** User inserts authenticated thumbdrive
**Destroyed when:** Thumbdrive removed or user executes BYE
**Each user session gets:**
- Own VM instance
- Own heap (memory arena)
- Own RAM blocks (0-2047)
- Own disk blocks mapped to thumbdrive (2048+)
- Capabilities as granted by signed capability block
**Capsule:** `user-shell.4th` or user-specified custom capsule
---
## 3. BLOCK SUBSYSTEM ARCHITECTURE
### 3.1 User's Logical View (Deterministic)
Every user sees the identical logical layout:
```
┌─────────────────────────────────────────────────────────────┐
│ Blocks 0-2047: RAM Drive │
│ - Fast, volatile working space │
│ - 2 MB total (2048 × 1024 bytes) │
│ - Cleared on session end │
├─────────────────────────────────────────────────────────────┤
│ Blocks 2048+: Physical Storage (Thumbdrive) │
│ - Persistent across sessions │
│ - User's "home" blocks │
│ - Block 2048 typically contains startup script │
└─────────────────────────────────────────────────────────────┘
```
**Guarantees:**
- Blocks 0-2047 are ALWAYS RAM (fast, volatile)
- Blocks 2048+ are ALWAYS physical storage (persistent)
- This layout is identical for every user on every machine
- User FORTH code is portable (same logical addresses everywhere)
### 3.2 Physical Storage Layout (Thumbdrive)
```
┌─────────────────────────────────────────────────────────────┐
│ Devblock 0: Volume Header (4 KiB) │
│ - Magic, version, geometry │
│ - BAM location pointers │
│ - Reserved range definitions │
├─────────────────────────────────────────────────────────────┤
│ Devblocks 1..B: Physical BAM │
│ - 1 bit per 1 KiB block │
│ - Tracks allocated/free status │
├─────────────────────────────────────────────────────────────┤
│ Devblocks B+1..X: System Region │
│ - NOT VISIBLE to user (no logical block maps here) │
│ - Identity sector (fingerprint source data) │
│ - Signed capability grants │
│ - ACL tables (variable size, chained blocks) │
│ - Audit log (grows over time) │
│ - Startup block pointer │
├─────────────────────────────────────────────────────────────┤
│ Devblocks X+1..end: User Block Region │
│ - Mapped to user's logical blocks 2048+ │
│ - User's FORTH code, data, definitions │
│ - Fully under user control (within ACL limits) │
└─────────────────────────────────────────────────────────────┘
```
### 3.3 Virtual BAM
Each user session has a **Virtual BAM** that maps only their accessible blocks:
```
User's Virtual BAM Physical Reality
───────────────── ─────────────────
Block 0 ─────────────────► RAM buffer[0]
Block 1 ─────────────────► RAM buffer[1]
...
Block 2047 ─────────────────► RAM buffer[2047]
Block 2048 ─────────────────► Thumbdrive devblock X+1
Block 2049 ─────────────────► Thumbdrive devblock X+2
...
```
**Key property:** System region devblocks have NO entry in the virtual BAM. From the user's FORTH perspective, they don't exist. No block number maps to them.
### 3.4 Block I/O Flow
User VM requests a block read:
```
User VM Messaging VM Block I/O VM
│ │ │
│ ─── MSG(READ,2048) ───► │ │
│ │ ─── MSG(READ) ─────► │
│ │ │
│ │ ├─ Check ACL
│ │ ├─ Map virtual→physical
│ │ ├─ Physical I/O
│ │ │
│ │ ◄─── MSG(DATA) ───── │
│ ◄─── MSG(DATA) ──────── │ │
│ │ │
```
**No direct hardware access** from user VMs. All I/O is mediated.
### 3.5 System Region Uses
The system region (invisible to user) stores variable-size metadata:
| Data | Size | Notes |
|------|------|-------|
| Identity sector | ~256 bytes | Source data for fingerprint |
| Fingerprint | 32-64 bytes | SHA-256 or SHA-512 hash |
| Capability grants | Variable | Signed by admin |
| ACL tables | Variable | Block-level permissions |
| Group memberships | Variable | For shared access |
| Audit log | Grows | Tracks significant events |
| Startup pointer | 8 bytes | Which block to LOAD at init |
| Custom metadata | Variable | Application-specific |
**Variable-size data** uses block chaining (existing `prev_block`, `next_block` fields in `blk_meta_t`).
---
## 4. USER IDENTITY MODEL
### 4.1 Physical Capability Model
**Identity IS the thumbdrive.** No passwords, no usernames.
```
┌─────────────────────────────────────────────────────────────┐
│ User's Thumbdrive │
├─────────────────────────────────────────────────────────────┤
│ Contains: │
│ - Identity data → SHA fingerprint │
│ - Signed capability grants │
│ - User's blocks (code, data) │
│ - Startup configuration │
├─────────────────────────────────────────────────────────────┤
│ Properties: │
│ - Fully portable (works on any LithosAnanke system) │
│ - Self-contained (no central user database) │
│ - Physical possession = authentication │
└─────────────────────────────────────────────────────────────┘
```
### 4.2 Fingerprint Generation
The SHA fingerprint uniquely identifies the user:
```
Fingerprint = SHA-256(identity_sector_data)
```
**Identity sector contents** (stored in system region):
- UUID (random, generated at user creation)
- Creation timestamp
- Optional: hardware serial numbers, user metadata
**The fingerprint:**
- Is computed by the kernel at thumbdrive insertion
- Is matched against the signed capability grant
- Cannot be forged (would need admin signature)
### 4.3 Session Lifecycle
```
┌─────────────────────────────────────────────────────────────┐
│ 1. THUMBDRIVE INSERTION │
├─────────────────────────────────────────────────────────────┤
│ • Hardware interrupt → Messaging VM │
│ • Messaging VM notifies Mama │
│ • Block I/O VM reads system region │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 2. IDENTITY VERIFICATION │
├─────────────────────────────────────────────────────────────┤
│ • Compute SHA fingerprint from identity sector │
│ • Read signed capability grant from system region │
│ • Verify signature against fused admin public key │
│ • If invalid: reject, log attempt, stop │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 3. VM BIRTH │
├─────────────────────────────────────────────────────────────┤
│ • Mama selects base capsule (user-shell or custom) │
│ • Mama births new VM │
│ • VM allocated: heap, RAM blocks (0-2047) │
│ • Disk blocks (2048+) mapped to thumbdrive │
│ • Capabilities granted: capsule ∩ signed_grant │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 4. INITIALIZATION │
├─────────────────────────────────────────────────────────────┤
│ • Base capsule executes (defines PERSONALITY) │
│ • User's startup block LOADed (e.g., 2048 LOAD) │
│ • Custom words, aliases, preferences applied │
│ • REPL ready for user input │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 5. ACTIVE SESSION │
├─────────────────────────────────────────────────────────────┤
│ • User interacts via REPL │
│ • All I/O through Block I/O VM │
│ • All comms through Messaging VM │
│ • ACLs enforced on every operation │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 6. SESSION END (thumbdrive removal OR BYE) │
├─────────────────────────────────────────────────────────────┤
│ • Hardware interrupt (removal) or explicit BYE │
│ • Messaging VM notifies Mama │
│ • Mama deaths the user VM │
│ • RAM blocks freed │
│ • Heap freed │
│ • Block mappings removed │
│ • Session over, no trace remains │
└─────────────────────────────────────────────────────────────┘
```
### 4.4 User Startup Customization
Users can customize their environment like `.bashrc`:
```forth
\ Block 2048 - User's startup script (example)
\ Load my personal library
2049 LOAD
\ Define aliases
: LS DIR ;
: PWD BLK@ . ;
\ Set preferences
DECIMAL
64 BASE-DEFAULT !
\ Greeting
." Welcome back, Captain!" CR
```
**Constraints:**
- Startup runs AFTER base capsule
- Can define words, load blocks, set variables
- CANNOT exceed capability grants
- CANNOT access blocks outside mapped range
- CANNOT send unauthorized messages
---
## 5. SECURITY MODEL
### 5.1 Root of Trust: Admin Fusing
**One-time, irreversible provisioning:**
```
┌─────────────────────────────────────────────────────────────┐
│ PROVISIONING MODE (First Boot Only) │
├─────────────────────────────────────────────────────────────┤
│ 1. System detects unprovisioned state │
│ 2. Generate Ed25519 keypair │
│ • Private key: 32 bytes │
│ • Public key: 32 bytes │
│ 3. FUSE public key to system │
│ • Written to NVRAM/secure storage │
│ • Cannot be changed after this │
│ 4. Create admin thumbdrive │
│ • Private key stored in system region │
│ • Root capability grant (all permissions) │
│ • Self-signed (bootstrap) │
│ 5. Exit provisioning mode FOREVER │
│ • Provisioning flag set │
│ • Cannot re-enter this mode │
└─────────────────────────────────────────────────────────────┘
```
**After fusing:**
- Public key is immutable (fused to system)
- Private key exists ONLY on admin thumbdrive
- No copies, no backups, no recovery mechanism
- Lose admin thumbdrive = cannot provision new users
### 5.2 Admin Thumbdrive
**The single point of administrative authority:**
```
Admin Thumbdrive (System Region)
├── Admin identity sector
├── Root capability grant (self-signed at provisioning)
├── Private signing key (Ed25519)
└── Admin blocks (tools, scripts)
Admin Capabilities:
├── CAP_SIGN_GRANTS - Can sign capability grants
├── CAP_CREATE_USERS - Can provision new thumbdrives
├── CAP_REVOKE_USERS - Can revoke capabilities
├── CAP_SYSTEM_CONFIG - Can modify system settings
├── CAP_ALL_BLOCKS - Can access any block
└── CAP_* - All capabilities
```
**Admin can:**
- Create new user thumbdrives
- Sign capability grants
- Revoke user capabilities (by not re-signing)
- Access any user's blocks (for recovery, audit)
- Modify system configuration
**Admin CANNOT:**
- Create another admin (no delegation)
- Recover lost admin thumbdrive
- Change the fused public key
### 5.3 Capability Grant Structure
```
┌─────────────────────────────────────────────────────────────┐
│ Capability Grant Block │
├─────────────────────────────────────────────────────────────┤
│ Header: │
│ Magic: "CPGR" (0x43504752) │
│ Version: 1 │
│ Grant ID: unique identifier │
│ Created: timestamp │
│ Expires: timestamp (0 = never) │
├─────────────────────────────────────────────────────────────┤
│ Identity: │
│ Fingerprint: SHA-256 (32 bytes) │
│ Must match computed fingerprint at auth time │
├─────────────────────────────────────────────────────────────┤
│ Capabilities (bitmap + parameters): │
│ CAP_BLOCK_READ - Can read blocks │
│ CAP_BLOCK_WRITE - Can write blocks │
│ CAP_BLOCK_RANGE - Allowed block range (start, end) │
│ CAP_MSG_SEND - Can send messages │
│ CAP_MSG_RECEIVE - Can receive messages │
│ CAP_MSG_TARGETS - Allowed message targets (VM list) │
│ CAP_CAPSULE_LOAD - Can request capsule loading │
│ CAP_CUSTOM_CAPSULE - Can use custom init capsule │
│ CAP_GROUP_MEMBER - Group membership list │
│ CAP_RESOURCE_QUOTA - CPU, memory, block limits │
├─────────────────────────────────────────────────────────────┤
│ Signature: │
│ Ed25519 signature over all above fields │
│ Signed by admin private key │
│ 64 bytes │
└─────────────────────────────────────────────────────────────┘
```
### 5.4 Capability Verification Flow
```
Thumbdrive inserted
│
▼
┌───────────────────────────────────────┐
│ 1. Read identity sector │
│ 2. Compute fingerprint = SHA-256(id) │
└───────────────────────────────────────┘
│
▼
┌───────────────────────────────────────┐
│ 3. Read capability grant block │
│ 4. Extract grant.fingerprint │
│ 5. Compare: computed == grant.fp? │
│ NO → Reject (identity mismatch) │
└───────────────────────────────────────┘
│ YES
▼
┌───────────────────────────────────────┐
│ 6. Extract grant.signature │
│ 7. Verify: Ed25519_verify( │
│ fused_public_key, │
│ grant_data, │
│ signature) │
│ FAIL → Reject (tampered/forged) │
└───────────────────────────────────────┘
│ PASS
▼
┌───────────────────────────────────────┐
│ 8. Check expiry timestamp │
│ EXPIRED → Reject │
└───────────────────────────────────────┘
│ VALID
▼
┌───────────────────────────────────────┐
│ 9. Grant capabilities to new VM │
│ effective_caps = grant.caps │
└───────────────────────────────────────┘
```
### 5.5 Capability Enforcement Points
**At VM Birth:**
- Capabilities extracted from signed grant
- Stored in VM registry entry
- Cannot be modified after birth
**At Message Send:**
- Messaging VM checks: Does sender have CAP_MSG_SEND?
- Messaging VM checks: Is target in CAP_MSG_TARGETS?
- Reject if not authorized
**At Block Access:**
- Block I/O VM checks: Does requestor have CAP_BLOCK_READ/WRITE?
- Block I/O VM checks: Is block in CAP_BLOCK_RANGE?
- Reject if not authorized
**At Capsule Load:**
- Mama checks: Does requestor have CAP_CAPSULE_LOAD?
- Mama checks: Does requestor have CAP_CUSTOM_CAPSULE (if custom)?
- Reject if not authorized
### 5.6 Effective Capabilities
Final capabilities are the intersection:
```
effective_caps = capsule.declared ∩ grant.allowed ∩ system.policy
```
**capsule.declared:** What the init capsule requests
**grant.allowed:** What the signed grant permits
**system.policy:** System-wide limits (optional)
A user cannot gain capabilities not in their signed grant, even if the capsule requests them.
### 5.7 Revocation
**Capability revocation works by:**
1. Admin creates new grant with reduced capabilities
2. Admin re-signs the grant
3. New grant replaces old on thumbdrive
4. Next session uses new (reduced) capabilities
**Immediate revocation:**
- Admin can maintain a revocation list
- Checked at session start
- Fingerprint on revocation list = reject
**No online revocation checking** - this is a standalone system. Revocation requires physical access to update the thumbdrive or revocation list.
### 5.8 Attack Surface Analysis
**Attacks PREVENTED:**
| Attack | Prevention |
|--------|------------|
| Password guessing | No passwords exist |
| Credential theft | Must steal physical device |
| Privilege escalation | Capabilities signed, immutable |
| Block range escape | Virtual BAM has no mapping |
| Direct hardware access | All I/O through service VMs |
| Message spoofing | Source VM verified by Messaging VM |
| Capability forging | Ed25519 signature required |
| Admin impersonation | Admin private key on single device |
| Recovery mode exploit | No recovery mode exists |
| Backup compromise | No backups exist |
**Attacks NOT PREVENTED:**
| Attack | Notes |
|--------|-------|
| Physical theft of thumbdrive | Physical security is user's responsibility |
| Physical theft of admin drive | Total compromise, must reflash |
| Cold boot attack | RAM contents may be recoverable |
| Evil maid (hardware tampering) | Outside scope, requires secure boot |
| Side channel timing | Not addressed in current design |
### 5.9 Security Boundaries
```
┌─────────────────────────────────────────────────────────────┐
│ KERNEL SPACE │
│ • Mama VM │
│ • VM scheduler │
│ • Fused public key │
│ • Hardware abstraction │
└─────────────────────────────────────────────────────────────┘
│
│ Capsule-defined boundary
▼
┌─────────────────────────────────────────────────────────────┐
│ PRIVILEGED VM SPACE │
│ • Messaging VM (CAP_ROUTE_MESSAGES) │
│ • Block I/O VM (CAP_BLOCK_PHYSICAL) │
└─────────────────────────────────────────────────────────────┘
│
│ Capability-checked boundary
▼
┌─────────────────────────────────────────────────────────────┐
│ USER VM SPACE │
│ • User session VMs │
│ • Constrained by signed capability grants │
│ • Cannot access other user's blocks │
│ • Cannot send unauthorized messages │
└─────────────────────────────────────────────────────────────┘
```
---
## 6. MESSAGING BACKBONE
### 6.1 Message Structure
```
┌─────────────────────────────────────────────────────────────┐
│ Message Header │
├─────────────────────────────────────────────────────────────┤
│ msg_id: uint64 Unique message identifier │
│ source_vm: uint32 Sender VM ID │
│ target_vm: uint32 Recipient VM ID │
│ msg_type: uint16 Message type code │
│ flags: uint16 Delivery flags │
│ payload_len: uint32 Payload size in bytes │
│ timestamp: uint64 Send timestamp (monotonic) │
├─────────────────────────────────────────────────────────────┤
│ Message Payload │
│ (variable length, message-type specific) │
└─────────────────────────────────────────────────────────────┘
```
### 6.2 Message Types
**System Messages:**
- `MSG_VM_BIRTH` - Mama→Messaging: new VM created
- `MSG_VM_DEATH` - Mama→Messaging: VM terminated
- `MSG_HARDWARE_EVENT` - Messaging→Mama: USB insert/remove
**Block I/O Messages:**
- `MSG_BLOCK_READ` - Request block read
- `MSG_BLOCK_WRITE` - Request block write
- `MSG_BLOCK_DATA` - Block data response
- `MSG_BLOCK_ACK` - Write acknowledgment
- `MSG_BLOCK_ERROR` - I/O error response
**User Messages:**
- `MSG_USER_DEFINED` - Application-specific messages
### 6.3 Message Routing
```
┌─────────────────────────────────────────────────────────────┐
│ Messaging VM │
├─────────────────────────────────────────────────────────────┤
│ Inbound Queue ──► │ Route │ ──► Per-VM Outbound Queues │
│ │ │ ├── Mama Queue │
│ │ │ ├── Block I/O Queue │
│ │ │ ├── User VM 1 Queue │
│ │ │ ├── User VM 2 Queue │
│ │ │ └── ... │
├─────────────────────────────────────────────────────────────┤
│ Routing Logic: │
│ 1. Verify source VM exists and is alive │
│ 2. Check source has CAP_MSG_SEND │
│ 3. Check target is in source's CAP_MSG_TARGETS │
│ 4. Enqueue to target's outbound queue │
│ 5. Wake target VM if sleeping │
└─────────────────────────────────────────────────────────────┘
```
### 6.4 Synchronous vs Asynchronous
**Asynchronous (default):**
- Sender posts message and continues
- Message queued for recipient
- Response comes as separate message
**Synchronous (CALL pattern):**
- Sender posts message and blocks
- Waits for response message
- Useful for RPC-style operations (e.g., block read)
---
## 7. BOOT SEQUENCE
### 7.1 Full Boot Flow
```
┌─────────────────────────────────────────────────────────────┐
│ PHASE 0: Hardware Initialization │
├─────────────────────────────────────────────────────────────┤
│ • UEFI boot │
│ • Memory map from firmware │
│ • Jump to kernel entry │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ PHASE 1: Kernel Initialization │
├─────────────────────────────────────────────────────────────┤
│ • Console init (serial UART) │
│ • PMM init (physical memory manager) │
│ • VMM init (virtual memory manager) │
│ • Heap init (kernel malloc) │
│ • IDT init (interrupt descriptor table) │
│ • APIC init (interrupt controller) │
│ • Timer init (APIC timer, TSC calibration) │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ PHASE 2: Provisioning Check │
├─────────────────────────────────────────────────────────────┤
│ • Check provisioning flag in NVRAM │
│ • If NOT provisioned: enter provisioning mode │
│ └── Generate keypair, fuse, create admin drive, exit │
│ • If provisioned: load fused public key, continue │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ PHASE 3: Mama VM Creation │
├─────────────────────────────────────────────────────────────┤
│ • Allocate Mama heap │
│ • Create Mama VM (VM ID 0) │
│ • Load mama-init capsule │
│ • Execute Mama init │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ PHASE 4: System VM Birth │
├─────────────────────────────────────────────────────────────┤
│ • Mama births Messaging VM from messaging.4th capsule │
│ • Mama births Block I/O VM from block-io.4th capsule │
│ • System services now running │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ PHASE 5: Device Enumeration │
├─────────────────────────────────────────────────────────────┤
│ • USB controller initialization │
│ • Storage device discovery │
│ • Messaging VM registers for hardware events │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ PHASE 6: Ready for Users │
├─────────────────────────────────────────────────────────────┤
│ • System idle, waiting for thumbdrive insertion │
│ • Console shows: "Insert thumbdrive to login" │
│ • Or admin thumbdrive for administration │
└─────────────────────────────────────────────────────────────┘
```
### 7.2 Capsule Boot Order
```
Capsule Directory (in kernel or on system storage)
├── mama-init.4th ← Loaded first, by kernel
├── messaging.4th ← Birthed by Mama, second
├── block-io.4th ← Birthed by Mama, third
├── user-shell.4th ← Birthed on user login
└── (other capsules) ← As needed
```
---
## 8. IMPLEMENTATION PRIORITIES
Based on dependencies:
### 8.1 Critical Path
```
1. USB Driver (XHCI)
└── Need to detect thumbdrives
2. Block I/O Kernel Port
└── Need to read thumbdrive system region
3. SHA-256 in Kernel
└── Need to compute fingerprints
4. Ed25519 in Kernel
└── Need to verify signatures
5. Messaging Backbone VM
└── Need for all inter-VM communication
6. Block I/O VM
└── Need for all storage access
7. Capability Enforcement
└── Need to check grants at every operation
8. User Session VM
└── Finally can have interactive users
```
### 8.2 Estimated Complexity
| Component | LOC Estimate | Complexity |
|-----------|--------------|------------|
| XHCI driver | ~2500 | High |
| USB mass storage | ~800 | Medium |
| Block subsystem port | ~500 | Medium (reuse existing) |
| SHA-256 | ~400 | Low (well-known algorithm) |
| Ed25519 | ~600 | Medium (or use library) |
| Capability grant parser | ~300 | Low |
| Messaging VM | ~1000 | Medium |
| Block I/O VM | ~800 | Medium |
| Capability enforcement | ~500 | Medium |
| User session management | ~600 | Medium |
| Admin provisioning | ~400 | Medium |
**Total: ~8,400 LOC** for core security infrastructure
---
## 9. DATA STRUCTURES SUMMARY
### 9.1 VM Registry Entry (Extended)
```c
typedef struct {
uint32_t vm_id; // Unique VM identifier
uint32_t state; // VMState enum
uint64_t birth_capsule_id; // Which capsule birthed this
uint64_t birth_timestamp_ns; // When born
uint64_t birth_dict_hash; // Dictionary hash at birth
// Security fields (new)
uint8_t fingerprint[32]; // SHA-256 of user identity
uint64_t capability_bitmap; // Granted capabilities
uint32_t block_range_start; // First allowed block
uint32_t block_range_end; // Last allowed block
uint32_t msg_target_count; // Number of allowed targets
uint32_t msg_targets[16]; // Allowed message target VMs
// Resource tracking
uint64_t heap_base; // VM's heap start
uint64_t heap_size; // VM's heap size
uint64_t ram_blocks_base; // RAM block allocation
uint32_t flags;
uint32_t reserved;
} VMRegistryEntry;
```
### 9.2 Capability Bitmap
```c
// Capability flags
#define CAP_BLOCK_READ (1ULL << 0) // Can read blocks
#define CAP_BLOCK_WRITE (1ULL << 1) // Can write blocks
#define CAP_MSG_SEND (1ULL << 2) // Can send messages
#define CAP_MSG_RECEIVE (1ULL << 3) // Can receive messages
#define CAP_CAPSULE_LOAD (1ULL << 4) // Can request capsule load
#define CAP_CUSTOM_CAPSULE (1ULL << 5) // Can use custom capsule
#define CAP_SIGN_GRANTS (1ULL << 6) // Can sign capability grants (admin)
#define CAP_CREATE_USERS (1ULL << 7) // Can provision users (admin)
#define CAP_REVOKE_USERS (1ULL << 8) // Can revoke capabilities (admin)
#define CAP_SYSTEM_CONFIG (1ULL << 9) // Can modify system config (admin)
#define CAP_ALL_BLOCKS (1ULL << 10) // Can access any block (admin)
#define CAP_HARDWARE_IRQ (1ULL << 11) // Can receive hardware interrupts
#define CAP_ROUTE_MESSAGES (1ULL << 12) // Can route messages (messaging VM)
#define CAP_BLOCK_PHYSICAL (1ULL << 13) // Direct hardware I/O (block VM)
#define CAP_VM_ENUMERATE (1ULL << 14) // Can query VM registry
#define CAP_BLOCK_MAP (1ULL << 15) // Can create block mappings
#define CAP_ACL_ENFORCE (1ULL << 16) // Can enforce ACLs
// Convenience
#define CAP_USER_DEFAULT (CAP_BLOCK_READ | CAP_BLOCK_WRITE | \
CAP_MSG_SEND | CAP_MSG_RECEIVE)
#define CAP_ADMIN_ALL (0xFFFFFFFFFFFFFFFFULL)
```
### 9.3 Signed Capability Grant (On-disk)
```c
typedef struct __attribute__((packed)) {
// Header (16 bytes)
uint32_t magic; // "CPGR" = 0x43504752
uint16_t version; // Grant format version
uint16_t flags; // Grant flags
uint64_t grant_id; // Unique grant identifier
// Timestamps (16 bytes)
uint64_t created_ns; // Creation timestamp
uint64_t expires_ns; // Expiry (0 = never)
// Identity (32 bytes)
uint8_t fingerprint[32]; // SHA-256 that must match
// Capabilities (32 bytes)
uint64_t capability_bitmap; // Granted capabilities
uint32_t block_range_start; // First allowed block (logical)
uint32_t block_range_end; // Last allowed block (logical)
uint32_t msg_target_count; // Number of target VMs
uint32_t msg_targets[4]; // Allowed targets (expandable)
// Quotas (16 bytes)
uint32_t max_heap_mb; // Maximum heap size
uint32_t max_ram_blocks; // Maximum RAM blocks
uint32_t max_disk_blocks; // Maximum disk blocks
uint32_t reserved;
// Signature (64 bytes)
uint8_t signature[64]; // Ed25519 signature
} CapabilityGrant; // Total: 176 bytes
```
---
## 10. GLOSSARY
| Term | Definition |
|------|------------|
| **Mama VM** | Root VM that births and deaths all other VMs |
| **Baby VM** | Any VM birthed by Mama |
| **Capsule** | Immutable init block sequence defining VM personality |
| **PERSONALITY** | VM's immutable identity after birth (from capsule) |
| **DOMAIN** | Mama's construction space (never visible to babies) |
| **Fingerprint** | SHA-256 hash of identity sector, identifies user |
| **Capability Grant** | Signed document specifying user's permissions |
| **Fused Key** | Admin public key permanently written at provisioning |
| **Virtual BAM** | Per-user block allocation map (subset view) |
| **System Region** | Thumbdrive area invisible to user (metadata, ACLs) |
| **Admin Thumbdrive** | Single device with signing authority |
---
## 11. OPEN QUESTIONS
1. **Multiple simultaneous users?** Can multiple thumbdrives be inserted at once, with multiple concurrent sessions?
2. **Shared blocks between users?** Can users share specific blocks with each other? (Would require group capabilities)
3. **Network authentication?** Future extension for remote login?
4. **Audit log rotation?** System region audit log grows forever - need rotation policy?
5. **Capability delegation?** Can a user grant subset of their capabilities to another? (Currently: no)
6. **Time-limited sessions?** Automatic logout after idle timeout?
---
## 12. REVISION HISTORY
| Version | Date | Changes |
|---------|------|---------|
| 0.1 | 2026-01-27 | Initial architecture draft |
---
**Document Status:** Draft - Pending Review
**Authors:** Claude (AI Assistant), Robert A. James
**Next Steps:**
1. Review and refine security model
2. Prototype USB detection
3. Implement Ed25519/SHA-256 in kernel
4. Design message format in detail
5. Implement Block I/O VM capsule
docs/working/archive/historical/2026-06-03-test-gap-audit.md
+228
View File
@@ -0,0 +1,228 @@
> **HISTORICAL (docs triage 2026-10-14).** A test-coverage audit of StarForth v3.0.3 / LithosAnanke v1.0.8, 2026-06-03; POST has grown since (1052 tests).
> Instead: v3/src/test_runner/ and the POST line of any current boot. Moved here from `docs/TEST_GAPS.md`.
# StarForth Dictionary Test Coverage — Gap Audit
**Branch:** `lithosananke` | **StarForth v3.0.3 / LithosAnanke v1.0.8**
**Audited:** 2026-06-03
---
## Summary
| Metric | Count |
|--------|-------|
| Total registered words | ~230 |
| Words with at least one test | ~115 |
| Words with **no** test | ~115 |
| Test files | 20 |
| Untested word categories | 5 (editor, inference, log, physics-diag, q48) |
Coverage is solid for the foundational FORTH-79 core but has critical holes
in control-flow completeness, format output, memory ops, and the S" word
just added in v3.0.3.
---
## Priority 1 — FORTH-79 Core: Missing Critical Words
These are standard FORTH-79 words used in virtually all real programs.
Missing tests here are compliance failures.
| Word(s) | Source file | Why critical |
|---------|-------------|--------------|
| `1+` `1-` `2+` `2-` `2*` `2/` | arithmetic_words.c | Shorthand increment/shift ops; used in almost every program. Arithmetic test only covers `/MOD` variants, not these. |
| `I` `J` | control_words.c | Loop index access. Every `DO...LOOP` that reads the index uses `I`; nested loops need `J`. No test at all. |
| `EXIT` | control_words.c | Return from word mid-execution. Essential for early-return patterns. No test. |
| `LEAVE` | control_words.c | Early exit from `DO...LOOP`. No test. |
| `UNLOOP` | control_words.c | Required to clean the return stack before `EXIT` inside a loop. Omitting it is a stack-corruption bug; no test exercises this. |
| `AGAIN` | control_words.c | `BEGIN...AGAIN` infinite loop. Common pattern, completely untested. |
| `CASE` `OF` `ENDOF` `ENDCASE` | control_words.c | Full `CASE` structure. Entire construct untested. |
| `+!` `-!` | memory_words.c | Modify-in-place. The single most common memory idiom in FORTH after `@` and `!`. No test. |
| `FILL` `MOVE` `ERASE` | memory_words.c | Bulk memory primitives. Used by string ops, block subsystem, array initialisation. No test. |
| `2!` | memory_words.c | Store double/address pair. `2@` is tested; its store partner is not. |
---
## Priority 2 — S" (New in v3.0.3) — Urgent
`S"` was added as a proper IMMEDIATE primitive in this release. It has zero
test coverage. Regression risk is high since the implementation is new.
| Word(s) | Source file | Gap |
|---------|-------------|-----|
| `S"` | string_words.c | No test for interpret-mode push (c-addr u), compile-mode inline (`(s")` runtime), or use inside a `:` definition. |
| `(s")` | string_words.c | Internal runtime for compiled `S"`. Exercised only by compiled `S"` — needs a compile-mode test. |
| `BL` | string_words.c | Space-character constant. Trivial but untested; often paired with `S"`. |
| `SOURCE` `>IN` | string_words.c | Input stream state. `SOURCE` returns ( c-addr u ); `>IN` is the offset. Both untested. |
---
## Priority 3 — Format / Output Words
`format_words.c` has 19 registered words. Only `.` and `BASE`/`DECIMAL` are
touched (from io_words_test.c). The pictured numeric output group is
completely dark.
| Word(s) | Source file | Gap |
|---------|-------------|-----|
| `HEX` `OCTAL` | format_words.c | Base switching. Used constantly in systems work. No test. |
| `.S` | format_words.c | Print entire stack without consuming it. Core debugging primitive. No test. |
| `?` | format_words.c | Print cell at address (` @ . `). Heavily used. No test. |
| `.R` `U.` `U.R` | format_words.c | Right-justified and unsigned output. No test. |
| `D.` `D.R` | format_words.c | Double-number output. No test. |
| `<#` `#` `#S` `#>` `HOLD` `SIGN` | format_words.c | Pictured numeric output group. Entire 6-word subsystem untested. |
| `DUMP` | format_words.c | Memory hex dump. No test. |
---
## Priority 4 — Logical / Comparison Gaps
logical_words_test.c covers the common cases but misses several FORTH-79 words.
| Word(s) | Source file | Gap |
|---------|-------------|-----|
| `0<>` | logical_words.c | Not-equal-to-zero. Symmetric counterpart to `0=`. No test. |
| `U<` `U>` | logical_words.c | Unsigned comparison. Critical for address arithmetic and char comparisons. No test. |
| `WITHIN` | logical_words.c | Range test. No test. |
| `TRUE` `FALSE` | logical_words.c | Boolean constants. Untested (though they're trivially `0` and `-1`). |
---
## Priority 5 — Stack Word Gaps
stack_words_test.c covers DUP DROP SWAP OVER ROT DEPTH PICK ROLL.
Two standard words missing:
| Word(s) | Source file | Gap |
|---------|-------------|-----|
| `?DUP` | stack_words.c | Conditional dup (dup only if non-zero). Common idiom. No test. |
| `-ROT` | stack_words.c | Reverse rotate. Tested implicitly via `ROT` but never directly. |
---
## Priority 6 — Double Number Gaps
double_words_test.c is thorough but misses:
| Word(s) | Source file | Gap |
|---------|-------------|-----|
| `S>D` | double_words.c | Convert single to double. The entry point for all `D`-word arithmetic. No test. |
| `D2*` `D2/` | double_words.c | Double arithmetic shifts. No test. |
| `2>R` `2R>` `2R@` | double_words.c | Double-number return stack. No test. |
---
## Priority 7 — Mixed Arithmetic Gap
| Word(s) | Source file | Gap |
|---------|-------------|-----|
| `M-` | mixed_arithmetic_words.c | `M+` and `M*` are tested; `M-` is not. |
---
## Priority 8 — Compilation / Meta Words
These are used to build user-defined IMMEDIATE words and to manipulate the
dictionary at compile time. None are tested.
| Word(s) | Source file | Gap |
|---------|-------------|-----|
| `COMPILE` `[COMPILE]` | defining_words.c | Used to force compilation of words inside IMMEDIATE definitions. No test. |
| `[']` | string_words.c | Compile-time version of `'` (tick). Returns XT of next word at compile time. No test. |
| `LIT` | defining_words.c | Internal compile-time literal push. No test. |
| `STATE` | defining_words.c / dictionary_manipulation_words.c | Dual-registered. Compile/interpret switch. No test. |
| `ALIGN` | dictionary_words.c | Align `HERE` to cell boundary. No test. |
| `'` (TICK) | dictionary_manipulation_words.c | Fetch execution token of named word. Core of `[']` and `EXECUTE`. No test. |
| `LITERAL` `[LITERAL]` | string_words.c | Compile a literal at runtime / inside definitions. Partially touched in io test but no dedicated test. |
---
## Priority 9 — Dictionary Traversal Words
These enable introspection of the dictionary structure. No tests at all.
| Word(s) | Source file | Gap |
|---------|-------------|-----|
| `>NAME` `NAME>` | dictionary_manipulation_words.c | Convert between CFA and NFA. No test. |
| `>LINK` `LINK>` | dictionary_manipulation_words.c | Navigate dictionary linked list. No test. |
| `CFA` `NFA` `LFA` `PFA` | dictionary_manipulation_words.c | Classic field-access words. No test. |
| `TRAVERSE` | dictionary_manipulation_words.c | Walk dictionary chain. No test. |
| `INTERPRET` | dictionary_manipulation_words.c | The outer interpreter. No test. |
---
## Priority 10 — System Words
| Word(s) | Source file | Gap |
|---------|-------------|-----|
| `ABORT"` `(ABORT")` | system_words.c | Conditional abort with message. Used in error handling. No test. |
| `WORDS` `VLIST` | system_words.c | Dictionary listing. No test. |
| `SEE` | system_words.c | Decompiler. No test. |
| `COLD` `WARM` | system_words.c | System restart/reinit. No test (understandably risky to test directly). |
| `QUIT` | system_words.c | Outer interpreter restart. No test. |
---
## Priority 11 — Block Word Gap
| Word(s) | Source file | Gap |
|---------|-------------|-----|
| `-->` | block_words.c | Chain-load next block. Used in multi-block definitions. No test. |
---
## Priority 12 — Lifecycle / Capsule Words
Capsule-BIRTH/KILL/PAUSE/RESUME/USE have partial coverage from mama_forth tests
but direct functional tests are missing.
| Word(s) | Source file | Gap |
|---------|-------------|-----|
| `KILL` `PAUSE` `RESUME` | lifecycle_words_hosted.c | Capsule lifecycle management. No direct test. |
| `USE` | lifecycle_words_hosted.c | Switch active VM. No test. |
---
## Priority 13 — Entire Untested Categories
These word groups have **zero test coverage**. Lower urgency than gaps above
because they are StarForth-specific extensions (not FORTH-79 compliance),
but they should be ticketed.
| Category | Source file | Word count | Notes |
|----------|-------------|------------|-------|
| Q48 fixed-point | q48_words.c | 21 | POST unit tests exist for the C layer but no FORTH-word tests |
| Inference | inference_words.c | 19 | Physics adaptive engine; complex to test deterministically |
| Log words | log_words.c | 7 | LOG-ERROR … LOG-LEVEL@; simple but untested |
| Physics diagnostics | physics_diagnostic_words.c | 4 | PHYSICS-WORD-METRICS etc. |
| Physics freeze | physics_freeze_words.c | 9 | FREEZE-WORD, HEAT@, etc. |
| Physics benchmark | physics_benchmark_words.c | 6 | BENCH-DICT-LOOKUP etc. |
| Physics pipelining diag | physics_pipelining_diagnostic_words.c | 6 | PIPELINING-SHOW-STATS etc. |
| Editor | editor_words.c | 4 | L, S, SHOW, EDIT — block editor; very low priority |
---
## Work Order (suggested sequence)
```
1. [ ] P1a arithmetic_words_test.c — add 1+ 1- 2+ 2- 2* 2/
2. [ ] P1b control_words_test.c — add I J EXIT LEAVE UNLOOP AGAIN
3. [ ] P1c control_words_test.c — add CASE OF ENDOF ENDCASE
4. [ ] P1d memory_words_test.c — add +! -! 2! FILL MOVE ERASE
5. [ ] P2 string_words_test.c — add S" (s") BL SOURCE >IN
6. [ ] P3a format_words_test.c — new file: HEX OCTAL .S ? .R U. U.R D. D.R
7. [ ] P3b format_words_test.c — add <# # #S #> HOLD SIGN DUMP
8. [ ] P4 logical_words_test.c — add 0<> U< U> WITHIN TRUE FALSE
9. [ ] P5 stack_words_test.c — add ?DUP -ROT
10.[ ] P6 double_words_test.c — add S>D D2* D2/ 2>R 2R> 2R@
11.[ ] P7 mixed_arithmetic_words_test.c — add M-
12.[ ] P8a defining_words_tests.c — add COMPILE [COMPILE] LIT STATE
13.[ ] P8b dictionary_manipulation_words_test.c — add ' ['] ALIGN LITERAL
14.[ ] P9 dictionary_manipulation_words_test.c — add >NAME NAME> CFA NFA LFA PFA TRAVERSE INTERPRET
15.[ ] P10 system_words_test.c — new file: ABORT" WORDS VLIST SEE
16.[ ] P11 block_words_test.c — add -->
17.[ ] P12 lifecycle capsule tests — add KILL PAUSE RESUME USE
18.[ ] P13a q48_words_test.c — new file: all 21 Q48 FORTH words
19.[ ] P13b log_words_test.c — new file: LOG-ERROR … LOG-LEVEL@
20.[ ] P13c physics words tests — new files as needed
```
docs/working/archive/historical/2026-06-10-next-session-plan.md
+218
View File
@@ -0,0 +1,218 @@
> **HISTORICAL (docs triage 2026-10-14).** A session hand-off plan from 2026-06-10, on the pre-split `lithosananke` branch; every item in it is long done or overtaken.
> Instead: .claude/CLAUDE.md and docs/fabric/FABRIC-3.md for current status. Moved here from `docs/NEXT_SESSION.md`.
# Next Session Plan
Generated at end of session `cdcb1844` (2026-06-10).
Branch at close: `lithosananke` (merged and pushed; all three arches boot to `ok>`)
---
## What Was Accomplished This Session
- Fixed SysV ABI stack alignment bug in `kernel_entry.S` (`sub rax, 8`)
- Removed redundant kmalloc stack switch from `kernel_main.c`
- Fixed GOT/PE crash in `apply_sgr` — `FB_ANSI_PALETTE` got
`__attribute__((visibility("hidden")))` in both `framebuffer.h` and
`framebuffer.c`; EXEC-DOE now runs to full completion (160,673 rows)
- Added framebuffer host render test (`tools/fbtest.c`) and QEMU screendump
(`scripts/qemu_screenshot.sh`) to the amd64 `qemu` target
- CLAUDE.md: one-process-at-a-time build rule
- Design doc: `docs/reference/kernel-args/DESIGN.md` — UEFI
command-line argument system (KernelArgs)
---
## Task 0 — Fix doe.4th EXEC regression (do first)
All three architectures boot to `ok>` but `S" doe.4th" EXEC` fails at
runtime with "EXEC: failed: doe.4th" on every arch. Root cause: merge
conflict resolution took the remote's self-executing `doe.4th` (480 runs)
and `init.4th` (calls `S" doe.4th" EXEC`), but the capsule fails at runtime.
`EXEC-DOE` word is also absent (removed from `init.4th` by the remote).
Investigation starting point:
- Check `capsule_exec_init` return path — add diagnostic to distinguish
NOT_FOUND vs HASH_MISMATCH vs runtime error
- Verify new `doe.4th` block structure (correct block numbers, no undefined words)
- Consider restoring `EXEC-DOE` as an alias once doe.4th runs correctly
---
## Task 1 — Carry-over acceptance tests (do first, one at a time)
All three architectures now confirmed on `lithosananke`:
amd64 ✓ aarch64 ✓ riscv64 ✓
These were completed at end of session. No action needed unless a new
change is made — then re-run all three before merging.
```bash
make -f Makefile.starkernel ARCH=aarch64 clean qemu
make -f Makefile.starkernel ARCH=riscv64 clean qemu
```
Commit logs after each run. Do not proceed to any implementation task
until all three architectures have a clean `ok>` log committed.
---
## Task 2 — Memory size audit
Before tuning anything, measure the actual headroom:
- BSS stack (`g_kernel_stack`): currently 2 MB — log the deepest RSP
observed during EXEC-DOE and compare against the stack base
- VM data stack / return stack depths — check `vm.h` for current limits
- Rolling window buffer (`ROLLING_WINDOW_SIZE=4096`) — verify it isn't
being silently truncated under DoE load
- `KERNEL_HEAP_SIZE` (2 GB allocated, 1 GB physical in QEMU) — confirm
kmalloc never hits the limit during a full DoE run
There are no hard rules on these sizes; increase freely if headroom is
tight. Document findings in `docs/lithosananke/memory-audit.md`.
---
## Task 3 — Implement KernelArgs
Design doc: `docs/reference/kernel-args/DESIGN.md`
Implementation order (one commit per step):
1. `include/starkernel/kernel_args.h` — struct + default macros
2. `include/starkernel/boot_info_offsets.h` — assembly-visible offsets
with `_Static_assert` checks in C
3. `include/starkernel/uefi.h` — add `kernel_stack_base`,
`kernel_stack_size`, `KernelArgs args` to `BootInfo`
4. `include/starkernel/cmdline.h` + `src/starkernel/boot/cmdline.c` —
UCS-2→ASCII, tokeniser, parser
5. `src/starkernel/boot/uefi_loader.c` — call `cmdline_parse`, allocate
dynamic stack via `AllocatePages`
6. `src/starkernel/arch/amd64/kernel_entry.S` — dynamic stack switch,
fall back to BSS if `kernel_stack_base == 0`
7. `src/starkernel/kernel_main.c` — use `args.heap_size`, honour
`args.run_doe` and `args.log_level`
8. `Makefile.starkernel` — pass `--doe` via OVMF boot entry; remove socat
EXEC-DOE injection
Run all three arch acceptance tests after step 8.
---
## Task 4 — ISO real-hardware boot audit
The ISO built by xorriso works in QEMU but has a known gap for real
hardware (xorriso warns: "no directory /EFI/BOOT"). Real commodity UEFI
firmware expects `BOOTX64.EFI` present in the ISO *filesystem* at
`/EFI/BOOT/BOOTX64.EFI`, not just as an El Torito boot image.
Targets:
- **amd64** — commodity x86-64 machine: add `/EFI/BOOT/BOOTX64.EFI` to
the ISO filesystem tree in the Makefile xorriso invocation
- **aarch64 / RPi4** — needs `BOOTAA64.EFI`; RPi4 requires the
tianocore RPi4 UEFI firmware layer on the SD card/USB first; the ISO
sits on top
- **aarch64 / BeagleBone AI-64** (aspirational) — same EFI path as RPi4
Document the verified boot procedure for each target in
`docs/lithosananke/hardware-boot.md`.
---
## Task 5 — `REBOOT` word
A FORTH word that optionally accepts a next-boot argument string and
performs a cold reset via UEFI Runtime Services.
Key design decisions (agreed this session):
- **Not interpreter-only at the C level** — no `STATE` check, no special
flag
- **ACL-controlled** — the word-level ACL system (Task 6) governs access
- **On ACL denial**: emit warning, return to REPL; system keeps running
- **No abort** — consistent with how all sensitive words will work under ACL
Implementation sketch:
```forth
S" --doe --stack=8M" REBOOT \ reboot with next-boot args
REBOOT \ plain cold reboot
```
Under the hood:
1. If a string is on the stack: write it to a UEFI variable
(`StarForthNextArgs` under a custom GUID) via `RT->SetVariable`
2. Call `RT->ResetSystem(EfiResetCold, EFI_SUCCESS, 0, NULL)`
3. Loader reads the variable on next boot, merges with `LoadOptions`;
clears the variable after reading (one-shot)
`REBOOT` with no args is a plain cold reboot.
---
## Task 6 — `USE` word and resilient capsule loading
A general FORTH word (not capsule-specific) for loading external blobs.
```forth
S" hires-video.4th" USE
S" sound-card-1" USE \ fails → line gets commented out
S" sound-card-2" USE
```
On failure, `USE`:
1. Locates the calling line in the block buffer
2. Prepends `\ ` to comment it out (writes to RAM block)
3. Continues — one failure does not stop subsequent `USE` calls
4. Up to 16 lines per block may be commented out
**Self-healing property:** over multiple boots the system converges to a
stable configuration. Hardware that isn't present or drivers that don't
work are silently excluded. Hardware that works is locked in.
`REBOOT` integrates naturally:
```forth
S" hires-video.4th" USE
REBOOT \ apply hardware init needing a cold reset
hires-video-init? NOT IF
\ didn't take — USE mechanism will comment it out on next boot
THEN
```
---
## Task 7 — Forward word declarations
A FORTH analog of C prototypes: declare a word's name and stack effect
before its implementing capsule is loaded. Enables circular dependencies
between capsules without load-order brittleness.
```forth
FORWARD: init-network ( -- ) \ stub; filled when net capsule loads
FORWARD: video-mode? ( -- f ) \ stub; filled when video capsule loads
```
If a forward-declared word is called before its definition arrives, it
emits a warning and returns without crashing. When the implementing
capsule loads and defines the word, the stub is replaced.
This is a general VM feature — not kernel or capsule specific.
---
## Task 8 — Word-level ACL system (existing design, next to implement)
Design doc: `docs/reference/word-acl/DESIGN.md`
This is already designed. It is the policy engine that makes Tasks 5–7
safe. Implement before or alongside Task 5 (`REBOOT`) since `REBOOT`
depends on ACL for its safety guarantees.
---
## Pointer for Next Session
Start with Task 1 (aarch64 + riscv64 acceptance). Read this file and the
design docs listed above before touching any code. Present the task order
to Captain Bob for approval before each implementation step.
All development stays on the working branch; never merge to master.
docs/working/archive/historical/2026-06-30-break-me-report.md
+82
View File
@@ -0,0 +1,82 @@
> **HISTORICAL (docs triage 2026-10-14).** A generated diagnostic report from one test run, 2026-06-30.
> Instead: the boot POST in any current logs/ run. Moved here from `docs/BREAK_ME_REPORT.md`.
# 🔥 STARFORTH BREAK-ME DIAGNOSTIC REPORT 🔥
## Executive Summary
**Generated:** 2026-06-30 16:53:09
**Test Mode:** ULTRA-COMPREHENSIVE DIAGNOSTIC
**Purpose:** This report documents the results_run_01_2025_12_08 of the most exhaustive testing suite ever run on StarForth. Every word, every edge case, every pathological scenario has been tested to ensure rock-solid reliability and FORTH-79 standard compliance.
---
## 📊 Performance Metrics
| Metric | Value |
|--------|-------|
| Test Duration | 0.00 seconds |
| Total Tests | 1003 |
| Tests/Second | inf |
## 🖥️ System Information
| Component | Specification |
|-----------|---------------|
| VM Architecture | Direct-threaded |
| Standard | FORTH-79 + StarForth Extensions |
| Cell Size | 8 bytes |
| Stack Size | 1024 cells |
| Memory | 5242880 bytes |
---
## 🎉 EASTER EGG: THE STARFORTH CHALLENGE 🎉
```
⭐ C O N G R A T U L A T I O N S ⭐
You've discovered the hidden StarForth challenge!
Your system just survived 1003 exhaustive tests.
But can YOU survive the ULTIMATE FORTH CHALLENGE?
🏆 THE STARFORTH MASTER CHALLENGE 🏆
Write a Forth program that:
1. Prints the Fibonacci sequence up to 1000
2. Uses ONLY the following words:
: ; DUP OVER DROP SWAP . CR < IF THEN DO LOOP
3. Fits in exactly 79 characters (FORTH-79 tribute!)
Submit your solution to the StarForth community!
Hint: Think recursively, but remember FORTH-79
doesn't have RECURSE... 😉
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
BONUS SECRET: If you ran --break-me, you're already
in the top 1% of Forth enthusiasts. Welcome to the
elite club! Your system just proved it can handle
anything you throw at it.
May your stacks be balanced and your words defined,
-- The StarForth Development Team ⭐
```
### 🎮 More Easter Eggs to Find:
1. Try defining a word called `STARSHIP` and see what happens
2. Run `42 ENTROPY@ .` to see the universe's favorite number
3. Type `WORDS` and count how many contain 'STAR'
4. Check what happens when you `FORGET FORGET` (just kidding, don't!)
---
*Generated by StarForth --break-me mode*
*Copyright (c) 2025 Robert A. James - StarshipOS Forth Project*
docs/working/archive/historical/2026-07-amd64-isr-fix/README.md
+100
View File
@@ -0,0 +1,100 @@
> **HISTORICAL (docs triage 2026-10-14).** The write-up of one fix (APIC timer ISR, amd64), done.
> Instead: the code itself (kernel/src/arch/amd64/). Moved here from `docs/lithosananke/amd64-isr-fix/README.md`.
# APIC Timer ISR Fix — amd64
**Date**: 2026-06-03
**Branch**: `claude/starforth-architecture-review-oubN0`
**Commits**: `fb0c256`, `901f827`
---
## Problem
The LithosAnanke amd64 kernel was triple-faulting on the first APIC timer interrupt,
causing QEMU to exit immediately after printing `ok>`. The adaptive heartbeat never
fired and `Heartbeat: N ticks` never appeared in serial logs.
## Root Cause
The kernel is compiled with `-fPIC -fvisibility=hidden` (required for UEFI PE
relocation). Assembly symbols `isr_stub_table` and `isr_stub0` in `isr.S` were
declared `.global` with **default visibility**. The C compiler therefore accessed
them through GOT indirection. However, the linker — building a final non-shared
binary — collapsed the GOT references to point directly at the symbols themselves.
The net effect: "load from GOT[`isr_stub_table`]" loaded the 8-byte instruction
bytes of `isr_stub0` (`0x006a90eb006a006a`) rather than its address. Both
`link_time_addr` and `runtime_addr` in `arch_interrupts_init()` became the same
wrong value (instruction bytes), so `reloc_offset = 0` and all 256 IDT entries
received garbage handler addresses. IDT[32] was set to `0x6a006afffffe91e9`
(non-canonical), which triggered #GP → double fault → triple fault on the first
timer delivery.
The bug was confirmed by disassembling `arch_interrupts_init()` in the loader PE:
the compiler emitted a `mov` (load from memory) for `&isr_stub0` instead of a
`lea` (compute address), reading the function's instruction bytes.
## Fix
Two changes, one line each:
**`src/starkernel/arch/amd64/isr.S`** — Added `.hidden` directives for all symbols
referenced from C:
```asm
.hidden isr_stub_table
.hidden isr_stub0
.hidden isr_stub32
.hidden isr_stub_default
.hidden isr_common_entry
```
**`src/starkernel/arch/amd64/interrupts.c`** — Added visibility attributes to the
`extern` declarations:
```c
extern void *isr_stub_table[IDT_ENTRIES] __attribute__((visibility("hidden")));
extern void isr_stub0(void) __attribute__((visibility("hidden")));
```
With hidden visibility, the compiler uses direct RIP-relative `lea` instead of GOT
indirection, yielding correct runtime addresses for all 256 IDT entries.
## Result
| Criterion | Before | After |
|---|---|---|
| IDT[32] handler | `0x6a006afffffe91e9` (corrupt) | `0x3d90251c` (valid canonical address) |
| First timer interrupt | #GP → triple fault → QEMU exit | ISR fires cleanly, returns via `iretq` |
| `PARITY:OK` | ✅ (unaffected) | ✅ |
| `Init: init.4th OK` | ✅ (unaffected) | ✅ |
| `ok>` REPL | ✅ reached, then immediate crash | ✅ runs indefinitely |
| `Heartbeat: N ticks` | Never | Every second at 100 Hz, TIME-TRUST ≈ 0.999 |
## Secondary Fix
`repl_print_dec32`, `repl_lapic_ccr`, `repl_print_hex32`, and `idle_iters` in
`repl.c` are x86-64 specific (LAPIC MMIO, `pushfq`). They were unguarded, causing
`-Werror=unused-function` / `unused-variable` failures on aarch64 (clang) and
riscv64 (gcc). All moved inside `#ifdef ARCH_AMD64`.
## Acceptance Testing
All three architectures must boot to `ok>` with `PARITY:OK` and `Init: init.4th OK`.
amd64 additionally shows `Heartbeat: N ticks TIME-TRUST≈0.999` repeating at 1 Hz.
```bash
make -f Makefile.starkernel ARCH=amd64 clean qemu
make -f Makefile.starkernel ARCH=aarch64 clean qemu
make -f Makefile.starkernel ARCH=riscv64 clean qemu
```
Verify each log:
```bash
grep "PARITY:OK" logs/qemu-<arch>-*.log
grep "Init: init.4th OK" logs/qemu-<arch>-*.log
grep "ok>" logs/qemu-<arch>-*.log
grep "Heartbeat:" logs/qemu-amd64-*.log | head -5
```
docs/working/archive/historical/2026-07-birthing/PLAN.md
+412
View File
@@ -0,0 +1,412 @@
> **HISTORICAL (docs triage 2026-10-14).** The child-VM birthing plan and its acceptance record; built and live since (M7.1).
> Instead: docs/reference/M7.1.md and docs/fabric/. Moved here from `docs/birthing/PLAN.md`.
# Birthing Plan: Child VMs Born of Hera
**Status:** DRAFT — requires Captain Bob approval before any implementation begins
**Branch:** `birthing` (off `lithosananke`)
**Authors:** Claude Code + Captain Bob
**Date:** 2026-05-24
---
## 1. Vision
Three named VMs form the initial constellation of LithosAnanke:
| VM Name | Role |
|---------|------|
| **Hera** | Mama VM — orchestrates all other VMs; runs `capsules/init.4th` |
| **Hermes** | Messaging stratum — pub/sub between VMs; runs `capsules/hermes/init.4th` |
| **Artemis** | Storage and memory manager; runs `capsules/artemis/init.4th` |
**Hades** remains the kernel + HAL substrate beneath all of them.
For now, Hera's job is simply: birth, start, stop, and kill VM instances.
Concurrent scheduling, inter-VM messaging, and storage management are later milestones.
Every VM — including Hermes and Artemis — carries the same lifecycle primitives.
A VM can birth and manage other VMs. The infrastructure is uniform.
---
## 2. Hard Constraints
Fixed by the execution model. Hera's implementation is not constrained — she expands
as the design requires.
| Constraint | Implication |
|------------|-------------|
| Same process as Hera | No `fork()`, no `pthread_create()` |
| No threads | Scheduling is cooperative, inside the interpreter loop |
| No OS processes | VMs share Hera's address space |
| ANSI C99, `-Wall -Werror` | No GNU extensions |
| Hades is the substrate | All I/O, timing, and memory from Hades (kernel+HAL) |
---
## 3. VM Identity and Naming
### 3.1 Every VM Has a Name
The current `VMRegistryEntry` tracks VMs by numeric ID only. This must be extended with a
symbolic name field. A VM's name:
- Is set at birth (derived from the capsule namespace — see §5)
- Is displayed in every log line emitted by that VM
- Is the argument to all lifecycle words: `S" Artemis" BIRTH`, `S" Artemis" KILL`, etc.
- Is unique within a session (no two live VMs share a name)
- Is case-preserved but case-insensitive for lookup
### 3.2 VM Registry Extension
`VMRegistryEntry` needs a `name` field (at minimum `char name[64]`). The name is:
- Set by `BIRTH` at VM creation time
- Cleared on `KILL` (slot becomes reusable)
- Logged in every parity record alongside the numeric VM ID
### 3.3 Log Prefix — Critical
Every line of console output emitted by a VM is prefixed with its name:
```
[Hera] ok
[Hermes] pub/sub ready
[Artemis] storage initialized
[Hera] 2 VMs alive
```
This applies to `EMIT`, `TYPE`, `.`, `.S`, `CR`, and all I/O words. The prefix is injected
at the HAL console layer — the VM does not format it; Hades prints it automatically based
on the current active VM context.
---
## 4. The Five Primitive Forth Words
These words are **built-in C primitives** registered in every VM's dictionary at startup.
They are not defined in Forth — they are part of the base word set, available to all VMs.
### Stack signatures use the counted-string convention: `( c-addr u -- )`
---
### `BIRTH ( c-addr u -- )`
Birth a new VM from the capsule whose namespace matches the given name.
**Mapping:** `S" Artemis"` → lowercase `"artemis"` → capsule name `artemis:init.4th`
(the standard capsule for a named VM is `<name>:init.4th`).
**Sequence:**
1. Lowercase the name string
2. Construct capsule name: `<name>:init.4th`
3. Look up capsule in the directory by name
4. Verify capsule is `FLAG_PRODUCTION` + `FLAG_ACTIVE` and not `FLAG_REVOKED`
5. Allocate a new VM context (full VM — Hera expands her pool as needed)
6. Execute the capsule init (via existing `capsule_exec_init` path)
7. Register VM under the symbolic name in the extended `VMRegistryEntry`
8. Log parity record: `PARITY:BIRTH name=Artemis vm_id=N capsule_id=X`
9. On success: push 0; on failure: push error code
**If a VM named "Artemis" is already alive:** check its health. If healthy, skip the birth
and log the fact. Do not error, do not replace. BIRTH is idempotent for healthy VMs.
---
### `KILL ( c-addr u -- )`
Destroy a named VM. The VM's resources are freed and its registry slot is cleared.
**Sequence:**
1. Look up VM by name in registry
2. If not found: push error code, return
3. Forcibly terminate any pending execution in that VM
4. Free the VM's stacks, dictionary, and memory
5. Clear registry slot (name + state = DEAD)
6. Log parity record: `PARITY:KILL name=Artemis vm_id=N`
7. Push 0 on success
**Note:** KILL is unconditional. It does not wait for the VM to reach a yield point.
At this phase (no concurrent execution), the VM is idle when KILL is called.
---
### `START ( c-addr u -- )`
Begin or resume execution of a named VM.
**At this phase:** `START` runs the VM to completion (or until it calls `STOP` on itself)
before returning control to the caller. Concurrent interleaved execution is a later milestone.
**Sequence:**
1. Look up VM by name
2. If state is EMBRYO or STOPPED: set state to LIVE, transfer execution
3. If state is LIVE: error (already running)
4. On VM completion or self-STOP: return control to the calling VM
5. Push 0 on success
---
### `STOP ( c-addr u -- )`
Suspend a named VM. Its execution state is preserved; `START` will resume it.
Any VM can STOP any other. Authority comes from identity — a VM may assume another
identity for failsafe and recovery. The identity assumption mechanism is a future
milestone; in Phase 1, cross-VM STOP is permitted without restriction.
**Sequence:**
1. Verify name matches the current VM
2. Save execution state (IP, stacks)
3. Set state to STOPPED
4. Return control to the VM that called `START`
---
### `USE ( c-addr u -- )`
Make a named VM the active console target. Subsequent REPL input and output is directed
to and from that VM until another `USE` call or the VM dies.
```forth
S" Hermes" USE \ REPL now talks to Hermes
: send-test S" hello" PUBLISH ;
S" Hera" USE \ REPL returns to Hera
```
USE is **system-wide** — it is not scoped to a capsule or identity. All REPL input and
output redirects to the named VM until the next `USE` call. The `[Name]` prefix in the
console reflects the active USE target.
**Sequence:**
1. Look up VM by name in registry
2. Set as the active system-wide REPL/console VM
3. Console prefix changes to `[VMName]`
4. Stack unchanged (no return value)
---
## 5. Capsule Namespace Convention
### 5.1 Directory Structure
Each named VM owns a subdirectory under `capsules/`:
```
capsules/
├── init.4th ← Hera's capsule ONLY (root level, no subdir)
├── hermes/
│ └── init.4th ← Hermes' capsule
└── artemis/
└── init.4th ← Artemis' capsule
```
**`capsules/init.4th` is Hera's exclusive init.** No other VM uses the root-level init.
Do not place general capsules there. Do not recreate the old `conf/` directory.
### 5.2 How mkcapsule Names Capsules
`tools/mkcapsule.c` scans `capsules/` recursively. It encodes relative paths by replacing
`/` with `:`. So:
| File path | Capsule name |
|-----------|-------------|
| `capsules/init.4th` | `init.4th` |
| `capsules/hermes/init.4th` | `hermes:init.4th` |
| `capsules/artemis/init.4th` | `artemis:init.4th` |
### 5.3 BIRTH Name → Capsule Name Mapping
`BIRTH` lowercases the VM name and appends `:init.4th`:
| Forth call | Capsule looked up |
|------------|------------------|
| `S" Hera" BIRTH` | `init.4th` (special case — root) |
| `S" Hermes" BIRTH` | `hermes:init.4th` |
| `S" Artemis" BIRTH` | `artemis:init.4th` |
| `S" MyWorker" BIRTH` | `myworker:init.4th` |
The root-level `init.4th` is a special case: `S" Hera" BIRTH` is how a future VM would
re-birth Hera; at bootstrap, Hera boots from it automatically.
### 5.4 mkcapsule Flag Mapping
The capsule flag (`FLAG_PRODUCTION` vs `FLAG_EXPERIMENT`) is determined by the directory
prefix — **not the file name**. Under the current `mkcapsule` logic:
- `capsules/hermes/init.4th` → does not match `production:*`, `core:*`, or `domains:*`
→ currently gets `FLAG_EXPERIMENT`
**All capsules carry both `FLAG_PRODUCTION` and `FLAG_EXPERIMENT`.** Birth eligibility is
not gated on the flag type — any capsule can be used for any identity. mkcapsule will be
updated to set both flags on every capsule it generates. The distinction between production
and experiment remains in the flags for audit/logging purposes but does not block BIRTH.
---
## 6. Capsule Content (Stubs)
Hermes and Artemis capsule directories exist but are empty (`.gitkeep` only). They need
`init.4th` stubs before the system can BIRTH them.
### `capsules/hermes/init.4th` (stub)
Should:
- Print `[Hermes] initializing` (or similar boot message)
- Define Hermes' pub/sub vocabulary (placeholder for now)
- Leave Hermes in a known state (not just empty)
### `capsules/artemis/init.4th` (stub)
Should:
- Print `[Artemis] initializing`
- Define Artemis' storage/memory vocabulary (placeholder)
**These stubs must be written before `make -f Makefile.starkernel` is run, because
mkcapsule runs at build time and produces `capsule_generated.c` from whatever is in
`capsules/` at that moment.**
---
## 7. VM Lifecycle Model
```
(capsule exists in directory)
│
▼
[BIRTH]
│
▼
EMBRYO ──────────────────[KILL]──────► DEAD
│ ▲
[START] │
│ [KILL]
▼ │
LIVE/RUNNING ──────────[STOP]──► STOPPED─┘
│
(completes)
│
▼
DEAD (natural)
```
States:
- **EMBRYO** — VM born (capsule executed) but `START` not yet called
- **LIVE** — executing (at this phase: running synchronously inside `START`)
- **STOPPED** — suspended; state saved; resumable via `START`
- **DEAD** — killed or naturally completed; slot reclaimable
---
## 8. Scheduling Model (Phase 1)
At this phase, scheduling is **sequential and explicit**:
- `START` runs a VM synchronously — the calling VM suspends while the started VM runs
- No round-robin, no time-slicing, no preemption
- The REPL is always in exactly one VM's context at a time
- `USE` switches the REPL/console target
**This is intentionally simple.** Cooperative concurrent scheduling (multiple VMs
interleaved in the interpreter loop) is a future milestone, designed separately after
the basic lifecycle works.
---
## 9. Dictionary and Namespace
The five lifecycle words (BIRTH, KILL, START, STOP, USE) are C primitives in **every VM's
base dictionary**. They are registered at VM initialization, before any capsule init runs.
A child VM born by Hermes or Artemis also has these words — all VMs can manage other VMs.
**Dictionary isolation:** Each VM has its own dictionary. Words defined by Artemis's
`init.4th` do not appear in Hera's dictionary or Hermes' dictionary. VMs communicate
through the messaging layer (Hermes), not shared dictionary state.
---
## 10. Hades Interface
All VMs access Hades through the same HAL interface. At this phase:
- No capability bitmask enforcement
- All VMs can call any HAL function (I/O, memory, time)
- The log prefix (`[VMName]`) is injected by the HAL console layer based on the currently
executing VM context — not by the VM itself
Capability enforcement (limiting what a child VM can do in Hades) is a future milestone.
---
## 11. Physics / SSM Integration
Each VM has its own independent SSM/L8 Jacquard state — this is already the natural
consequence of each VM being a full VM instance with its own `VM` struct. No shared physics.
The heartbeat runs per-VM. Hera's SSM does not observe Hermes' or Artemis' execution.
---
## 12. Decisions — RESOLVED
| # | Decision | Resolution |
|---|----------|-----------|
| D1 | BIRTH name collision | **Idempotent — check health of existing VM; if healthy, skip and log. No error, no replace.** |
| D2 | Production flag | **All capsules carry both `FLAG_PRODUCTION` and `FLAG_EXPERIMENT`. Birth eligibility is not gated on flag type. mkcapsule updated accordingly.** |
| D3 | `BIRTH` return value | **Nothing — BIRTH is a pure side-effect word, leaves stack clean.** |
| D4 | `USE` scope | **REPL only, but system-wide. Not bound to any specific capsule or identity. All REPL I/O redirects to the named VM.** |
| D5 | STOP across VMs | **Any VM may assume another identity for failsafe and recovery purposes, enabling cross-VM STOP. Identity assumption mechanism is a future milestone; Phase 1 allows any VM to STOP any other.** |
| D6 | Max named VMs | **Dynamic — Hera allocates as needed. No fixed ceiling.** |
| D7 | Log prefix format | **`[Name]`** — e.g., `[Hera]`, `[Hermes]`, `[Artemis]` |
---
## 13. What This Is NOT
- **Not a thread scheduler.** No preemption. Sequential execution at this phase.
- **Not a process model.** VMs share Hera's address space.
- **Not a security boundary yet.** No capability enforcement in this phase.
- **Not constrained by Hera's current implementation.** Hera expands as the design requires.
---
## 14. Implementation Phases
No code until Captain Bob approves the plan.
| Phase | Scope | Deliverable |
|-------|-------|-------------|
| 0 | Design approval | This document, signed off |
| 1 | VM registry naming | Add `name[64]` to `VMRegistryEntry`; name lookup functions |
| 2 | Log prefix | HAL console layer emits `[VMName]` prefix per active VM |
| 3 | Capsule stubs | Write `capsules/hermes/init.4th`, `capsules/artemis/init.4th` |
| 4 | mkcapsule flag update | Set both `FLAG_PRODUCTION` and `FLAG_EXPERIMENT` on all capsules |
| 5 | BIRTH word | C primitive; name→capsule lookup; health-check idempotence; full VM alloc |
| 6 | KILL word | C primitive; teardown; registry clear |
| 7 | START word | C primitive; synchronous execution hand-off |
| 8 | STOP word | C primitive; cross-VM; state save; return to caller |
| 9 | USE word | C primitive; system-wide REPL/console redirect |
| 10 | Hera init.4th update | Use new words to birth Hermes and Artemis at boot |
| 11 | Integration test | Boot sequence: Hera → BIRTH Hermes → BIRTH Artemis → `[Name]` in logs |
---
## 15. Mythological Framing Reference
| Name | Role |
|------|------|
| **Hades** | Kernel + HAL — the underworld substrate |
| **Hera** | Mama VM — queen, orchestrator, eternal; `capsules/init.4th` |
| **Hermes** | Messenger — pub/sub stratum; `capsules/hermes/init.4th` |
| **Artemis** | Hunter/keeper — storage and memory; `capsules/artemis/init.4th` |
| **BIRTH** | The word that brings a new VM into being |
| **KILL** | The word that destroys a VM |
| **START** | The word that gives a VM its first breath (or resumes it) |
| **STOP** | The word that suspends a VM |
| **USE** | The word that shifts the speaker's attention to a VM |
---
*This document is a planning artifact only. No code changes have been made.
Captain Bob's approval of the open decisions in §12 is required before Phase 1 begins.*
docs/working/archive/historical/2026-07-birthing/README.md
+25
View File
@@ -0,0 +1,25 @@
> **HISTORICAL (docs triage 2026-10-14).** The child-VM birthing plan and its acceptance record; built and live since (M7.1).
> Instead: docs/reference/M7.1.md and docs/fabric/. Moved here from `docs/birthing/README.md`.
# docs/working/archive/historical/2026-07-birthing/
Design and status for VM "birthing" — the mechanism by which Hera (the Tripod
governor) spawns Hermes and Artemis as child VMs at boot. See
`docs/working/archive/superseded/2026-08-tripod/TRIPOD.md` for the architectural contract this
work implements.
- **[PLAN.md](PLAN.md)** — Birthing plan for the initial three-VM constellation
(Hera/Hermes/Artemis). Marked DRAFT, requires Captain Bob approval before
implementation.
- **[STATUS.md](STATUS.md)** — Current work status: execution model (single
OS process, no threads/fork, VMs share one heap and address space, the C
call stack doubles as the execution stack), the birth-graph vs.
execution-stack distinction, and phase completion tracking.
- **[acceptance-logs/](acceptance-logs/)** — Three-architecture QEMU
acceptance test logs and summary for the lifecycle words (`BIRTH KILL
START STOP USE`).
## See also
- [`docs/working/archive/superseded/2026-08-tripod/README.md`](../03-architecture/tripod/README.md)
- [Project root](../../README.md)
docs/working/archive/historical/2026-07-birthing/STATUS.md
+307
View File
@@ -0,0 +1,307 @@
> **HISTORICAL (docs triage 2026-10-14).** The child-VM birthing plan and its acceptance record; built and live since (M7.1).
> Instead: docs/reference/M7.1.md and docs/fabric/. Moved here from `docs/birthing/STATUS.md`.
# Birthing — Work Status and Context
**Branch:** `birthing` (off `lithosananke`)
**Last updated:** 2026-05-27
**Current phase:** Phases 1–11 complete (kernel QEMU acceptance passed all three architectures)
---
## Execution Model — Single Process, VM Graph
**Hard constraints:**
- One OS process. No threads. No fork. No shared memory between processes.
- All VMs live in the same heap, same address space.
- The C call stack IS the execution stack.
**The graph:**
VMs are nodes. Edges are relationships — birth (who created whom) and execution (who
STARTed whom). These are distinct:
```
Birth graph (static, ownership): Execution stack (dynamic, runtime):
Hera C call stack:
/ \ hera_interp()
Hermes Artemis └─ vm_start("Hermes")
└─ hermes_interp()
└─ vm_start("Artemis")
└─ artemis_interp()
└─ STOP → unwind
```
**`START` = push.** Calling `S" Hermes" START` from Hera literally calls Hermes's
interpreter loop from within Hera's C execution context. Hera blocks in the `vm_start()`
call until Hermes returns.
**`STOP` = pop.** A VM stops itself — execution returns up the C call stack to whoever
called `vm_start()`. In Phase 1, STOP is always self-stop. Cross-VM STOP (via identity
assumption) is a future milestone.
**`BIRTH` does not equal `START`.** BIRTH allocates and initializes the VM. START runs it.
A VM can be birthed and left dormant.
**`USE` is orthogonal.** It redirects the REPL I/O independently of which VM is currently
executing. It does not affect the C call stack.
**Any VM can birth sub-VMs.** Hermes can call `S" Companion" BIRTH` — it is not Hera's
exclusive privilege. This makes the birth graph a true directed graph (rooted at Hera),
not a strict two-level tree.
---
## What This Branch Is
We are designing and implementing the multi-VM birth system for LithosAnanke. Hera (the
primary Forth VM) gains the ability to birth, name, start, stop, and kill peer VMs. All
VMs run inside the same process — no threads, no OS processes.
---
## The Three Named VMs
| VM | Role | Capsule |
|----|------|---------|
| **Hera** | Mama VM — orchestrates all others | `capsules/init.4th` (root, exclusive) |
| **Hermes** | Messaging stratum — pub/sub | `capsules/hermes/init.4th` |
| **Artemis** | Storage and memory manager | `capsules/artemis/init.4th` |
**Hades** = kernel + HAL substrate beneath all VMs.
---
## The Five Forth Primitives (C-level, in every VM's dictionary)
All words take a counted string `( c-addr u -- )` and leave the stack clean.
| Word | Behavior |
|------|----------|
| `BIRTH` | Birth a VM from `<name>:init.4th`. Idempotent — if healthy VM exists, skip + log. |
| `KILL` | Destroy a named VM unconditionally. Free resources, clear registry slot. |
| `START` | Begin or resume a named VM. Runs synchronously (Phase 1). |
| `STOP` | Suspend a named VM. Any VM can STOP any other. Saves execution state. |
| `USE` | System-wide REPL/console redirect to the named VM. Not capsule-scoped. |
Example usage:
```forth
S" Artemis" BIRTH
S" Artemis" START
S" Artemis" USE
S" Artemis" STOP
S" Artemis" KILL
```
---
## Resolved Decisions
| # | Decision | Resolution |
|---|----------|-----------|
| D1 | BIRTH name collision | Idempotent — healthy VM: skip and log. No error, no replace. |
| D2 | Capsule production flag | All capsules get both `FLAG_PRODUCTION` + `FLAG_EXPERIMENT`. Birth not gated on flag type. |
| D3 | BIRTH return value | Nothing — stack clean. Pure side-effect word. |
| D4 | USE scope | System-wide REPL redirect. Not bound to any capsule or identity. |
| D5 | STOP authority | Any VM can STOP any other. Identity assumption (future) adds nuance. |
| D6 | Max VMs | Dynamic — Hera allocates as needed. No fixed ceiling. |
| D7 | Log prefix format | `[Name]` — e.g., `[Hera]`, `[Hermes]`, `[Artemis]` |
---
## Capsule Namespace Convention
`mkcapsule` encodes relative paths by replacing `/` with `:`.
| File | Capsule name | Used by |
|------|-------------|---------|
| `capsules/init.4th` | `init.4th` | Hera only |
| `capsules/hermes/init.4th` | `hermes:init.4th` | Hermes |
| `capsules/artemis/init.4th` | `artemis:init.4th` | Artemis |
`BIRTH` maps `S" Artemis"` → lowercase → `artemis:init.4th`.
`capsules/hermes/` and `capsules/artemis/` currently exist but contain only `.gitkeep`.
Stub `init.4th` files must be written before the kernel build runs `mkcapsule`.
---
## Implementation Checklist
### Phase 0 — Design and Approval
- [x] `birthing` branch created off `lithosananke` HEAD
- [x] `docs/working/archive/historical/2026-07-birthing/PLAN.md` written
- [x] All seven decisions resolved with Captain Bob
- [x] `docs/working/archive/historical/2026-07-birthing/STATUS.md` created (this file)
- [x] **Captain Bob approves plan — gate for Phase 1**
---
### Phase 1 — VM Registry Naming
- [x] Add `char name[VM_NAME_MAX]` + `VM_STATE_STOPPED` to `VMRegistryEntry` (`include/starkernel/capsule_run.h`)
- [x] Replace static `vm_registry[64]` with dynamic kmalloc linked list (`src/starkernel/capsule/capsule_birth.c`)
- [x] Hera (VM 0) registered as `"Hera"` in `capsule_vm_registry_init()`
- [x] Add `capsule_vm_find_by_name()` — walks linked list, copies entry on match
- [x] Add `capsule_vm_registry_set_name()` — sets symbolic name by vm_id
- [x] Declare both new functions in `include/starkernel/capsule_birth.h`
- [x] Kernel build (`make -f Makefile.starkernel ARCH=amd64 STARFORTH_ENABLE_VM=1`) passes clean
- [ ] Tests: find by name, not-found returns -1, name persists across states
---
**Gap 1 (deferred to Phase 4):** `CAPSULE_MODE_VALID` enforces XOR between `FLAG_PRODUCTION`
and `FLAG_EXPERIMENT`. Captain Bob's answer about D2 points at a runtime logger/mode
selector (`LOG-DOE`) that does not yet exist. Design of `LOG-DOE` and the runtime mode
selection is a separate task before Phase 4 (mkcapsule flag update) can proceed.
---
### Phase 2 — Log Prefix `[Name]`
- [x] Identify the HAL console output path (`src/starkernel/hal/console.c`)
- [x] Add "active VM" context pointer to the HAL console layer
- [x] Every `hal_console_putchar` / `hal_console_puts` prefixes `[VMName] ` at line start
- [x] Hera's boot output shows `[Hera]` from first character
- [ ] QEMU verification pending Captain Bob
---
### Phase 3 — Capsule Stubs
- [x] Write `capsules/hermes/init.4th` (boot message + placeholder vocabulary)
- [x] Write `capsules/artemis/init.4th` (boot message + placeholder vocabulary)
- [ ] Verify `mkcapsule` generates descriptors for both (requires kernel build env)
- [ ] Verify capsule names resolve correctly (`hermes:init.4th`, `artemis:init.4th`)
---
### Phase 4 — mkcapsule Flag Update
- [ ] Update `tools/mkcapsule.c` to set both `FLAG_PRODUCTION` and `FLAG_EXPERIMENT` on
every capsule (remove the directory-prefix-based flag logic, or extend it to set both)
- [ ] Rebuild and verify `capsule_generated.c` shows both flags on all entries
- [ ] Confirm birth eligibility check in `capsule_birth.c` no longer blocks on flag type
---
### Phase 5 — `BIRTH` Word
- [x] Register `BIRTH` as a C primitive in `register_mama_forth_words()` (mama_forth_words.c)
- [x] Lowercase the name string from the stack
- [x] Construct capsule name: `<name>:init.4th` (special case: `hera` → rejected)
- [x] Look up capsule in directory by name
- [x] Health check: if VM with that name is alive and healthy, log and return (idempotent)
- [x] Allocate new VM (dynamic — no fixed pool ceiling)
- [x] Execute capsule init via `capsule_birth_baby` path
- [x] Register VM under symbolic name in `VMRegistryEntry`
- [x] Emit parity log via `capsule_parity_log_birth`
- [x] Stack clean on exit
- [x] Fixed caddr+1 offset bug — now reads chars directly at caddr per FORTH-79 S"
---
### Phase 6 — `KILL` Word
- [x] Register `KILL` as a C primitive
- [x] Look up VM by name (case-insensitive)
- [x] If not found: log, return (no stack effect)
- [x] Tear down VM, free via `vm_cleanup` + `sf_free`
- [x] Clear registry slot (state → DEAD, name cleared)
- [x] Emit parity log via `capsule_parity_log_kill`
- [x] Stack clean on exit
- [x] Hera kill rejected
---
### Phase 7 — `START` Word
- [x] Register `START` as a C primitive
- [x] Look up VM by name
- [x] If EMBRYO or STOPPED: set state → LIVE
- [x] Call `sk_repl_run(target)` — blocks until target->halted
- [x] On target halt: state → STOPPED, caller resumes
- [x] If target already LIVE: log error, return clean
- [x] Stack clean on exit
---
### Phase 8 — `STOP` Word
- [x] Register `STOP` as a C primitive
- [x] Self-stop: sets `vm->halted = 1`, sk_repl_run loop exits
- [x] State updated to STOPPED by START word after REPL returns
- [x] Stack clean (STOP takes no arguments)
---
### Phase 9 — `USE` Word
- [x] Register `USE` as a C primitive
- [x] Look up VM by name
- [x] Set active system-wide REPL VM via `sk_repl_set_active_vm`
- [x] HAL console prefix changes to `[VMName]` via `console_set_vm_name`
- [x] Stack clean on exit
---
### Phase 10 — Hera `init.4th` Update
- [x] `S" Hermes" BIRTH` in `capsules/init.4th`
- [x] `S" Artemis" BIRTH` in `capsules/init.4th`
- [x] S" fixed to FORTH-79 ( -- c-addr u ) — stores at HERE, no count-byte prefix
- [x] Boot sequence QEMU verified — all three architectures (amd64, aarch64, riscv64)
---
### Phase 11 — Integration Test (kernel QEMU)
- [x] Boot QEMU: kernel loads, Hera starts, births Hermes and Artemis — all three architectures
- [x] Serial log shows `[Hera]`, `[Hermes]`, `[Artemis]` prefixes
- [x] `."` compile-mode fixed (do-string inline approach) — "Hermes Up" / "Artimis Up" print correctly
- [x] 734 passed / 0 failed / 0 errors on amd64, aarch64, riscv64
- [ ] `S" Hermes" USE` switches console to Hermes — interactive test (requires Captain Bob)
- [ ] `S" Hera" USE` switches back — interactive test (requires Captain Bob)
- [ ] KILL + re-BIRTH a VM in the same session — interactive test (requires Captain Bob)
**Logs:** `docs/working/archive/historical/2026-07-birthing/acceptance-logs/` — kernel QEMU serial logs for all three architectures
---
## Key Files
| File | Purpose |
|------|---------|
| `docs/working/archive/historical/2026-07-birthing/PLAN.md` | Full architectural plan |
| `docs/working/archive/historical/2026-07-birthing/STATUS.md` | This file — living checklist |
| `tools/mkcapsule.c` | Build-time capsule packager |
| `capsules/init.4th` | Hera's init (root, exclusive) |
| `capsules/hermes/init.4th` | Hermes stub (to be written) |
| `capsules/artemis/init.4th` | Artemis stub (to be written) |
| `src/starkernel/capsule/capsule_birth.c` | C-level VM birth logic |
| `src/starkernel/capsule/capsule_loader.c` | Capsule load + exec |
| `src/starkernel/capsule/capsule_registry.c` | Capsule directory |
| `src/starkernel/hal/console.c` | HAL console — add `[Name]` prefix here |
| `include/starkernel/capsule.h` | `CapsuleDesc`, `VMRegistryEntry` structs |
| `include/starkernel/capsule_birth.h` | Birth protocol headers |
---
## Branch and Commit History
```
birthing (HEAD)
d2766d6 plan: lock all seven decisions into birthing plan
ba27226 plan: overhaul birthing plan — named VMs, five Forth primitives, capsule namespacing
d258b12 plan: update birthing plan — Hera is not constrained
84e1c61 plan: add birthing plan for child VMs born of Hera
91e0b3d (lithosananke HEAD at branch point) docs: comprehensive CLAUDE.md update
```
---
## Notes and Open Threads
- **Identity assumption (D5 future):** Cross-VM STOP requires a VM to assume another's
identity. Not in scope for current phases — design separately. Phase 1 STOP is self-stop.
- **Cooperative scheduling / YIELD:** The single-process graph model means VMs run one at a
time (depth-first down the call stack). True cooperative interleaving (Hermes and Artemis
sharing time while Hera waits) would require an explicit `YIELD` mechanism and a trampoline
scheduler — future milestone, designed separately.
- **Graph depth — CONFIRMED unlimited:** Any VM can BIRTH sub-VMs at any depth. The birth
graph is a true rooted directed graph. Deep graphs consume C stack space — note for kernel
stack budgeting. No enforced limit; document safe depth in practice.
- **Hermes pub/sub and Artemis storage:** Their full vocabularies are future work. Current
phase only needs stub `init.4th` files.
- **`capsules/hermes/` and `capsules/artemis/`** currently contain only `.gitkeep`. Stub
`init.4th` files must exist before `make -f Makefile.starkernel` calls `mkcapsule`.
docs/working/archive/historical/2026-07-birthing/acceptance-logs/README.md
+22
View File
@@ -0,0 +1,22 @@
> **HISTORICAL (docs triage 2026-10-14).** The child-VM birthing plan and its acceptance record; built and live since (M7.1).
> Instead: docs/reference/M7.1.md and docs/fabric/. Moved here from `docs/birthing/acceptance-logs/README.md`.
# docs/working/archive/historical/2026-07-birthing/acceptance-logs/
Kernel QEMU acceptance test results for the VM lifecycle words (`BIRTH KILL
START STOP USE`) and the `."` compile-mode fix, run across all three
supported architectures.
- **[SUMMARY.md](SUMMARY.md)** — Consolidated results table (800 tests run,
734 passed, 0 failed/errored on amd64/aarch64/riscv64), key serial-log
evidence, list of source files changed, and the remaining interactive
tests that require Captain Bob (`S" Hermes" USE`, KILL + re-BIRTH in the
same session).
- **[amd64.log](amd64.log)** — Raw QEMU serial log, amd64.
- **[arm64.log](arm64.log)** — Raw QEMU serial log, aarch64.
- **[riscv64.log](riscv64.log)** — Raw QEMU serial log, riscv64.
## See also
- [`docs/working/archive/historical/2026-07-birthing/README.md`](../README.md)
- [Project root](../../../README.md)
docs/working/archive/historical/2026-07-birthing/acceptance-logs/SUMMARY.md
+66
View File
@@ -0,0 +1,66 @@
> **HISTORICAL (docs triage 2026-10-14).** The child-VM birthing plan and its acceptance record; built and live since (M7.1).
> Instead: docs/reference/M7.1.md and docs/fabric/. Moved here from `docs/birthing/acceptance-logs/SUMMARY.md`.
# Birthing — Kernel QEMU Acceptance Test Summary
**Date:** 2026-05-27
**Branch:** `birthing`
**Feature:** Lifecycle words (BIRTH KILL START STOP USE) + `."` compile-mode fix
## Scope
These logs cover the **kernel UEFI build** (`make -f Makefile.starkernel ARCH=<arch> clean qemu`).
All three architectures booted to `ok>`, ran POST, and executed `capsules/init.4th` successfully.
## Kernel QEMU Results
| Arch | Tests | Passed | Failed | Errors | BIRTH Hermes | BIRTH Artemis | `."` |
|------|-------|--------|--------|--------|-------------|---------------|------|
| amd64 | 800 | 734 | 0 | 0 | `[Hermes] Hermes Up` / live | `[Artemis] Artimis Up` / live | PASS |
| aarch64 | 800 | 734 | 0 | 0 | `[Hermes] Hermes Up` / live | `[Artemis] Artimis Up` / live | PASS |
| riscv64 | 800 | 734 | 0 | 0 | `[Hermes] Hermes Up` / live | `[Artemis] Artimis Up` / live | PASS |
## Key Evidence (all three architectures identical)
```
[Hera] POST: PASSED
[Hera] Init: loading init.4th...
[Hermes] Hermes UpBIRTH: Hermes live
[Artemis] Artimis UpBIRTH: Artemis live
[Hera] Init: init.4th OK
```
`."` in compile mode now uses the `(do-string)` inline runtime — string bytes live
inside the threaded code body, not at `body_addr`. The previous LIT-addr-LIT-len-TYPE
approach overwrote the start of the word body with string bytes, causing a triple fault
when `execute_colon_word` read them as DictEntry pointers.
## What Changed
- `src/word_source/io_words.c` — `."` compile mode replaced with `(do-string)` inline
approach; `(do-string)` runtime word reads inline string via return-stack IP, prints,
advances IP past padded block
- `src/starkernel/capsule/mama_forth_words.c` — BIRTH KILL START STOP USE kernel
implementations; fixed caddr+1 offset bug to align with FORTH-79 S"
- `src/word_source/string_words.c` — S" fixed to FORTH-79 standard ( -- c-addr u )
- `src/word_source/lifecycle_words_hosted.c` — hosted Linux stubs (guarded #ifndef __STARKERNEL__)
- `src/main.c` — register_lifecycle_words(&vm) wired in for hosted build
- `capsules/init.4th` — S" Hermes" BIRTH + S" Artemis" BIRTH
- `capsules/hermes/init.4th` — stub: `: UP ." Hermes Up" ; UP`
- `capsules/artemis/init.4th` — stub: `: UP ." Artimis Up" ; UP`
## Serial Logs
| Arch | Log file |
|------|----------|
| amd64 | `logs/qemu-amd64-20260526-234957.log` |
| aarch64 | `logs/qemu-aarch64-20260526-235056.log` |
| riscv64 | `logs/qemu-riscv64-20260527-000127.log` |
(`logs/` is gitignored — run locally to reproduce)
## Remaining Interactive Tests (require Captain Bob)
- `S" Hermes" USE` — switches system console to Hermes
- `S" Hera" USE` — switches back
- KILL + re-BIRTH a VM in the same session
docs/working/archive/historical/2026-07-birthing/acceptance-logs/amd64.log
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2dbe1a4abd9342a0ec96fbd4c4c15ab039385b2af9639c7489eda6fe9dfc6fa4
size 10216
docs/working/archive/historical/2026-07-birthing/acceptance-logs/arm64.log
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:25dc9dcce2403f30ef6b2c0d14e5462d7e17e42070c1afa443f92cf85985cd99
size 9889
docs/working/archive/historical/2026-07-birthing/acceptance-logs/riscv64.log
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2af00462812d21a44807da17be38ad5fb5e97ca56b4c2b883cf192b7ef1778db
size 9885
docs/working/archive/historical/2026-08-riscv64-stack-fix/README.md
+186
View File
@@ -0,0 +1,186 @@
> **HISTORICAL (docs triage 2026-10-14).** The write-up of one fix (kernel stack trampoline, riscv64), done.
> Instead: the code itself (kernel/src/arch/riscv64/). Moved here from `docs/lithosananke/riscv64-stack-fix/README.md`.
# Kernel Stack Trampoline Fix — riscv64
**Date**: 2026-08-02
**Branch**: `master`
**Commit**: `7366275`
---
## Problem
The riscv64 kernel crashed partway through VM bootstrap during the 2026-08-01
DoE campaign run, well after amd64 and aarch64 had both booted cleanly to
`ok>` in the same session. The crash was a wild jump: execution landed at an
address nowhere inside the kernel's own linked image, then almost immediately
faulted trying to load from a small, near-null address.
```
Registering FORTH-79 Standard word set...
Registering FORTH-79 arithmetic words...
FORTH-79 Standard word set registration complete
*** EXCEPTION (riscv64) ***
scause = 0x000000000000000d
sepc = 0x00000000be27c3c2
stval = 0x00000000000074d8
```
`scause = 0xd` is a supervisor-mode load page fault. `sepc` (~0xbe27c3c2) sat
near the top of QEMU's 1 GB RAM window — nowhere close to the kernel's own
`.text`/`.data`/`.bss` footprint (`0x400000`–`0xcdf9a0`, confirmed via `nm`/
`readelf` on the exact build that produced the log). `stval` (0x74d8, ~30 KB)
stayed fixed across rebuilds even as `sepc` drifted — the signature of a
corrupted return address rather than a fixed logic bug at one source line.
## Investigation
Two dead ends were ruled out before the real cause surfaced:
1. **ELF relocation / buffer overlap in the split-build loader.** The
codebase has a `!MONOLITHIC_BUILD` path (`elf_load_kernel()` in
`elf_loader.c`) that loads `kernel.elf` from the ESP into a UEFI
`AllocateAnyPages` buffer. This looked like a plausible overlap candidate
at first, but `Makefile.starkernel` defaults to `MONOLITHIC=1`, meaning
`kernel_main` is linked directly into the loader and that code path never
runs. Confirmed by checking the built binary for the investigation's own
diagnostic strings — absent, because the compilation unit was built with
`-DMONOLITHIC_BUILD` and the code was preprocessed out.
2. **`RAW_LOG()` silently no-ops on every non-amd64 arch.** All of
`load_kernel_from_esp()`'s narration (and an early ConOut-based
diagnostic) never reached the captured serial log, on riscv64, in any
build. `RAW_LOG` was defined only for `ARCH_AMD64`; everywhere else it
expanded to `((void)0)`. This made straightforward diagnosis impossible
until a real raw-UART writer was added for riscv64 (see Secondary Fix).
With real output available, `console_println()` checkpoints were bisected
through `vm_init_with_host()` and `sk_vm_bootstrap_parity()`. Every checkpoint
inside `vm_init_with_host()` printed successfully, including the very last
line of the function — but the checkpoint immediately after its *return*
never appeared. Re-running with a trivial, unrelated local-variable addition
shifted the crash earlier and changed the exception type entirely (from a
load fault to an instruction-fetch fault at `sepc == stval`, i.e. the CPU
tried to *execute* an unmapped address). A crash whose location and even
category shift under unrelated code changes is the standard signature of
stack corruption, not a fixed bug at a specific call site.
## Root Cause
`kernel_main.c` documented the gap directly, once read closely:
> On amd64, `kernel_entry.S` switches the stack from UEFI's default to a
> 2 MiB zero-initialised BSS stack and tail-calls this function as
> `kernel_main_impl`. On aarch64 and riscv64 the assembly trampoline is not
> yet implemented and the UEFI loader calls `kernel_main` directly.
amd64 has carried a dedicated stack-switch trampoline
(`src/starkernel/arch/amd64/kernel_entry.S`) since early on, with its own
comment explaining why: *"UEFI hands control to kernel_main() running on a
small UEFI-provided stack (typically 128 KB–256 KB). The FORTH interpreter +
DOE experiment loop can easily exceed that depth."* riscv64 never had the
equivalent. It ran the entire VM bootstrap — 27 chained word-registration
modules, physics/SSM init, Tripod capsule birth — directly on whatever small
stack EDK2's riscv64 firmware handed the loader, overflowing it and
corrupting a return address. aarch64 has the same theoretical exposure but
happens not to trip it, apparently because its firmware's default boot stack
is larger — incidental, not a guarantee.
## Fix
**`src/starkernel/arch/riscv64/kernel_entry.S`** (new) — RISC-V stack-switch
trampoline mirroring amd64's:
```asm
.section .bss
.align 4
.global g_kernel_stack
g_kernel_stack:
.space 0x200000
g_kernel_stack_top:
.global g_kernel_stack_top
.section .text
.extern kernel_main_impl
.global kernel_main
kernel_main:
ld t0, BOOT_INFO_KERNEL_STACK_BASE_OFFSET(a0)
bnez t0, .Ldynamic_stack
la t0, g_kernel_stack_top
j .Lstack_ready
.Ldynamic_stack:
ld t1, BOOT_INFO_KERNEL_STACK_SIZE_OFFSET(a0)
add t0, t0, t1
.Lstack_ready:
andi t0, t0, -16
mv sp, t0
mv s0, zero
tail kernel_main_impl
```
**`src/starkernel/kernel_main.c`** — riscv64 now builds `kernel_main_impl`
(invoked via the trampoline) instead of `kernel_main` directly, same pattern
as amd64:
```c
#if defined(__x86_64__) || defined(__riscv)
void kernel_main_impl(BootInfo *boot_info) {
#else
void kernel_main(BootInfo *boot_info) {
#endif
```
**`Makefile.starkernel`** — wires the new file into the riscv64 loader build:
```makefile
ifeq ($(ARCH),riscv64)
LOADER_ASM += $(KERNEL_SRC)/arch/$(ARCH)/kernel_entry.S
endif
```
## Secondary Fix
`RAW_LOG()` in `src/starkernel/boot/uefi_loader.c` was a no-op on every arch
except amd64, silencing all of the loader's own diagnostic messages on
riscv64 (and aarch64). Added a real raw-UART writer for riscv64, targeting
QEMU virt's 16550-compatible `uart8250` at MMIO `0x10000000` (matches
`Domain0 Region03` in the OpenSBI boot banner) — the same byte-polling
protocol as the existing amd64 port-I/O version, just MMIO instead of `outb`/
`inb`. Existing and future `RAW_LOG()` call sites in the loader now actually
produce output on riscv64.
## Result
| Criterion | Before | After |
|---|---|---|
| `sepc` after word registration | Wild address outside kernel image (`0xbe27c3c2`-ish, drifting per build) | N/A — no longer occurs |
| Exception | Load or instruction-fetch page fault, non-deterministic between builds | None |
| `[Hera] ok>` (riscv64) | Never reached | ✅ reached, Tripod fleet (Hera/Hermes/Artemis) runs to completion |
| `PASS: E2E msg flow` | Never logged | ✅ |
| `PARITY:MAMA_INIT` / `Init: Mama birth OK` | Never logged | ✅ |
| Loader `RAW_LOG()` output on riscv64 | Silent no-op | ✅ visible in serial log |
## Acceptance Testing
All three architectures must boot to `ok>`, run the Tripod fleet, and reach
`[Hera] ok>`. Per repo convention, run in this exact order, one QEMU instance
at a time, always `clean` before `qemu`:
```bash
make -f Makefile.starkernel ARCH=amd64 clean qemu
make -f Makefile.starkernel ARCH=aarch64 clean qemu
make -f Makefile.starkernel ARCH=riscv64 clean qemu
```
Verify each log:
```bash
grep "ok>" logs/<session>/<arch>/qemu-<arch>-*.log
grep "PASS: E2E msg flow" logs/<session>/<arch>/qemu-<arch>-*.log
grep "PARITY:MAMA_INIT" logs/<session>/<arch>/qemu-<arch>-*.log
grep "EXCEPTION" logs/<session>/<arch>/qemu-<arch>-*.log # must be empty
```
All three architectures confirmed clean on 2026-08-02 (see
`logs/20260802-064050/riscv64/`, `logs/20260802-064424/amd64/`,
`logs/20260802-064656/aarch64/`, committed alongside this fix).
docs/working/archive/historical/2026-10-lithosananke-orientation.md
+198
View File
@@ -0,0 +1,198 @@
> **HISTORICAL (docs triage 2026-10-14).** An orientation page for the old docs/lithosananke/ folder, flagged partly stale on 2026-10-09; the folder is gone (its parts are in docs/reference/ and this archive).
> Instead: docs/README.md. Moved here from `docs/lithosananke/README.md`.
# LithosAnanke — StarForth Kernel Branch
> **⚠ PARTIALLY STALE (flagged 2026-10-09).** This page predates the repo split and the
> Tripod/kernel reshuffle. There is no `lithosananke` branch in this repo — `master` is the
> sole production line — and the VM fleet is now **Hera / Artemis / Hestia** (the Hermes VM
> was removed; messaging moved into the kernel as kernel-Hermes). M7.1 is **live**, not
> merely designed. For authoritative status see `.claude/CLAUDE.md` and
> `docs/fabric/FABRIC-2.md` / `FABRIC-3.md` / `FABRIC-3.5.md` / `FABRIC-3.6.md`; for the
> milestone plan see `ROADMAP.md` (this file is an orientation page).
**Line:** `master` (sole production line post-split)
**Status:** M7 Complete (VM Parity) · M7.1 Live (capsules, Tripod, Stadium, word-level ACL Phases 1–7)
---
## What is LithosAnanke?
LithosAnanke is the kernel branch of StarForth — a UEFI-bootable microkernel with the StarForth VM as its native execution engine.
**Etymology:** *Lithos* (stone/foundation) + *Ananke* (necessity/inevitability)
The name reflects the project's nature: a necessary foundation for StarshipOS.
---
## Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ UEFI Firmware │
│ • ExitBootServices → handoff to kernel │
└─────────────────────┬───────────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ LithosAnanke Boot (M0) │
│ • UEFI loader collects BootInfo │
│ • Memory map, ACPI tables, framebuffer │
└─────────────────────┬───────────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ LithosAnanke HAL (M1-M6) │
│ • PMM: Physical Memory Manager (bitmap) │
│ • VMM: Virtual Memory Manager (4-level paging) │
│ • IDT: Interrupt Descriptor Table │
│ • APIC: Timer + IRQ routing │
│ • Heap: kmalloc/kfree │
│ • Console: Serial UART │
└─────────────────────┬───────────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ StarForth VM (M7+) │
│ • 295 FORTH-79 words │
│ • Parity validation (hash=0x684bbf2fa1d96d55) │
│ • Physics-driven adaptive runtime │
│ • Init capsule system (M7.1) │
└─────────────────────────────────────────────────────────────┘
```
---
## Current Milestones
| Milestone | Description | Status |
|-----------|-------------|--------|
| M0 | UEFI Boot + Serial Output | Complete |
| M1 | Physical Memory Manager | Complete |
| M2 | Virtual Memory Manager | Complete |
| M3 | IDT + Interrupts | Complete |
| M4 | APIC Timer | Complete |
| M5 | Timer Calibration | Complete |
| M6 | Kernel Heap | Complete |
| **M7** | **VM Parity Validation** | **Complete** |
| **M7.1** | **Capsules · Tripod · Stadium · ACL** | **Live** (see FABRIC docs) |
| M8 | REPL (keyboard input) | Superseded — see FABRIC-0 §25.5 item 4.4 |
| M9 | Block Storage / Block I/O | Live (virtio-blk + block subsystem) |
---
## Build & Run
```bash
# Build kernel with VM integration (loader + kernel)
make -f kernel/Makefile ARCH=amd64 STARFORTH_ENABLE_VM=1
# Run in QEMU
make -f kernel/Makefile ARCH=amd64 STARFORTH_ENABLE_VM=1 qemu
# Other architectures
make -f kernel/Makefile ARCH=aarch64 qemu
make -f kernel/Makefile ARCH=riscv64 qemu
```
**Output artifacts:**
- `build/amd64/kernel/starkernel_loader.efi` — UEFI loader
- `build/amd64/kernel/starkernel_kernel.elf` — Kernel binary
---
## Documentation
### Core Design
| Document | Description |
|----------|-------------|
| [M7.1.md](M7.1.md) | Init Capsule Architecture |
| [ROADMAP.md](ROADMAP.md) | Milestone roadmap |
| [amd64-isr-fix/README.md](amd64-isr-fix/README.md) | APIC timer ISR fix + kernel 3-arch acceptance test |
| [hosted-acceptance-test/README.md](hosted-acceptance-test/README.md) | Hosted VM 3-arch acceptance test (amd64 native, aarch64/riscv64 under QEMU user-mode) |
### HAL (Hardware Abstraction Layer)
| Document | Description |
|----------|-------------|
| [hal/overview.md](hal/overview.md) | HAL architecture and principles |
| [hal/interfaces.md](hal/interfaces.md) | HAL interface specifications |
| [hal/lithosananke-integration.md](hal/lithosananke-integration.md) | Kernel-specific HAL implementation |
| [hal/platform-implementations.md](hal/platform-implementations.md) | Platform implementation guide |
| [hal/migration-plan.md](hal/migration-plan.md) | Migration from hosted to kernel |
---
## Key Concepts
### One Truth Per VM
Every VM has exactly one birth capsule — its identity and provenance.
```
PARITY:BIRTH vm_id=42 capsule_id=0xA13F mode=p capsule_hash=0x... dict_hash=0x...
```
### DOMAIN vs PERSONALITY
**DOMAIN** is a construction concept that exists **only in Mama FORTH**. It's how Mama classifies, composes, and selects capsules. DOMAIN is never visible to baby VMs.
**PERSONALITY** is the result of executing a production `(p)` INIT capsule. It's what the baby VM *is* — its vocabulary, invariants, capabilities. PERSONALITY is immutable and defines the VM's entire worldview.
### Production (p) vs Experiment (e)
- **(p) capsules** are truth-bearing — they can birth VMs
- **(e) capsules** are Mama-only workloads — experiments that never touch babies
A baby VM never sees, executes, or knows about (e) capsules.
### Content-Addressed Capsules
Capsules are identified by content hash, not names. Names lie. Hashes don't.
### Mama FORTH
The root VM that holds all truths, manages child VMs, and enforces the birth protocol. Mama:
- Maintains the capsule directory
- Selects (p) capsules for birth
- Executes (e) capsules for experimentation
- Logs all births and runs for provenance
---
## Directory Structure
```
src/starkernel/
├── boot/ # UEFI loader, ELF loader
├── hal/ # Console, host services
├── memory/ # PMM, VMM, kmalloc
├── arch/amd64/ # x86_64 specific (APIC, IDT, timer)
├── vm/ # VM integration
│ ├── arena.c # VM memory arena
│ ├── parity.c # Parity validation
│ ├── bootstrap/ # sk_vm_bootstrap
│ └── host/ # Host services shim
└── kernel_main.c # Entry point
```
---
## Contributing
The contributing rules are:
1. Maintain parity validation (M7 baseline)
2. Pass the three-arch QEMU boot test (`ARCH={amd64,aarch64,riscv64}`)
3. Not break the hosted VM build (plain `make` at the repo root)
---
## References
- [StarForth README](../../README.md) — Main project
- [FORTH-79 Standard](http://www.forth.org/forth-79.html)
- [OSDev Wiki](https://wiki.osdev.org/)
- [UEFI Specification](https://uefi.org/specifications)
docs/working/archive/session-logs/2026-07-24-punch-list.md
+2 -2
View File
@@ -47,7 +47,7 @@ that actually matters).
## 2. Fix the broken riscv64 command in this branch's acceptance-test doc
`docs/lithosananke/hosted-acceptance-test/README.md` still documents the
`docs/reference/hosted-acceptance-test/README.md` still documents the
riscv64 leg using `riscv64-linux-gnu-gcc`, which cannot build this tree
(`nanosleep` visibility failure under `-std=c99`, unrelated to any project
change). This was already fixed in `master`'s `.claude/CLAUDE.md` (the
@@ -152,6 +152,6 @@ now — just flagging.
open item if anyone wants to chase it down.
- Today's aarch64/riscv64 `.hidden` ISR hardening (commit `5fe188bd`) only
has a CHANGELOG entry, not a dedicated writeup the way
`docs/lithosananke/amd64-isr-fix/README.md` documents the original amd64
`docs/working/archive/historical/2026-07-amd64-isr-fix/README.md` documents the original amd64
fix. Not urgent — the CHANGELOG entry covers it — but noting the
asymmetry in case someone later wants doc parity.
docs/working/archive/superseded/2025-12-14-starforth-roadmap.md
+666
View File
@@ -0,0 +1,666 @@
> **SUPERSEDED (docs triage 2026-10-14).** The StarForth-era VM -> kernel -> OS -> FPGA plan of 2025-12-14, flagged obsolete 2026-08-26.
> Instead: docs/reference/ROADMAP.md (the LithosAnanke roadmap and versioning policy). Moved here from `docs/ROADMAP.md`.
# StarForth Roadmap: VM → Kernel → OS → FPGA
> **OBSOLETE (flagged 2026-08-26, Captain Bob's call — this is more than "stale," the
> architecture, branch topology, and terminology it describes no longer exist).** This is a
> StarForth-era plan dated 2025-12-14, written before the repo split, before
> Tripod/Stadium/word-level ACL existed, and before "Phase 1 Starting (HAL)" had any of the
> meaning it has now. Kept for historical record only. For current status, see
> `docs/reference/ROADMAP.md` (repo-specific, LithosAnanke roadmap) and
> `FABRIC-0.md`/`FABRIC-1.md`/`FABRIC-2.md` (design history and current work) — not this
> document.
**Version**: 1.0
**Date**: 2025-12-14
**Timeline**: 2025-2028 (3.5 years)
**Status**: Phase 0 Complete (VM), Phase 1 Starting (HAL)
---
## VISION
**From**: Adaptive FORTH VM running on hosted OS (Linux)
**To**: Self-hosting operating system running on custom FPGA hardware
```
2025 (NOW) 2026 2027 2028
│ │ │ │
V V V V
┌─────────┐ ┌──────────┐ ┌────────────┐ ┌──────────┐
│StarForth│ → │StarKernel│ → │ StarshipOS │ → │ FPGA │
│ VM │ │(Bare Metal) │(Self-hosting) │ Hardware │
└─────────┘ └──────────┘ └────────────┘ └──────────┘
COMPLETE 6-8 months 12-18 months 12-18 months
```
**Key Principle**: Each phase builds on the previous. No skipping steps.
---
## CURRENT STATE (Phase 0: COMPLETE ✅)
### What We Have
**StarForth VM** - Production-ready adaptive FORTH interpreter:
- ✅ FORTH-79 compliant (780+ tests passing)
- ✅ Adaptive runtime (7 feedback loops, 0% algorithmic variance)
- ✅ Platforms: Linux (HISTORICAL: also L4Re/Fiasco.OC through mid-2026; removed as an active target)
- ✅ Performance: 100M iterations/sec, profile-guided optimization
- ✅ Quality: 9.2/10 codebase rating, strict ANSI C99
- ✅ Validation: Peer-reviewed, 90-run statistical proof
- ✅ Patent pending: Adaptive runtime mechanisms
**Documentation**:
- ✅ 121-page academic paper
- ✅ Scientific defense documents (9 files)
- ✅ Ontology & formal definitions
- ✅ HAL architecture (documented, not implemented)
**Infrastructure**:
- ✅ Make-based build system (40+ targets)
- ✅ DoE (Design of Experiments) framework
- ✅ Reproducibility guarantees
- ✅ CI-ready test suite
### What's Missing
**See**: [docs/GAP_ANALYSIS.md](docs/GAP_ANALYSIS.md) for complete list (37 gaps identified)
**Critical Blockers**:
1. HAL abstraction layer (8 components) - **Blocks everything**
2. StarKernel implementation (13 components)
3. StarshipOS subsystems (12 components)
4. FPGA feasibility & design (19 components)
---
## PHASE 1: HAL IMPLEMENTATION (Months 1-2)
**Goal**: Make StarForth platform-agnostic via Hardware Abstraction Layer
**Why**: HAL is the foundation for all future work. Without HAL, cannot build kernel or port to FPGA.
### Deliverables
| # | Component | LOC | Duration | Status |
|---|-----------|-----|----------|--------|
| 1.1 | Define HAL interfaces (6 headers) | 500 | 1 week | 📋 Planned |
| 1.2 | Implement Linux HAL | 1500 | 2 weeks | 📋 Planned |
| 1.3 | Migrate VM core to HAL | 800 | 2 weeks | 📋 Planned |
| 1.4 | Migrate physics subsystems | 300 | 1 week | 📋 Planned |
| 1.5 | Validate determinism (DoE) | Testing | 2 days | 📋 Planned |
**Total**: ~3,100 LOC, 6-7 weeks
### Success Criteria
- ✅ All HAL interface headers exist (`hal_time.h`, `hal_interrupt.h`, etc.)
- ✅ Linux HAL implementation complete
- ✅ VM core has zero direct platform dependencies
- ✅ All 780+ tests pass on HAL
- ✅ 0% algorithmic variance maintained
- ✅ No measurable performance regression
### Milestones
**Week 2**: HAL headers defined, Linux implementation started
**Week 4**: Linux HAL complete, VM migration started
**Week 6**: All tests passing, determinism validated
**Blocker Risk**: Low (well-understood refactoring)
---
## PHASE 2: STARKERNEL (Months 3-5)
**Goal**: Boot StarForth on bare metal (UEFI → kernel → "ok" prompt)
**Why**: Kernel enables true OS development and demonstrates platform independence.
### Architecture
```
┌─────────────────────────────────────────────┐
│ UEFI Firmware │
│ • Memory map, ACPI, framebuffer │
└────────────────┬────────────────────────────┘
│ ExitBootServices()
▼
┌─────────────────────────────────────────────┐
│ StarKernel Boot (boot/uefi_loader.c) │
│ • Collect BootInfo │
│ • Jump to kernel_main() │
└────────────────┬────────────────────────────┘
▼
┌─────────────────────────────────────────────┐
│ StarKernel HAL (platform/kernel/) │
│ • CPU init (GDT, IDT, interrupts) │
│ • Memory (PMM, VMM, heap) │
│ • Time (TSC, HPET, APIC timer) │
│ • Console (UART + framebuffer) │
└────────────────┬────────────────────────────┘
▼
┌─────────────────────────────────────────────┐
│ StarForth VM │
│ • vm_create() │
│ • Physics subsystems │
│ • REPL → "ok" prompt │
└─────────────────────────────────────────────┘
```
### Deliverables
| # | Component | LOC | Duration | Status |
|---|-----------|-----|----------|--------|
| 2.1 | UEFI boot loader | 800 | 2 weeks | 📋 Planned |
| 2.2 | Physical memory manager (PMM) | 1200 | 2 weeks | 📋 Planned |
| 2.3 | Virtual memory manager (VMM) | 1500 | 2 weeks | 📋 Planned |
| 2.4 | Kernel heap (kmalloc) | 800 | 1 week | 📋 Planned |
| 2.5 | CPU initialization (GDT/IDT) | 1000 | 1 week | 📋 Planned |
| 2.6 | UART driver (serial console) | 600 | 1 week | 📋 Planned |
| 2.7 | Time subsystem (TSC/HPET) | 1000 | 2 weeks | 📋 Planned |
| 2.8 | APIC interrupt controller | 1200 | 2 weeks | 📋 Planned |
| 2.9 | Framebuffer driver | 800 | 1 week | 📋 Planned |
| 2.10 | ACPI parsing | 600 | 1 week | 📋 Planned |
| 2.11 | HAL implementation (kernel) | 1000 | 2 weeks | 📋 Planned |
| 2.12 | Build system integration | 500 | 1 week | 📋 Planned |
**Total**: ~11,000 LOC, 12-14 weeks
### Success Criteria
- ✅ UEFI boots on QEMU + OVMF
- ✅ Serial console prints "StarKernel booting..."
- ✅ PMM/VMM allocate memory correctly
- ✅ Kernel heap works (kmalloc/kfree)
- ✅ Interrupts and timers functional
- ✅ VM starts and prints "ok" prompt
- ✅ Can execute simple FORTH code (`1 2 + .` → `3`)
- ✅ Physics subsystems operational on bare metal
### Milestones
**Month 3**: UEFI boot + serial output working
**Month 4**: Memory managers complete, VM starts
**Month 5**: Full HAL working, "ok" prompt on QEMU
**Blocker Risk**: Medium (complex low-level code, but well-documented)
### Testing Strategy
**Development**: QEMU with OVMF (UEFI firmware)
**Validation**: Physical hardware (Intel NUC or similar)
**Automation**: QEMU headless testing in CI
---
## PHASE 3: STARSHIPOS (Months 6-18)
**Goal**: Self-hosting operating system (compiles itself, runs applications)
**Why**: Demonstrates complete independence from Linux/POSIX, enables FORTH-native ecosystem.
### Architecture
```
┌─────────────────────────────────────────────────────────┐
│ StarshipOS Layers │
├─────────────────────────────────────────────────────────┤
│ Userland │
│ • Shell, editors, compilers, tools │
├─────────────────────────────────────────────────────────┤
│ System Services │
│ • Filesystem (FAT32 or custom) │
│ • Networking (TCP/IP stack) │
│ • Process manager, scheduler │
├─────────────────────────────────────────────────────────┤
│ Drivers │
│ • Storage (AHCI, NVMe) │
│ • Network (VirtIO, E1000) │
│ • Block/char device abstraction │
├─────────────────────────────────────────────────────────┤
│ StarKernel (from Phase 2) │
│ • CPU, memory, time, interrupts │
└─────────────────────────────────────────────────────────┘
```
### Deliverables (Staged)
#### Stage 3A: Storage (Months 6-9)
| # | Component | LOC | Duration | Status |
|---|-----------|-----|----------|--------|
| 3.1 | AHCI driver | 2500 | 1 month | 📋 Planned |
| 3.2 | NVMe driver | 2000 | 1 month | 📋 Planned |
| 3.3 | Block device abstraction | 1500 | 2 weeks | 📋 Planned |
| 3.4 | FAT32 filesystem | 3000 | 1 month | 📋 Planned |
| 3.5 | VFS layer | 2000 | 2 weeks | 📋 Planned |
**Subtotal**: ~11,000 LOC, 3-4 months
**Success Criteria**:
- ✅ Read/write files to disk
- ✅ Boot from hard drive (not just RAM)
- ✅ Persistent FORTH definitions
---
#### Stage 3B: Networking (Months 10-12)
| # | Component | LOC | Duration | Status |
|---|-----------|-----|----------|--------|
| 3.6 | VirtIO-net driver | 2000 | 1 month | 📋 Planned |
| 3.7 | E1000 driver | 1500 | 2 weeks | 📋 Planned |
| 3.8 | lwIP TCP/IP stack (port) | 8000 | 2 months | 📋 Planned |
| 3.9 | Socket API (FORTH-native) | 1500 | 2 weeks | 📋 Planned |
**Subtotal**: ~13,000 LOC, 3-4 months
**Success Criteria**:
- ✅ Send/receive packets
- ✅ TCP connections work
- ✅ Simple HTTP server in FORTH
---
#### Stage 3C: Process Model (Months 13-15)
| # | Component | LOC | Duration | Status |
|---|-----------|-----|----------|--------|
| 3.10 | FORTH task abstraction | 2000 | 1 month | 📋 Planned |
| 3.11 | Scheduler (round-robin) | 1500 | 2 weeks | 📋 Planned |
| 3.12 | IPC mechanism | 2500 | 1 month | 📋 Planned |
| 3.13 | System call interface | 1000 | 2 weeks | 📋 Planned |
**Subtotal**: ~7,000 LOC, 2-3 months
**Success Criteria**:
- ✅ Multiple concurrent tasks
- ✅ Task switching works
- ✅ Tasks can communicate (send/receive)
---
#### Stage 3D: Self-Hosting (Months 16-18)
| # | Component | LOC | Duration | Status |
|---|-----------|-----|----------|--------|
| 3.14 | FORTH compiler (meta-compilation) | 3000 | 1 month | 📋 Planned |
| 3.15 | Assembler (FORTH-hosted) | 2000 | 2 weeks | 📋 Planned |
| 3.16 | Linker/loader | 1500 | 2 weeks | 📋 Planned |
| 3.17 | Shell & core utilities | 2500 | 1 month | 📋 Planned |
| 3.18 | Build system (FORTH-based) | 1000 | 2 weeks | 📋 Planned |
**Subtotal**: ~10,000 LOC, 2-3 months
**Success Criteria**:
- ✅ StarshipOS compiles itself
- ✅ No dependency on Linux toolchain
- ✅ Can build applications entirely in FORTH
---
**Phase 3 Total**: ~41,000 LOC, 12-15 months
### Milestones
**Month 9**: Persistent storage working, boot from disk
**Month 12**: Network stack operational, HTTP demo
**Month 15**: Multitasking working, IPC functional
**Month 18**: Self-hosting achieved, OS compiles itself
**Blocker Risk**: High (large scope, complex subsystems)
**De-risking Strategy**: Port existing code where possible (lwIP for TCP/IP, FAT32 reference)
---
## PHASE 4: FPGA FEASIBILITY (Months 6-7)
**Goal**: Determine if FPGA implementation is worth pursuing
**Why**: FPGA is a massive investment (12-18 months). Need go/no-go decision before committing.
### Feasibility Study
| # | Task | Duration | Deliverable |
|---|------|----------|-------------|
| 4.1 | Survey existing FORTH FPGA cores | 1 week | Comparison table |
| 4.2 | Estimate resource usage (LUTs, BRAMs) | 3 days | Resource budget |
| 4.3 | Select target FPGA | 3 days | FPGA selection rationale |
| 4.4 | Prototype stack machine (Verilog) | 2 weeks | Minimal stack machine |
| 4.5 | Synthesize & test on FPGA board | 1 week | Synthesis report |
| 4.6 | Performance analysis | 3 days | Speed vs. software |
| 4.7 | Cost/benefit analysis | 2 days | Go/no-go recommendation |
**Total**: 5-6 weeks
### Decision Criteria
**GO if**:
- ✅ Fits in affordable FPGA (~$200-500 board)
- ✅ Performance ≥ 50 MHz clock (faster than software?)
- ✅ Adaptive runtime feasible in hardware (critical!)
- ✅ Development time justified by benefits
**NO-GO if**:
- ❌ Too large for affordable FPGAs
- ❌ Slower than software implementation
- ❌ Adaptive runtime not synthesizable
- ❌ >18 months to implement (opportunity cost too high)
### Prototype Spec
**Minimal Stack Machine** (proof of concept):
- Data stack (16 entries × 64-bit)
- Return stack (16 entries × 64-bit)
- Basic operations: +, -, *, /, DUP, DROP, SWAP
- Dictionary ROM (256 entries)
- Simple UART I/O
**Target**: Prove concept works in hardware, get resource usage numbers
---
## PHASE 5: FPGA IMPLEMENTATION (Months 8-24)
**Status**: CONDITIONAL (only if Phase 4 says GO)
**Goal**: Hardware FORTH processor on FPGA
### Architecture
```
┌────────────────────────────────────────────────┐
│ FPGA Top Module │
├────────────────────────────────────────────────┤
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Stack Machine│ │ Memory │ │
│ │ • Data stack│ │ Controller │ │
│ │ • Return stk│ │ • Dict ROM │ │
│ │ • ALU │ │ • RAM │ │
│ └──────────────┘ └──────────────┘ │
├────────────────────────────────────────────────┤
│ ┌──────────────┐ ┌──────────────┐ │
│ │ UART │ │ Timers │ │
│ │ IP Core │ │ Counters │ │
│ └──────────────┘ └──────────────┘ │
├────────────────────────────────────────────────┤
│ ┌──────────────────────────────────────────┐ │
│ │ Adaptive Runtime (if feasible) │ │
│ │ • Execution counters │ │
│ │ • Hot-word cache │ │
│ │ • Hardware decay (?) │ │
│ └──────────────────────────────────────────┘ │
└────────────────────────────────────────────────┘
```
### Deliverables
#### Stage 5A: Core Implementation (Months 8-13)
| # | Component | LOC (HDL) | Duration | Status |
|---|-----------|-----------|----------|--------|
| 5.1 | Data stack (16 deep) | 500 | 2 weeks | 📋 Conditional |
| 5.2 | Return stack (16 deep) | 500 | 2 weeks | 📋 Conditional |
| 5.3 | ALU (arithmetic/logic ops) | 2000 | 1 month | 📋 Conditional |
| 5.4 | Dictionary memory (ROM) | 1000 | 2 weeks | 📋 Conditional |
| 5.5 | RAM controller | 1500 | 3 weeks | 📋 Conditional |
| 5.6 | Instruction decoder | 2000 | 1 month | 📋 Conditional |
| 5.7 | Control FSM | 1500 | 3 weeks | 📋 Conditional |
**Subtotal**: ~9,000 lines Verilog, 4-5 months
---
#### Stage 5B: I/O & Peripherals (Months 14-16)
| # | Component | LOC (HDL) | Duration | Status |
|---|-----------|-----------|----------|--------|
| 5.8 | UART transmitter/receiver | 800 | 2 weeks | 📋 Conditional |
| 5.9 | Timer/counter IP cores | 600 | 1 week | 📋 Conditional |
| 5.10 | Interrupt controller | 1000 | 2 weeks | 📋 Conditional |
| 5.11 | Memory-mapped I/O bus | 1200 | 3 weeks | 📋 Conditional |
**Subtotal**: ~3,600 lines Verilog, 2-3 months
---
#### Stage 5C: Adaptive Runtime (Months 17-20)
**CRITICAL UNCERTAINTY**: Can we synthesize this?
| # | Component | LOC (HDL) | Duration | Status |
|---|-----------|-----------|----------|--------|
| 5.12 | Execution frequency counters | 1500 | 1 month | 📋 Conditional |
| 5.13 | Hot-word cache (top-K) | 2000 | 1 month | 📋 Conditional |
| 5.14 | Hardware decay logic | 1000 | 2 weeks | 📋 Conditional |
| 5.15 | Transition tracking | 1500 | 3 weeks | 📋 Conditional |
**Subtotal**: ~6,000 lines Verilog, 3-4 months
**Fallback**: If adaptive runtime doesn't synthesize well, implement in software on soft-core CPU (MicroBlaze/NIOS II) alongside FORTH core.
---
#### Stage 5D: Validation & Optimization (Months 21-24)
| # | Task | Duration | Deliverable |
|---|------|----------|-------------|
| 5.16 | Testbench (all modules) | 2 months | 100% coverage |
| 5.17 | Formal verification (critical paths) | 1 month | Proofs |
| 5.18 | Synthesis optimization | 1 month | Timing closure |
| 5.19 | Hardware-in-loop testing | 2 weeks | Physical validation |
| 5.20 | Performance tuning | 2 weeks | Final optimizations |
**Subtotal**: 4-5 months
---
**Phase 5 Total**: ~18,600 lines Verilog + testbenches, 15-18 months
### Target FPGAs (Options)
| FPGA | Cost | Resources | Speed | Notes |
|------|------|-----------|-------|-------|
| **Xilinx Zynq-7000** | $300 | 85K LUTs, ARM core | 200 MHz | **Recommended**: Hybrid ARM+FPGA |
| Intel Cyclone V | $250 | 77K LEs | 150 MHz | Good alternative |
| Lattice iCE40 UP5K | $50 | 5K LUTs | 50 MHz | Budget option, slower |
| Xilinx Artix-7 | $200 | 33K LUTs | 200 MHz | Pure FPGA, no ARM |
**Recommendation**: Zynq-7000 (ARM + FPGA fabric)
- Run StarshipOS on ARM
- Offload FORTH core to FPGA fabric
- Best of both worlds
### Success Criteria
- ✅ FPGA boots, prints "ok" via UART
- ✅ Can execute FORTH code
- ✅ Performance ≥ software on same workload
- ✅ Adaptive runtime functional (or gracefully degraded)
- ✅ 780+ tests pass on hardware
- ✅ 0% algorithmic variance maintained (if adaptive works)
### Milestones
**Month 13**: Core stack machine working in simulation
**Month 16**: I/O functional, boots on real FPGA
**Month 20**: Adaptive runtime implemented (or fallback decided)
**Month 24**: Full validation, production-ready FPGA core
**Blocker Risk**: VERY HIGH
- HDL expertise required
- Timing closure can be unpredictable
- Adaptive runtime may not be synthesizable
- Resource constraints on affordable FPGAs
---
## INTEGRATION TIMELINE
### Parallel Paths
**Months 1-5**: Sequential (HAL → Kernel)
**Months 6-18**: **Parallel** (OS + FPGA feasibility)
**Months 8-24**: **Conditional** (FPGA impl if feasible)
```
2025 2026 2027 2028
│ │ │ │
V V V V
Phase 1 Phase 2 Phase 3 Phase 5
HAL Kernel OS FPGA
(2m) (3m) ┌──────────────────(12m)────────────┐ (16m)
│ │ │ │
│ Phase 4: Feasibility (2m) │ │
│ └─────┬─────────────────────────────┘ │
│ │ │
└────────────────┴──GO/NO-GO Decision───────────────┘
│
If NO-GO: Focus on OS only
If GO: Parallel OS + FPGA work
```
### Resource Allocation (Solo Development)
**Months 1-5** (HAL + Kernel): 100% focus
**Months 6-7** (Feasibility): 80% OS, 20% FPGA study
**Months 8-18** (Parallel): 70% OS, 30% FPGA (if GO)
**Months 19-24** (FPGA finish): 50% OS, 50% FPGA
**Community Contributions**: Welcomed at all stages (see CONTRIBUTING.md)
---
## RISK MITIGATION
### High-Risk Items
| Risk | Probability | Impact | Mitigation |
|------|-------------|--------|------------|
| **HAL breaks determinism** | Low | Critical | Extensive testing, DoE validation |
| **Kernel never boots** | Medium | Critical | Incremental dev, QEMU testing |
| **FPGA too slow** | Medium | High | Feasibility study gates investment |
| **Adaptive runtime unsynthesizable** | High | High | Software fallback on ARM core |
| **Project too large (solo)** | High | Medium | Community outreach, phased delivery |
| **FPGA cost too high** | Low | Medium | Target affordable boards ($200-500) |
### Fallback Plans
**If HAL migration fails**: Continue hosted-only, defer kernel
**If OS scope too large**: Deliver minimal components (storage + network only)
**If FPGA infeasible**: Software-only path is still valuable
---
## DELIVERABLES SUMMARY
### By End of 2025 (Months 1-2)
- ✅ HAL implementation complete
- ✅ StarForth platform-agnostic
- ✅ All tests passing on HAL
### By Mid-2026 (Months 3-5)
- ✅ StarKernel boots to "ok" on QEMU
- ✅ Physics subsystems work on bare metal
### By End of 2026 (Months 6-12)
- ✅ Storage subsystem operational
- ✅ Network stack functional
- ✅ FPGA feasibility determined (GO/NO-GO)
### By Mid-2027 (Months 13-18)
- ✅ Process model working
- ✅ StarshipOS self-hosting
- ✅ (If FPGA GO): Prototype FPGA core functional
### By End of 2027 (Months 19-24)
- ✅ StarshipOS production-ready
- ✅ (If FPGA GO): FPGA core validated
### By 2028 (Future)
- 🎯 Ecosystem growth (applications, libraries)
- 🎯 FPGA optimizations & variants
- 🎯 Community-driven features
---
## SUCCESS METRICS
### Technical Metrics
| Metric | Target | Measurement |
|--------|--------|-------------|
| **HAL overhead** | <5% | Benchmark comparison |
| **Kernel boot time** | <1 second | QEMU timing |
| **OS compile speed** | <10 min | Self-hosting build |
| **FPGA clock speed** | ≥50 MHz | Synthesis report |
| **Test pass rate** | 100% (780+) | All platforms |
| **Determinism** | 0% CV | DoE validation |
### Project Health Metrics
| Metric | Target | Current |
|--------|--------|---------|
| **Code quality** | ≥9.0/10 | 9.2/10 ✅ |
| **Test coverage** | ≥80% | ~90% ✅ |
| **Documentation** | Complete | Strong ✅ |
| **Community** | ≥10 contributors | 1 (solo) |
| **GitHub stars** | ≥500 | TBD |
---
## COMMUNITY & ECOSYSTEM
### How to Contribute
See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
**High-value contributions**:
1. HAL implementations (new platforms)
2. Kernel drivers (AHCI, NVMe, network)
3. FPGA expertise (Verilog, synthesis)
4. Testing & validation
5. Documentation improvements
### Seeking Collaborators
**Needed expertise**:
- Kernel development (UEFI, memory management)
- FPGA design (Verilog/VHDL)
- OS design (filesystems, networking)
- Formal verification (HDL proofs)
**Contact**: rajames440@gmail.com (Robert A. James)
---
## CONCLUSION
**This roadmap is ambitious but achievable.**
**Key Success Factors**:
1. ✅ Strong foundation (Phase 0 complete)
2. ✅ Clear milestones (phased delivery)
3. ✅ Risk mitigation (fallback plans)
4. ⚠️ Community growth (critical for timeline)
5. ⚠️ FPGA decision (feasibility gates investment)
**Realistic Timeline**:
- Solo: 3.5 years (42 months)
- With 3-5 contributors: 2 years (24 months)
- With 10+ contributors: 1.5 years (18 months)
**The vision is clear. The path is defined. Now we build.**
---
**Next Steps**:
1. Read [docs/GAP_ANALYSIS.md](docs/GAP_ANALYSIS.md) for detailed gaps
2. Start Phase 1: HAL implementation
3. Join the project (see CONTRIBUTING.md)
**License**: See ./LICENSE
docs/working/archive/superseded/2026-07-08-hermes-message-block-storage-design.md
+397
View File
@@ -0,0 +1,397 @@
> **SUPERSEDED (docs triage 2026-10-14).** Storage design for the Hermes VM's messages and channels; the Hermes VM no longer exists -- messages are kernel-Hermes's.
> Instead: docs/fabric/FABRIC-3.5.md and FABRIC-3.6.md. Moved here from `docs/working/architecture/HERMES-MESSAGE-BLOCK-STORAGE-DESIGN-20260708.md`.
# Hermes Message/Channel Block-Backed Storage — Design Doc
**Date:** 2026-07-08 (rev 2026-07-09d)
**Branch:** `lithosananke`
**Status:** Design only. Not implemented. This is the dependency flagged
in `HERMES-MESSAGE-CHANNEL-PHYSICS-DESIGN-20260708.md`'s Status header —
that doc's heat-window sizing is provisional until this one lands.
**Author:** Captain Bob / Claude Code
**Revision history** (kept honest per this repo's convention — see
`VM-PHYSICS-DYNAMIC-FLEET-DESIGN-20260705.md` for the pattern):
- **rev a** (original): proposed Hermes duplicating Artemis's free-map
allocator shape verbatim, in isolation, as a standalone Hermes-only
change. Wrong framing — see rev b.
- **rev b**: corrected rev a toward a two-VM "shared Physical BAM /
Logical BAM effort" — Hermes and Artemis jointly building toward a
common logical layer. **Also wrong.** Captain Bob's direct correction:
*"no direct coupling between Hermes and Artemis... Hermes knows NOTHING
about Artemis or a bam of any kind."* Hermes's only inter-VM capability
is messaging (send/receive on the common channel, request → ACK/NACK)
— never reaching into another VM's storage, and never a BAM concept
shared across VMs.
- **rev c** (this revision): the actual confirmed architecture, reached
after grounding in how Artemis concretely lays out its own disk today.
**Every VM manages its own storage independently** — own RAM blocks,
then own ramdrive blocks, then its own dedicated appended virtio-blk
disk, exactly mirroring Artemis's already-proven pattern but with zero
knowledge of or coupling to Artemis (or to any other VM). No shared
allocator, no coordinator VM. "Physical BAM" and "Logical BAM" are
per-VM concepts — each VM eventually has its own of each — not a
cross-VM split. Confirmed verbatim: *"each VM uses its ram blocks and
ramdisk blocks then appends the virtual drive"* / *"yes, that's
correct"* / *"it's going to look like PBAM and LBAM are the same right
now. this will change when we start figuring out how to use and shape
the tripod."*
- **rev d** (this revision, 2026-07-09): rev c is corrected. A full
Tripod-docs audit (reading `.claude/TRIPOD.md`/`HERMES.md`/`ARTEMIS.md`
in full, which should have happened before rev a was ever written)
found that rev c's "every VM independently, no coordinator" picture is
not how the Tripod is actually specified. `TRIPOD.md`'s dispatch table
is explicit: `Block I/O → Artemis`, reached via `VM-EXEC` naming the
VM by capability, not heat or convenience. `ARTEMIS.md` is direct:
*"Artemis is memory. She owns the external attached virtual disk image
and everything that lives on it."* Hermes does not grow its own disk.
**Corrected architecture: Artemis is sole storage authority for the
fleet.** Hermes's message-block persistence need is satisfied by
*asking* Artemis, not by building parallel storage machinery — this
supersedes rev c's "own dedicated appended virtio-blk disk" tier for
Hermes entirely (see "Storage tiering," corrected below).
This does **not** reopen rev b's rejection of direct coupling. *"There
is to be no tight coupling between ANY VM"* — reaffirmed by Captain Bob
in the same breath as naming Artemis storage authority — and it isn't
in tension with that at all: `TRIPOD.md`'s capability-based dispatch
(`VM-EXEC` by name, request/ACK-NACK) is already the Tripod's normal,
loose, message-mediated interaction model. Hermes asking Artemis to
store a message's payload is the same shape as any other cross-VM
request — it was only ever a contradiction under the mistaken
assumption that "Artemis as authority" meant Hermes reaching into
Artemis's BAM or address space directly. It doesn't. Hermes never
learns a physical LBN, never touches `FM-*`/`BLK-ALLOC`, never knows a
zone exists. See "The logical-reference gap, resolved" below for the
concrete mechanism.
- **rev e** (this revision, 2026-07-09): rev d overcorrected. Captain
Bob's direct correction: *"ALL VMs regardless have ram backed blocks
zero to 2047, ramdrive from 2048 to its end. that's a VMs state and
identity hard requirement."* Rev c's tiers 1–2 (own RAM 0–2047, own
ramdrive 2048+, per VM) were **right** — rev d was wrong to fold them
into "Artemis owns everything." They're not storage Hermes borrows or
requests; they're part of what makes a VM a VM at all, same category as
its own dictionary and stacks. Corroborating evidence, not invented:
`capsule_birth_baby()`'s own comment — *"PERSONALITY: per-VM block
storage is M9 scope; no-op until then"* — names this as a real, planned
milestone, just not built yet.
**What actually changes under Artemis-as-storage-authority is narrower
than rev d claimed: only the tier beyond a VM's own ramdrive.** Rev c's
tier 3 ("own dedicated appended virtio-blk disk") is the one thing that
doesn't survive — no VM other than Artemis appends a persistent disk.
Once Hermes's own ramdrive is exhausted, *that's* when it asks Artemis,
not before. Sections below are corrected to reflect three tiers again:
own RAM, own ramdrive, then (not own disk, but) ask Artemis.
---
## Execution rules
Binding for all implementation work under this doc, matching the
discipline `VM-PHYSICS-DYNAMIC-FLEET-DESIGN-20260705.md` established and
Captain Bob's explicit intent for this specific piece of work — this is
as much a Tripod discovery exercise as it is a storage redesign:
1. **Small units.** One new word, one call-site migration, one allocator
primitive at a time — not the whole redesign in one pass.
2. **Clean + make at checkpoints.** `make -f Makefile.starkernel clean`
then build after each unit that touches buildable code.
3. **QEMU when applicable**, one ISA at a time (amd64 first, fastest under
TCG), reserving the full three-arch pass for milestones — new
primitives wired end-to-end, deletions landed, before declaring a
phase implemented. This redesign is expected to surface real Tripod
gaps the same way the `LOAD-DOE` bug did during the `doe-campaign.4th`
migration — that's a feature of doing this incrementally with real
boots, not a risk to design around.
4. **Never modify `FM-TEST`/`FM-SET`/`FM-CLR`/`FM-FIND-FREE`/`BLK-ALLOC`/
`BLK-FREE` in `artemis/init.4th`.** They're registered, tested,
HARD LOCKED (blocks 4110–4113, `capsules/MANIFEST.md`) — Artemis's own
disk-tier allocator. Per rev e, Hermes never touches these, and never
builds an equivalent for Artemis's disk; Hermes's own RAM/ramdrive
tiers (own M9 allocator, once that lands) are a separate, independent
implementation over Hermes's own blocks, not a modification, extension,
or generalization of Artemis's disk-tier primitives.
---
## Problem
`hermes/init.4th`'s message and channel storage is a fixed in-memory pool:
`CREATE MSG-ARENA MSG-MAX MSG-CELLS * CELLS ALLOT` (Block 4101), a
32-slot, 9-cell-per-message (`MSG-CELLS`) free-list array; `CH-ARENA` is
the same shape at 16 slots. A fixed compile-time array can't grow without
recompiling, and Hermes's own comments already flag real exhaustion risk
(`FLEET-DOE-WORKLOAD`'s block 2058 comment: `BROADCAST-TEST`... would
exhaust Hermes's fixed member/message arenas after ~20 runs").
Separately, the heat-physics sibling doc's window-sizing already depends
on knowing the real capacity messages will be observed at (see that doc's
Status header) — an arena bound that can silently change relative to
`MSG-MAX`/`CH-MAX` compile-time constants makes that reasoning fragile.
## Goal
Give Hermes's messages and channels a way to outgrow the fixed 32/16
in-memory arenas — first into Hermes's own per-VM RAM/ramdrive blocks
(M9 scope), and only past that, **by asking Artemis to hold what Hermes's
own storage can't** — matching `TRIPOD.md`'s dispatch table exactly:
*"Block I/O → Artemis... `VM-EXEC` takes a VM name. The caller names the
correct VM for the work type. There is no routing algorithm."* Hermes
does grow its own block-addressed storage (own RAM, own ramdrive — that's
its per-VM identity, not optional); what it never grows is a *disk* of
its own. Past its own ramdrive, it sends a message; Artemis does the
storing.
**Explicitly not the goal:** Hermes appending its own persistent disk
(rev c's tier 3 — superseded, see revision history). Also not the goal:
designing Hermes's own per-VM RAM/ramdrive block layout (M9 scope,
separate work) or Artemis's own storage internals (zones, BAM cell
layout, migration policy) — the latter already specified, in more detail
than this doc would add, in `.claude/ARTEMIS.md`'s Future Material
section, and explicitly not build-authorized yet ("Do not implement any
of this without Captain Bob explicitly reopening it").
## The C/FORTH boundary question
Tiers 1–2 (Hermes's own RAM/ramdrive) are M9 scope — genuinely deferred,
not this doc's to design, so this question doesn't apply to them here.
For tier 3 (asking Artemis), the question doesn't arise for Hermes at
all — Hermes only ever calls `VM-EXEC` (or the equivalent message-send
word) naming Artemis, the same capability-based dispatch every other
Tripod interaction uses. Artemis's own block-storage subsystem is already
proven pure FORTH (`artemis/init.4th`'s free-map bitmap, allocator,
`LE32!`/`LE64!` arch-neutral I/O, all composed over `BLOCK`/`BUFFER`/
`C@`/`C!`, zero new C) — that precedent matters for whoever eventually
builds out Artemis's Future Material, not for anything in this doc.
## Storage model: three tiers — own RAM, own ramdrive, then ask Artemis
Every VM — Hermes included — has its own RAM-backed blocks 0–2047 and its
own ramdrive from 2048 to its end, as a hard requirement of what a VM is
(`capsule_birth_baby()`'s own "PERSONALITY: per-VM block storage is M9
scope" comment names this as planned, not yet built). `ARTEMIS.md`'s
*"Artemis is memory. She owns the external attached virtual disk image
and everything that lives on it"* is about the **external persistent
disk** specifically — it doesn't reach into a VM's own RAM or ramdrive,
which are that VM's own identity, not storage Artemis administers.
1. **Own RAM blocks first (0–2047).** Hermes's own private RAM-backed
block range, part of its M9 per-VM identity, once that milestone
lands. Today's `MSG-ARENA`/`CH-ARENA` (`CREATE...ALLOT`) are the
pre-M9 stand-in for this tier — an in-VM-memory working set, not yet
block-addressed. Fast, synchronously touchable, no round trip — the
right home for a message actively being delivered, ACKed, or NACKed.
2. **Own ramdrive next (2048 to its end)**, once tier 1 isn't enough.
Also part of Hermes's own per-VM identity under M9 — Hermes's own
ramdrive, not a shared or Artemis-administered one. This is where
`MSG-ARENA`/`CH-ARENA` graduate to once they outgrow the fixed
32/16-slot compile-time arrays, block-addressed within Hermes's own
range, still entirely within Hermes.
3. **Ask Artemis, only once tier 2 is exhausted.** This is the one place
rev c was wrong and rev d's correction actually applies: Hermes does
not append its own persistent disk. Once its own ramdrive runs out (or
a message needs to survive past Hermes's own reap), it sends Artemis a
store request and gets back a handle — see "The logical-reference gap,
resolved," below.
**Escalation between tiers is compudynamics-driven, not a size
threshold.** `ARTEMIS.md`'s own Migration Policy already establishes the
pattern for exactly this kind of question: *"No zone is special... when a
zone has pressure... the coldest block migrates first because it is
coldest... no magic constants... everything else is self-organizing."*
That's not specific to Artemis's disk zones — it's the same heat/cool/
reap model already governing word heat, VM heat, and (once
`msg_decay_slope_q48` lands) message heat. Tier 1→2 pressure and tier
2→3 pressure work the same way: the coldest message under RAM-tier
pressure is what migrates to Hermes's own ramdrive, and the coldest
ramdrive-tier message under pressure is what gets sent to Artemis — not
an arbitrary byte-count cutoff invented for this doc. The concrete
numbers (window depth, pressure threshold) are still M9/Future-Material
scope, unspecified — but the *mechanism* isn't an open question, it's
already-established compudynamics, applied one level up.
`capsules/MANIFEST.md`'s existing Hermes ranges (4100–4153) stay what
they've always been — capsule source organization for `hermes/init.4th`'s
word definitions, not a claim on tiers 1–2's future per-VM block ranges
(those are a distinct, not-yet-designed M9 concern, out of scope here —
see Open questions).
## The logical-reference gap, resolved
The open question flagged when rev c was superseded: `ARTEMIS.md`'s
already-specified Logical BAM (`ARTEMIS.md`, "Two BAMs" — 4 cells:
physical LBN, heat, XXHash64 content identity, flags) is keyed by
content. That's the right key for what it's explicitly scoped for —
*"Content-addressing, ACL records, and cold capsule storage live here...
Identity is derived from block content — same content, same identity...
Mutation is detectable: hash changes if content changes."* A live Hermes
message is exactly the case that breaks that key: its content can change
during its lifetime (composed, delivered, ACKed), so a content hash
cannot be the stable handle Hermes holds onto across a request/fetch
round trip — the hash itself would move whenever the content did.
The resolution doesn't require inventing a new mechanism — `HERMES.md`
already has the right shape for a stable, VM-minted reference, used today
for exactly this purpose: *"`channel_id` = owner VM identity + monotonic
sequence number. Unforgeable by non-owners. Hermes mints it. No VM
constructs its own channel_id."* A message or channel that needs
Artemis-backed storage gets stored under that same kind of handle —
Hermes's own minted ID, opaque to Artemis, stable for the object's
lifetime regardless of what its content is doing or where Artemis
physically keeps it. Artemis's storage-request protocol accepts that
opaque ID as the key for this class of entry — necessarily a *different*
internal bookkeeping path from the content-addressed Logical BAM entries
`ARTEMIS.md` already specifies for cold/immutable data, since the two
have genuinely different identity semantics (stable-opaque-handle vs.
content-hash). Designing that internal path is Artemis's own future
work, not this doc's — flagged here only so the gap has a named, grounded
answer instead of sitting open.
## Physical BAM / Logical BAM — per-VM, both Hermes's and Artemis's own
Rev c had this right and rev d wrongly discarded it: Physical BAM
(free-map + block-heat allocator) and Logical BAM (normalizing that same
VM's own heat for fleet-conservation participation) are **per-VM
concepts**, not something only Artemis has. Under M9, Hermes eventually
needs its own Physical/Logical BAM over its own RAM/ramdrive blocks
(tiers 1–2) — a new, independent implementation, out of scope for this
doc (M9 is separate work), but real and not something this doc should
keep implying doesn't exist.
What rev d got right and stays true under rev e: Hermes's own BAM (once
built) has nothing to do with, and never touches, Artemis's disk-tier
BAM. `ARTEMIS.md`'s Future Material section already specifies Artemis's
own version at a level of detail this doc doesn't need to restate:
Physical BAM (2 cells — physical LBN, free flag; "the raw substrate... no
heat, no identity"), Logical BAM (4 cells — physical LBN, heat, content
identity, flags), three thermal zones with a self-organizing migration
policy. That whole section is explicitly marked Future Material in
`ARTEMIS.md`, not build-authorized without Captain Bob reopening it —
unrelated to whatever shape Hermes's own future per-VM BAM eventually
takes, and `artemis/init.4th`'s `FM-*`/`BLK-ALLOC`/`BLK-FREE` (blocks
4110–4113, ★ HARD LOCKED) remain untouched by Hermes either way.
## The mechanism (sketch — tier 3 protocol shape only)
Tiers 1–2 (Hermes's own RAM/ramdrive) are M9 scope, not sketched here —
that's per-VM block storage work shared by every Tripod VM, not specific
to Hermes's messages, and belongs in its own doc when M9 is reopened.
What follows is tier 3 only: what happens once Hermes's own ramdrive is
exhausted and it has to ask Artemis.
### 1. A store/fetch request, dispatched the same way any Tripod work is
`TRIPOD.md`'s dispatch model already covers this — Hermes names Artemis
by capability, same as any Block I/O work: something like `ART-STORE
( addr len logical-id -- ack/nack )` and `ART-FETCH ( logical-id --
addr len )`, sent via `VM-EXEC` (or the message-passing equivalent once
Hermes's async model, `HERMES.md`'s G1/G8 items, supports a return
value). `logical-id` is Hermes's own minted ID — see "The
logical-reference gap, resolved," above.
### 2. What Hermes sends is a payload, not a block
Hermes doesn't decide block geometry, packing, or free-map granularity —
that's Artemis's Future Material to design (`ARTEMIS.md`'s "Physical BAM
Block Geometry" section already sketches a 3-data+1-metadata packing
scheme for variable-length records generally, which a Hermes message
would be one instance of, once that chapter is reopened). This doc stops
at "Hermes hands Artemis a payload and an ID" — how Artemis packs,
allocates, and migrates it is out of scope here by design, not by gap.
### 3. Synchronous limitation, already named in `HERMES.md`
`HERMES.md`'s G8 gap applies directly: *"Wiring [a cross-VM value] into
`K-FLEET` in Hera requires a cross-VM return value — something the
current synchronous `VM-EXEC` model cannot deliver."* `ART-FETCH`
returning a payload has the same shape problem. Until the async
inter-VM model lands, a Hermes→Artemis fetch request is bounded by
whatever `VM-EXEC`'s synchronous call/return can actually carry — not
redesigned here, just inherited as a precondition.
## Consumer code migration
Deferred to implementation, for the same reason rev c deferred it: the
concrete request/response word names, and everything on Artemis's side
of the boundary, depend on Future Material that isn't build-authorized
yet (`ARTEMIS.md`: *"Do not begin implementation without explicit
instruction from Captain Bob"*). A follow-up pass once Artemis's flat-pool
Immediate Goal is done and this chapter is explicitly reopened should add
this section for real.
## Verification approach
- Three-arch acceptance as usual once any unit reaches boot-behavior-
affecting completeness, per the Execution Rules above.
- Real message-storm test: repeat `BROADCAST-TEST`-style traffic well
past the ~20-run point that exhausts today's 32-slot arena (per the
comment already in `hermes/init.4th`), confirm messages that overflow
to Artemis are still retrievable and don't silently drop — the
concrete, observable proof this redesign did something, not just moved
the same limit elsewhere.
- `TRIPOD-TEST`'s existing `PASS: broadcast reached all 3` /
`PASS: msg queued` / `PASS: E2E msg flow` checks should continue
passing unchanged throughout — this redesign changes what happens on
arena overflow, not Hermes's external message-delivery behavior.
## Explicitly out of scope
- **The heat-decay mechanism** (`msg_decay_slope_q48`) — sibling doc,
`HERMES-MESSAGE-CHANNEL-PHYSICS-DESIGN-20260708.md`, independent of
this doc's rev d/e corrections.
- **Hermes's own per-VM RAM/ramdrive BAM (tiers 1–2, M9 scope).** Real,
confirmed (`capsule_birth_baby()`'s own comment names it), and not
designed in this doc — a separate, larger piece of work shared in
shape by every Tripod VM, not specific to Hermes's messages.
- **Channel storage** (`CH-ARENA`) — this doc focuses on messages, the
higher-volume and more clearly arena-exhaustion-prone of the two per
the existing `hermes/init.4th` comment. Whether channels need the same
overflow-to-Artemis treatment is a question for after messages are
done, not decided here.
- **Artemis's own storage internals** (zones, BAM cell layout, packing,
migration policy). Fully specified as Future Material in `ARTEMIS.md`
already; explicitly not build-authorized without Captain Bob reopening
that chapter. This doc's protocol sketch (section 1-3 above) doesn't
depend on those internals being resolved to be a correct *shape* — it
just can't be implemented before they are.
- **Finishing Artemis's own `K-FLEET` wiring gap.** Tracked in
`ARTEMIS-BAM-ACCEPTANCE-20260703.md`'s Deferred/Next table and
`ARTEMIS-BLOCK-PHYSICS-DESIGN-20260708.md`. Not this doc's to close.
- **Artemis block heat generally** — sibling doc,
`ARTEMIS-BLOCK-PHYSICS-DESIGN-20260708.md`.
## Open questions (flagged, not guessed at)
1. **The concrete word names and wire shape** for the store/fetch
protocol — sketched at the level of "a request with a logical ID,"
not specified further, since it depends on Artemis's own
not-yet-authorized storage internals.
2. **Target message capacity** — "bigger than 32" is not a number.
Needs a real target, ideally informed by the message-storm test's
observed exhaustion behavior at various loads.
3. **How Artemis's internal store distinguishes opaque-handle entries
(Hermes messages) from content-addressed entries (cold capsules, ACL
certs)** — flagged in "The logical-reference gap, resolved," above,
as Artemis's own design work.
4. **The synchronous-call-return limitation** (section 3 above) — whether
this doc's protocol waits for `HERMES.md`'s async model, or an interim
shape is worth designing sooner.
5. **Whether anything here genuinely needs a new C primitive** — no
evidence either way yet, since Artemis's side of this isn't designed.
Precedent (`artemis/init.4th`'s existing block subsystem is pure
FORTH) suggests no, but that's Artemis's question to answer when this
chapter reopens, not this doc's to assume.
6. **The concrete pressure threshold for tier escalation** — the
*mechanism* is settled (compudynamics heat-pressure, matching
`ARTEMIS.md`'s Migration Policy — see "Storage model," above); the
*numbers* (how much ramdrive pressure triggers a tier-2→3 request)
depend on M9's own design, which doesn't exist yet.
7. **`ARTEMIS-FREEMAP-CAPSULE-RESERVATION-GAP-20260710.md` applies
here too.** Artemis's live `BLK-ALLOC` doesn't yet reserve the LBN
range her own capsule-loaded content occupies — a gap in Artemis's
current allocator, not something this doc introduces. But a future
`ART-STORE` request from Hermes (section 1 above) would hand out
blocks from that same unprotected pool, so this doc's tier-3
implementation inherits that gap and shouldn't be built before (or
without accounting for) it being closed.
docs/working/archive/superseded/2026-07-08-hermes-message-channel-physics-design.md
+324
View File
@@ -0,0 +1,324 @@
> **SUPERSEDED (docs triage 2026-10-14).** Heat physics for the Hermes VM's messages and channels; the Hermes VM no longer exists.
> Instead: docs/fabric/FABRIC-3.5.md and FABRIC-3.6.md. Moved here from `docs/working/architecture/HERMES-MESSAGE-CHANNEL-PHYSICS-DESIGN-20260708.md`.
# Hermes Message/Channel Heat Physics — Design Doc
**Date:** 2026-07-08 (rev 2026-07-08a)
**Branch:** `lithosananke`
**Status:** Design only. Not implemented. **Depends on a sibling
architectural doc**: `HERMES-MESSAGE-BLOCK-STORAGE-DESIGN-20260708.md`
(rev c), which plans to move Hermes's message area off its current
in-memory `CREATE...ALLOT` arena (`MSG-ARENA`/`CH-ARENA`, `MSG-MAX`=32/
`CH-MAX`=16) onto a dedicated, LBN-backed block range within the kernel
ramdrive — a Hermes-owned physical BAM over Hermes's own blocks, built
independently of and with zero coupling to Artemis (Artemis's own
disk-block model is precedent for the allocator *shape*, not something
Hermes's storage shares or depends on). That redesign is out of scope for
this doc — this doc's window-sizing reasoning (section 2) is provisional
against the *current* in-memory scale and will need revisiting once the
block-backed redesign lands, since block-backed storage will likely
change both the arena's real capacity and its churn characteristics.
Sibling doc to
`VM-PHYSICS-DYNAMIC-FLEET-DESIGN-20260705.md`'s "Fold-in question" section,
which settled that messages and blocks get their own independent decay
mechanisms — not merged into `VMPhysics`, not derived from its slope —
each fit from its own rolling window, in separate follow-on docs. This is
that doc for Hermes (messages + channels). See
`ARTEMIS-BLOCK-PHYSICS-DESIGN-20260708.md` for the block-heat sibling.
**Author:** Captain Bob / Claude Code
**Correction (2026-07-09):** this doc's original Goal section (rev a)
claimed message/channel heat has no conservation invariant, citing a
"settled fold-in decision" from `VM-PHYSICS-DYNAMIC-FLEET-DESIGN-
20260705.md`. That claim was wrong — it was derived without first reading
`.claude/HERMES.md`, which is authoritative and states plainly: *"Every
object Hermes manages — messages and channels — participates in K≡1.0.
Fleet K includes the thermal contribution of in-flight messages and open
channels."* A full Tripod-docs audit (`TRIPOD.md`/`HERMES.md`/
`ARTEMIS.md`) confirmed this directly contradicts what's below. The Goal
section is corrected in place. The rest of this doc's mechanism — an
independently-fit decay *rate* for messages and channels, its own
`HermesHeatWindow` struct, its own rolling window, separate from
word-level and VM-fleet physics — is unaffected: `HERMES.md` requires
message/channel heat to *participate in* K≡1.0, not that it share
VM-fleet's inference mechanism. Those are separable, and only the
conservation claim was wrong.
---
## Problem
`capsules/hermes/init.4th` hardcodes `65208 CONSTANT Q-DECAY` (Block 4100)
and applies it identically to two different entities via straight
multiplicative shrink:
- `MSG-COOL-ONE`/`MSG-COOL-ALL` (Block 4108): `heat = heat * Q-DECAY` for
every live message in the 32-slot `MSG-ARENA`.
- `CH-COOL-ALL` (Block 4114): same formula, walked over the live channel
linked list (16-slot `CH-ARENA`).
This is the exact "before" picture `VM-PHYSICS-DYNAMIC-FLEET-DESIGN`
already diagnosed for VM heat: `65208` was independently re-declared as
an identical hand-picked constant in three separate files
(`compudynamics.4th`, `hermes/init.4th`, `artemis/init.4th`) — copy-pasted,
uncoordinated, with no regression anywhere. The VM-heat instance of this
was fixed (`capsule_vm_physics.c`). Hermes's and Artemis's instances
weren't — this doc is Hermes's fix.
No evidence exists that `65208` (≈0.9950 per cool-call, i.e. a message or
channel's heat halves after roughly 138 `MSG-COOL-ALL` calls) was ever
tuned against real message/channel churn. It's the same number VM heat
used, which was itself never tuned against VM touch frequency either.
## Goal
Replace the hardcoded `Q-DECAY` with a `msg_decay_slope_q48` inferred from
observed cooling behavior, mirroring the *shape* of the word-level and
VM-fleet physics engines — same mechanics (rolling window, inferred rate,
skip-rather-than-substitute-a-default when unwarmed) applied to a third
kind of entity — without merging into either existing engine's *state* or
*inference mechanism*.
**Message and channel heat DOES participate in K≡1.0** — per
`.claude/HERMES.md`'s Compudynamic Invariant: *"Fleet K includes the
thermal contribution of in-flight messages and open channels... Hermes
reaping a message or channel must rebalance K correctly."* This doc does
not design that redistribution mechanism — `HERMES.md` already establishes
it conceptually (ACK → clean death, K redistributed; reap → K rebalanced),
and `HERMES.md`'s own G8 gap tracks the one known incompleteness (`HERMES-
K` exists and totals live message/channel heat, but wiring it into Hera's
`K-FLEET` is deferred pending an async inter-VM return-value model). What
this doc *does* design is orthogonal to that: **the rate** at which a
live message or channel's heat decays between creation and reap. Whether
that decay is later credited back to the fleet correctly is `HERMES.md`'s
concern and G8's tracked gap, not this doc's.
## Why this can't just copy either existing mechanism verbatim
**Not word-level's shape (rolling window of `word_id`s + trajectory
replay against current heat):** words are permanent dictionary entries —
`extract_heat_trajectory` (`inference_engine.c`) replays a window of
`word_id`s and looks up each one's *current* heat, exploiting that a word
never stops existing. Messages and channels are transient: `MSG-REAP`
frees a message's arena slot back to the free list once its heat cools
past whatever threshold or it's explicitly consumed, and a freed slot's
"current heat" is meaningless — it may already belong to an unrelated
message by the time anyone would replay a window containing its old ID.
Per-entity trajectory replay does not work here.
**Not VM-fleet's shape either, but structurally the closest analog:**
`capsule_vm_physics.c` (rev o) doesn't curve-fit a trajectory — it
recovers the transfer rate directly, because the transfer law is known
exactly (`amount = elapsed_us * slope_q48 >> 16`) and any unclamped touch
can be inverted for an exact per-sample rate; the fleet-wide
`fleet_transfer_slope_q48` is the *median* of recovered per-sample rates
over a 64-deep window of aggregate touch samples, not per-VM. The same
"known law, recover the rate, aggregate not per-entity" shape applies
here: `heat_new = heat_old * Q-DECAY` is *also* a fully known multiplicative
law, so there is nothing to curve-fit — what's actually being inferred is
not "what is the decay law" (known) but "is the current decay *rate*
tuned right for observed churn," which is a different question than either
existing engine answers, addressed below.
## The mechanism
### 1. What "inferring the rate" actually means here
Because the decay law itself is known and fixed-shape (`heat *= rate`),
there's no unknown functional relationship to recover the way VM-fleet
recovers `fleet_transfer_slope_q48` from touch samples, or the way Loop #6
recovers `decay_slope_q48` from a heat trajectory. What *can* legitimately
vary and be worth observing: **how fast messages/channels are actually
being reaped relative to how fast they're being created.** If churn is
high (many `MSG-ALLOC`/`MSG-REAP` cycles per cool-sweep), a slow decay
rate lets dead-weight messages linger, consuming arena slots that new
messages need. If churn is low, an aggressive decay rate discards heat
signal (e.g. NACK-retry priority, per `hermes/init.4th`'s own
`MSG-REDELIVER-NACKED`) before it's useful.
`msg_decay_slope_q48` should therefore be a function of **observed reap
throughput relative to arena occupancy**, not a re-derivation of the
`Q-DECAY` law itself. Concretely: a rolling window of samples taken once
per `MSG-COOL-ALL` call, each recording `(live_count, reaped_since_last)`;
periodically re-fit `msg_decay_slope_q48` so that, at the observed churn
rate, an average message crosses the reap threshold within a target
window (e.g. "most messages should be reaped within N cool-sweeps of
going idle" — the specific target is an open question, see Resolved
Items).
This is a genuinely different inference problem than either existing
engine solves, which is why it doesn't reduce to copying `infer_decay_slope_q48`
or `vm_physics_tick`'s median-of-rates — it's closer in spirit to Loop #5's
window-width inference (which also asks "is the current parameter well-
matched to observed dynamics," not "what is the true decay curve").
### 2. `HermesHeatWindow` — new struct, two instances (message, channel)
```c
typedef struct {
uint32_t live_count; /* live entities at this sample */
uint32_t reaped_since_last; /* entities freed since the prior sample */
} HermesHeatSample;
typedef struct {
HermesHeatSample samples[HERMES_HEAT_WINDOW_DEPTH];
uint32_t head;
uint32_t count; /* saturates at HERMES_HEAT_WINDOW_DEPTH */
int is_warm; /* count >= HERMES_HEAT_WINDOW_DEPTH */
} HermesHeatWindow;
```
Two instances — one for messages, one for channels — since they have
independent arenas, independent churn rates, and are proposed to have
independent decay slopes as this doc's own conservative default (see
Open questions). Not one shared window; `MSG-MAX`=32
and `CH-MAX`=16 are different enough scales that conflating them risks
exactly the kind of unjustified shared-constant problem this doc exists
to fix.
**`HERMES_HEAT_WINDOW_DEPTH`: proposed 16, not yet settled, and provisional
against a scale that's about to change.** `MSG-MAX`=32 and `CH-MAX`=16 are
today's *in-memory arena* limits — far smaller than word-level's millions
of executions or even VM-fleet's `VM_FLEET_WINDOW_DEPTH`=64, so a window
deeper than the arena itself provides no additional information at
*current* scale. But per this doc's Status header, the message area is
planned to move onto a dedicated block range, which will likely raise
that capacity substantially (Artemis's equivalent, `ART-DATA-BLKS`, is
22,998 — three orders of magnitude larger than `MSG-MAX`=32). 16 is a
starting guess against *today's* scale, not derived from anything, and
should be treated as provisional until the block-backed redesign's real
capacity is known (see Resolved Items).
### 3. Recording — passive, hooked at the existing cool-sweep call sites
`MSG-COOL-ALL` and `CH-COOL-ALL` already sweep their entire arena once per
call; a new `hermes_heat_sample(HermesHeatWindow*, uint32_t live_count,
uint32_t reaped_since_last)` call at the end of each sweep is the only new
call site needed — no new dispatch points, matching both existing engines'
"passive observer, hooked at what's already there" discipline.
**`reaped_since_last` bookkeeping is the one real implementation question
this doc leaves open.** `MSG-REAP` and channel reaping aren't currently
instrumented to report a count to the cool-sweep — that's new plumbing,
not free. Whether it's a shared counter incremented at each reap site and
read-and-reset by the cool-sweep, or something else, is unresolved (see
Resolved Items) — flagged rather than guessed at.
### 4. Inference — heartbeat-gated, same skip-don't-substitute discipline
`hermes_heat_tick()`, called from `HERMES-TICK`'s existing heartbeat path
(mirroring `vm_physics_heartbeat_tick`'s hook into each VM's own tick):
when a window is warm, re-fits that window's decay slope; when unwarmed,
leaves the current slope untouched rather than substituting a default —
same philosophy both existing engines already use.
The actual fit function (how `(live_count, reaped_since_last)` samples
become a new `msg_decay_slope_q48`) is intentionally not specified in
concrete formula form here — see Resolved Items. Writing a specific
formula now, without the real observed-churn data either existing engine's
design had before it settled on its final shape (VM-fleet went through
revisions b through o before landing on median-of-rates), would be
inventing content rather than designing it.
### 5. Application — replaces `Q-DECAY Q.*` at both cool-sweep sites
`MSG-COOL-ONE` and `CH-COOL-ALL`'s `Q-DECAY Q.*` becomes
`msg_decay_slope_q48 Q.*` / `ch_decay_slope_q48 Q.*` (or a single
`msg_decay_slope_q48` if the message/channel split above turns out
unwarranted after real observation — see Resolved Items). No change to
the multiplicative decay *shape* itself, only to where the rate comes
from.
## Consumer code migration
- **`hermes/init.4th` Block 4100**: `65208 CONSTANT Q-DECAY` deleted.
- **`hermes/init.4th` Block 4108** (`MSG-COOL-ONE`/`MSG-COOL-ALL`):
`Q-DECAY` reference replaced with a call to fetch the current
`msg_decay_slope_q48` (new C primitive, e.g. `MSG-DECAY-SLOPE@`);
`MSG-COOL-ALL` gains the sample-recording call described in section 3.
- **`hermes/init.4th` Block 4114** (`CH-COOL-ALL`): same pattern for
channel heat.
- New kernel-only file pair, matching `capsule_vm_physics.c`'s own scope
(Hermes-specific state, no hosted-build equivalent since the hosted
build has no Hermes):
- `src/starkernel/capsule/hermes_heat_physics.c`
- `include/starkernel/hermes_heat_physics.h`
- New FORTH-visible primitives (backed by the new file): `MSG-DECAY-SLOPE@`,
`CH-DECAY-SLOPE@` (or one shared accessor if sections above's split is
resolved differently), plus a status word mirroring `VM-PHYSICS-STATUS`'s
shape for diagnostics.
## What gets deleted
- `hermes/init.4th` Block 4100: `65208 CONSTANT Q-DECAY`.
Nothing else — `MSG-COOL-ONE`/`MSG-COOL-ALL`/`CH-COOL-ALL` keep their
existing structure, only the decay-rate source changes.
## Verification approach
- Three-arch acceptance as usual (kernel-only code,
`#ifdef __STARKERNEL__` gates per `.claude/CLAUDE.md`).
- A targeted test: drive real message/channel churn (repeated
`SEND-BROADCAST-TEST`-style traffic) at two different rates across
separate boot runs, confirm `msg_decay_slope_q48` converges to
different values reflecting the different observed churn — the direct
analog of confirming VM-fleet's `fleet_transfer_slope_q48` actually
responds to real touch frequency rather than sitting at its seed value.
- Confirm `HermesHeatWindow`'s dead-entity handling (a message reaped
mid-window) doesn't fault or corrupt the sample count — same class of
test as VM-fleet's "kill-during-warm-up" test
(`VM-PHYSICS-DYNAMIC-FLEET-DESIGN-20260705.md`).
## Explicitly out of scope
- **Moving Hermes's message/channel storage onto a dedicated LBN-backed
block range** (mirroring Artemis's `ART-DATA-BLKS`). Flagged in this
doc's Status header as a real dependency, but the redesign itself —
which blocks, what the on-disk layout looks like, how it interacts with
`MSG-ARENA`/`CH-ARENA`'s current in-memory shape — is a separate,
not-yet-written architectural doc, not part of this heat-decay design.
- **Artemis block heat** — sibling doc, `ARTEMIS-BLOCK-PHYSICS-DESIGN-20260708.md`.
- **Designing how K is credited/debited at message and channel alloc/reap**
— already governed conceptually by `HERMES.md`'s Compudynamic Invariant
section; the one known incompleteness (wiring `HERMES-K` into Hera's
`K-FLEET`) is `HERMES.md`'s own tracked G8 gap, not something this doc
redesigns. This doc's decay-rate inference operates on top of that
redistribution, not in place of it.
- **The multi-level DoE rewrite** (word/VM-fleet/message/block as four
independent metric spaces) — this doc gives message/channel heat its
mechanism; observing it as part of a real experiment is separate,
larger work already on the punch list.
- **Implementation** — this is a design doc. Writing
`hermes_heat_physics.c` is follow-on work, not part of this pass.
## Resolved items
None yet — this is a first-pass design doc, not an iterated one. Unlike
`VM-PHYSICS-DYNAMIC-FLEET-DESIGN-20260705.md`'s extensive revision history
(rev a through t, each resolving a question found during real
implementation and three-arch verification), this doc has had no
implementation pass yet to surface and resolve open questions against.
## Open questions (explicitly not resolved — flagged, not guessed at)
1. **`HERMES_HEAT_WINDOW_DEPTH` value.** Proposed 16 (matching `CH-MAX`),
not derived from real observed churn data. Needs a real implementation
pass with instrumented boot logs (same discovery method VM-fleet used:
`doe_log.c`'s per-VM heat CSV columns exposed the dead-flat-trajectory
bug that led to VM-fleet's rev-f seed-value fix) before treating this
as settled.
2. **The concrete fit function** for turning `(live_count,
reaped_since_last)` samples into `msg_decay_slope_q48`. Section 4
deliberately stops short of a formula. A first candidate worth
prototyping: target reap-latency-in-cool-sweeps, adjust the slope
proportionally to the ratio of observed to target latency — but this
is a starting hypothesis to implement and test against, not a
specified design.
3. **`reaped_since_last` bookkeeping mechanism** — where the reap-count
plumbing lives and how it's reset per sample window.
4. **Whether messages and channels genuinely need independent slopes**,
or whether observed data shows one shared `msg_decay_slope_q48`
suffices — section 2 assumes independence as this doc's own
conservative default (matching the pattern word-level and VM-fleet
physics each use their own independently-fit rate), but this should
be revisited once real data exists.
docs/working/archive/superseded/2026-08-02-artemis-hermes-compudynamics-plan.md
+379
View File
@@ -0,0 +1,379 @@
> **SUPERSEDED (docs triage 2026-10-14).** An implementation plan for Artemis and the Hermes VM; the Hermes half is gone with the VM, the Artemis half was carried by the FABRIC work.
> Instead: docs/fabric/FABRIC-3.5.md and FABRIC-3.6.md. Moved here from `docs/working/architecture/ARTEMIS-COMPUDYNAMICS-IMPLEMENTATION-PLAN-20260802.md`.
# Artemis (and Hermes) Compudynamics — Implementation Plan
**Date:** 2026-08-02
**Branch:** `master`
**Status:** Plan approved in shape, **not final and not started.** Captain Bob
has more to discuss on the design before implementation begins — this document
is expected to change as a result. Do not treat it as settled, and do not
start coding from it as written. The six Open Questions are the known gaps;
the pending discussion may add or reframe others.
**Author:** Captain Bob / Claude Code
**Siblings:** `ARTEMIS-BLOCK-PHYSICS-DESIGN-20260708.md` (the decay-rate design
this implements), `HERMES-MESSAGE-CHANNEL-PHYSICS-DESIGN-20260708.md`,
`ARTEMIS-BAM-ACCEPTANCE-20260703.md` (which tracks the `ART-TICK heartbeat`
gap this closes), `VM-PHYSICS-DYNAMIC-FLEET-DESIGN-20260705.md` (pattern source).
---
## Preamble: how this plan came to be written
This plan exists because of a wasted day, and the failure is worth recording
so it isn't repeated.
On 2026-08-02 a 30-replicate Artemis surface-stress campaign was run across
all three architectures to confirm the `block_words.c` stale-pointer
cache-aliasing fix held at scale. It did: **4500 trials, zero failures,
amd64/aarch64/riscv64**. That result is real and stands.
While reviewing the campaign's heartbeat CSV, the assistant observed the
`hera_heat_q48` / `hermes_heat_q48` / `artemis_heat_q48` columns sitting
perfectly flat (`65536, 0, 0`) across all 51,092 ticks, concluded this was a
defect, and rebuilt the campaign driver to route every stress-test phase
through Hera via `VM-EXEC` so that fleet heat would move.
That was wrong three separate ways:
1. **Wrong metric.** Those columns are *VM-fleet* heat
(`capsule_vm_physics.c`, `vm_physics_heat_of()`). The quantity of interest
for Artemis is *block* heat (`BLK-HEAT` / `ART-K-TOTAL`). Different
mechanisms, no connection between them.
2. **Wrong method.** Manufacturing cross-VM calls to force a physics reading
is explicitly prohibited: `TRIPOD.md` — *"No VM manages another VM's
internal heartbeat"*, *"There is no 'send to the hottest VM.' That model is
wrong and must not be implemented"*; `capsule_vm_physics.h:35-37` — *"a
passive observer of VM activity, never a driver of it."*
3. **Wrong premise.** Block heat cannot move today under any workload,
because nothing ages it (see below). No amount of driving would have
produced a signal.
All three were avoidable by reading `.claude/TRIPOD.md` and `.claude/ARTEMIS.md`
completely before touching Tripod code — which both files instruct in their own
headers, and which `.claude/CLAUDE.md` also instructs. They were skimmed by
keyword instead. Keyword search cannot surface a prohibition, because
prohibitions do not contain the term you are searching for.
The capsule changes from that detour have been reverted. **Standing rule
going forward: read the governing documents in full before starting any major
piece of work in this repo.**
---
## The actual defect
Block heat and message heat do not decay. Not "decay incorrectly" — **do not
decay at all**, unless something manually calls a sweep word.
- `ART-TICK` (`ART-COOL ART-REAP`, `capsules/artemis/init.4th` block 4140) has
**zero call sites** anywhere in the tree — no C, no FORTH, no capsule.
- `HERMES-TICK` (`capsules/hermes/init.4th` block 4116) has **zero C call
sites**. Every caller is Hera poking it manually via `VM-EXEC`
(`capsules/init.4th` lines 31, 47, 66, 83, 130).
`BLK-HEAT@` and `MSG-HEAT@` read a stored number that nothing ages. Heat is
frozen between manual pokes — neither lazy nor scheduled, simply absent.
Consequences:
- Blocks never reap by cooling. `ART-REAP` frees blocks at heat exactly 0;
nothing ever brings a block to 0 except an explicit `BLK-FREE`.
- Message TTL never expires on its own, contradicting `HERMES.md`'s stated
model (*"A message that nobody answers simply cools to death"*).
- `Q-DECAY` (65208) is hardcoded and hand-copied into three separate capsules
(`artemis/init.4th`, `hermes/init.4th`, and the now-deleted
`compudynamics.4th`) with no shared derivation — flagged in the block
physics design doc as the "before" picture.
`ARTEMIS-BAM-ACCEPTANCE-20260703.md` already tracks this as a named deferred
item ("ART-TICK heartbeat — wire into compudynamic tick loop"). This plan
closes it.
---
## The governing model: mirror how words work
Per Captain Bob's direction: **word-level execution physics is the reference
implementation.** Messages decay on **TTL**; blocks decay on **access
frequency**. Two distinct behaviours, one shared mechanism shape.
The word model, verified in source, has four parts:
**1. Storage.** `DictPhysics` (`include/vm.h:257-268`) carries
`last_decay_tick`, commented in-source as *"the only clock Loop #3 decay
reads."* The two `*_ns` fields beside it are explicitly marked **diagnostics
only, never read for decay** — the residue of the wall-clock bug fixed in
`VM-FLEET-ATTRACTOR-DESIGN-20260705.md` rev t.
**2. Lazy decay at every access point.** `physics_pre_execute()`
(`src/physics_execution_hooks.c:85-89`), `physics_on_lookup()` (`:203-215`),
and the kernel's inline duplicates (`src/starkernel/vm/vm_core.c:676, 870,
880`) all do exactly this:
```c
uint64_t elapsed_ticks = vm->heartbeat.tick_count - word->physics.last_decay_tick;
physics_metadata_apply_linear_decay(word, elapsed_ticks, vm);
word->physics.last_decay_tick = vm->heartbeat.tick_count;
/* then accumulate: */
physics_execution_heat_increment(word);
```
**3. Bounded background sweep with a resumable cursor.**
`vm_tick_apply_background_decay()` (`src/starkernel/vm/vm_runtime.c:357-388`)
processes at most `HEARTBEAT_DECAY_BATCH` dictionary entries per tick and
stores `vm->heartbeat_decay_cursor_id` to resume on the next tick.
**This is the answer to "sweeping 22,998 blocks per tick is too expensive."**
An earlier fleet-DoE attempt stalled outright on exactly that cost. The word
engine never sweeps its whole space in one tick — it amortizes. Artemis must
do the same.
**4. Decay math.** `physics_metadata_apply_linear_decay()`
(`src/physics_metadata.c:320-386`): `decay = (elapsed_ticks * slope_q48) >> 16`,
subtractive, clamped at zero. The slope is adaptive (`vm->decay_slope_q48`,
tuned by Loop #6), and the whole path is gated by
`ssm_config->L3_linear_decay` so L8 can switch it off.
### Two model differences to resolve, not assume
| | Words today | Artemis today | Consequence |
|---|---|---|---|
| Decay shape | **linear** subtract | **multiplicative** (`heat * 65208/65536`) | Multiplicative approaches zero asymptotically but may never reach exactly 0 in Q48.16; `ART-REAP` fires only at exactly 0. |
| On access | **increments** heat | `BLK-FETCH` **resets** to `Q.1` | Reset is recency-only; increment is frequency-sensitive. Bob named *frequency* as the block signal. |
Recommendation: match the word model on both counts. Confirm first (O2, O4) —
this changes when blocks reap.
---
## Design decisions
### D1. Block heat becomes C-side state
Today `BLK-HEAT` is a FORTH `CREATE ... ALLOT` array (block 4137, marked
immutable in `MANIFEST.md`). Word heat by contrast lives in C
(`DictEntry.execution_heat` + `DictPhysics`), with FORTH words reading it.
Mirror that. Block heat plus `last_decay_tick` move into
`src/starkernel/capsule/artemis_heat_physics.c`, with FORTH primitives for the
observational surface.
This is the decision that makes everything else fall out: the background sweep
becomes plain C called from `vm_tick()`, and **no C→FORTH bridge is required.**
That matters more than it might appear. **A mechanism for C to invoke a named
FORTH word on a tick does not exist in this codebase** — there is no
`vm_execute_word_by_name`, and `capsule_vm_hooks.c` carries only birth-time
hooks (exec, dict-hash, vm-alloc). Building one would be a substantial new
mechanism with its own risks. This design avoids needing it.
Per the ruling recorded in `VM-FLEET-ATTRACTOR-DESIGN-20260705.md` rev s —
*"TRIPOD.md's/ARTEMIS.md's 'StarForth dialect ONLY' applies to a thin
administrative/observational word surface; the underlying mechanics are meant
to be C99"* — this is in bounds. **Confirm the ruling still holds (O5);** D1
depends entirely on it.
### D2. Each VM's own heartbeat tick is the clock
`HeartbeatState` is embedded per-VM (`include/vm.h:280-298`). `tick_count`
advances only from `vm_tick()`, driven by that VM's own word executions
(`src/physics_execution_hooks.c:172-177`). Artemis's blocks therefore age on
Artemis's own tick. No cross-VM coupling, nothing injected, nothing to drive.
One clock, as the standing rule requires.
**Known consequence:** an idle VM does not tick, so its blocks and messages do
not age. For words this is explicitly intentional and documented as such. For
**messages it is a genuine problem** — a TTL that stops while Hermes is idle
is not a TTL. See O3.
### D3. The insertion point already exists
`vm_tick()` (`src/starkernel/vm/vm_runtime.c:145-149`) carries two
commented-out "future plugin" slots. That is the natural home for
`artemis_heat_tick()` and later `hermes_heat_tick()`.
---
## Phase 1 — Artemis block heat
**Goal:** block heat ages against Artemis's heartbeat, accumulates on access,
and reaps at zero, with an inferred slope replacing the hardcoded constant.
1. **New kernel-only files** (`#ifdef __STARKERNEL__`), following
`capsule_vm_physics.c`'s established shape:
- `include/starkernel/artemis_heat_physics.h`
- `src/starkernel/capsule/artemis_heat_physics.c`
State: per-block `{ heat_q48, last_decay_tick }` across `ART-DATA-BLKS`
(22,998 entries), plus the `ArtemisHeatWindow` the design doc specifies —
`ArtemisHeatSample { uint32_t live_count; uint32_t reaped_since_last; }`,
`ARTEMIS_HEAT_WINDOW_DEPTH` proposed 64.
2. **Lazy decay on access** — resolve heat at `BLK-FETCH` / `BLK-HEAT@` with
the exact `elapsed_ticks` pattern above, then apply the accumulation term.
3. **Bounded sweep** — `artemis_heat_tick()`, called from `vm_tick()`, gated
to Artemis's VM, processing a fixed batch per tick with a resumable cursor.
Mirror `vm_tick_apply_background_decay()` directly. This is what allows
reap to happen at all without an O(n) stall.
4. **Adaptive slope** — `blk_decay_slope_q48` fit from the window; when the
window is unwarmed, **skip the update rather than substituting a default**
(the discipline both existing engines use, and the specific trap that
froze VM-fleet physics at zero twice — see rev l and rev m of the fleet
doc).
5. **New FORTH primitives** — `BLK-DECAY-SLOPE@`, plus a status word
mirroring `VM-PHYSICS-STATUS`. Registration pattern is
`void (*)(VM *vm)` + `register_word(vm, "NAME", fn)`; see
`src/word_source/q48_words.c:76,204` for the minimal example and
`mama_forth_words.c:1091` for the kernel/MAMA variant (note it registers
into both the FORTH and MAMA vocabularies — follow that duplication).
6. **Capsule changes** — `ART-COOL` (block 4139) stops being a full sweep;
`ART-TICK` (block 4140) is retained for manual diagnostics. Both blocks are
editable — the hard-locked set is 4110–4113 plus 4128 and 4137. Any new
blocks belong in **4200–4299** (genuinely unclaimed; 4175–4199 is nominally
Hermes extension space).
### Available Q48.16 vocabulary
`Q.+ Q.- Q.* Q./ Q.ABS Q.NEG Q.LOG Q.EXP Q.SQRT Q.FROM-INT Q.TO-INT Q.1 Q.0
Q.SCALE Q.= Q.< Q.> Q.0= Q.MAX Q.MIN Q.PRINT` — all registered from
`src/word_source/q48_words.c:200-223`. There is **no** `Q.MOD` or remainder
word; if decay must be exactly conservative, that arithmetic has to come from
C. (Note `src/word_source/q48_16_words.c` exists but has no register function
and is not referenced by `word_registry.c` — dead code, do not follow it.)
---
## Phase 2 — Hermes message and channel heat
Same mechanism, after Artemis is proven. Per
`HERMES-MESSAGE-CHANNEL-PHYSICS-DESIGN-20260708.md`: `hermes_heat_physics.c`,
primitives `MSG-DECAY-SLOPE@` / `CH-DECAY-SLOPE@`, sampling hooked at
`MSG-COOL-ALL` (block 4108) and `CH-COOL-ALL` (block 4114).
**Behavioural difference from blocks:** messages are pure TTL — born hot,
decay only, never refreshed on access. No accumulation term. That is the whole
distinction Bob drew between the two.
**Correct that design doc before implementing it.** It proposes calling
`hermes_heat_tick()` from *"`HERMES-TICK`'s existing heartbeat path"* and
contrasts this against Artemis as having the harder problem for lacking one.
**That premise is false** — verified 2026-08-02: `HERMES-TICK` has no
heartbeat path either. Both VMs are in identical positions, and the doc's
sizing of the two efforts is wrong as written.
Message nodes are 8 cells (`HERMES.md:305-313`) with heat at cell 5. A
`last_decay_tick` needs either a 9th cell or C-side storage — C-side, per D1.
---
## Phase 3 — K participation (deferred; needs design, not wiring)
`.claude/ARTEMIS.md` requires that blocks participate in K≡1.0: *"Every
logical block's heat contributes to Artemis's K total. Reap must credit K
back. Alloc must charge K correctly."* Today nothing does, and closing it is
not a small job:
- **The current model is not conservative by construction.** `BLK-ALLOC` and
`BLK-FETCH` mint `Q.1` from nothing, `ART-COOL` destroys heat, `BLK-FREE`
zeroes it. 22,998 blocks at `Q.1` sums to 22,998.0, not 1.0.
- **`K-FLEET`, `K-LOCAL@`, and `K-CONSERVED?` no longer exist** — deleted in
commit `9323f776` along with `fleet-k.4th` and `compudynamics.4th`. The BAM
acceptance doc's "wire `ART-K-TOTAL` into `K-FLEET`" item refers to words
that are gone. Fleet conservation today is C-side only
(`vm_physics_conserved()`, 5% tolerance, `capsule_vm_physics.c:456`).
- Proper normalization is the **Logical BAM**, which is `FUTURE MATERIAL` in
`ARTEMIS.md` and gated behind Captain Bob explicitly reopening it.
Contrast with VM-fleet physics, which *is* conservative by construction via a
single balanced primitive `vm_physics_transfer(from, to, amount)` — the sum
invariant holds by induction rather than by enforcement. Blocks have no
equivalent primitive. Designing one is real work.
**Stale text to fix:** `VM-PHYSICS-DYNAMIC-FLEET-DESIGN-20260705.md`'s
"Fold-in question" section states that conservation does *not* generalize to
messages or blocks. The 2026-07-09 corrections on both sibling docs overturn
this, citing `.claude/ARTEMIS.md` and `.claude/HERMES.md` as authoritative.
The corrections win; the stale passage should be amended so it stops
contradicting them.
---
## Open questions — answer before writing code
**O1.** `Q-DECAY` lives in block **4110, marked ★ HARD LOCKED** in
`MANIFEST.md`. The block physics design doc says delete it. The manifest says
don't touch that block. Which wins — delete, or leave it and shadow it?
**O2.** Linear-subtractive or multiplicative decay for blocks? The word model
is linear; Artemis is currently multiplicative. This determines whether reap
ever fires.
**O3.** Idle-VM TTL. If Hermes executes no words it accrues no ticks, so
messages never expire. Accept this (TTL means "ticks of that VM's own
execution"), or does message aging need a different clock source? Note that
any answer involving a second clock conflicts with the one-clock rule.
**O4.** `BLK-FETCH` currently *resets* heat to `Q.1`. Change to increment
(frequency-sensitive, matches words, matches "how frequently" as the stated
signal) or keep reset (recency-only)?
**O5.** Confirm the C99-mechanics / FORTH-surface split still holds. D1 —
and therefore the whole no-bridge-needed argument — depends on it.
**O6.** `ARTEMIS_HEAT_WINDOW_DEPTH` (proposed 64) and the concrete slope-fit
function are both explicitly unresolved in the design docs. Start with the
proposed values and instrument, or settle them up front?
---
## Known risk
`ARTEMIS-BAM-ACCEPTANCE-20260703.md` records an **unfixed riscv64
heartbeat/compile race**: identical binaries alternately pass and fail
`TRIPOD-TEST` with `execute_colon_word: NULL cell in 'K-PUSH' after
'(start)'`, discriminated by boot-time heartbeat tick alignment (DoE row 59
reading `...,107,10,33,...` passing versus `...,107,9,32,...` failing).
This plan adds work inside `vm_tick()`, which shifts tick alignment. **Expect
to encounter this race.** It was reported and never fixed — "awaiting Captain
Bob's direction."
---
## Verification
1. `build/tools/mkcapsule --lint capsules` → 26 files, 0 violations.
2. Hosted build clean, zero warnings under `-Wall -Werror`.
3. Three-arch QEMU acceptance — `make -f Makefile.starkernel ARCH=<arch> clean
qemu` for amd64, aarch64, riscv64, one at a time, foreground. This is the
only acceptance authority for kernel changes.
4. **Behavioural proof that decay is live:** allocate blocks, execute
unrelated Artemis words to advance Artemis's own tick count *without ever
calling `ART-TICK` manually*, then confirm heat has dropped and cold blocks
have reaped. This is impossible today — nothing ages — so it is a genuine
discriminator.
5. **Slope responds to real churn** (the design doc's own test): drive block
churn at two different rates across separate boots and confirm
`blk_decay_slope_q48` converges to different values rather than sitting at
its seed.
6. **`dict_hash` must stay byte-identical across all three architectures.**
Established baseline: Hera `0x83c2c109100e2ed6`, Artemis
`0x5284ea5cd0f9983c`, Hermes `0x4159dcb326d79759`, Mama
`0xc88c3c1db6ef601b`. Any divergence means a wall-clock dependency has
crept in — precisely the bug class rev t of the fleet doc fixed.
## Not in scope
- Logical BAM / Physical BAM split, thermal zones (Phase 3 blocker,
`FUTURE MATERIAL`, requires explicit reopening)
- ACL Phase 8 / PKI — standing instruction is **STOP before starting**
- Any commit without Captain Bob asking for one
---
*Governing documents for this work, to be read completely before starting:*
`.claude/ARTEMIS.md`, `.claude/TRIPOD.md`, `.claude/HERMES.md`,
`.claude/CLAUDE.md`, and the four sibling design docs named in the header.
docs/working/archive/superseded/2026-08-tripod/03-architecture-README.md
+22
View File
@@ -0,0 +1,22 @@
> **SUPERSEDED (docs triage 2026-10-14).** The index of the old docs/03-architecture/ folder; its parts moved (word-acl to docs/reference/, tripod here).
> Instead: docs/README.md. Moved here from `docs/03-architecture/README.md`.
# docs/03-architecture/
Architecture-constraint documents for LithosAnanke subsystems that are still
being designed or built. These are binding constraints for anyone (human or
Claude Code) touching the affected code, not general background reading.
- **[tripod/](tripod/)** — Architectural contracts for the three Tripod VMs
(Hera, Hermes, Artemis): roles, dispatch rules, the compudynamic K≡1.0
invariant, and explicit prohibitions.
- **[word-acl/](word-acl/)** — Design doc for the word-level ACL system
(`capsules/ACL.4th`, `capsules/zuse.4th`, `src/word_source/acl_words.c`).
## See also
- [`docs/working/archive/historical/2026-07-birthing/`](../birthing/) — VM birthing plan/status that implements
the Tripod contract described in `tripod/TRIPOD.md`.
- [`docs/working/archive/historical/2026-10-lithosananke-orientation.md`](../lithosananke/README.md) — kernel-branch
overview these designs build on.
- [Project root](../../README.md)
docs/working/archive/superseded/2026-08-tripod/ARTEMIS.md
+190
View File
@@ -0,0 +1,190 @@
> **SUPERSEDED (docs triage 2026-10-14).** The Tripod as Hera/Hermes/Artemis; the Hermes VM was removed (messaging moved into the kernel as kernel-Hermes) and the Tripod is Hera/Artemis/Hestia.
> Instead: docs/fabric/FABRIC-3.5.md and FABRIC-3.6.md. Moved here from `docs/03-architecture/tripod/ARTEMIS.md`.
# ARTEMIS.md — Artemis VM Architecture
# StarshipOS / StarForth — Captain Bob (Robert Allan James)
# This document constrains Claude Code behavior. Read it completely before touching Artemis.
---
## Language Constraint — Non-Negotiable
All Artemis implementation is in StarForth dialect ONLY.
This includes:
- Data structures
- Constants
- Variables
- Control flow
- Everything
C99 is forbidden unless explicitly blocked in StarForth AND explicit written permission
is given by Captain Bob for that specific construct. "I could not figure out how to do
this in StarForth" is not permission. Ask first. Wait for the answer.
If you are about to write a C struct, a #define, or a C variable — stop.
Implement it in StarForth or ask.
---
## When Stuck — Non-Negotiable
If you are stuck, STOP.
Do not confabulate.
Do not invent dictionary words that do not exist.
Do not fabricate primitives.
Do not guess at behavior.
Do not work around the problem silently.
STOP. Report exactly what is blocking you. Wait for instruction.
Silent confabulation has caused git resets on a project with a hard patent deadline.
It is not acceptable under any circumstances.
---
## What Artemis Is
Artemis is memory. She owns the external attached virtual disk image and everything
that lives on it. She manages block storage compudynamically.
Artemis does **not** move messages — that is Hermes.
Artemis does **not** make VM lifecycle decisions — that is Hera.
Artemis does **not** own the internal ramdisk — that is kernel territory.
---
## Storage Topology
### Internal Ramdisk — Not Artemis Scope
- Baked into each VM at build time by mkcapsule.c
- LBN 0–3071
- Kernel territory
- Every VM carries its own ramdisk
- Artemis does not touch it, manage it, or know about it
### External Attached Virtual Disk Image — Artemis Territory
- Mounted virtual disk image, attached to the QEMU instance
- Artemis owns this exclusively
- LBN boundaries to be determined as Artemis is built
- Nothing persists between runs yet — that is a future chapter
### Nothing Persists Between Runs Yet
Persistence across runs is a future chapter. Do not implement it yet.
Do not design around it. The current model is: VM starts, Artemis initializes
her free map, VM stops, everything is gone. That is correct for now.
---
## What Artemis Owns
- The free map over her LBN range on the external disk image
- Block fetch and persist operations on the external disk image
- ACL lists — structured as Artemis-managed blocks (see ACL section)
- Cold capsule store — evicted from Hermes cache (future chapter)
- Block metadata
---
## Compudynamic Block Lifecycle
Artemis manages blocks compudynamically. Same model as everything else in StarshipOS:
- Block enters active use — born hot
- Cools over time with each heartbeat tick
- Frequently accessed blocks stay warm (heat refreshed on access)
- Cold blocks are candidates for eviction or reuse
- Block reap must rebalance K correctly
Artemis is the hardest VM to design because she has three interleaved
compudynamic concerns:
1. The free map — which LBNs are available
2. Block thermal state — how hot is each active block
3. K≡1.0 — fleet conservation across all block operations
These must be designed together. Do not implement one without the others.
Do not begin implementation without explicit instruction from Captain Bob.
---
## ACL — Access Control Lists
### Current State
ACL is a toggle. It is currently OFF.
A `\` comment disables ACL enforcement. When commented out, zuse is the
implicit and only user. No enforcement occurs.
```forth
\ ACL-ENFORCE ( disabled — zuse only )
```
### Structure
ACL lists must be structured as Artemis-managed blocks from day one.
Even before Artemis is fully built, ACL data structures must assume
they will live in Artemis-managed storage. No shortcuts that require
redesign later.
### When ACL Turns On
ACL enforcement becomes mandatory when VMs outside the Tripod arrive.
Until then the toggle stays off.
When enforcement is enabled:
- Every VM has an identity
- Every operation is checked against the ACL
- Artemis is the authority — she holds the lists
### Security Model
Security grows with the threat surface. Not before it.
The architecture is always structurally correct — enforcement is simply
not active yet. No retrofit needed when the time comes.
Toggling ACL on must require zero structural changes to the codebase.
Only the `\` comment is removed. If toggling ACL requires anything else,
the implementation is wrong.
---
## What Is Not Artemis
- **Internal ramdisk** — kernel territory, not Artemis scope
- **Message routing** — that is Hermes
- **VM lifecycle** — that is Hera
- **Persistence across runs** — future chapter, not now
- **Cold capsule eviction from Hermes** — future chapter, not now
---
## Compudynamic Invariant
Every block Artemis manages participates in K≡1.0.
Block fetch, persist, and reap operations must rebalance K correctly.
Artemis's thermal contribution to the fleet is the sum of her active block heat.
Do not write code that breaks K≡1.0 and add a comment explaining why it's okay.
---
## Current Scope
Artemis is not yet started. Before any implementation begins:
1. Captain Bob must give explicit instruction to begin
2. The three compudynamic concerns (free map, block thermal state, K≡1.0)
must be designed together before any code is written
3. ACL data structures must be Artemis-block-shaped from first line of code
4. Storage boundaries (LBN ranges on external disk image) must be confirmed
**Do not begin implementation without explicit instruction.**
Future chapters (do not implement):
- Persistence across runs
- Cold capsule store from Hermes
- Full ACL enforcement
- LBN range expansion
---
*This document is authoritative. If it conflicts with something in the codebase,
the codebase is wrong.*
docs/working/archive/superseded/2026-08-tripod/HERMES.md
+253
View File
@@ -0,0 +1,253 @@
> **SUPERSEDED (docs triage 2026-10-14).** The Tripod as Hera/Hermes/Artemis; the Hermes VM was removed (messaging moved into the kernel as kernel-Hermes) and the Tripod is Hera/Artemis/Hestia.
> Instead: docs/fabric/FABRIC-3.5.md and FABRIC-3.6.md. Moved here from `docs/03-architecture/tripod/HERMES.md`.
# HERMES.md — Hermes VM Architecture
# StarshipOS / StarForth — Captain Bob (Robert Allan James)
# This document constrains Claude Code behavior. Read it completely before touching Hermes.
---
## Language Constraint — Non-Negotiable
All Hermes implementation is in StarForth dialect ONLY.
This includes:
- Data structures
- Constants
- Variables
- Control flow
- Everything
C99 is forbidden unless explicitly blocked in StarForth AND explicit written permission
is given by Captain Bob for that specific construct. "I could not figure out how to do
this in StarForth" is not permission. Ask first. Wait for the answer.
If you are about to write a C struct, a #define, or a C variable — stop.
Implement it in StarForth or ask.
---
## What Hermes Is
Hermes is the messenger. He moves messages between VMs and manages channels.
That is his complete contract.
Hermes does **not** store persistent data — that is Artemis.
Hermes does **not** make lifecycle decisions — that is Hera.
Hermes does **not** route by load or heat — dispatch is capability-based, always.
Capsules are baked into the OS blob by mkcapsule.c at build time.
Hermes has no capsule cache, no fetch logic, no repository concern.
That is a future chapter.
---
## The Event Loop
Hermes runs a compudynamic event loop. Every message is a compudynamic object:
- Born hot (full heat at creation)
- Cools each heartbeat tick
- ACK received → clean death, K redistributed to fleet
- NACK received → requeue with reduced heat, or reaped if heat floor reached
- TTL expires (heat → 0) → Hermes reaps the message, K rebalanced
There is no special timeout logic. The physics handles it.
A message that nobody answers simply cools to death.
---
## Message Structure
Every message carries:
```
type — work type tag (determines recipient)
sender — originating VM identity
recipient — destination VM identity
payload — message body
heat — current thermal value (compudynamic)
sequence — monotonic sequence number
```
Point-to-point only. Sender addresses recipient directly.
Hermes delivers and tracks thermal state. He does not inspect payload.
---
## ACK/NACK Protocol
- ACK → message dies cleanly, K redistributed
- NACK → message requeued with reduced heat
- No response → message cools naturally to death via TTL
The binary receipt of a response IS the ACK where applicable.
No separate acknowledgement frame needed in those cases.
---
## PubSub — Channels
Hermes implements a publish/subscribe model. Topics are called **channels**.
### The Common Channel
There is one permanent channel: `COMMON`.
- Always exists, always hot
- Any VM can publish to it
- Used for negotiation and channel establishment only
- Not for business transactions — those happen on ephemeral channels
### Ephemeral Channels
Ephemeral channels are compudynamic objects — born hot, reaped when cold or closed.
**Channel structure:**
```
channel_id — owner VM identity + monotonic sequence number
owner — VM that created the channel (always the responder)
members[] — participant list
heat — compudynamic, born hot at creation
state — NEGOTIATING | OPEN | CLOSING
```
### The Negotiation Protocol
```
1. X publishes request on COMMON — born hot, TTL ticking
2. Y sees request, responds yes/no on COMMON
— No response → message cools to death → implicit NACK to X
3. Y accepts → Y creates ephemeral channel
→ Y publishes channel_id to X over COMMON
4. X and Y conduct business on ephemeral channel
5. Business concluded → owner Y reaps channel → K rebalanced
```
Y owns channel creation and destruction. Always the responder, never the requester.
A channel dying of cold (abandoned transaction) is not an error — the physics handles it.
### Multi-Party Channels
Channels are bilateral by default. The owner may invite additional VMs:
```
1. Owner publishes invite to candidate VM on COMMON
2. Candidate accepts or declines on COMMON — same TTL/cool protocol
3. Accept → candidate added to members[]
4. Decline or timeout → candidate never joins, channel continues bilateral
```
Multi-party is designed for but not implemented yet.
Do not implement invite logic until explicitly instructed.
The members[] structure must exist from the start to avoid future redesign.
### Channel Identity
`channel_id` = owner VM identity + monotonic sequence number.
Unforgeable by non-owners. Hermes mints it. No VM constructs its own channel_id.
### Compudynamic Invariant for Channels
Channels participate in K≡1.0.
An open channel with active messages holds heat.
Channel reap must redistribute K correctly to the fleet.
A channel cannot be reaped while messages on it are still in flight —
drain first, then reap.
---
## What Is Not Hermes
- **Capsule cache** — future chapter, not now
- **Remote fetch** — future chapter, not now
- **UDP server / client** — future chapter, not now
- **Instance authentication** — future chapter, not now
- **Persistent block storage** — that is Artemis
- **VM lifecycle decisions** — that is Hera
- **Heat-based routing** — wrong model, never implement
---
## Compudynamic Invariant
Every object Hermes manages — messages and channels — participates in K≡1.0.
Fleet K includes the thermal contribution of in-flight messages and open channels.
Hermes reaping a message or channel must rebalance K correctly.
Do not write code that breaks K≡1.0 and add a comment explaining why it's okay.
---
## Current Scope
1. Compudynamic event loop — message emit, drain, TTL, ACK/NACK, reap
2. COMMON channel — permanent, always hot, negotiation only
3. Ephemeral channels — compudynamic lifecycle, owner=responder, bilateral default
4. Channel negotiation protocol — request on COMMON, Y creates, Y reaps
5. members[] structure — present from the start, multi-party invite not yet implemented
Capsule management is a future chapter.
Remote repository is a future chapter.
Multi-party channel invite is a future chapter.
---
## v1 Block Map — LOCKED
Blocks 4110–4113 are Artemis. NEVER touch them.
```
4100 Constants: event codes, channel states, node sizes, arena sizes
4101 Arena CREATE: MSG-ARENA CH-ARENA MBR-ARENA; free-list vars; MSG-SEQ CH-ACTIVE
4102 MSG-INIT-FREE + CH-INIT-FREE
4103 MBR-INIT-FREE + MSG-ALLOC + MSG-FREE-NODE
4104 CH-ALLOC + CH-FREE-NODE + MBR-ALLOC + MBR-FREE-NODE
4105 Message field accessors: MSG-TYPE@/! MSG-FROM@/! MSG-TO@/! MSG-PADDR@/! MSG-PLEN@/! MSG-HEAT@/! MSG-SEQ@/!
4106 Channel+member accessors: CH-ID@/! CH-OWNER@/! CH-STATE@/! CH-HEAT@/! CH-MBRS@/! CH-NEXT@/! MBR-NEXT@ MBR-VM@
4107 IDX>NAME + MSG-DELIVER + MSG-SEND
4108 VARIABLE MSG-SCAN + MSG-COOL-ONE + MSG-COOL-ALL
4109 MSG-REAP
---- 4110–4113: ARTEMIS — DO NOT TOUCH ----
4114 VARIABLE CH-SCAN + CH-COOL-ALL
4115 CH-REAP-SAFE
4116 VARIABLE COMMON-CH + COMMON-INIT + HERMES-TICK
4117 EVENT-EMIT + EVENT-WAIT + EVENT-DRAIN (backward compat)
4118 HERA-NOTIFY-SPAWN + HERA-NOTIFY-KILL + HERA-DISPATCH-ONE + HERA-DISPATCH
4119 CH-REQUEST + CH-ACCEPT + CH-CLOSE
4120 WELCOME + CD-INIT
```
### Node layouts (cells)
**Message node — 7 cells:**
- 0: type (in-use) / next-free ptr (free)
- 1: sender VM index
- 2: recipient VM index
- 3: payload addr (FORTH string addr)
- 4: payload len
- 5: heat (Q48.16)
- 6: seq
**Channel node — 6 cells:**
- 0: channel-id (in-use) / next-free ptr (free)
- 1: owner VM index
- 2: state (0=NEGOTIATING 1=OPEN 2=CLOSING)
- 3: heat (Q48.16)
- 4: members-head (→ member list)
- 5: next-active (active channel list link)
**Member node — 2 cells:**
- 0: next-ptr
- 1: vm-id
### Physics
Cooling constant: Q-DECAY (65208) from compudynamics.4th — same as VM-DECAY-ONE.
Messages and VMs cool at identical rates. Fleet is thermodynamically consistent.
---
*This document is authoritative. If it conflicts with something in the codebase,
the codebase is wrong.*
docs/working/archive/superseded/2026-08-tripod/README.md
+31
View File
@@ -0,0 +1,31 @@
> **SUPERSEDED (docs triage 2026-10-14).** The Tripod as Hera/Hermes/Artemis; the Hermes VM was removed (messaging moved into the kernel as kernel-Hermes) and the Tripod is Hera/Artemis/Hestia.
> Instead: docs/fabric/FABRIC-3.5.md and FABRIC-3.6.md. Moved here from `docs/03-architecture/tripod/README.md`.
# docs/working/archive/superseded/2026-08-tripod/
Architectural constraints for the three StarForth VMs that make up the
Tripod. These documents are binding — read the relevant one in full before
touching any Tripod code. Violating the mythological contract they describe
is treated as an architectural error, not a style preference.
- **[TRIPOD.md](TRIPOD.md)** — The Tripod as a whole: Hera (VM-HERA=0,
governor — spawns/manages/reaps VMs), Hermes (VM-HERMES=1, messenger —
moves events), Artemis (VM-ARTEMIS=2, memory — owns block storage).
Covers capability-based dispatch (no heat-based routing), the CD-TICK
heartbeat, the K≡1.0 compudynamic conservation invariant, the DoE test
fixture's role, and explicit prohibitions (no heat-based VM routing, no
primitives added without instruction, no `BLOCK` calls outside the
0–3071 ramdrive range).
- **[HERMES.md](HERMES.md)** — Hermes VM architecture: the messenger role.
- **[ARTEMIS.md](ARTEMIS.md)** — Artemis VM architecture: the memory/block-
storage role.
All three documents share a non-negotiable constraint: Tripod implementation
is StarForth dialect only — no C99 without Captain Bob's explicit written
permission for that specific construct.
## See also
- [`docs/working/archive/historical/2026-07-birthing/`](../../birthing/) — birthing plan and QEMU acceptance
logs for the Hera-spawns-Hermes/Artemis constellation described here.
- [Project root](../../../README.md)
docs/working/archive/superseded/2026-08-tripod/TRIPOD.md
+191
View File
@@ -0,0 +1,191 @@
> **SUPERSEDED (docs triage 2026-10-14).** The Tripod as Hera/Hermes/Artemis; the Hermes VM was removed (messaging moved into the kernel as kernel-Hermes) and the Tripod is Hera/Artemis/Hestia.
> Instead: docs/fabric/FABRIC-3.5.md and FABRIC-3.6.md. Moved here from `docs/03-architecture/tripod/TRIPOD.md`.
# TRIPOD.md — Architectural Constraints for the Tripod VMs
# StarshipOS / StarForth — Captain Bob (Robert Allan James)
# This document constrains Claude Code behavior. Read it completely before touching any Tripod code.
---
## The Tripod: What It Is
Three StarForth VMs constitute the Tripod. They are not generic workers. Each holds a
specific architectural contract expressed by its mythological name. Violating the
mythological contract is an architectural error, not a style preference.
| VM | ID | Contract |
|------------|-------------|-----------------------------------------------|
| Hera | VM-HERA=0 | Governor. Spawns, manages, and reaps VMs. |
| Hermes | VM-HERMES=1 | Messenger. Moves events between VMs. |
| Artemis | VM-ARTEMIS=2| Memory. Owns block storage and persistence. |
**Hera is queen. She does not do work. She governs who exists and who doesn't.**
**Hermes moves messages. He does not store anything.**
**Artemis holds things. She does not route anything.**
If you find yourself putting event dispatch logic in Artemis, or storage logic in Hermes,
or work execution logic in Hera — stop. You have the wrong VM.
---
## The Immediate Goal
Hera must spawn Hermes and Artemis at boot, and all three must cooperate
compudynamically under K≡1.0 conservation.
That is the complete scope. Nothing else.
Acceptance criteria:
1. Hera boots and initializes her own compudynamic state.
2. Hera spawns Hermes. Hermes initializes his own compudynamic state.
3. Hera spawns Artemis. Artemis initializes her own compudynamic state.
4. All three VMs are running. Fleet K = sum of individual VM K values. K≡1.0 holds.
5. Hera can reap (cleanly terminate) a child VM.
6. The system is stable across a long-running test workload.
---
## Compudynamics: Dual Role — Read This Carefully
Compudynamics serves TWO roles in this system. Do not conflate them.
### Role 1: Intra-VM Physics Engine
Inside each VM, Compudynamics drives the adaptive heartbeat:
- 7-loop rolling window
- K conservation arithmetic
- Phase operators on the heartbeat signal
Every VM — Hera, Hermes, Artemis — runs its own compudynamic engine internally.
This is autonomous. No VM manages another VM's internal heartbeat.
### Role 2: Hera's VM Lifecycle Driver
Hera observes the fleet K picture and uses it to make **lifecycle decisions**:
- Is a child VM healthy?
- Is a child VM starved?
- Does a child VM need to be spawned, suspended, or reaped?
Hera does NOT use fleet K to route work. She uses it to govern existence.
**The fleet K view is lifecycle telemetry. It is not a dispatch mechanism.**
---
## Dispatch: Capability-Based, Always
Work routing is determined by work type. It is not determined by heat, K value,
VM load, or any compudynamic signal.
| Work type | Destination |
|-----------------|-------------|
| Block I/O | Artemis |
| Event emit/drain| Hermes |
| VM lifecycle | Hera |
`VM-EXEC` takes a VM name. The caller names the correct VM for the work type.
There is no routing algorithm. There is no dispatch table keyed on heat.
There is no "send to the hottest VM." That model is wrong and must not be implemented.
---
## CD-TICK: What It Is and Is Not
CD-TICK advances the compudynamic heartbeat. That is all it does.
CD-TICK is **not**:
- A scheduler
- A timeslicer
- A run-queue manager
- A work dispatcher
- A router
If you are calling CD-TICK to decide where work goes, you are using it wrong.
---
## The DoE (Design of Experiments) Test Fixture
The DoE campaign (PHASE6-TEST, K-conservation CSV logging, 38,400-run Latin Square)
is a **stress and measurement fixture**. It exists to:
- Validate K≡1.0 conservation under long-running diverse workloads
- Generate telemetry for the SSRN paper
- Provide a long-runtime acceptance test for the compudynamic physics
The DoE is **not**:
- A description of what the system does in production
- A feature
- A model for how work gets dispatched
`DOE-WORK` must never appear in the production dispatch path.
If you see `DOE-WORK` being dispatched by anything other than the test fixture, remove it.
The DoE is a good test precisely because it is long-running and diverse.
It exercises the physics. It does not define the system.
---
## What Not To Do — Explicit Prohibitions
1. **Do not implement heat-based VM routing.** It causes starvation. It is wrong.
2. **Do not pin Hera's heat floor.** That is a band-aid on a wrong model.
3. **Do not put DOE-WORK in the production dispatch path.**
4. **Do not call CD-TICK a scheduler** in comments, docs, or variable names.
5. **Do not call the fleet K view a router** in comments, docs, or variable names.
6. **Do not add primitives to StarForth without explicit instruction.**
NIP is not registered. TUCK is not registered. Assume nothing is available
that is not in the verified primitive table. Use SWAP DROP instead of NIP.
7. **Do not call BLOCK on LBNs outside the ramdrive range (0–3071).**
LBN 4501–4599 are Artemis metadata space. They are not in the kernel ramdrive.
Use ART-STATUS to touch the free map. Do not call ART-SELF-TEST in acceptance tests.
8. **Do not exceed 64 bytes per line or 16 lines per block in capsule files.**
mkcapsule.c enforces this at build time. Violations will fail the build.
---
## The Invariant
K≡1.0 is conserved across the fleet at all times.
Fleet K = sum of all VM K values.
This is a physics constraint, not a software convention.
Do not write code that breaks it and then add a comment explaining why it's okay.
---
*This document is authoritative. If it conflicts with something in the codebase,
the codebase is wrong.*
---
## Language Constraint — Non-Negotiable
All Tripod implementation is in StarForth dialect ONLY.
This includes:
- Data structures
- Constants
- Variables
- Control flow
- Everything
C99 is forbidden unless explicitly blocked in StarForth AND explicit written permission
is given by Captain Bob for that specific construct. "I could not figure out how to do
this in StarForth" is not permission. Ask first. Wait for the answer.
If you are about to write a C struct, a #define, or a C variable — stop.
Implement it in StarForth or ask.
---
## When Stuck — Non-Negotiable
If you are stuck, STOP.
Do not confabulate.
Do not invent dictionary words that do not exist.
Do not fabricate primitives.
Do not guess at behavior.
Do not work around the problem silently.
STOP. Report exactly what is blocking you. Wait for instruction.
Silent confabulation has caused git resets on a project with a hard patent deadline.
It is not acceptable under any circumstances.
docs/working/archive/superseded/L8_WIRING_PLAN.md
+358
View File
@@ -0,0 +1,358 @@
> **SUPERSEDED (docs triage 2026-10-14).** Flagged superseded 2026-10-09: the plan for wiring L8, since built.
> Instead: docs/working/experiments/02-experiments/L8_WIRING_IMPLEMENTATION_SUMMARY.md. Moved here from `docs/working/experiments/02-experiments/L8_WIRING_PLAN.md`.
<!-- Moved from docs/02-experiments/L8_WIRING_PLAN.md to docs/working/archive/superseded/L8_WIRING_PLAN.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ SUPERSEDED (flagged 2026-10-09).** The wiring plan was executed; superseded by
> `L8_WIRING_IMPLEMENTATION_SUMMARY.md` and `L8_WIRING_CODE_REVIEW_GUIDE.md`. Kept as a
> record of the plan.
# L8 Control Wiring Implementation Plan
**Status**: Infrastructure exists but execution paths are not gated
**Date**: 2025-12-05
**Issue**: L8 Jacquard mode selector computes 4-bit control signals (L2/L3/L5/L6) but feedback loops ignore them
---
## Executive Summary
The L8 Steady State Machine (SSM) infrastructure is **90% complete** but the final 10% is critical:
✅ **Complete**:
- L8 state machine (`ssm_jacquard.c`) computes 4-bit mode (C0-C15) based on entropy/CV/temporal metrics
- `ssm_apply_mode()` sets L2/L3/L5/L6 flags in `vm->ssm_config`
- `vm_heartbeat_update_l8()` updates L8 state every heartbeat cycle (lines 993-1045 in `vm.c`)
- Mode transitions logged with debug info
❌ **Missing**:
- Feedback loop execution paths don't check `ssm_config` flags before running
- L2/L3/L5/L6 run unconditionally despite L8 setting mode bits
- Result: L8 is a "supervisor without authority"
---
## Current Architecture (From DoE Analysis)
From `Makefile:112-127` and `ssm_jacquard.h:32-51`:
| Loop | Name | Status | Control |
|------|------|--------|---------|
| L1 | Heat Tracking | ALWAYS OFF | Disabled (harmful in 86% of configs) |
| L2 | Rolling Window | **NEEDS GATING** | L8 bit 3 (entropy-driven) |
| L3 | Linear Decay | **NEEDS GATING** | L8 bit 2 (temporal locality) |
| L4 | Pipelining Metrics | ALWAYS OFF | Disabled (harmful in 100% of configs) |
| L5 | Window Inference | **NEEDS GATING** | L8 bit 1 (CV-driven) |
| L6 | Decay Inference | **NEEDS GATING** | L8 bit 0 (CV + temporal) |
| L7 | Adaptive Heartrate | ALWAYS ON | Enabled (beneficial in 71% of configs) |
| L8 | Jacquard Selector | **IMPLEMENTED** | 4-bit mode controller (C0-C15) |
**Key Insight**: L1 and L4 are compile-time disabled. L7 is always-on. **L2/L3/L5/L6 need runtime gating** based on L8 mode bits.
---
## Required Changes (4 Locations)
### 1. L2: Rolling Window Recording (src/rolling_window_of_truth.c)
**Current Code** (`rolling_window_record_execution()` line 181):
```c
int rolling_window_record_execution(RollingWindowOfTruth* window, uint32_t word_id)
{
if (!window || !window->execution_history)
return -1;
window->execution_history[window->window_pos] = word_id;
// ... always executes
}
```
**Required Change**:
Add check at function entry or call site. Two options:
**Option A** (Preferred - Clean separation):
Modify call site in `src/vm.c:1652`:
```c
// Before (line 1652):
rolling_window_record_execution(&vm->rolling_window, word_id);
// After:
if (vm->ssm_config && ((ssm_config_t*)vm->ssm_config)->L2_rolling_window) {
rolling_window_record_execution(&vm->rolling_window, word_id);
}
```
**Option B** (Alternative - Gate inside function):
Modify `rolling_window_record_execution()` to accept VM pointer and check internally.
**Files to modify**:
- `src/vm.c:1652` - Call site in `execute_word()`
- `src/rolling_window_of_truth.c:181` - Consider adding VM* parameter for internal check
- `include/rolling_window_of_truth.h` - Update function signature if Option B chosen
---
### 2. L3: Linear Decay Application (src/physics_metadata.c)
**Current Code** (line 176):
```c
/* L8 FINAL INTEGRATION: L3 linear decay is always-on */
void physics_metadata_apply_linear_decay(DictEntry *entry, uint64_t elapsed_ns, VM *vm) {
if (!entry || !vm) {
return;
}
/* Frozen words do not decay */
if (entry->flags & WORD_FROZEN) {
return;
}
/* Don't bother with insignificant time intervals */
if (elapsed_ns < DECAY_MIN_INTERVAL) {
return;
}
// ... decay logic always executes
}
```
**Required Change**:
Add L3 gate check after frozen/interval checks:
```c
/* L8 FINAL INTEGRATION: L3 linear decay - runtime gated by Jacquard */
void physics_metadata_apply_linear_decay(DictEntry *entry, uint64_t elapsed_ns, VM *vm) {
if (!entry || !vm) {
return;
}
/* Frozen words do not decay */
if (entry->flags & WORD_FROZEN) {
return;
}
/* Don't bother with insignificant time intervals */
if (elapsed_ns < DECAY_MIN_INTERVAL) {
return;
}
/* L8 GATE: Check if L3 (linear decay) is enabled by Jacquard mode */
if (vm->ssm_config) {
ssm_config_t *config = (ssm_config_t*)vm->ssm_config;
if (!config->L3_linear_decay) {
return; /* L3 disabled by L8 mode selector */
}
}
// ... existing decay logic
}
```
**Files to modify**:
- `src/physics_metadata.c:176` - Add L3 gate check (line ~188-193)
- Comment update at line 175 to reflect runtime gating
**Call sites** (no changes needed, gate is internal):
- `src/vm.c:737` - `vm_tick_apply_background_decay()` (heartbeat loop)
- Other call sites inherit the gate
---
### 3. L5: Window Width Inference (src/inference_engine.c)
**Current Code** (line 538):
```c
/* === PHASE 2C: Window Width Inference === */
/* L8 FINAL INTEGRATION: Loop #5 always-on */
uint32_t inferred_width = find_variance_inflection(
trajectory,
traj_len,
current_variance
);
```
**Required Change**:
Gate inference execution with conditional block:
```c
/* === PHASE 2C: Window Width Inference === */
/* L8 FINAL INTEGRATION: Loop #5 runtime-gated by Jacquard */
uint32_t inferred_width = outputs->adaptive_window_width; /* Keep previous value as default */
if (inputs->vm && inputs->vm->ssm_config) {
ssm_config_t *config = (ssm_config_t*)inputs->vm->ssm_config;
if (config->L5_window_inference) {
/* L5 enabled - run inference */
inferred_width = find_variance_inflection(
trajectory,
traj_len,
current_variance
);
}
/* else: L5 disabled, keep previous adaptive_window_width */
}
```
**Files to modify**:
- `src/inference_engine.c:538-544` - Add L5 conditional gate
- Comment update at line 539 to reflect runtime gating
---
### 4. L6: Decay Slope Inference (src/inference_engine.c)
**Current Code** (line 546):
```c
/* === PHASE 2D: Decay Slope Inference === */
/* L8 FINAL INTEGRATION: Loop #6 always-on */
uint64_t inferred_slope = infer_decay_slope_q48(trajectory, traj_len);
```
**Required Change**:
Gate inference execution with conditional block:
```c
/* === PHASE 2D: Decay Slope Inference === */
/* L8 FINAL INTEGRATION: Loop #6 runtime-gated by Jacquard */
uint64_t inferred_slope = outputs->adaptive_decay_slope; /* Keep previous value as default */
if (inputs->vm && inputs->vm->ssm_config) {
ssm_config_t *config = (ssm_config_t*)inputs->vm->ssm_config;
if (config->L6_decay_inference) {
/* L6 enabled - run inference */
inferred_slope = infer_decay_slope_q48(trajectory, traj_len);
}
/* else: L6 disabled, keep previous adaptive_decay_slope */
}
```
**Files to modify**:
- `src/inference_engine.c:546-548` - Add L6 conditional gate
- Comment update at line 547 to reflect runtime gating
---
## Implementation Order (Recommended)
1. **L3 (Decay)** - Simplest, internal to one function
- Modify `src/physics_metadata.c:176-220`
- Add gate check after line 188
- Test with mode C0 (all off) vs C4 (L3 on)
2. **L5 (Window Inference)** - Medium complexity
- Modify `src/inference_engine.c:538-544`
- Preserve previous value when disabled
- Test early-exit behavior with mode C0 vs C2
3. **L6 (Decay Inference)** - Similar to L5
- Modify `src/inference_engine.c:546-548`
- Preserve previous value when disabled
- Test with mode C0 vs C1
4. **L2 (Rolling Window)** - Most complex (data structure)
- Decide on Option A (call-site gate) vs Option B (internal gate)
- Modify call site `src/vm.c:1652` OR function signature
- Test with mode C0 vs C8 (high entropy)
---
## Testing Strategy
### Unit Tests
For each loop, verify:
1. **Mode C0 (0000)**: All loops disabled → function returns early or skips logic
2. **Single-bit modes**: Enable one loop at a time (C1, C2, C4, C8)
3. **Top 5% modes**: C4, C7, C9, C11, C12 work correctly
### Integration Tests
1. **Mode transitions**: L8 changes mode → loops respond correctly
2. **Hysteresis**: Mode doesn't flap rapidly (5-tick delay)
3. **Metrics**: Verify entropy/CV/temporal drive correct mode selection
### Regression Tests
1. Run existing 936+ test suite with different L8 modes
2. Verify deterministic behavior (no variance across runs)
3. Check performance matches DoE predictions
---
## Validation Checklist
After implementation, verify:
- [ ] L2: `rolling_window_record_execution()` only executes when L2 bit set
- [ ] L3: `physics_metadata_apply_linear_decay()` returns early when L3 bit clear
- [ ] L5: `find_variance_inflection()` only called when L5 bit set
- [ ] L6: `infer_decay_slope_q48()` only called when L6 bit set
- [ ] L8: Mode transitions logged in debug output
- [ ] All existing tests pass with default mode (C0 or C12)
- [ ] DoE experiment can reproduce 128 configurations via L8 mode forcing
---
## Impact Assessment
**Lines of code changed**: ~40-60 lines across 3 files
**Risk**: Low (adding guards, not changing logic)
**Performance**: Neutral to positive (skips work when disabled)
**Correctness**: High confidence (L8 already validated via DoE)
---
## Open Questions
1. **Default mode**: Should VM initialize with C0 (minimal) or C12 (DoE optimal)?
- Current: `ssm_l8_init(..., SSM_MODE_C0)` in `vm.c:387`
- Top 5%: C4, C7, C9, C11, C12
- Recommendation: Start C0, let L8 adapt
2. **Mode override**: Should FORTH words exist to force L8 mode?
- `L8-MODE!` to set mode manually
- `L8-MODE@` to query current mode
- Useful for debugging/benchmarking
3. **Metrics exposure**: Should heartbeat CSV export include L8 mode?
- Currently exports: executions, heat, variance, slope
- Add: `l8_mode` column (0-15 integer)
---
## Success Criteria
**Definition of Done**:
1. All 4 feedback loops check `ssm_config` flags before executing
2. L8 mode selector controls L2/L3/L5/L6 runtime behavior
3. Mode transitions visible in debug logs
4. Existing test suite passes with default L8 mode
5. Can reproduce DoE configurations by forcing L8 modes 0-15
**Verification**:
```bash
# Build with L8 control wired
make clean && make fastest
# Test mode C0 (all off) - should be fastest, least adaptive
./build/amd64/fastest/starforth --benchmark 1000
# Test mode C15 (all on) - should be slowest, most adaptive
# (requires word to set L8 mode or environment variable)
# Run full test suite
make test
```
---
## Next Steps
**Before coding**:
1. Review this plan with stakeholders
2. Decide on L2 implementation strategy (Option A vs B)
3. Choose default L8 mode (C0 vs C12)
**During implementation**:
1. Follow order: L3 → L5 → L6 → L2
2. Test each loop individually before moving to next
3. Update comments to reflect runtime gating
**After completion**:
1. Run full DoE validation (128 configs × 300 reps)
2. Verify 0% algorithmic variance maintained
3. Document L8 control API for future developers
docs/working/archive/superseded/factorial-doe/guide.md
+265
View File
@@ -0,0 +1,265 @@
> **SUPERSEDED (docs triage 2026-10-14).** The 2^6 factorial DoE guide, flagged obsolete 2026-10-09.
> Instead: experiments/bare_metal/README.md for the DoE as it runs now. Moved here from `docs/working/experiments/02-experiments/factorial-doe/guide.md`.
<!-- Moved from docs/02-experiments/factorial-doe/guide.md to docs/working/archive/superseded/factorial-doe/guide.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ OBSOLETE (flagged 2026-10-09).** Describes the retired per-loop Makefile-toggle
> architecture, replaced by the L8 Jacquard runtime mode selector. Kept for historical record.
# StarForth Complete 2^6 Factorial Design of Experiments
## Overview
**One-and-done comprehensive factorial testing** - collect all data systematically, analyze separately.
This guide describes how to run a **complete 2^6 factorial design of experiments** across all 6 feedback loops, yielding:
- **64 unique configurations** (all binary combinations)
- **1,920+ runs** (30 samples per configuration)
- **Single unified dataset** for factorial analysis
## The Why: Factorial DoE
Instead of incrementally tuning parameters, we collect **comprehensive experimental data** in one go:
### Traditional Approach (Old)
- Run baseline
- Run with Loop #1
- Run with Loops #1-2
- Run with Loops #1-3
- ... (sequential, incremental)
- **Problem:** Can't see interactions, trade-offs, or surprises
### Factorial Approach (New)
- Run ALL 2^6 = 64 combinations in randomized order
- Collect all data at once
- Analyze factorial effects, interactions, and main effects
- **Advantage:** Complete picture of how loops interact
## The 6 Feedback Loops
Each loop is a binary toggle in the Makefile:
| Loop | Makefile Flag | Purpose |
|------|---|---|
| **#1** | `ENABLE_LOOP_1_HEAT_TRACKING` | Count word execution frequency |
| **#2** | `ENABLE_LOOP_2_ROLLING_WINDOW` | Capture execution history (circular buffer) |
| **#3** | `ENABLE_LOOP_3_LINEAR_DECAY` | Age words by exponential decay over time |
| **#4** | `ENABLE_LOOP_4_PIPELINING_METRICS` | Track word-to-word transitions |
| **#5** | `ENABLE_LOOP_5_WINDOW_INFERENCE` | Infer optimal window width (Levene's test) |
| **#6** | `ENABLE_LOOP_6_DECAY_INFERENCE` | Infer decay slope via linear regression |
## Configuration Naming
Each of 64 configurations is named by its loop status:
```
L1_L2_L3_L4_L5_L6
Where each digit is 0 (OFF) or 1 (ON)
Examples:
000000 = Baseline (all loops OFF - pure FORTH-79)
000001 = Only heat tracking
111110 = All except decay inference
111111 = All loops ON (likely optimal)
101010 = Alternating pattern (for research)
```
## Quick Start
### Minimal Test (validate script)
```bash
./scripts/run_factorial_doe.sh --runs-per-config 1 TEST_RUN
# 64 builds × 1 run = 64 total runs
# Runtime: ~30-45 minutes
# Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/TEST_RUN/
```
### Standard Comprehensive Run
```bash
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FACTORIAL
# 64 builds × 30 runs = 1,920 total runs
# Runtime: ~2-4 hours
# Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FACTORIAL/
```
### High-Precision Run
```bash
./scripts/run_factorial_doe.sh --runs-per-config 100 HIGH_PRECISION_DOE
# 64 builds × 100 runs = 6,400 total runs
# Runtime: ~6-12 hours
# Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/HIGH_PRECISION_DOE/
```
## Output Structure
```
/home/rajames/CLionProjects/StarForth-DoE/experiments/<EXPERIMENT_LABEL>/
├── experiment_results.csv # All measurements (1920+ rows)
├── experiment_summary.txt # Timing, metadata, statistics
├── configuration_manifest.txt # All 64 configs documented
├── test_matrix.txt # Randomized execution order
└── run_logs/
├── 000000_run_1.log
├── 000000_run_2.log
├── 000001_run_1.log
├── ...
└── 111111_run_30.log # 1,920 total log files
```
## CSV Schema
Each row in `experiment_results.csv` contains:
```
timestamp,configuration,run_number,
<35+ metrics from test harness>,
enable_loop_1_heat_tracking,
enable_loop_2_rolling_window,
enable_loop_3_linear_decay,
enable_loop_4_pipelining_metrics,
enable_loop_5_window_inference,
enable_loop_6_decay_inference
```
This lets you correlate every metric against every loop combination.
## Analysis Strategy
After collection, analyze in the separate `StarForth-DoE` repository:
### Step 1: Load Data
```python
import pandas as pd
df = pd.read_csv("experiment_results.csv")
print(df.shape) # Should be (1920, 38+)
print(df.columns)
```
### Step 2: Separate Metrics from Factors
```python
factor_cols = ['enable_loop_1_heat_tracking', ..., 'enable_loop_6_decay_inference']
metric_cols = ['total_lookups', 'cache_hit_percent', ..., 'vm_workload_duration_ns_q48']
factors = df[factor_cols]
metrics = df[metric_cols]
```
### Step 3: Factorial Analysis
```python
# Main effects: What's the average change when Loop X goes from 0→1?
main_effects = {}
for loop in factor_cols:
main_effects[loop] = metrics[factors[loop]==1].mean() - metrics[factors[loop]==0].mean()
# Interactions: Do loops work better/worse together?
# Can use statsmodels.formula.api for ANOVA/regression
# Optimal subset: Which combinations of loops give best throughput?
best_configs = df.nlargest(10, 'vm_workload_duration_ns_q48')
```
## Key Design Choices
### 1. **Rebuild Every Pass** ✓
- Each configuration requires a `make clean && make`
- Ensures clean, isolated state
- No cross-contamination
- True ground truth for each variant
### 2. **Randomized Execution**
- All 1,920 runs randomized into single matrix (not 64 separate runs)
- Prevents thermal ramp, temporal bias, or systematic trends
- DoE principle: randomization eliminates confounds
### 3. **Full Factorial** (not fractional)
- Testing ALL 2^6 = 64 combinations
- Can detect unexpected interactions
- Most expensive, most informative
### 4. **One-and-Done Collection**
- Single experiment, all data
- No progressive tuning during collection
- Separates measurement from analysis (good practice)
## Expected Results
### Hypothetical Output
Assuming typical performance distribution:
```
Configuration | Avg Time (ns) | Speed vs Baseline
─────────────────────────────────────────────────
000000 (base) | 10,000,000 | 1.0x
000001 (L1) | 9,950,000 | 1.005x
111111 (all) | 8,500,000 | 1.176x ← Likely best
```
### Expected Patterns
1. **Loop #1 (heat tracking)**: ~2-5% improvement alone
2. **Loop #2 (rolling window)**: ~1-3% improvement (depends on L1)
3. **Loops #3-6**: Increasingly synergistic (work better together)
4. **Interactions**: Loop #5 + #6 likely synergistic (inference loops)
## Troubleshooting
### Build Fails for a Configuration
```bash
# Check specific config build log
cat /home/rajames/CLionProjects/StarForth-DoE/experiments/<LABEL>/run_logs/build_<CONFIG>.log
```
### Test Harness Crashes on Some Configs
- Check if loop combination is invalid (e.g., Loop #5 without Loop #2)
- Verify Makefile flags are correctly parsed
- See: `include/vm.h` for interdependencies
### Results Look Suspicious
- Check `test_matrix.txt` - ensure randomization is present
- Verify CSV row counts match expected total
- Check for missing runs in log directory
## Performance Notes
### Timing Estimates
- **Minimal (1 run/config)**: 30-45 minutes
- **Standard (30 runs/config)**: 2-4 hours
- **High-precision (100 runs/config)**: 6-12 hours
- **Depends on:** CPU speed, disk I/O, test harness overhead
### Optimization
- Run overnight for high-precision experiments
- Use `screen` or `tmux` to keep session alive
- Monitor disk space (~100 MB for results + logs)
## Next Steps
1. **Run the experiment:**
```bash
./scripts/run_factorial_doe.sh 2025_11_19_COMPLETE_FACTORIAL
```
2. **Monitor progress:**
```bash
tail -f /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_COMPLETE_FACTORIAL/experiment_summary.txt
```
3. **Move to analysis repo** (once complete):
```bash
cp -r /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_COMPLETE_FACTORIAL \
/home/rajames/CLionProjects/StarForth-DoE-Analysis/data/
cd /home/rajames/CLionProjects/StarForth-DoE-Analysis
python3 analyze_factorial.py data/2025_11_19_COMPLETE_FACTORIAL/experiment_results.csv
```
4. **Generate reports:**
- Main effects plots
- Interaction matrices
- Pareto optimal configurations
- Recommendations for production builds
---
**Version:** 1.0
**Last Updated:** 2025-11-19
**Author:** Claude Code (with Robert A. James)
docs/working/archive/superseded/factorial-doe/incompatible-configs.md
+404
View File
@@ -0,0 +1,404 @@
> **SUPERSEDED (docs triage 2026-10-14).** The 2^6 factorial DoE guide, flagged obsolete 2026-10-09.
> Instead: experiments/bare_metal/README.md for the DoE as it runs now. Moved here from `docs/working/experiments/02-experiments/factorial-doe/incompatible-configs.md`.
<!-- Moved from docs/02-experiments/factorial-doe/incompatible-configs.md to docs/working/archive/superseded/factorial-doe/incompatible-configs.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ OBSOLETE (flagged 2026-10-09).** Build-time incompatibility handling for the retired 2^6
> toggle experiment, replaced by the L8 Jacquard mode selector. Kept for historical record.
# Handling Incompatible Configurations in Factorial DoE
## Overview
Your intuition was correct: **some loop combinations can't coexist**. Rather than failing the entire experiment, the updated script treats incompatibility as valuable data.
This document explains how the script detects and records incompatible configurations during the factorial experiment.
---
## What is an Incompatible Configuration?
An incompatible configuration is a combination of feedback loop toggles that:
1. **Fails to compile** - Makefile target produces build errors
2. **Causes StarForth to crash** - Binary exits abnormally (segfault, signal, etc.)
3. **Produces invalid output** - Test harness runs but produces no metrics
Examples (hypothetical):
- Loop 5 (window inference) without Loop 2 (rolling window) = no data to infer from
- Loop 6 (decay inference) without Loop 3 (linear decay) = no slope to measure
- Certain combinations trigger memory conflicts or uninitialized states
---
## How the Script Handles It
### Detection Strategy
**Build Phase:**
```
Configuration X attempted
├─ make TARGET=fastest ENABLE_LOOP_1=1 ...ENABLE_LOOP_6=0
├─ Check exit code
├─ If success: mark as "OK", proceed to testing
└─ If failure: mark as "BUILD_FAILED", skip all runs for this config
```
**Execution Phase:**
```
Run configuration X, iteration N
├─ Execute: ./starforth --doe-experiment
├─ Check exit code
├─ Try to extract metrics from output
├─ If success: mark as "OK", record metrics
└─ If failure: mark as "CRASH", record incompatibility
```
### CSV Recording
Incompatible configurations are recorded with status flags:
**Successful run (viable config):**
```
timestamp,configuration,run_number,build_status,run_status,
2025-11-19T10:30:45,1_1_0_1_0_0,5,OK,OK,
<35+ metrics>,1,1,0,1,0,0
```
**Build failure (incompatible):**
```
timestamp,configuration,run_number,build_status,run_status,
2025-11-19T10:35:20,0_1_1_1_1_0,1,BUILD_FAILED,<skipped>,
<empty>,0,1,1,1,1,0
```
**Runtime crash (incompatible):**
```
timestamp,configuration,run_number,build_status,run_status,
2025-11-19T10:40:15,1_0_0_1_1_1,3,OK,CRASH,
<empty>,1,0,0,1,1,1
```
### Experiment Flow
The experiment **continues** when incompatibility is detected:
```
Run 1: Config A (OK) → Test completes, metrics recorded
Run 2: Config B (BUILD_FAILED) → Build fails, skip all runs for B, continue
Run 3: Config A (OK) → Test completes, metrics recorded
Run 4: Config C (CRASH) → Crashes, record as incompatible, continue
...
Run N: Config D (OK) → Test completes, metrics recorded
SUMMARY:
✓ Successful runs: 847/1920
✗ Failed configs: 17 (incompatible combinations detected)
✓ Experiment completed despite incompatibilities
```
---
## CSV Schema
### Two New Columns
| Column | Values | Meaning |
|--------|--------|---------|
| `build_status` | OK, BUILD_FAILED | Did Makefile compilation succeed? |
| `run_status` | OK, CRASH | Did StarForth execute without crashing? |
### Analysis Filter Rules
```python
import pandas as pd
df = pd.read_csv('experiment_results.csv')
# Only viable configurations
viable = df[(df['build_status'] == 'OK') & (df['run_status'] == 'OK')]
# Incompatible configs (at compile time)
build_failures = df[df['build_status'] == 'BUILD_FAILED']
# Incompatible configs (at runtime)
runtime_crashes = df[df['run_status'] == 'CRASH']
# All incompatible
incompatible = df[(df['build_status'] != 'OK') | (df['run_status'] != 'OK')]
```
---
## Interpreting Results
### Experiment Summary
After completion, you'll see:
```
═══════════════════════════════════════════════════════════
EXPERIMENT COMPLETE (with incompatibility detection)
═══════════════════════════════════════════════════════════
✓ Successful runs: 1847/1920
✓ Failed configurations (incompatible): 17
→ Incompatible configurations detected:
✗ 0_1_1_1_1_0 (BUILD_FAILED)
✗ 1_0_0_1_1_1 (CRASH)
✗ 0_1_0_1_1_1 (CRASH)
... (total 17)
→ These configs are marked in CSV with build_status or run_status
→ They represent configurations where loops cannot coexist without conflict
```
### What This Means
- **1847 successful runs** = viable, measurable configurations
- **17 incompatible** = design space constraints discovered
- **Experiment completed** = data collection succeeded despite incompatibilities
- **CSV includes both** = full picture of parameter space
---
## Analysis Examples
### Example 1: Summary by Viability
```python
import pandas as pd
df = pd.read_csv('experiment_results.csv')
# Count by status
viable = len(df[(df['build_status'] == 'OK') & (df['run_status'] == 'OK')])
build_fail = len(df[df['build_status'] == 'BUILD_FAILED'].drop_duplicates('configuration'))
runtime_crash = len(df[df['run_status'] == 'CRASH'].drop_duplicates('configuration'))
print(f"Viable configurations: {viable} runs")
print(f"Build failures: {build_fail} configurations")
print(f"Runtime crashes: {runtime_crash} configurations")
print(f"Total tested: {64} configurations, {64*30} runs")
print(f"Success rate: {viable}/{len(df)} = {100*viable/len(df):.1f}%")
```
Expected output:
```
Viable configurations: 1847 runs
Build failures: 5 configurations
Runtime crashes: 12 configurations
Total tested: 64 configurations, 1920 runs
Success rate: 1847/1920 = 96.2%
```
### Example 2: Identify Problematic Loop Combinations
```python
# Find configs that always crash
crashes = df[df['run_status'] == 'CRASH'].drop_duplicates('configuration')
print("Configurations that crash:")
for config in crashes['configuration']:
parts = config.split('_')
loops_on = [f"Loop{i+1}" for i, bit in enumerate(parts) if bit == '1']
print(f" {config}: {', '.join(loops_on) or 'None'}")
```
Expected output:
```
Configurations that crash:
0_1_1_1_1_0: Loop2, Loop3, Loop4, Loop5
1_0_0_1_1_1: Loop1, Loop4, Loop5, Loop6
0_1_0_1_1_1: Loop2, Loop4, Loop5, Loop6
... etc
```
**Insight:** Loop 5 (window inference) appears in many crashes → suggests window inference has prerequisites or conflicts.
### Example 3: Feasibility Matrix
```python
# Which loop combinations are feasible?
df_viable = df[(df['build_status'] == 'OK') & (df['run_status'] == 'OK')].copy()
# For each config, mark as viable (1) or not (0)
feasible = {}
for config in df['configuration'].unique():
is_viable = len(df_viable[df_viable['configuration'] == config]) > 0
feasible[config] = 1 if is_viable else 0
# Show as binary matrix
print("Feasibility (1=viable, 0=incompatible):")
for i in range(64):
config = f"{i%2}_{(i//2)%2}_{(i//4)%2}_{(i//8)%2}_{(i//16)%2}_{(i//32)%2}"
print(f"{config}: {'✓' if feasible[config] else '✗'}")
```
Result: Feasibility map of 64 configurations for maintenance and design decisions.
---
## Maintenance Insights
### Why Some Combinations Are Incompatible
Common reasons:
1. **Missing Prerequisites:**
- Loop 5 (window inference) needs Loop 2 (rolling window) data
- Loop 6 (decay inference) needs Loop 3 (linear decay) to measure
2. **Memory/Resource Conflicts:**
- Multiple loops allocating same memory region
- Circular dependencies in initialization
3. **Semantic Contradictions:**
- Loop X disables feature that Loop Y requires
- Incompatible macro definitions
4. **Bugs in Specific Combinations:**
- Race conditions appearing only with certain feature combinations
- Uninitialized state when paths not taken
### Using This Information
**For Future Development:**
```
Findings: Loop 5 incompatible without Loop 2
→ Check: Does window inference code assume rolling window exists?
→ Action: Either add guard (if !LOOP_5 without LOOP_2) or document prerequisite
→ Test: Add compile-time or runtime check
```
**For Optimization:**
```
Result: All configs with Loop 6 crash if Loop 1 is OFF
→ Issue: Decay inference needs heat tracking for slope calculation
→ Fix: Add LOOP_1_REQUIRED_BY_LOOP_6 guard in code
→ Verify: Re-run factorial (expect fewer crashes)
```
**For Production:**
```
Feasible subset: {0_0_0_0_0_0, 1_1_0_0_0_0, 1_1_1_0_0_0, 1_1_1_1_1_1}
→ Recommendation: Use 1_1_1_1_1_1 (all compatible, best performance)
→ Fallback: Use 1_1_0_0_0_0 (compatible, simpler)
→ Baseline: Use 0_0_0_0_0_0 (no feedback, minimal overhead)
```
---
## Next Steps
1. **Run the experiment** to completion (incompatibilities are discovered organically)
2. **Analyze the CSV** to identify which configs are viable
3. **Examine incompatible configs** to understand why they fail
4. **Document prerequisites** (which loops require others)
5. **Fix or guard** the code to eliminate incompatibilities
6. **Re-run** experiment to validate fixes
---
## Technical Details
### Build Failure Detection
```bash
if make TARGET=fastest ENABLE_LOOP_1=1 ... > build_log 2>&1; then
# Build successful
BUILD_STATUS="OK"
else
# Build failed - mark incompatible
BUILD_STATUS="BUILD_FAILED"
# Skip all runs for this config
continue
fi
```
### Runtime Crash Detection
```bash
if "${BINARY}" --doe-experiment > run_log 2>&1; then
# Binary executed (exit code 0)
if extract_csv_row(run_log) succeeds:
RUN_STATUS="OK"
else:
RUN_STATUS="CRASH" # Binary ran but no output
else
# Binary crashed (non-zero exit)
RUN_STATUS="CRASH"
fi
```
### CSV Recording
```bash
# Viable run
printf '%s,%s,%s,%s,%s,%s,%s\n' \
"${timestamp}" "${config}" "${run_num}" \
"OK" "OK" \
"${csv_row}" "${flags}" >> results_run_01_2025_12_08.csv
# Incompatible run
printf '%s,%s,%s,%s,%s,%s\n' \
"${timestamp}" "${config}" "${run_num}" \
"${build_status}" "${run_status}" \
"${flags}" >> results_run_01_2025_12_08.csv
```
---
## Example: Real Incompatibility
**Scenario:** Loop 5 (window inference) without Loop 2 (rolling window)
```c
// Loop 5: Window Inference Code
void vm_tick_inference_engine(VM *vm) {
// Try to infer window width from rolling window
RollingWindowOfTruth *window = &vm->rolling_window; // LOOP_2 data
// BUG: If LOOP_2 is OFF, window is uninitialized!
uint32_t window_pos = window->window_pos; // CRASH: uninitialized read
}
```
**Symptom in CSV:**
```
Configuration 0_0_0_0_1_0 marked as CRASH
Loop1=0, Loop2=0, Loop3=0, Loop4=0, Loop5=1, Loop6=0
↑ ↑
Missing prerequisite Problematic loop
```
**Fix:**
```c
// Add guard in code
#if ENABLE_LOOP_5_WINDOW_INFERENCE
#if !ENABLE_LOOP_2_ROLLING_WINDOW
#error "Loop 5 requires Loop 2: window inference needs rolling window data"
#endif
#endif
```
**Result after fix:** Configuration 0_0_0_0_1_0 removed from viable set (prevented at compile time), reduces crash count.
---
## Summary
| Aspect | Approach |
|--------|----------|
| **Detection** | Build exit codes + runtime crash detection |
| **Recording** | New `build_status` and `run_status` columns |
| **Handling** | Skip incompatible config, continue experiment |
| **Impact** | Partial dataset (viable configs) + incompatibility map |
| **Analysis** | Filter by status, identify patterns |
| **Action** | Fix code, re-run, measure improvement |
This design enables discovering incompatibilities without sacrificing experimental continuity.
---
**Version:** 1.0
**Last Updated:** 2025-11-19
**Status:** Integrated into run_factorial_doe.sh
docs/working/archive/superseded/factorial-doe/index.md
+271
View File
@@ -0,0 +1,271 @@
> **SUPERSEDED (docs triage 2026-10-14).** The 2^6 factorial DoE guide, flagged obsolete 2026-10-09.
> Instead: experiments/bare_metal/README.md for the DoE as it runs now. Moved here from `docs/working/experiments/02-experiments/factorial-doe/index.md`.
<!-- Moved from docs/02-experiments/factorial-doe/index.md to docs/working/archive/superseded/factorial-doe/index.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ OBSOLETE (flagged 2026-10-09).** Index for the retired 2^6 factorial toggle
> architecture, replaced by the L8 Jacquard mode selector. Kept for historical record.
# StarForth Complete 2^6 Factorial DoE - Documentation Index
## Overview
This is a **one-and-done comprehensive factorial design of experiments** to systematically test all 64 possible combinations of 6 feedback loops.
**Key insight:** Don't toggle back-and-forth. Instead, systematically test ALL 2^6 combinations in a single experiment, then analyze the results afterward.
## Documentation Structure
### 1. **START HERE** 📋
- **File:** `docs/QUICK_START_FACTORIAL_DoE.md`
- **Purpose:** Entry point for quick reference
- **Read time:** 5 minutes
- **Contains:**
- 3 command options (quick/standard/high-precision)
- Time estimates
- What to expect during execution
- Basic troubleshooting
### 2. **Complete Guide** 📚
- **File:** `docs/FACTORIAL_DoE_GUIDE.md`
- **Purpose:** Full design documentation and rationale
- **Read time:** 20-30 minutes
- **Contains:**
- Why factorial DoE (vs. incremental tuning)
- 6 feedback loop definitions
- Configuration naming scheme (0_0_0_0_0_0 → 1_1_1_1_1_1)
- Design choices explained
- Expected results and patterns
- Troubleshooting guide
- Performance notes
### 3. **Analysis Workflow** 🔍
- **File:** `docs/FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
- **Purpose:** End-to-end pipeline from collection to insights
- **Read time:** 15-20 minutes (or 1-2 hours if coding along)
- **Contains:**
- Phase 1: Data collection (run script)
- Phase 2: Data transfer (to analysis repo)
- Phase 3: Statistical analysis (main effects, interactions)
- Phase 4: Reporting (generating insights)
- Python code examples:
* Load data and validate
* Compute main effects (individual loop impacts)
* Detect interactions (loop synergies)
* Optimize configuration subset
* Visualization code
- Example report structure
- Checklist for completion
### 4. **Main Script** ⚙️
- **File:** `scripts/run_factorial_doe.sh`
- **Purpose:** Executable for running the entire experiment
- **Read time:** 10 minutes (code + comments)
- **Contains:**
- Generates all 64 configurations
- Rebuilds for each unique configuration
- Randomizes execution order (1,920+ runs)
- Collects metrics to CSV
- Full instrumentation and progress reporting
---
## Reading Guide by Use Case
### "I just want to run the experiment"
1. Read: `QUICK_START_FACTORIAL_DoE.md` (5 min)
2. Execute: Choose command option (quick/standard/high-precision)
3. Wait: 30 min - 12 hours depending on option
4. Done!
### "I want to understand the design before running"
1. Read: `QUICK_START_FACTORIAL_DoE.md` (5 min)
2. Read: `FACTORIAL_DoE_GUIDE.md` (30 min)
3. Execute: Run with confidence
4. Analyze: Follow `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
### "I need to analyze the results"
1. Read: `QUICK_START_FACTORIAL_DoE.md` → understand naming
2. Read: `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md` → statistical methods
3. Code along: Use Python examples in Analysis Workflow
4. Interpret: Reference `FACTORIAL_DoE_GUIDE.md` for expected patterns
### "I want complete understanding"
Read in order:
1. `QUICK_START_FACTORIAL_DoE.md` (overview)
2. `FACTORIAL_DoE_GUIDE.md` (design rationale)
3. `scripts/run_factorial_doe.sh` (implementation)
4. `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md` (analysis methods)
---
## Quick Command Reference
### Run Experiment
```bash
# Option A: Quick validation (30 min)
./scripts/run_factorial_doe.sh --runs-per-config 1 TEST_RUN
# Option B: Standard production (2-4 hours)
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FULL_FACTORIAL
# Option C: High-precision overnight (6-12 hours)
./scripts/run_factorial_doe.sh --runs-per-config 100 HIGH_PRECISION_FACTORIAL
```
### Monitor Progress
```bash
# Check results_run_01_2025_12_08 CSV growth
watch -n 5 'wc -l /path/to/experiment_results.csv'
# View summary
tail /path/to/experiment_summary.txt
# Count completed configurations
ls /path/to/run_logs/build_*.log | wc -l
```
### Analyze Results
```bash
# Load and analyze in Python
cd /path/to/StarForth-DoE-Analysis
python3 << 'EOF'
import pandas as pd
df = pd.read_csv('data/2025_11_19_FULL_FACTORIAL/experiment_results.csv')
print(f"Loaded {len(df)} runs")
# See FACTORIAL_DoE_ANALYSIS_WORKFLOW.md for full analysis code
EOF
```
---
## Key Concepts
### 64 Configurations
Each configuration is a unique combination of 6 binary factors:
- **0_0_0_0_0_0** = Baseline (all loops OFF)
- **1_0_0_0_0_0** = Only heat tracking
- **1_1_0_0_0_0** = Heat + rolling window
- ...
- **1_1_1_1_1_1** = All loops ON (current optimal)
### Randomized Execution
All 1,920+ runs (64 configs × 30 samples) are randomized into a single execution matrix.
- Eliminates temporal bias
- Prevents thermal ramp (if CPU heats during night)
- Proper DoE methodology
### One-and-Done Collection
Collect all data in single experiment, analyze separately.
- No tuning during collection (avoid bias)
- Single pass through parameter space
- Separates measurement from analysis
### Main Effects
Individual impact of each loop:
```
Loop #1: Loop turns ON, average metric changes by X%
Loop #2: Loop turns ON, average metric changes by Y%
... etc
```
### Interactions
How loops work together (amplify or suppress each other):
```
Loop #1 + Loop #2: Do they work better together than independently?
```
---
## Files at a Glance
| File | Size | Purpose |
|------|------|---------|
| `scripts/run_factorial_doe.sh` | 19 KB | Main executable script |
| `docs/QUICK_START_FACTORIAL_DoE.md` | 4.9 KB | Quick reference |
| `docs/FACTORIAL_DoE_GUIDE.md` | 8.0 KB | Design documentation |
| `docs/FACTORIAL_DoE_ANALYSIS_WORKFLOW.md` | 9.6 KB | Analysis pipeline |
| `docs/FACTORIAL_DoE_INDEX.md` | This file | Documentation index |
---
## Execution Timeline
### Phase 1: Preparation (5-10 min)
- Read `QUICK_START_FACTORIAL_DoE.md`
- Choose execution option
- Run command
### Phase 2: Collection (30 min - 12 hours)
- Script builds all 64 configs (as needed)
- Randomized execution of 1,920+ runs
- Metrics collected to CSV
- Progress shown every run
### Phase 3: Transfer (5 min)
- Copy results to analysis repo
- Verify file integrity
### Phase 4: Analysis (1-2 hours)
- Load CSV in Python
- Compute main effects
- Detect interactions
- Find optimal configurations
- Generate visualizations
### Phase 5: Reporting (1-2 hours)
- Summarize findings
- Create report
- Recommendations for production
**Total time:** Collection (2-4 hrs) + Analysis (2-4 hrs) = **4-8 hours**
---
## The Design Philosophy
### Problem Statement
How do we systematically understand the performance contribution and interactions of 6 interdependent feedback loops?
### Solution
**Complete 2^6 factorial design of experiments**
### Why This Approach
1. **Complete:** Tests all 64 combinations (no gaps)
2. **Systematic:** Proper experimental design methodology
3. **Separable:** Collection ≠ Analysis (reduces bias)
4. **Reproducible:** Randomized execution, full logging
5. **Scalable:** Single script, extensible to more factors
### Alternatives Rejected
- ❌ Incremental tuning ("baseline → +loop1 → +loop1+2"): Misses interactions
- ❌ Dynamic toggle without rebuild: Introduces state contamination
- ❌ Partial factorial: Misses key interactions
- ❌ One-at-a-time testing: Expensive and confounded
---
## Next Steps
1. **Read** `QUICK_START_FACTORIAL_DoE.md` (5 minutes)
2. **Decide** which execution option (quick/standard/precise)
3. **Run** the command and let it collect data
4. **Monitor** progress in another terminal
5. **Analyze** results using `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
6. **Report** findings to stakeholders
---
## Questions?
- **"How long will this take?"** → See `QUICK_START_FACTORIAL_DoE.md`
- **"What does configuration X mean?"** → See `FACTORIAL_DoE_GUIDE.md`
- **"How do I analyze the results?"** → See `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
- **"Why this approach?"** → See `FACTORIAL_DoE_GUIDE.md` - "The Why"
- **"How do I run it?"** → `./scripts/run_factorial_doe.sh`
---
**Version:** 1.0
**Created:** 2025-11-19
**Status:** Ready for production use
docs/working/archive/superseded/factorial-doe/quick-start.md
+158
View File
@@ -0,0 +1,158 @@
> **SUPERSEDED (docs triage 2026-10-14).** Quick start for the same 2^6 factorial DoE, flagged obsolete 2026-10-09.
> Instead: experiments/bare_metal/README.md. Moved here from `docs/working/architecture/getting-started/quick-start/factorial-doe.md`.
<!-- Moved from docs/01-getting-started/quick-start/factorial-doe.md to docs/working/archive/superseded/factorial-doe/quick-start.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ OBSOLETE (flagged 2026-10-09).** Describes the retired per-loop `ENABLE_LOOP_1..6`
> compile-time toggle architecture and hardcoded local paths; the physics adaptation is now
> driven by the L8 Jacquard runtime mode selector. Kept for historical record.
# Quick Start: Complete 2^6 Factorial DoE
## TL;DR
Run a **complete, exhaustive 2^6 = 64 configuration experiment** in one go, collecting all data for later analysis.
## Three Command Options
### Option 1: Quick Validation (30 min)
Test the script and infrastructure:
```bash
./scripts/run_factorial_doe.sh --runs-per-config 1 QUICK_TEST
```
- 64 builds × 1 run = 64 total runs
- Fast feedback on whether setup works
- Output: `/home/rajames/CLionProjects/StarForth-DoE/experiments/QUICK_TEST/`
### Option 2: Standard Full DoE (2-4 hours)
Comprehensive factorial collection:
```bash
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FULL_FACTORIAL
```
- 64 builds × 30 runs = 1,920 total runs
- Good statistical power for main effects
- Standard production-grade experiment
- Output: `/home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/`
### Option 3: High-Precision Run (6-12 hours)
Maximum data for deep analysis:
```bash
./scripts/run_factorial_doe.sh --runs-per-config 100 HIGH_PRECISION_FACTORIAL
```
- 64 builds × 100 runs = 6,400 total runs
- Best for detecting subtle interactions
- Run overnight for best results
- Output: `/home/rajames/CLionProjects/StarForth-DoE/experiments/HIGH_PRECISION_FACTORIAL/`
## What Gets Generated
```
/home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/
├── experiment_results.csv ← All 1,920 rows of measurements
├── experiment_summary.txt ← Timing and metadata
├── configuration_manifest.txt ← All 64 configs listed
├── test_matrix.txt ← Randomized execution order
└── run_logs/
├── 0_0_0_0_0_0_run_1.log
├── 0_0_0_0_0_0_run_2.log
├── ...
└── 1_1_1_1_1_1_run_30.log
```
## What's Being Tested
6 **independent binary feedback loops**:
| Loop | Toggle | What It Does |
|------|--------|---|
| 1 | `ENABLE_LOOP_1_HEAT_TRACKING` | Count execution frequency |
| 2 | `ENABLE_LOOP_2_ROLLING_WINDOW` | Capture execution history |
| 3 | `ENABLE_LOOP_3_LINEAR_DECAY` | Age words over time |
| 4 | `ENABLE_LOOP_4_PIPELINING_METRICS` | Track word transitions |
| 5 | `ENABLE_LOOP_5_WINDOW_INFERENCE` | Infer window width |
| 6 | `ENABLE_LOOP_6_DECAY_INFERENCE` | Infer decay slope |
**All 2^6 = 64 combinations** tested:
- `0_0_0_0_0_0` = Baseline (all OFF)
- `1_0_0_0_0_0` = Only heat tracking
- `0_1_0_0_0_0` = Only rolling window
- ...
- `1_1_1_1_1_1` = All loops ON (current optimal)
## During Execution
You'll see progress like:
```
═══════════════════════════════════════════════════════════
STARFORTH COMPLETE 2^6 FACTORIAL DoE
═══════════════════════════════════════════════════════════
→ Factors: 6 feedback loops (LOOP_1 through LOOP_6)
→ Configurations: 2^6 = 64
→ Runs per config: 30
→ TOTAL RUNS: 1,920
→ Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL
>>> Generating randomized test matrix...
✓ Test matrix generated
→ Run 1/1920 - config 0_1_0_1_1_0 #14...
✓ Run 1/1920 completed (3s)
→ Run 2/1920 - config 1_1_1_0_0_1 #2...
✓ Run 2/1920 completed (2s)
...
```
The script:
1. **Builds** each of 64 configurations (rebuilds when config changes)
2. **Runs** tests in randomized order (no systematic bias)
3. **Collects** metrics to CSV
4. **Outputs** progress every run
## After Completion
Move to analysis repo to understand results:
```bash
cd /home/rajames/CLionProjects/StarForth-DoE-Analysis
python3 analyze_factorial.py \
data/2025_11_19_FULL_FACTORIAL/experiment_results.csv
```
Expected analysis outputs:
- **Main effects:** Which individual loops help most?
- **Interactions:** Do loops work better/worse together?
- **Optimal subset:** Best combination of loops?
- **Trade-offs:** Performance vs complexity?
## Key Design Decisions
✅ **Rebuild every pass** - ensures clean, isolated testing
✅ **Randomized execution** - eliminates temporal/thermal bias
✅ **Complete factorial** - captures all interactions
✅ **One-and-done** - collect all data, analyze later
## Estimated Time
| Runs/Config | Total | Time | When |
|---|---|---|---|
| 1 | 64 | 30-45 min | Quick test |
| 30 | 1,920 | 2-4 hours | Standard |
| 100 | 6,400 | 6-12 hours | Overnight |
Times depend on CPU, disk speed, and test harness overhead.
## Example Command
**Run it now:**
```bash
cd /home/rajames/CLionProjects/StarForth
./scripts/run_factorial_doe.sh 2025_11_19_FULL_FACTORIAL
```
**Monitor progress:**
```bash
tail -f /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_summary.txt
```
---
**Ready?** Run the command above and let the experiment collect comprehensive data for downstream analysis.
docs/working/archive/superseded/factorial-doe/workflow.md
+300
View File
@@ -0,0 +1,300 @@
> **SUPERSEDED (docs triage 2026-10-14).** The 2^6 factorial DoE guide, flagged obsolete 2026-10-09.
> Instead: experiments/bare_metal/README.md for the DoE as it runs now. Moved here from `docs/working/experiments/02-experiments/factorial-doe/workflow.md`.
<!-- Moved from docs/02-experiments/factorial-doe/workflow.md to docs/working/archive/superseded/factorial-doe/workflow.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ HISTORICAL (flagged 2026-10-09).** Documents the completed 2025-11-19 factorial
> campaign whose data informed the current L8 design; the toggle architecture it runs on is
> retired. Kept as a record.
# Factorial DoE Analysis Workflow
## End-to-End Process
```
StarForth Repo
└─ ./scripts/run_factorial_doe.sh
│
├─ Builds all 64 configurations
├─ Runs 1,920+ tests (randomized)
└─ Outputs: experiment_results.csv + logs
│
└─→ Copy to StarForth-DoE-Analysis repo
│
├─ Load CSV into Python/R
├─ Separate factors from metrics
├─ Compute main effects & interactions
├─ Identify optimal subsets
└─ Generate reports & visualizations
```
## Phase 1: Data Collection (StarForth Repo)
### Step 1: Run Experiment
```bash
cd /home/rajames/CLionProjects/StarForth
# Quick validation (optional)
./scripts/run_factorial_doe.sh --runs-per-config 1 TEST_01
# Full production run
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FULL_FACTORIAL
```
### Step 2: Wait for Completion
Monitor in another terminal:
```bash
watch -n 5 'wc -l /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_results.csv'
```
Expected progression:
- 0 min: 1 line (header)
- 30 min: ~500 lines
- 2 hours: ~1,500 lines
- 2.5 hours: 1,921 lines (complete)
### Step 3: Verify Completion
```bash
tail /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_results.csv
echo "---"
cat /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_summary.txt
```
## Phase 2: Data Transfer & Setup
### Step 1: Copy to Analysis Repo
```bash
RESULTS_DIR="/home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL"
ANALYSIS_REPO="/home/rajames/CLionProjects/StarForth-DoE-Analysis"
mkdir -p "$ANALYSIS_REPO/data"
cp -r "$RESULTS_DIR" "$ANALYSIS_REPO/data/2025_11_19_FULL_FACTORIAL"
```
### Step 2: Verify Transfer
```bash
ls -lah "$ANALYSIS_REPO/data/2025_11_19_FULL_FACTORIAL/"
wc -l "$ANALYSIS_REPO/data/2025_11_19_FULL_FACTORIAL/experiment_results.csv"
```
## Phase 3: Analysis (StarForth-DoE-Analysis Repo)
### Load Data (Python)
```python
import pandas as pd
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
# Load results_run_01_2025_12_08
df = pd.read_csv("data/2025_11_19_FULL_FACTORIAL/experiment_results.csv")
print(f"Loaded {len(df)} runs across {len(df.groupby('configuration'))} configurations")
# Separate factors and metrics
factors = df[['enable_loop_1_heat_tracking',
'enable_loop_2_rolling_window',
'enable_loop_3_linear_decay',
'enable_loop_4_pipelining_metrics',
'enable_loop_5_window_inference',
'enable_loop_6_decay_inference']]
metrics = df[['total_lookups', 'cache_hit_percent', 'vm_workload_duration_ns_q48', ...]]
print(f"Shape: {len(df)} rows × {len(df.columns)} columns")
print(f"Factors: {len(factors.columns)}")
print(f"Metrics: {len(metrics.columns)}")
```
### Main Effects Analysis
```python
# For each loop, compute average metric change when loop goes 0→1
main_effects = {}
for loop in factors.columns:
loop_name = loop.replace('enable_', '').replace('_', ' ').title()
metric_when_on = df[df[loop] == 1]['vm_workload_duration_ns_q48'].mean()
metric_when_off = df[df[loop] == 0]['vm_workload_duration_ns_q48'].mean()
effect = metric_when_on - metric_when_off
pct_change = 100 * (effect / metric_when_off)
main_effects[loop_name] = {
'effect': effect,
'pct_change': pct_change,
'on_mean': metric_when_on,
'off_mean': metric_when_off,
}
# Print sorted by impact
for loop, effect in sorted(main_effects.items(), key=lambda x: abs(x[1]['pct_change']), reverse=True):
print(f"{loop:30} {effect['pct_change']:+7.2f}% (ON: {effect['on_mean']:.0f}, OFF: {effect['off_mean']:.0f})")
```
Expected output:
```
Loop 6 Decay Inference +12.3% (ON: 8,800,000, OFF: 7,840,000)
Loop 5 Window Inference +8.5% (ON: 8,600,000, OFF: 7,920,000)
Loop 4 Pipelining Metrics +6.2% (ON: 8,400,000, OFF: 7,900,000)
Loop 3 Linear Decay +4.1% (ON: 8,200,000, OFF: 7,870,000)
Loop 2 Rolling Window +3.0% (ON: 8,100,000, OFF: 7,850,000)
Loop 1 Heat Tracking +2.1% (ON: 8,010,000, OFF: 7,850,000)
```
### Interaction Analysis
```python
# Two-way interactions: Do loops amplify/suppress each other?
interactions = {}
for i, loop_a in enumerate(factors.columns):
for j, loop_b in enumerate(factors.columns):
if i >= j:
continue # Only upper triangle
# Four cases: (0,0), (0,1), (1,0), (1,1)
case_00 = df[(df[loop_a] == 0) & (df[loop_b] == 0)]['vm_workload_duration_ns_q48'].mean()
case_01 = df[(df[loop_a] == 0) & (df[loop_b] == 1)]['vm_workload_duration_ns_q48'].mean()
case_10 = df[(df[loop_a] == 1) & (df[loop_b] == 0)]['vm_workload_duration_ns_q48'].mean()
case_11 = df[(df[loop_a] == 1) & (df[loop_b] == 1)]['vm_workload_duration_ns_q48'].mean()
# Interaction effect: (1,1) - (1,0) - (0,1) + (0,0)
interaction = (case_11 - case_10 - case_01 + case_00)
loop_a_name = loop_a.replace('enable_', '').replace('_', ' ')
loop_b_name = loop_b.replace('enable_', '').replace('_', ' ')
if abs(interaction) > 100000: # Only report large interactions
interactions[f"{loop_a_name} × {loop_b_name}"] = interaction
# Print sorted interactions
for pair, interaction in sorted(interactions.items(), key=lambda x: abs(x[1]), reverse=True):
print(f"{pair:40} {interaction:+12,.0f}")
```
### Optimal Configuration Search
```python
# Find best k configurations by performance metric
k = 10
best_configs = df.nlargest(k, 'vm_workload_duration_ns_q48')[
['configuration', 'vm_workload_duration_ns_q48', 'enable_loop_1_heat_tracking',
'enable_loop_2_rolling_window', 'enable_loop_3_linear_decay',
'enable_loop_4_pipelining_metrics', 'enable_loop_5_window_inference',
'enable_loop_6_decay_inference']
].drop_duplicates(subset=['configuration'])
print("Top 10 Configurations by Performance:")
print("─" * 100)
for idx, row in best_configs.iterrows():
loops = (
f"L1:{row['enable_loop_1_heat_tracking']} " +
f"L2:{row['enable_loop_2_rolling_window']} " +
f"L3:{row['enable_loop_3_linear_decay']} " +
f"L4:{row['enable_loop_4_pipelining_metrics']} " +
f"L5:{row['enable_loop_5_window_inference']} " +
f"L6:{row['enable_loop_6_decay_inference']}"
)
print(f"{row['configuration']:20} | {row['vm_workload_duration_ns_q48']:12,.0f} ns | {loops}")
```
### Visualization Example
```python
# Main effects plot
import matplotlib.pyplot as plt
fig, axes = plt.subplots(2, 3, figsize=(15, 10))
fig.suptitle('2^6 Factorial DoE Main Effects', fontsize=16)
loop_names = [f"Loop {i+1}" for i in range(6)]
metric_key = 'vm_workload_duration_ns_q48'
for idx, loop in enumerate(factors.columns):
ax = axes[idx // 3, idx % 3]
off_mean = df[df[loop] == 0][metric_key].mean()
on_mean = df[df[loop] == 1][metric_key].mean()
bars = ax.bar(['OFF', 'ON'], [off_mean, on_mean], color=['lightcoral', 'lightgreen'])
ax.set_ylabel('Workload Duration (ns)')
ax.set_title(f'{loop_names[idx]}')
ax.set_ylim([7000000, 10000000])
# Add value labels
for bar in bars:
height = bar.get_height()
ax.text(bar.get_x() + bar.get_width()/2., height,
f'{height/1e6:.1f}M', ha='center', va='bottom')
plt.tight_layout()
plt.savefig('main_effects.png', dpi=150)
print("Saved: main_effects.png")
```
## Phase 4: Reporting
### Key Outputs to Generate
1. **Main Effects Summary** (which loops matter most?)
- Table of individual loop impacts
- Sorted by effect size
2. **Interaction Matrix** (do loops amplify/suppress?)
- 6×6 table of two-way interactions
- Highlights synergies/conflicts
3. **Optimal Configuration** (best 10-20 subsets)
- Recommend top 3-5 production configs
- Show Pareto frontier (performance vs complexity)
4. **Statistical Summary**
- Variance by configuration
- Confidence intervals on estimates
- Regression model R²
### Example Report Structure
```markdown
# Factorial DoE Analysis Report - 2025-11-19
## Executive Summary
- Tested all 64 feedback loop combinations
- 1,920 experimental runs collected
- Key finding: Loops 5+6 show strong synergy (+18% combined)
## Main Effects
| Loop | Impact | Confidence | Recommendation |
|------|--------|-----------|---|
| Loop 6 (Decay Inference) | +12.3% | p<0.001 | **Enabled** |
| Loop 5 (Window Inference) | +8.5% | p<0.001 | **Enabled** |
| Loop 4 (Pipelining) | +6.2% | p<0.01 | Enabled |
| Loop 3 (Decay) | +4.1% | p<0.05 | Enabled |
| Loop 2 (Window) | +3.0% | p<0.05 | Enabled |
| Loop 1 (Heat) | +2.1% | p>0.05 | Optional |
## Recommended Configurations
1. **111111** (All loops): +30% vs baseline
2. **111110** (Except heat): +28% vs baseline, simpler
3. **011111** (Except heat, decay): +24% vs baseline
## Trade-off Analysis
- Full stack (111111): Best performance, highest code complexity
- Recommended (111110): 28% improvement, 95% of benefit
- Minimal set (001110): 18% improvement, 50% code overhead
---
```
## Checklist
- [ ] Run factorial DoE script to completion
- [ ] Verify 1,920 rows in experiment_results.csv
- [ ] Copy results to analysis repo
- [ ] Load data in Python/R
- [ ] Compute main effects (6 loops)
- [ ] Compute interactions (2-way or higher)
- [ ] Identify top 5-10 configurations
- [ ] Generate visualizations
- [ ] Write summary report
- [ ] Decide on production configuration
---
**Timeline:** Collection (2-4 hrs) + Analysis (2-4 hrs) = **6-8 hours total** for comprehensive factorial DoE.
docs/working/archive/superseded/feedback-loops-analysis.md
+365
View File
@@ -0,0 +1,365 @@
> **SUPERSEDED (docs triage 2026-10-14).** Flagged superseded 2026-10-09.
> Instead: its own header. Moved here from `docs/working/architecture/03-architecture/physics-engine/feedback-loops-analysis.md`.
<!-- Moved from docs/03-architecture/physics-engine/feedback-loops-analysis.md to docs/working/archive/superseded/feedback-loops-analysis.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ SUPERSEDED (flagged 2026-10-09).** Precursor bug-hunt audit covering six feedback loops;
> the canonical, current reference is `feedback-loops.md` (all seven loops, tuning knobs, and
> source files). Kept for historical record.
# StarForth Feedback Loops & Self-Optimization Analysis
## Executive Summary
StarForth implements **6 feedback loops** for automatic self-tuning during execution. Of these:
- **3 fully wired AND operational** ✅ (actively driving optimization decisions)
1. ✅ Hot-words cache promotion (1.78× speedup measured)
2. ✅ Rolling window adaptive shrinking (FIXED 2025-11-08, enforcement now works)
4. ✅ Pipelining transition speculation (WIRED 2025-11-08, speculative prefetch active)
- **1 wired AND utilized, but NOT validated** ⚠️ (running, but unsupported design choices)
3. ⚠️ Heat decay of execution history (arbitrary linear function, no tuning knob, unvalidated)
- **1 wired but NOT utilized** ⏳ (Phase 2 experimental, infrastructure ready)
5. Context-aware window tuning (binary chop algorithm stub, needs VM-level metrics aggregation)
- **1 observability-only** (metrics collected for formal verification)
6. Physics metadata temperature tracking
**CURRENT STATUS:**
- A_BASELINE, A_B_CACHE, A_C_FULL, A_B_C_FULL configurations now test complete optimization stack:
- Loop #1 (cache) enabled via A_B_CACHE flag
- Loop #2 (window shrinking) always enabled, active in all configs
- Loop #3 (decay) always enabled, active in all configs
- Loop #4 (speculation) disabled by default (ENABLE_PIPELINING=0), ready for A_C_FULL testing
- **BUT:** Loop #3 (heat decay) lacks principled design—slope is arbitrary, function choice unvalidated
- **Gap:** No measurements showing decay actually helps; DECAY_RATE has NO DoE knob to tune it
---
## FULLY WIRED & UTILIZED FEEDBACK LOOPS ✅
### ~~1. Hot-Words Cache Promotion Feedback Loop~~ ✅ DONE
**What's Measured:**
- `execution_heat` counter on each dictionary entry (incremented per execution)
**Trigger Condition:**
- When `execution_heat > HOTWORDS_EXECUTION_HEAT_THRESHOLD` (default: 50)
**Automatic Action:**
- `hotwords_cache_promote()` moves word to 32-entry LRU cache
- Subsequent lookups hit cache (1-2 ns vs 20-30 ns in bucket)
**Code Locations:**
- Metric tracking: `src/dictionary_management.c:170`
- Threshold check: `src/physics_hotwords_cache.c:155`
- Promotion logic: `src/physics_hotwords_cache.c:420-450`
**Observed Impact:**
- 1.78× speedup on cache-hit paths (measured in DoE phase 1)
- **Status: FULLY OPERATIONAL** ✅
---
### ~~2. Rolling Window Adaptive Shrinking Feedback Loop~~ ✅ FIXED
**What's Measured:**
- Pattern diversity: count of unique word transitions in execution history
**Measurement Function:**
```c
// src/rolling_window_of_truth.c:546-577 (FIXED)
uint64_t rolling_window_measure_diversity(const RollingWindowOfTruth* window)
{
// Count unique adjacent transitions (word_a → word_b)
// NOW RESPECTS effective_window_size - only scans recent entries
}
```
**Trigger Condition:**
- When pattern diversity growth rate < `ADAPTIVE_GROWTH_THRESHOLD` (default: 1%)
- Checked every `ADAPTIVE_CHECK_FREQUENCY` executions (default: 256)
**Automatic Action:**
- Shrink `effective_window_size` to `ADAPTIVE_SHRINK_RATE` percent of current (default: 75%)
- Window shrinks: 4096 → 3072 → 2304 → 1728 → ... → `ADAPTIVE_MIN_WINDOW_SIZE` (256)
**Code Locations:**
- Check function: `src/rolling_window_of_truth.c:589-645`
- Shrink calculation: Line 626
- Knobs: `include/rolling_window_knobs.h`
**Integration:**
- Called from `rolling_window_record_execution()` on every word execution
- **Status: FULLY OPERATIONAL** ✅
**Fix Applied (2025-11-08):**
- Line 559: Now limits scan to `effective_window_size` when window is warm
- Lines 561-564: Circular indexing ensures only recent entries within effective window are scanned
- **Effect:** New data arrives every tick, old data falls off after `effective_window_size` ticks (was: always 4096)
---
### 3. Linear Decay of Execution Heat Feedback Loop
**What's Measured:**
- `execution_heat` value per word (time-sensitive metric)
**Decay Formula (INTENTIONALLY BASELINE):**
- `H(t) = max(0, H_0 - decay_rate × Δt)` — **LINEAR decay**
- Configurable decay rate: `DECAY_RATE_PER_US_Q16` (default: 1 = 1/65536 heat/µs)
- Derived half-life: ~6-7 seconds for 100-heat word
**Automatic Action:**
- Stale words automatically lose heat weight
- Only frequently-executed words stay above promotion threshold
**Code Locations:**
- Decay logic: `src/physics_metadata.c:164-203` (`physics_metadata_apply_linear_decay()`)
- Integration: `src/vm.c:524` (called before each execution in inner loop)
**Design Rationale (From Phase 2 Specification):**
- Chose LINEAR over exponential for:
1. Performance (integer arithmetic, no floating-point)
2. Predictability (bounded convergence time)
3. Simplicity (fewer lines, easier to verify formally)
4. StarshipOS applicability (models "task context switch forces heat reset")
- **Explicitly noted as baseline:** "Can always switch to halflife after validating linear approach"
**STATUS: WIRED & UTILIZED, BUT NOT VALIDATED** ⚠️
**CRITICAL GAPS:**
1. **No empirical validation**: Does linear decay actually help? No measurements.
2. **Arbitrary initial slope**: Why 1 µs⁻¹? No evidence this is optimal.
3. **Wrong function?**: Could exponential, log, parabolic, or sinusoidal be better?
4. **Tuning blind**: DECAY_RATE_PER_US_Q16 has NO KNOB in DoE configurations
5. **No feedback loop closure**: Decay happens, but no measurement of whether it improved anything
---
## ✅ IMPLEMENTATION FIXES APPLIED
### Rolling Window Shrinking - FIXED
**The Bug (Now Fixed):**
The adaptive shrinking mechanism was shrinking the reported metric without enforcing the limit.
**Old Code (Line 557):**
```c
for (uint32_t i = 0; i < ROLLING_WINDOW_SIZE; i++) // ← ALWAYS full size (4096)
{
uint32_t current = window->execution_history[i];
// Count unique transitions...
}
```
**Fixed Code (Lines 559-574):**
```c
uint32_t scan_limit = (window->is_warm) ? window->effective_window_size : ROLLING_WINDOW_SIZE;
for (uint32_t i = 0; i < scan_limit; i++)
{
/* Index into the recent window (window_pos is the next write position) */
uint32_t idx = (window->window_pos + ROLLING_WINDOW_SIZE - scan_limit + i) % ROLLING_WINDOW_SIZE;
uint32_t current = window->execution_history[idx];
// Count unique transitions in recent entries only
}
```
**What Changed:**
- Now limits scan to `effective_window_size` when window is warm
- Uses circular indexing to ensure only RECENT entries are scanned
- New data arrives every tick, old data falls off after `effective_window_size` ticks
**User's Requirement Met:**
> "The window should advance with every tick so new data comes in and stale data falls off."
✅ Now data falls off after `effective_window_size` ticks, not always 4096
---
## ~~4. Pipelining Transition Metrics~~ ✅ NOW FUNCTIONAL
**What's Measured:**
- Transition patterns: (word_A → word_B) frequencies via `transition_metrics_record()`
- Probability calculations: P(B | A) in Q48.16 fixed-point format
- Prefetch accuracy: Hits vs. misses from speculative cache promotions
**Trigger Conditions:**
- `SPECULATION_THRESHOLD_Q48` (50% minimum confidence)
- `MIN_SAMPLES_FOR_SPECULATION` (10 minimum observations)
**Automatic Action (NOW WIRED):**
- After recording each transition, update probability cache
- Check: Does next word probability exceed threshold?
- If yes: Find dictionary entry and pre-promote to hotwords cache
- Effect: When next word is actually looked up, it's cache-warm
- Record: Track prefetch_attempts for feedback measurement
**Code Locations:**
- Metric collection: `src/physics_pipelining_metrics.c:92-109` (`transition_metrics_record()`)
- Probability calculation: `src/physics_pipelining_metrics.c:112-123`
- Decision logic: `src/physics_pipelining_metrics.c:164-186` (`transition_metrics_should_speculate()`)
- **Action wiring: `src/vm.c:544-583`** (execute_colon_word, speculative prefetch logic)
**Status: FULLY WIRED & FUNCTIONAL** ✅
**Wiring Details (2025-11-08):**
- Line 548: `transition_metrics_record()` - record the transition
- Line 551: `transition_metrics_update_cache()` - find most likely next word
- Line 554-555: `transition_metrics_should_speculate()` - check confidence threshold
- Lines 560-581: If confident, look up predicted word and promote to hotwords cache
- Line 580: Record the prefetch attempt for metrics feedback
**Loop Closure:**
1. ✅ Metrics collected: transition_heat array counts (word_B | word_A)
2. ✅ Probability calculated: Q48.16 format
3. ✅ Decision made: Confidence check against threshold
4. ✅ Action taken: Speculative promotion to hotwords cache
5. ✅ Feedback recorded: prefetch_attempts counter
### 5. Context-Aware Window Size Tuning (Phase 2 Placeholder)
**What's Measured:**
- Multi-word context patterns (sequences of 2-4 consecutive words)
**Intended Action (Not Connected):**
- Use binary search to find optimal `effective_window_size` for given workload
- Function: `transition_metrics_binary_chop_suggest_window()` (lines 340-357)
**Why Not Utilized:**
- Phase 2 research feature
- Experimental algorithm not yet validated
- No integration point in main execution path
**Status: PROTOTYPE, NOT INTEGRATED** ⏳
---
## OBSERVABILITY-ONLY (Not Decision-Driving)
### 6. Physics Metadata Temperature Tracking
**What's Measured:**
- `temperature_q8`: Exponential moving average of execution_heat
- Used for observability and formal verification model tracing
**Current Use:**
- Profiling and debugging (not driving optimization decisions)
- Forms foundation for Isabelle/HOL physics state machine proofs
**Status: WORKING AS INTENDED** (observability, not optimization)
---
## Bootstrap Seeding Mechanisms
These run once at startup to initialize feedback loop state:
### POST (Power-On Self-Test) Seeding
- **Function**: `rolling_window_seed_hotwords_cache()` (lines 258-329)
- **Action**: Records POST execution, promotes high-heat words to cache
- **Effect**: Cache is "warm" when user workload begins
### Pipelining Context Seeding
- **Function**: `rolling_window_seed_pipelining_context()` (lines 341-415)
- **Action**: Records POST sequences, establishes transition baseline
- **Effect**: Pipelining has initial pattern knowledge
---
## Configuration Knobs for Feedback Loops
All knobs are **tunable via Makefile**:
```bash
# Hot-words cache promotion
make HOTWORDS_EXECUTION_HEAT_THRESHOLD=75
# Adaptive window shrinking
make ADAPTIVE_SHRINK_RATE=75 # Keep 75%, shrink by 25%
make ADAPTIVE_MIN_WINDOW_SIZE=256 # Never shrink below 256
make ADAPTIVE_CHECK_FREQUENCY=256 # Check every 256 executions
make ADAPTIVE_GROWTH_THRESHOLD=1 # Shrink when growth < 1%
# Heat decay
make DECAY_RATE_PER_US_Q16=16384 # Decay rate (units: heat/μs)
make DECAY_MIN_INTERVAL=1000000 # Min interval between decays (ns)
# Pipelining (Phase 2, disabled by default)
make ENABLE_PIPELINING=1
make SPECULATION_THRESHOLD_Q48=$((50 << 16)) # 50% confidence
make MIN_SAMPLES_FOR_SPECULATION=10
```
---
## Summary Table: All Feedback Loops
| Loop | Measured | Threshold | Action | Wired? | Utilized? | Status |
|------|----------|-----------|--------|--------|-----------|--------|
| ~~1. Hot-words promotion~~ | `execution_heat` | > 50 | Add to cache | ✅ | ✅ | ✅ VALIDATED |
| ~~2. Adaptive shrinking~~ | `pattern_diversity` | growth < 1% | Shrink window | ✅ | ✅ | ✅ FIXED |
| 3. Heat decay | Time since last execution | ~6-7s half-life | Reduce heat | ✅ | ✅ | ⚠️ NOT VALIDATED |
| 4. Pipelining transitions | `transition_heat[]` | > 50% confidence | Speculative prefetch | ✅ | ❌ | ⏳ Phase 2 |
| 5. Context window tuning | Multi-word sequences | N/A | Binary chop | ✅ | ❌ | ⏳ Phase 2 |
| 6. Temperature tracking | `temperature_q8` | N/A | Observability only | ✅ | ✅ | ✅ OBSERVABILITY |
---
## Completed Actions ✅
### ✅ DONE: Fix Rolling Window Shrinking Implementation
**Status: APPLIED (2025-11-08)**
Modified `src/rolling_window_of_truth.c:546-577` (function `rolling_window_measure_diversity()`):
- Now respects `effective_window_size` when window is warm
- Uses circular indexing to scan only recent entries
- New data arrives every tick, old data falls off after `effective_window_size` ticks (not always 4096)
- A_B_C_FULL configuration now tests both hot-words cache + adaptive window shrinking together
---
## Next Steps for Phase 2 (If Proceeding)
### OPTIONAL 1: Connect Pipelining Decision Logic
**When `ENABLE_PIPELINING=1`:**
- Call `transition_metrics_should_speculate()` from vm inner loop
- If returns true: Pre-load next word from context prediction
- Measure cache miss reduction vs. A_BASELINE (Phase 2 experimental validation)
**Status:** Metrics collection ready, decision logic not wired
### OPTIONAL 2: Validate Context Window Tuning (Phase 2)
**For adaptive window size optimization:**
- Implement binary chop search algorithm in main execution path
- Tune `effective_window_size` dynamically based on context complexity
- Measure pattern capture efficiency vs. memory overhead
**Status:** Binary chop function exists, integration point not defined
---
## Files & Line Numbers Reference
| Purpose | File | Lines |
|---------|------|-------|
| Rolling window structure | `include/vm.h` | 58-70 |
| Rolling window implementation | `src/rolling_window_of_truth.c` | - |
| Adaptive shrinking | `src/rolling_window_of_truth.c` | 583-639 |
| **Bug location** | `src/rolling_window_of_truth.c` | **557** |
| Diversity measurement | `src/rolling_window_of_truth.c` | 546-571 |
| Shrinking knobs | `include/rolling_window_knobs.h` | - |
| Hot-words cache | `src/physics_hotwords_cache.c` | - |
| Promotion threshold | `src/physics_hotwords_cache.c` | 155 |
| Heat decay | `src/physics_metadata.c` | 164-203 |
| Pipelining metrics | `src/physics_pipelining_metrics.c` | - |
| Pipelining decision | `src/physics_pipelining_metrics.c` | 164-186 |
| Main integration | `src/vm.c` | 524, 542-548 |
---
**Analysis Complete:** 2025-11-08
**User Request:** Investigate feedback loops, identify implementation problems
**Finding:** 3 active + 2 dormant + 1 observability feedback loops; 1 critical bug in rolling window shrinking
docs/working/archive/superseded/heartbeat-doe-corrected-design.md
+344
View File
@@ -0,0 +1,344 @@
> **SUPERSEDED (docs triage 2026-10-14).** Flagged superseded 2026-10-09.
> Instead: its own header. Moved here from `docs/working/experiments/02-experiments/heartbeat-doe/corrected-design.md`.
<!-- Moved from docs/02-experiments/heartbeat-doe/corrected-design.md to docs/working/archive/superseded/heartbeat-doe-corrected-design.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ SUPERSEDED (flagged 2026-10-09).** Superseded by `phase2-corrected.md` (the
> "corrected and finalized" revision). Kept for historical record.
# StarForth Heartbeat DoE – CORRECTED Design (Heartrate Focus)
**Date:** November 20, 2025
**CORRECTION:** Previous design missing critical metric: **HEARTBEAT RATE (tick_ns)**
**Status:** Ready for implementation
---
## The Critical Insight You Caught
The heartbeat isn't just a **fixed tick**. The system has a **variable heartbeat rate** controlled by:
```c
/* HeartbeatWorker.tick_ns — this is DYNAMIC, not constant! */
uint64_t tick_ns; /* Nanoseconds between ticks — VARIES with load! */
```
**What this means:**
- Default: `HEARTBEAT_TICK_NS` (~1 millisecond)
- **But:** The inference engine can adaptively adjust `tick_target_ns`
- Different configurations may modulate heartbeat frequency differently
- **This is the KEY physics behavior we must measure!**
---
## Stage 1 Problem (What We Got Wrong)
Stage 1 baseline ran **without heartbeat thread** (HEARTBEAT_THREAD_ENABLED=0):
- Fixed 1ms tick (no modulation)
- No adaptive heartrate
- Doesn't reflect StarForth's intended physics-driven behavior
**Stage 1 conclusion:** "System is deterministic, no heartbeat variation"
**Stage 1 was incomplete** because it disabled the very mechanism that makes the system adaptive!
---
## Stage 2 Correction (What We Should Do)
### Primary Hypothesis
The 5 elite configurations differ in **how they modulate heartbeat rate**:
| Config | Expected Behavior |
|--------|-------------------|
| 1_0_1_1_1_0 | Tight, responsive heartrate (low jitter, fast modulation) |
| 1_0_1_1_1_1 | Similar, with decay inference slowing response? |
| 1_1_0_1_1_1 | Window-driven, may have different modulation pattern |
| 1_0_1_0_1_0 | Lean, may have sluggish response |
| 0_1_1_0_1_1 | Window-without-heat, baseline behavior |
### Key Metrics for Heartrate (Not Just Intervals)
**We need to measure:**
1. **Heartbeat Rate Modulation (NOT just interval jitter)**
```
heartbeat_rate_trajectory = [tick_ns_t0, tick_ns_t1, tick_ns_t2, ...]
What we measure:
- Mean heartbeat rate (expected: ~1,000,000 ns = 1 kHz baseline)
- Responsiveness to load (does rate slow under heavy load?)
- Convergence time (how long until rate stabilizes?)
- Modulation range (min to max tick_ns observed)
```
2. **Heartrate vs. Workload Coupling**
```
For each tick:
workload_intensity = lookups_this_tick / lookups_per_second_baseline
heartrate_hz = 1_000_000_000 / tick_ns
Measure: correlation(workload_intensity, heartrate_hz)
- Positive correlation: heartrate slows under load (good physics)
- Negative correlation: heartrate speeds up under load (bad physics)
- No correlation: heartrate doesn't respond to load (broken physics)
```
3. **Heartrate Stability (Not "jitter" but "rate consistency")**
```
- Does heartrate converge to a steady state?
- How quickly does it adapt to workload changes?
- Does it oscillate or settle smoothly?
```
4. **Heartrate Efficiency**
```
- Is the system running faster/slower with adapted heartrate?
- Workload duration: should improve with optimized heartrate
- Memory: should be more efficient with tuned heartrate
```
---
## Corrected Metrics Collection
### Per-Tick Collection (During Execution)
```c
/* In heartbeat_thread_main(), before nanosleep() */
struct HeartbeatTickData {
uint64_t tick_number;
uint64_t tick_ns; /* THIS IS THE HEARTBEAT RATE! */
uint64_t workload_ops_this_tick; /* Dictionary lookups in this tick */
uint64_t wall_clock_ns; /* Actual wall-clock time */
uint64_t hot_word_count; /* Physics state */
uint64_t total_heat; /* Physics state */
};
/* Store in circular buffer or rolling window */
```
### Post-Run Analysis Metrics
```c
typedef struct {
/* === Original Metrics === */
/* ... existing 38 metrics ... */
/* === CORRECTED: Heartbeat Rate Metrics === */
/* Heartbeat rate trajectory */
uint64_t total_ticks_executed;
double mean_heartbeat_rate_hz; /* Hz = 1e9 / mean_tick_ns */
double heartbeat_rate_stddev_hz; /* Variation in Hz */
double heartbeat_rate_cv; /* Coefficient of variation */
uint64_t heartbeat_rate_min_ns; /* Slowest (highest tick_ns) */
uint64_t heartbeat_rate_max_ns; /* Fastest (lowest tick_ns) */
double heartbeat_rate_range_hz; /* max_hz - min_hz */
/* Heartrate vs Load Coupling */
double load_to_heartrate_correlation; /* Does rate respond to load? */
int load_response_direction; /* +1 = slower under load (good) */
double heartrate_adaptation_latency_ticks; /* Ticks until rate responds */
double heartrate_settling_time_ticks; /* Ticks to settle after change */
/* Heartrate Convergence */
double heartrate_convergence_time_ms; /* Time to reach stable rate */
int heartrate_converged; /* 1 = yes, 0 = still oscillating */
double heartrate_oscillation_amplitude_hz; /* Peak-to-peak variation */
/* Overall Heartrate Stability Score */
double heartrate_stability_score; /* 0-100: how stable is heartrate? */
} DoeMetrics_CORRECTED;
```
---
## The Missing Piece: Heartbeat Rate Logging
**Critical code addition needed:**
```c
/* In vm.c heartbeat_thread_main() - CAPTURE tick_ns BEFORE and AFTER */
static inline void capture_heartbeat_rate_sample(VM *vm, uint64_t tick_ns, uint64_t workload_ops) {
/* Store sample in rolling buffer */
if (!vm->heartbeat_rate_samples) {
vm->heartbeat_rate_samples = calloc(HEARTBEAT_RATE_SAMPLE_MAX, sizeof(HeartbeatRateSample));
}
HeartbeatRateSample *sample = &vm->heartbeat_rate_samples[vm->heartbeat_rate_sample_count];
sample->tick_number = vm->heartbeat.tick_count;
sample->tick_ns = tick_ns; /* THE HEARTBEAT RATE! */
sample->workload_ops = workload_ops;
sample->timestamp_ns = platform_monotonic_ns();
vm->heartbeat_rate_sample_count++;
}
/* In heartbeat_thread_main() */
static void* heartbeat_thread_main(void *arg) {
VM *vm = (VM*)arg;
while (!vm->heartbeat.worker->stop_requested) {
uint64_t tick_start_ns = platform_monotonic_ns();
vm_heartbeat_run_cycle(vm); /* May adjust tick_ns */
uint64_t tick_ns = vm->heartbeat.worker->tick_ns;
uint64_t workload_ops = vm->execution_heat_current_tick; /* Need to track */
/* CAPTURE THE HEARTBEAT RATE! */
capture_heartbeat_rate_sample(vm, tick_ns, workload_ops);
/* Sleep */
nanosleep(...);
}
}
```
---
## Corrected Analysis (R Script)
```r
# Load heartbeat rate samples from CSV
doe_data <- read.csv("experiment_results_heartbeat.csv") %>%
mutate(
# Convert heartbeat rates (tick_ns → Hz)
mean_heartbeat_hz = 1e9 / mean_heartbeat_rate_ns,
heartrate_stability = 1 / (1 + heartbeat_rate_cv), # Lower CV = higher score
)
# PRIMARY ANALYSIS: Heartrate Stability
stability_by_heartrate <- doe_data %>%
group_by(configuration) %>%
summarize(
mean_heartrate_hz = mean(mean_heartbeat_hz),
mean_heartrate_stability = mean(heartrate_stability),
mean_load_coupling = mean(load_to_heartrate_correlation),
mean_workload_ms = mean(vm_workload_duration_ns_q48 / 1e6)
) %>%
mutate(
# Golden config = best heartrate stability + strongest load coupling
golden_score = (heartrate_stability * 0.5) + (load_coupling * 0.3) + (workload_improvement * 0.2)
)
# Golden Config = Highest golden_score
```
---
## Why This Matters
### Stage 1 Got Lucky
Stage 1 found 5 good configs **by accident** because it measured final performance, not physics.
### Stage 2 Should Be Different
Stage 2 should measure **whether the physics is working correctly**:
- **Can the system modulate heartbeat rate** in response to load?
- **Is the modulation smooth** or does it oscillate?
- **Does heartrate modulation improve overall performance?**
- **Which configuration has the most responsive, stable heartrate?**
**The golden configuration should be the one where heartbeat rate modulation is:**
1. ✓ **Responsive** (correlates with workload)
2. ✓ **Smooth** (doesn't oscillate wildly)
3. ✓ **Efficient** (results in best overall performance)
---
## Revised DoE Experiment Design
### What Actually Gets Measured
Instead of just "jitter in fixed 1ms tick intervals", measure:
**Per-Run Heartbeat Rate Trajectory:**
```
Tick 0: tick_ns = 1,000,000 (1 ms baseline)
Tick 1: tick_ns = 1,500,000 (slowing under load)
Tick 2: tick_ns = 2,000,000 (further slowdown)
Tick 10: tick_ns = 1,200,000 (adapting back)
Tick 100: tick_ns = 1,050,000 (converged to steady state)
```
**What we analyze:**
1. Did heartrate change at all? (some configs might keep it fixed)
2. How much did it vary? (CV of tick_ns, not CV of fixed intervals)
3. Did it respond to load? (correlation with workload)
4. Did it converge smoothly? (no oscillation)
5. Did it improve performance? (faster execution with adaptive heartrate?)
---
## Implementation Checklist (Revised)
- [ ] Add `HeartbeatRateSample` struct to capture `tick_ns` per tick
- [ ] Modify `heartbeat_thread_main()` to log `tick_ns` trajectory
- [ ] Extend `DoeMetrics` with heartrate-specific fields (not just interval jitter)
- [ ] Update CSV schema to include heartrate trajectory metrics
- [ ] Modify R analysis to focus on **heartrate modulation patterns**
- [ ] Update golden config selection: prioritize **load responsiveness**
---
## Expected Results (Corrected)
### Hypothesis: Configs Differ in Heartrate Modulation
**Config 1 (1_0_1_1_1_0) - Expected WINNER:**
- Heartrate slowly increases under load (tick_ns increases, Hz decreases)
- Smooth adaptation (no oscillation)
- Converges within 1000 ticks
- Performance improves (faster overall execution)
- Load coupling: 0.8+ (strong)
**Config 5 (0_1_1_0_1_1) - Expected LOSER:**
- Heartrate doesn't respond to load (fixed tick_ns)
- No correlation with workload
- Performance degrades (why? system can't adapt)
---
## Key Realization
The heartbeat isn't just a **timer**. It's a **feedback mechanism**:
```
Load increases
↓
Inference engine detects change
↓
Adjusts tick_target_ns (slower heartbeat = more thinking time)
↓
Heartbeat rate modulates (tick_ns increases)
↓
Physics engine gets more time to tune parameters
↓
System adapts and performance improves
```
**We need to measure whether this feedback loop is working!**
---
## Next Steps
1. **Extend HeartbeatWorker:** Add rate sample collection
2. **Modify heartbeat_thread_main():** Log `tick_ns` per tick
3. **Update DoeMetrics:** Heartrate trajectory analysis
4. **Rewrite CSV schema:** Focus on heartrate modulation
5. **Rewrite R analysis:** Measure load-response coupling and heartrate stability
6. **Re-run experiment:** Focus on configs that modulate heartrate best
---
**This changes the entire experiment focus.**
Instead of measuring "jitter in a fixed heartbeat", we're measuring "**how well does the system modulate its heartbeat in response to load?**"
That's the real physics story.
docs/working/archive/superseded/heartbeat-doe-design.md
+434
View File
@@ -0,0 +1,434 @@
> **SUPERSEDED (docs triage 2026-10-14).** Flagged superseded 2026-10-09.
> Instead: its own header. Moved here from `docs/working/experiments/02-experiments/heartbeat-doe/design.md`.
<!-- Moved from docs/02-experiments/heartbeat-doe/design.md to docs/working/archive/superseded/heartbeat-doe-design.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ SUPERSEDED (flagged 2026-10-09).** Pre-correction draft; superseded first by
> `corrected-design.md`, then by `phase2-corrected.md` (the finalized design). Kept for
> historical record.
# StarForth Factorial DoE with Heartbeat Integration
**Date:** November 20, 2025
**Status:** Design Phase
**Experiment:** 2^6 Factorial with Heartbeat-Aware Metrics Collection
---
## Overview
This document describes the **second-generation factorial DoE** that incorporates heartbeat observability into metric collection. Building on the baseline DOE (Stage 1: 3200 runs, 6 factors, no heartbeat), this experiment captures **per-tick heartbeat dynamics** to identify configurations with optimal stability characteristics.
### Key Innovation
Instead of measuring static performance, this experiment measures **temporal stability**:
- How quickly does the system converge under load changes?
- How much does heartbeat interval modulate in response to workload?
- What is the jitter envelope after settling?
- Which configuration achieves smoothest load→heartbeat coupling?
---
## Design Points (Top 5 Configurations from Stage 1)
Based on Stage 1 analysis, the elite configurations are:
1. **Config 1:** `1_0_1_1_1_0` (F1=ON, F2=OFF, F3=ON, F4=ON, F5=ON, F6=OFF)
2. **Config 2:** `1_0_1_1_1_1` (F1=ON, F2=OFF, F3=ON, F4=ON, F5=ON, F6=ON)
3. **Config 3:** `1_1_0_1_1_1` (F1=ON, F2=ON, F3=OFF, F4=ON, F5=ON, F6=ON)
4. **Config 4:** `1_0_1_0_1_0` (F1=ON, F2=OFF, F3=ON, F4=OFF, F5=ON, F6=OFF)
5. **Config 5:** `0_1_1_0_1_1` (F1=OFF, F2=ON, F3=ON, F4=OFF, F5=ON, F6=ON)
**Factors Mapping:**
- F1 = ENABLE_LOOP_1_HEAT_TRACKING
- F2 = ENABLE_LOOP_2_ROLLING_WINDOW
- F3 = ENABLE_LOOP_3_LINEAR_DECAY
- F4 = ENABLE_LOOP_4_PIPELINING_METRICS
- F5 = ENABLE_LOOP_5_WINDOW_INFERENCE
- F6 = ENABLE_LOOP_6_DECAY_INFERENCE
---
## New Heartbeat Metrics (Per-Tick Collection)
### 1. Heartbeat Interval Dynamics
```c
struct HeartbeatTickMetrics {
uint64_t tick_count; /* Absolute tick number */
uint64_t tick_interval_ns; /* Wall-clock duration of this tick */
uint64_t instantaneous_workload_ops; /* Dictionary lookups in this tick */
double instantaneous_load_ratio; /* (ops this tick) / (avg ops/tick) */
};
```
**Purpose:** Capture how heartbeat frequency modulates in response to workload intensity.
### 2. Cache Performance (Per-Tick Window)
```c
struct CacheWindowMetrics {
uint64_t tick_start;
uint64_t cache_hits_window; /* Hits in last 100 ticks */
uint64_t cache_attempts_window; /* Attempts in last 100 ticks */
double cache_hit_percent_window; /* Rolling window %, 0-100 */
};
```
**Purpose:** Measure cache stability, not just final hit rate.
### 3. Rolling Window Diagnostics
```c
struct WindowDiagnostics {
uint32_t effective_window_size; /* Current adaptive size */
uint64_t pattern_diversity; /* Unique word IDs observed */
double diversity_growth_rate; /* % new patterns per 1000 executions */
int is_warm; /* 1 if window is representative */
};
```
**Purpose:** Track window convergence and saturation.
### 4. Jitter Envelope
```c
struct JitterMetrics {
double tick_interval_mean_ns; /* Average heartbeat interval */
double tick_interval_stddev_ns; /* StdDev of interval */
double tick_interval_cv; /* Coefficient of variation (σ/μ) */
uint64_t outlier_count; /* Ticks >3σ from mean */
double outlier_ratio; /* Outliers / total ticks */
};
```
**Purpose:** Measure temporal smoothness and jitter control.
### 5. Decay Slope Tracking
```c
struct DecayTrajectory {
uint64_t tick_sample_count; /* How many ticks sampled */
double decay_slope_observed; /* Fitted slope from observations */
double slope_convergence_rate; /* Speed of slope convergence */
int slope_direction; /* -1=decreasing, 0=stable, +1=increasing */
};
```
**Purpose:** Monitor whether decay slope inference is tracking correctly.
### 6. Load-Response Coupling
```c
struct LoadResponseCoupling {
double correlation_load_to_interval; /* Correlation(workload, heartbeat_interval) */
double response_latency_ticks; /* Ticks until interval reacts to load change */
double overshoot_ratio; /* Peak response / steady-state response */
double settling_time_ticks; /* Ticks to settle within 10% of target */
};
```
**Purpose:** Measure how quickly and smoothly heartbeat responds to workload shifts.
---
## Extended DoeMetrics Struct
Add to `include/doe_metrics.h`:
```c
typedef struct {
/* === Original Metrics (preserved) === */
/* ... existing 35+ metrics ... */
/* === NEW: Heartbeat Stability Metrics === */
/* Tick dynamics */
uint64_t total_heartbeat_ticks; /* Total ticks during run */
double tick_interval_mean_ns; /* Average heartbeat interval */
double tick_interval_stddev_ns; /* StdDev of heartbeat interval */
double tick_interval_cv; /* Coefficient of variation */
uint64_t tick_outlier_count_3sigma; /* Ticks >3σ from mean */
double tick_outlier_ratio; /* Outliers / total ticks */
/* Cache stability window */
double cache_hit_percent_rolling_mean; /* Rolling avg, last 100 ticks */
double cache_hit_percent_rolling_stddev; /* Rolling stddev */
double cache_stability_index; /* How stable is cache hit %? */
/* Window convergence */
uint32_t window_final_effective_size; /* Final adaptive window size */
double pattern_diversity_final; /* Final unique patterns observed */
double diversity_growth_final; /* Growth rate at end */
int window_warmth_achieved; /* 1 = warm, 0 = still cold */
/* Decay slope convergence */
double decay_slope_fitted_final; /* Final fitted decay slope */
double decay_slope_convergence_rate; /* How fast did slope converge? */
int decay_slope_stable; /* 1 = stable direction, 0 = oscillating */
/* Load-response */
double load_interval_correlation; /* Correlation(load, heartbeat_interval) */
double response_latency_ticks; /* Ticks until response to workload change */
double overshoot_ratio; /* Peak response / steady-state */
double settling_time_ticks; /* Time to 10% settling band */
/* Aggregate stability score */
double overall_stability_score; /* Composite metric: (low jitter + fast convergence + tight correlation) */
} DoeMetrics; /* Now extends to ~65+ columns */
```
---
## Heartbeat Data Collection Architecture
### Per-Tick Logging (Lightweight)
During VM execution, capture minimal per-tick data:
```c
/* In vm.c vm_tick_heartbeat_worker() */
void vm_tick_heartbeat_worker(VM *vm) {
HeartbeatSnapshot snapshot = {
.published_tick = vm->heartbeat.tick_count,
.published_ns = platform_monotonic_ns(),
.window_width = vm->rolling_window.effective_window_size,
.decay_slope_q48 = vm->decay_slope_q48,
.hot_word_count = vm->hot_word_count_at_check,
.stale_word_count = vm->stale_word_count_at_check,
.total_heat = vm->total_heat_at_last_check,
};
/* Publish snapshot (double-buffered, lock-free read) */
int next_slot = 1 - vm->heartbeat.snapshot_index;
vm->heartbeat.snapshots[next_slot] = snapshot;
vm->heartbeat.snapshot_index = next_slot;
}
```
### Post-Run Analysis (Batch Processing)
After test execution completes, analyze collected tick data:
```c
/* In src/doe_metrics.c metrics_from_vm() */
void analyze_heartbeat_trajectory(VM *vm, DoeMetrics *metrics) {
HeartbeatSnapshot *snapshots = vm->heartbeat.snapshots;
uint64_t total_ticks = vm->heartbeat.tick_count;
/* 1. Compute tick interval statistics */
double intervals[total_ticks];
for (size_t i = 1; i < total_ticks; i++) {
intervals[i] = snapshots[i].published_ns - snapshots[i-1].published_ns;
}
metrics->tick_interval_mean_ns = compute_mean(intervals, total_ticks);
metrics->tick_interval_stddev_ns = compute_stddev(intervals, total_ticks);
metrics->tick_interval_cv = metrics->tick_interval_stddev_ns / metrics->tick_interval_mean_ns;
/* 2. Compute outlier ratio (ticks >3σ from mean) */
double threshold_3sigma = metrics->tick_interval_mean_ns + 3.0 * metrics->tick_interval_stddev_ns;
uint64_t outliers = 0;
for (size_t i = 0; i < total_ticks; i++) {
if (intervals[i] > threshold_3sigma) outliers++;
}
metrics->tick_outlier_ratio = (double)outliers / total_ticks;
/* 3. Compute decay slope convergence */
double slopes[total_ticks];
for (size_t i = 100; i < total_ticks; i++) { /* Skip warm-up */
slopes[i] = snapshots[i].decay_slope_q48 / 65536.0;
}
metrics->decay_slope_fitted_final = slopes[total_ticks - 1];
metrics->decay_slope_convergence_rate = compute_convergence_rate(slopes, total_ticks);
/* 4. Compute overall stability score */
double stability = 0.0;
stability += (100.0 - metrics->tick_interval_cv); /* Lower CV = higher score */
stability += (100.0 - metrics->tick_outlier_ratio * 100.0); /* Fewer outliers = higher */
stability += metrics->decay_slope_convergence_rate; /* Fast convergence = higher */
metrics->overall_stability_score = stability / 3.0;
}
```
---
## CSV Output Schema (Extended)
**New columns (per heartbeat experiment run):**
```
timestamp,configuration,run_number,
[... original 38 columns ...],
total_heartbeat_ticks,
tick_interval_mean_ns,tick_interval_stddev_ns,tick_interval_cv,
tick_outlier_count_3sigma,tick_outlier_ratio,
cache_hit_percent_rolling_mean,cache_hit_percent_rolling_stddev,cache_stability_index,
window_final_effective_size,pattern_diversity_final,diversity_growth_final,window_warmth_achieved,
decay_slope_fitted_final,decay_slope_convergence_rate,decay_slope_stable,
load_interval_correlation,response_latency_ticks,overshoot_ratio,settling_time_ticks,
overall_stability_score
```
**Total:** ~60 columns per run
---
## R Analysis Template
```r
library(tidyverse)
library(ggplot2)
library(gridExtra)
# Load heartbeat DoE data
doe_data <- read.csv("experiment_results_with_heartbeat.csv") %>%
mutate(
configuration = as.factor(configuration),
timestamp = as.POSIXct(timestamp)
)
# ========== MAIN ANALYSIS ==========
# 1. Stability Ranking
stability_rank <- doe_data %>%
group_by(configuration) %>%
summarize(
mean_stability_score = mean(overall_stability_score),
mean_jitter_cv = mean(tick_interval_cv),
mean_convergence_rate = mean(decay_slope_convergence_rate),
mean_settling_time = mean(settling_time_ticks),
n_runs = n()
) %>%
mutate(
rank = rank(-mean_stability_score, ties.method = "min")
) %>%
arrange(rank)
print("=== STABILITY RANKINGS ===")
print(stability_rank)
# 2. Visualization: Stability Scores
ggplot(doe_data, aes(x = reorder(configuration, overall_stability_score, FUN=median),
y = overall_stability_score,
fill = configuration)) +
geom_boxplot(alpha = 0.7) +
geom_jitter(width = 0.2, alpha = 0.4) +
coord_flip() +
theme_minimal() +
labs(title = "Overall Stability Scores by Configuration",
x = "Configuration",
y = "Stability Score (0-100)")
ggsave("01_stability_scores.png", width = 10, height = 6)
# 3. Visualization: Jitter Control (CV)
ggplot(doe_data, aes(x = reorder(configuration, tick_interval_cv, FUN=median),
y = tick_interval_cv,
fill = configuration)) +
geom_boxplot(alpha = 0.7) +
coord_flip() +
theme_minimal() +
labs(title = "Heartbeat Jitter (Coefficient of Variation)",
x = "Configuration",
y = "CV (lower = less jitter)")
ggsave("02_jitter_control.png", width = 10, height = 6)
# 4. Visualization: Convergence Speed
ggplot(doe_data, aes(x = reorder(configuration, decay_slope_convergence_rate, FUN=median),
y = decay_slope_convergence_rate,
fill = configuration)) +
geom_boxplot(alpha = 0.7) +
coord_flip() +
theme_minimal() +
labs(title = "Decay Slope Convergence Speed",
x = "Configuration",
y = "Convergence Rate (ticks)")
ggsave("03_convergence_speed.png", width = 10, height = 6)
# 5. Correlation: Load Response Coupling
ggplot(doe_data, aes(x = load_interval_correlation,
y = overall_stability_score,
color = configuration)) +
geom_point(alpha = 0.6, size = 2) +
facet_wrap(~configuration) +
theme_minimal() +
labs(title = "Load-Response Coupling vs Stability",
x = "Correlation(load, interval)",
y = "Overall Stability Score")
ggsave("04_load_response_coupling.png", width = 12, height = 8)
# 6. Statistical Test: Is top config significantly better?
top_config <- stability_rank$configuration[1]
other_configs <- stability_rank$configuration[-1]
top_data <- doe_data %>% filter(configuration == top_config) %>% pull(overall_stability_score)
other_data <- doe_data %>% filter(configuration %in% other_configs) %>% pull(overall_stability_score)
t_test <- t.test(top_data, other_data)
print(sprintf("\nT-test: %s vs others\np-value: %.4f", top_config, t_test$p.value))
# 7. Golden Configuration Summary
cat("\n=== GOLDEN CONFIGURATION CANDIDATE ===\n")
golden <- stability_rank %>% slice(1)
print(golden)
cat("\nInterpretation:")
cat(sprintf("\n- Lowest jitter (CV): %.4f", golden$mean_jitter_cv))
cat(sprintf("\n- Fastest convergence: %.1f ticks", golden$mean_convergence_rate))
cat(sprintf("\n- Settling time: %.1f ticks", golden$mean_settling_time))
cat(sprintf("\n- Recommendation: Use '%s' as golden configuration\n", golden$configuration))
```
---
## Execution Plan
### Phase 1: Implementation (3-4 hours)
1. Extend `DoeMetrics` struct with heartbeat columns
2. Implement `analyze_heartbeat_trajectory()` in `doe_metrics.c`
3. Capture snapshot data during heartbeat ticks
4. Update CSV schema
### Phase 2: Experiment Execution (2-4 hours)
1. Create `run_factorial_doe_with_heartbeat.sh` script
2. Run 5 elite configurations × 50 runs each = 250 total runs
3. Collect heartbeat + performance data
### Phase 3: Analysis (2-3 hours)
1. Load CSV in R
2. Compute stability rankings
3. Generate visualizations
4. Identify golden configuration
---
## Expected Outcomes
**Primary Question:** Which of the 5 elite configurations exhibits:
- Lowest heartbeat jitter (CV)?
- Fastest convergence to steady state?
- Strongest load-response correlation?
- Cleanest temporal signature?
**Expected Winner:** Configuration with balanced F-factors, likely:
- F1=1 (heat tracking required)
- F3=1 or F5=1 (decay or inference, but not both)
- F6=0 or F6=1 (decay inference optional)
**Golden Configuration:** Becomes baseline physics for MamaForth & PapaForth downstream work.
---
## Success Criteria
✓ All 5 configurations build and run without segfault
✓ Heartbeat data collected for all 250 runs
✓ Stability rankings computed and visualized
✓ Golden configuration identified with p < 0.05 significance
✓ R analysis report generated with 5+ plots
---
**Version:** 1.0
**Status:** Ready for implementation
**Next Step:** Implement heartbeat metrics capture and extended DoE script
docs/working/archive/superseded/heartbeat-rate-logging-plan.md
+545
View File
@@ -0,0 +1,545 @@
> **SUPERSEDED (docs triage 2026-10-14).** Flagged superseded 2026-10-09.
> Instead: its own header. Moved here from `docs/working/architecture/03-architecture/heartbeat-system/implementation-plan.md`.
<!-- Moved from docs/03-architecture/heartbeat-system/implementation-plan.md to docs/working/archive/superseded/heartbeat-rate-logging-plan.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ SUPERSEDED (flagged 2026-10-09).** Superseded by `instrumentation-plan.md`. Proposes the
> earlier `HeartbeatRateSample` design; the live target is `HeartbeatTickSnapshot`
> (`include/vm.h`) with a `tick_buffer` and `heartbeat_export_csv()`. Kept for historical
> record.
# Implementation Plan: Heartbeat Rate Logging
**Date:** November 20, 2025
**Priority:** BLOCKING — Must implement before running Phase 2 DoE
**Complexity:** Medium (requires code additions, CSV schema update)
---
## Overview
We need to capture **tick_ns trajectory** (the actual heartbeat rate changes) during execution, then analyze how each configuration modulates heartrate in response to workload.
---
## Step 1: Add Heartbeat Rate Sample Struct
### File: `include/vm.h`
Add after `HeartbeatSnapshot` definition:
```c
/* === Heartbeat Rate Trajectory Collection === */
#define HEARTBEAT_RATE_SAMPLE_MAX 100000 /* Maximum samples per run */
typedef struct {
uint64_t tick_number; /* Absolute tick count */
uint64_t tick_ns; /* ← HEARTBEAT RATE (nanoseconds) */
uint64_t workload_ops_this_tick; /* Dictionary lookups this tick */
uint64_t timestamp_ns; /* Wall-clock timestamp */
uint64_t hot_word_count; /* Physics state: hot words */
uint64_t total_heat; /* Physics state: total execution heat */
} HeartbeatRateSample;
```
### File: `include/vm.h` (in VM struct)
Add to the HeartbeatState section:
```c
/** @name Heartbeat Rate Trajectory (Phase 2 DoE)
* @{
*/
HeartbeatRateSample* heartbeat_rate_samples; /* Circular buffer of rate samples */
uint64_t heartbeat_rate_sample_count; /* Number of samples collected */
/** @} */
```
---
## Step 2: Initialize Heartbeat Rate Collection
### File: `src/vm.c` in `vm_init()`
Add after heartbeat initialization:
```c
/* Initialize heartbeat rate sampling for Phase 2 DoE */
vm->heartbeat_rate_samples = calloc(HEARTBEAT_RATE_SAMPLE_MAX, sizeof(HeartbeatRateSample));
vm->heartbeat_rate_sample_count = 0;
```
### File: `src/vm.c` in `vm_cleanup()`
Add cleanup:
```c
if (vm->heartbeat_rate_samples) {
free(vm->heartbeat_rate_samples);
vm->heartbeat_rate_samples = NULL;
}
```
---
## Step 3: Capture Heartbeat Rate Per Tick
### File: `src/vm.c` - Add helper function
```c
/**
* @brief Capture current heartbeat rate sample for analysis
*
* Called by heartbeat_thread_main() to log the tick_ns (heartbeat rate)
* along with contemporaneous workload metrics.
*
* @param vm Pointer to VM
* @param tick_ns Current heartbeat interval (nanoseconds)
* @param workload_ops Dictionary lookups executed this tick
*/
static void capture_heartbeat_rate_sample(VM *vm, uint64_t tick_ns, uint64_t workload_ops)
{
if (!vm || !vm->heartbeat_rate_samples)
return;
if (vm->heartbeat_rate_sample_count >= HEARTBEAT_RATE_SAMPLE_MAX)
return; /* Buffer full, stop sampling */
HeartbeatRateSample *sample = &vm->heartbeat_rate_samples[vm->heartbeat_rate_sample_count];
sample->tick_number = vm->heartbeat.tick_count;
sample->tick_ns = tick_ns;
sample->workload_ops_this_tick = workload_ops;
sample->timestamp_ns = platform_monotonic_ns();
/* Capture current physics state */
sample->hot_word_count = vm->hot_word_count_at_check;
sample->total_heat = vm->total_heat_at_last_check;
vm->heartbeat_rate_sample_count++;
}
```
---
## Step 4: Log Workload Per Tick
### File: `src/vm.c` - Track lookups per tick
Add to VM struct (in `include/vm.h`):
```c
/** @name Per-Tick Workload Metrics
* @{
*/
uint64_t workload_ops_current_tick; /* Dictionary lookups in current tick */
/** @} */
```
### File: `src/vm.c` - In `vm_heartbeat_run_cycle()`
**Before calling inference engine:**
```c
void vm_heartbeat_run_cycle(VM *vm)
{
if (!vm || !vm->heartbeat.heartbeat_enabled)
return;
vm->heartbeat.tick_count++;
/* Reset per-tick workload counter */
uint64_t lookups_this_tick = vm->workload_ops_current_tick;
vm->workload_ops_current_tick = 0; /* Reset for next tick */
/* ... existing inference logic ... */
}
```
### File: `src/vm.c` - Increment lookup counter
In the main word execution loop (in `vm_interpret()` or inner interpreter):
```c
/* After successful dictionary lookup */
vm->workload_ops_current_tick++;
```
---
## Step 5: Capture in Heartbeat Thread
### File: `src/vm.c` in `heartbeat_thread_main()`
Replace the existing sleep logic:
```c
static void* heartbeat_thread_main(void *arg)
{
VM *vm = (VM*)arg;
if (!vm || !vm->heartbeat.worker)
return NULL;
HeartbeatWorker *worker = vm->heartbeat.worker;
worker->running = 1;
while (!worker->stop_requested && vm->heartbeat.heartbeat_enabled)
{
vm_heartbeat_run_cycle(vm);
/* GET THE CURRENT HEARTBEAT RATE */
uint64_t tick_ns = worker->tick_ns ? worker->tick_ns : HEARTBEAT_TICK_NS;
uint64_t workload_ops = vm->workload_ops_current_tick;
/* ↓ ADD THIS ↓ CAPTURE HEARTBEAT RATE SAMPLE */
capture_heartbeat_rate_sample(vm, tick_ns, workload_ops);
/* ↑ ADD THIS ↑ */
struct timespec req;
req.tv_sec = (time_t)(tick_ns / 1000000000ULL);
req.tv_nsec = (long)(tick_ns % 1000000000ULL);
struct timespec rem;
while (!worker->stop_requested && vm->heartbeat.heartbeat_enabled &&
nanosleep(&req, &rem) == -1 && errno == EINTR)
{
req = rem;
}
}
worker->running = 0;
return NULL;
}
```
---
## Step 6: Extend DoeMetrics
### File: `include/doe_metrics.h`
Add to `DoeMetrics` struct:
```c
/* === HEARTBEAT RATE MODULATION METRICS (NEW) === */
/* Per-run heartbeat rate statistics */
uint64_t total_heartbeat_ticks;
double mean_heartbeat_rate_hz; /* Average frequency (Hz) */
double heartbeat_rate_stddev_hz; /* Variation in Hz */
double heartbeat_rate_cv; /* CV of heartbeat rate */
uint64_t heartbeat_rate_min_ns; /* Slowest tick observed */
uint64_t heartbeat_rate_max_ns; /* Fastest tick observed */
double heartbeat_rate_modulation_range_hz; /* max_hz - min_hz */
/* Load-Heartrate Response */
double load_to_heartrate_correlation; /* Correlation(workload, heartrate) */
int load_response_direction; /* +1 if slower under load, -1 if faster */
double heartrate_adaptation_latency_ticks; /* Ticks until rate responds to load */
double heartrate_settling_time_ticks; /* Ticks to settle after adaptation */
/* Heartrate Convergence */
double heartrate_convergence_time_ms; /* Time to reach stable rate */
int heartrate_converged; /* 1 = converged, 0 = oscillating */
double heartrate_oscillation_amplitude_hz; /* Peak-to-peak variation in final state */
/* Overall Heartrate Quality Score */
double heartrate_responsiveness_score; /* 0-100: how well does it respond? */
double heartrate_stability_score; /* 0-100: how smooth is adaptation? */
```
---
## Step 7: Implement Metrics Extraction
### File: `src/doe_metrics.c`
Add new function:
```c
/**
* @brief Analyze heartbeat rate trajectory and compute metrics
*
* @param vm Pointer to VM with heartbeat_rate_samples
* @param metrics Pointer to DoeMetrics to fill
*/
static void analyze_heartbeat_rate_trajectory(VM *vm, DoeMetrics *metrics)
{
if (!vm || !vm->heartbeat_rate_samples || vm->heartbeat_rate_sample_count == 0) {
metrics->mean_heartbeat_rate_hz = 0;
metrics->load_to_heartrate_correlation = 0;
return;
}
uint64_t count = vm->heartbeat_rate_sample_count;
HeartbeatRateSample *samples = vm->heartbeat_rate_samples;
/* 1. Compute heartbeat rate statistics (tick_ns → Hz) */
double *heartrate_hz = malloc(count * sizeof(double));
double *workload_intensity = malloc(count * sizeof(double));
double min_ns = samples[0].tick_ns;
double max_ns = samples[0].tick_ns;
double sum_hz = 0;
double sum_workload = 0;
for (uint64_t i = 0; i < count; i++) {
double tick_ns = samples[i].tick_ns;
heartrate_hz[i] = 1e9 / tick_ns;
workload_intensity[i] = samples[i].workload_ops_this_tick;
sum_hz += heartrate_hz[i];
sum_workload += workload_intensity[i];
if (tick_ns < min_ns) min_ns = tick_ns;
if (tick_ns > max_ns) max_ns = tick_ns;
}
/* Mean heartbeat rate */
metrics->mean_heartbeat_rate_hz = sum_hz / count;
metrics->heartbeat_rate_min_ns = (uint64_t)min_ns;
metrics->heartbeat_rate_max_ns = (uint64_t)max_ns;
metrics->heartbeat_rate_modulation_range_hz = (1e9 / min_ns) - (1e9 / max_ns);
/* 2. Compute standard deviation */
double variance = 0;
for (uint64_t i = 0; i < count; i++) {
double delta = heartrate_hz[i] - metrics->mean_heartbeat_rate_hz;
variance += delta * delta;
}
metrics->heartbeat_rate_stddev_hz = sqrt(variance / count);
metrics->heartbeat_rate_cv = metrics->heartbeat_rate_stddev_hz / metrics->mean_heartbeat_rate_hz;
/* 3. Compute correlation(workload, heartrate) — THE KEY METRIC */
double workload_mean = sum_workload / count;
double cov = 0;
double workload_var = 0;
for (uint64_t i = 0; i < count; i++) {
double workload_delta = workload_intensity[i] - workload_mean;
double heartrate_delta = heartrate_hz[i] - metrics->mean_heartbeat_rate_hz;
cov += workload_delta * heartrate_delta;
workload_var += workload_delta * workload_delta;
}
if (workload_var > 0 && metrics->heartbeat_rate_stddev_hz > 0) {
metrics->load_to_heartrate_correlation = cov / sqrt(workload_var *
(metrics->heartbeat_rate_stddev_hz * metrics->heartbeat_rate_stddev_hz * count));
}
/* 4. Determine if heartrate responds correctly to load */
metrics->load_response_direction = (metrics->load_to_heartrate_correlation > 0.3) ? 1 : -1;
/* 5. Estimate convergence time (when heartrate stabilizes) */
double threshold = 0.15; /* 15% of final value */
metrics->heartrate_converged = 1;
for (uint64_t i = count / 2; i < count; i++) {
double current_cv = 0;
/* Compute CV of last N samples */
for (uint64_t j = i; j < (i + 100 < count ? i + 100 : count); j++) {
double delta = heartrate_hz[j] - metrics->mean_heartbeat_rate_hz;
current_cv += delta * delta;
}
if (sqrt(current_cv) > threshold * metrics->mean_heartbeat_rate_hz) {
metrics->heartrate_converged = 0;
metrics->heartrate_convergence_time_ms = i; /* Approximate */
break;
}
}
/* 6. Compute overall responsiveness score */
/* Higher correlation + faster convergence + lower CV = higher score */
metrics->heartrate_responsiveness_score =
(metrics->load_to_heartrate_correlation + 1.0) * 50 *
(1.0 / (1.0 + metrics->heartbeat_rate_cv));
metrics->heartrate_stability_score =
(100.0 * (1.0 - metrics->heartbeat_rate_cv)) *
(metrics->heartrate_converged ? 1.0 : 0.5);
free(heartrate_hz);
free(workload_intensity);
}
```
### Update `metrics_from_vm()`
Call the new analysis function:
```c
DoeMetrics metrics_from_vm(VM *vm, ...) {
/* ... existing metrics extraction ... */
/* NEW: Analyze heartbeat rate trajectory */
analyze_heartbeat_rate_trajectory(vm, &metrics);
return metrics;
}
```
---
## Step 8: Update CSV Schema
### File: `src/doe_metrics.c` in `metrics_write_csv_header()`
Add new columns:
```c
void metrics_write_csv_header(FILE *out) {
{
printf '[...existing columns...]');
printf('total_heartbeat_ticks,mean_heartbeat_rate_hz,heartbeat_rate_stddev_hz,heartbeat_rate_cv,'
'heartbeat_rate_min_ns,heartbeat_rate_max_ns,heartbeat_rate_modulation_range_hz,'
'load_to_heartrate_correlation,load_response_direction,'
'heartrate_convergence_time_ms,heartrate_converged,'
'heartrate_responsiveness_score,heartrate_stability_score\n');
} > output_csv;
}
```
### File: `src/doe_metrics.c` in `metrics_write_csv_row()`
Add the values:
```c
void metrics_write_csv_row(FILE *out, const DoeMetrics *metrics) {
fprintf(out, "[...existing fields...],"
"%lu,%lf,%lf,%lf,%lu,%lu,%lf,%lf,%d,%lu,%d,%lf,%lf\n",
metrics->total_heartbeat_ticks,
metrics->mean_heartbeat_rate_hz,
metrics->heartbeat_rate_stddev_hz,
metrics->heartbeat_rate_cv,
metrics->heartbeat_rate_min_ns,
metrics->heartbeat_rate_max_ns,
metrics->heartbeat_rate_modulation_range_hz,
metrics->load_to_heartrate_correlation,
metrics->load_response_direction,
metrics->heartrate_convergence_time_ms,
metrics->heartrate_converged,
metrics->heartrate_responsiveness_score,
metrics->heartrate_stability_score);
}
```
---
## Step 9: Updated R Analysis
### File: `scripts/analyze_heartbeat_stability_REVISED.R`
```r
library(tidyverse)
# Load data
doe_data <- read.csv("experiment_results_heartbeat.csv") %>%
mutate(
configuration = as.factor(configuration),
# New metrics available now
load_response = ifelse(load_to_heartrate_correlation > 0.5, "Responsive", "Sluggish")
)
# PRIMARY METRIC: Load-Heartrate Correlation
# This tells us which config best responds to workload changes
ranking <- doe_data %>%
group_by(configuration) %>%
summarize(
mean_correlation = mean(load_to_heartrate_correlation, na.rm = TRUE),
mean_responsiveness = mean(heartrate_responsiveness_score, na.rm = TRUE),
mean_stability = mean(heartrate_stability_score, na.rm = TRUE),
mean_convergence_time = mean(heartrate_convergence_time_ms, na.rm = TRUE),
golden_score = (mean(load_to_heartrate_correlation) * 0.5) +
(mean(heartrate_responsiveness_score) * 0.3) +
(mean(heartrate_stability_score) * 0.2)
) %>%
arrange(-golden_score)
print("=== HEARTBEAT RATE RESPONSIVENESS RANKING ===")
print(ranking)
# Plot 1: Load-Heartrate Correlation (KEY METRIC)
ggplot(doe_data, aes(x = reorder(configuration, load_to_heartrate_correlation, FUN=median),
y = load_to_heartrate_correlation,
fill = configuration)) +
geom_boxplot() +
coord_flip() +
labs(title = "Load-Heartrate Correlation (Higher = Better)",
y = "Correlation(workload, heartrate)")
ggsave("01_load_heartrate_correlation.png")
# Plot 2: Heartrate Responsiveness Score
ggplot(doe_data, aes(x = reorder(configuration, heartrate_responsiveness_score, FUN=median),
y = heartrate_responsiveness_score,
fill = configuration)) +
geom_boxplot() +
coord_flip() +
labs(title = "Heartrate Responsiveness Score (0-100)")
ggsave("02_heartrate_responsiveness.png")
```
---
## Compilation Steps
1. **Update headers:** Add structs and fields to `include/vm.h`, `include/doe_metrics.h`
2. **Implement collection:** Add capture function in `src/vm.c`
3. **Implement analysis:** Add analysis function in `src/doe_metrics.c`
4. **Update CSV:** Modify CSV header/row writers
5. **Compile:** `make clean && make fastest`
6. **Test:** `./build/starforth -c "1 2 + . BYE"`
---
## Verification
After implementation, verify:
✓ Binary compiles without errors
✓ Heartbeat rate samples are captured (check sample_count > 0 after run)
✓ CSV contains new heartrate columns
✓ Correlation metrics are computed
✓ R analysis generates ranking by load-response
---
## Expected Results
When properly implemented, you'll see:
```
=== HEARTBEAT RATE RESPONSIVENESS RANKING ===
configuration mean_correlation mean_responsiveness golden_score
1_0_1_1_1_0 0.82 78.5 0.795
1_0_1_1_1_1 0.78 75.2 0.761
1_1_0_1_1_1 0.65 62.3 0.641
1_0_1_0_1_0 0.45 48.1 0.451
0_1_1_0_1_1 0.12 15.8 0.148
```
**Golden config:** 1_0_1_1_1_0 (strongest load-heartrate coupling)
---
## Timeline
- **Implementation:** 2-3 hours
- **Testing:** 1 hour
- **Re-run Phase 2:** 2-4 hours
- **Analysis:** 30 min
- **Total:** ~6-9 hours
---
## Files Modified
- `include/vm.h` — Add HeartbeatRateSample struct and VM fields
- `include/doe_metrics.h` — Add heartrate metrics to DoeMetrics
- `src/vm.c` — Add sample capture, logging, initialization
- `src/doe_metrics.c` — Add analysis function, CSV updates
- `scripts/analyze_heartbeat_stability_REVISED.R` — Rewritten for load-response focus
---
**This implementation captures the actual heartbeat rate modulation and measures how well each config responds to load.**
This is the correct physics story. 🚀
docs/working/archive/superseded/loop-5-sketch.md
+278
View File
@@ -0,0 +1,278 @@
> **SUPERSEDED (docs triage 2026-10-14).** Flagged superseded 2026-10-09 (an early design sketch of Loop #5).
> Instead: its own header; Loop #5 as built is v3/src/inference_engine.c. Moved here from `docs/working/architecture/03-architecture/adaptive-systems/loop-5-sketch.md`.
<!-- Moved from docs/03-architecture/adaptive-systems/loop-5-sketch.md to docs/working/archive/superseded/loop-5-sketch.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ SUPERSEDED (flagged 2026-10-09).** Superseded by `window-inference-redesign.md`
> (Levene's-test window inference — the shipped Loop #5 mechanism). This undated sketch ended
> in an open question and was never approved as designed. Kept for historical record.
# Loop #5: Context-Aware Window Tuning - Design Sketch
## The Challenge
**Current state:** Window size shrinks based on pattern diversity, but we don't know if shrinking improves or hurts **pipelining prediction accuracy**.
**Goal:** Use actual prefetch hit rate to guide window size tuning via binary chop search.
---
## What Loop #5 Should Do
### High-Level Flow
1. **Measure**: Track global pipelining metrics across ALL words
- Total speculative prefetch attempts
- Successful prefetch hits
- Current accuracy = hits / attempts
2. **Trigger**: Periodically (e.g., every 1000 executions when warm)
- Calculate prediction accuracy at current effective_window_size
- Call binary chop to suggest next size to try
3. **Adapt**: Shrink or grow effective_window_size based on accuracy trend
- If smaller window → better accuracy: keep shrinking
- If smaller window → worse accuracy: grow back up
- Converge to sweet spot
4. **Measure**: Record the accuracy at each window size tried
- Build history of (window_size → accuracy) pairs
- Use this history to accelerate convergence
---
## Architecture Needed
### 1. VM-Level Metrics (New)
**File:** `include/vm.h` (add to VM struct)
```c
typedef struct {
uint64_t prefetch_attempts; /* Total speculative prefetch calls */
uint64_t prefetch_hits; /* Successful hits (word was looked up next) */
uint64_t window_tuning_checks; /* How many times we've checked window size */
uint32_t last_checked_window_size;
double last_checked_accuracy;
uint32_t suggested_next_size; /* What binary chop recommends trying */
} PipelineGlobalMetrics;
```
**Add to VM struct:**
```c
struct VM {
// ... existing fields ...
PipelineGlobalMetrics pipeline_metrics; /* NEW: aggregated metrics */
};
```
### 2. Update Prefetch Feedback (in vm.c)
**When speculative prefetch succeeds:**
```c
// In execute_colon_word(), after promotion:
if (spec_entry && vm->hotwords_cache && ENABLE_HOTWORDS_CACHE)
{
spec_entry->execution_heat = HOTWORDS_EXECUTION_HEAT_THRESHOLD + 1;
hotwords_cache_promote(vm->hotwords_cache, spec_entry);
// NEW: Record the prefetch attempt globally
vm->pipeline_metrics.prefetch_attempts++;
prev_word->transition_metrics->prefetch_attempts++; // Per-word too
}
```
**When word is actually looked up (check if it was speculated):**
```c
// In vm_interpret_word(), during dictionary lookup:
if (entry && ENABLE_PIPELINING && vm->hotwords_cache)
{
// NEW: Check if this word was in speculated cache (dirty bit or timestamp)
if (word_was_speculatively_promoted_recently(entry))
{
vm->pipeline_metrics.prefetch_hits++; // Global metric
// per_word_metrics->prefetch_hits++; // Already done in prefetch logic
}
}
```
### 3. Window Tuning Decision Loop
**New function:** `physics_pipelining_metrics.c`
```c
/**
* Suggest window size via binary chop based on prefetch accuracy trend.
* Called periodically (every 1000 executions when pipelining is enabled).
*/
uint32_t loop_5_binary_chop_suggest_window(
const PipelineGlobalMetrics *global,
uint32_t current_window_size,
uint32_t min_window,
uint32_t max_window)
{
if (global->prefetch_attempts == 0)
return current_window_size; /* Not enough data yet */
double current_accuracy = (double)global->prefetch_hits / (double)global->prefetch_attempts;
/* Binary chop: try smaller window first (aggressive shrinking)
* If accuracy drops, back off and try larger windows */
if (global->window_tuning_checks == 0)
{
/* First check: try shrinking by 25% */
return (current_window_size * 75) / 100;
}
/* Compare current accuracy to last check */
double accuracy_delta = current_accuracy - global->last_checked_accuracy;
if (accuracy_delta > 0.01) /* Improvement threshold: 1% */
{
/* Accuracy improved! Try shrinking more */
uint32_t smaller = (current_window_size * 75) / 100;
return (smaller > min_window) ? smaller : min_window;
}
else if (accuracy_delta < -0.01)
{
/* Accuracy degraded. Try growing instead */
uint32_t larger = (current_window_size * 133) / 100; /* Grow by ~33% */
return (larger < max_window) ? larger : max_window;
}
else
{
/* Plateau: stick with current size */
return current_window_size;
}
}
```
### 4. Integration Point
**In rolling_window_of_truth.c: `rolling_window_check_adaptive_shrink()`**
```c
void rolling_window_check_adaptive_shrink(RollingWindowOfTruth* window)
{
// ... existing adaptive shrinking code ...
/* Loop #5: Binary chop window tuning (Phase 2 IMPLEMENTATION) */
if (ENABLE_PIPELINING && window->is_warm)
{
window->adaptive_check_count++;
/* Check every 1000 executions for tuning opportunity */
if (window->adaptive_check_count % 1000 == 0)
{
PipelineGlobalMetrics *metrics = &vm->pipeline_metrics;
/* Calculate current accuracy */
double current_accuracy = (metrics->prefetch_attempts > 0)
? (double)metrics->prefetch_hits / (double)metrics->prefetch_attempts
: 0.0;
/* Ask binary chop for next window size to try */
uint32_t suggested_size = loop_5_binary_chop_suggest_window(
metrics,
window->effective_window_size,
ADAPTIVE_MIN_WINDOW_SIZE,
ROLLING_WINDOW_SIZE
);
/* Apply if different */
if (suggested_size != window->effective_window_size)
{
log_message(LOG_INFO,
"Loop #5: Binary chop tuning: %u → %u (accuracy %.2f%%, %lu/%lu hits)",
window->effective_window_size,
suggested_size,
current_accuracy * 100.0,
metrics->prefetch_hits,
metrics->prefetch_attempts);
window->effective_window_size = suggested_size;
}
/* Record this check for next iteration */
metrics->last_checked_window_size = window->effective_window_size;
metrics->last_checked_accuracy = current_accuracy;
metrics->window_tuning_checks++;
}
}
}
```
---
## Key Design Decisions
### A. Accuracy Measurement
- **Count**: Successful speculative promotions that were actually used
- **Challenge**: How to know if lookup was "predicted" vs "natural"?
- Option 1: Mark speculatively-promoted entries with a timestamp/flag
- Option 2: Track word IDs predicted and check if they match next lookup
- Option 3: Use per-word prefetch metrics already collected
### B. Tuning Frequency
- **Every 1000 executions** (when window is warm)
- Could be tunable: `make WINDOW_TUNING_FREQUENCY=1000`
### C. Search Strategy
- **Greedy shrinking first**: Try smaller windows aggressively
- **Backoff on degradation**: If accuracy drops, grow back up
- **Convergence**: Stop when oscillating around sweet spot
### D. Knobs Needed (Makefile)
```bash
make ENABLE_PIPELINING=1 # Must be enabled for Loop #5
make WINDOW_TUNING_FREQUENCY=1000 # How often to check
make WINDOW_TUNING_ACCURACY_THRESHOLD=1 # % improvement to continue shrinking
```
---
## Validation Approach
Once Loop #5 is wired, DoE would test:
- **A_BASELINE**: No optimizations
- **A_B_CACHE**: Cache only
- **A_C_FULL**: Window tuning with pipelining only
- **A_B_C_FULL**: Cache + window tuning + pipelining
**Metrics to collect:**
- Final effective_window_size (what did it converge to?)
- Final prefetch accuracy (%)
- Throughput improvement vs baseline
---
## Risks & Unknowns
1. **Overhead**: Checking accuracy every 1000 executions adds latency
2. **Noise**: If accuracy is noisy, binary chop might oscillate
3. **Workload-dependent**: Optimal window size might vary by workload
4. **False positives**: Speculative lookup might not correlate with prediction
---
## Sketch Summary
**Loop #5 would:**
1. Track global prefetch accuracy (hits/attempts)
2. Every 1000 executions, ask binary chop: "Is current window size optimal?"
3. Binary chop suggests shrinking, holding, or growing based on accuracy trend
4. Apply suggestion and record for next iteration
5. Over time, converge to window size that maximizes prediction accuracy
**Code locations if implemented:**
- VM struct: Add PipelineGlobalMetrics field
- vm.c: Update global metrics on prefetch attempts/hits
- physics_pipelining_metrics.c: `loop_5_binary_chop_suggest_window()`
- rolling_window_of_truth.c: Call binary chop from adaptive shrink check
**Effort**: ~150 LOC across 3-4 files
---
**Question for user:** Does this architecture align with what you had in mind for Loop #5?
docs/working/archive/superseded/steady-state-machine.md
+223
View File
@@ -0,0 +1,223 @@
> **SUPERSEDED (docs triage 2026-10-14).** Flagged obsolete 2026-10-09.
> Instead: its own header. Moved here from `docs/working/architecture/03-architecture/physics-engine/steady-state-machine.md`.
<!-- Moved from docs/03-architecture/physics-engine/steady-state-machine.md to docs/working/archive/superseded/steady-state-machine.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ OBSOLETE (flagged 2026-10-09).** Conceptual precursor to the L8 Jacquard mode-selector
> system; it never mentions Jacquard/L8, the attractor buckets, or the six named modes, and
> was never reconciled with the implemented `ssm_jacquard.c` system. The canonical current
> reference for the physics/feedback model is `feedback-loops.md`. Kept for historical record.
# 1. Introduction
A Steady State Machine (SSM) is a novel computational model in which the system’s operational state is defined not by discrete symbolic transitions, but by convergence to an optimal runtime equilibrium.
Unlike traditional models such as Finite State Machines (FSMs), Pushdown Automata, or Turing Machines — where “state” is explicit and symbolic — a Steady State Machine derives its behavior from continuous thermodynamic variables such as:
* execution_heat
* entropy (diversity)
* pipeline readiness
* heat decay
* steady-state slope
* localized temperature gradients
* adaptive strategy selection
An SSM observes itself, adapts, and settles into stable performance basins called steady states.
This makes the Steady State Machine the first computational model where the attractor defines the machine, not the symbol.
# 2. Core Definition
Definition 1 — Steady State Machine
> A Steady State Machine is a computational system defined by:
A set of internal physical metrics
* 𝐻 = execution_heats
* 𝐸 = entropy/diversity measures
* 𝑃 = pipeline activation thresholds
* Θ = heat decay profile
* 𝑊 = sliding execution window
A set of adaptive policies
* lookup strategies
* caching regimes
* bucket reorganizations
* pipeline decisions
* inference-based mode switches
* A state-space composed of attractor basins, not symbolic states
* steady attractor
* quasi-steady attractor
* transient attractor
* chaotic attractor
* A governing dynamic
### 𝑀𝑡+1=𝑓(𝑀𝑡,Δ𝐻,Δ𝑊,Δ𝐸) where 𝑓 pushes the machine toward a stable basin.
The machine is defined by its convergence behavior, not its instruction sequence.
# 3. The Fundamental Insight
In an SSM, performance is the state.
The system is constantly measuring:
* how hot words are
* how often patterns recur
* how pipeline-able a sequence is
* how diverse the window is
* how much the dictionary should reorganize
And as these stabilize, the machine settles into a steady state — a self-maintaining optimal regime.
This regime is preserved until the environment changes (e.g., new workloads, diversity spikes), at which point the SSM moves into a transient or chaotic regime and eventually finds a new equilibrium.
# 4. Phases of a Steady State Machine
## 1. Transient Phase
- System warms up
- Execution patterns are sporadic
- Heat gradients are noisy
- Pipeline inactive or unstable
## 2. Quasi-Steady Phase
- Some patterns emerge
- Lookups begin to bias
- Hot words form local attractors
- Dictionary reorders stabilize
## 3. True Steady State
- Heat surface smooth
- Pipeline active
- Diversity window predictable
- Lookups converge to optimal
- Runtime > baseline VM
- System remains stable unless perturbed
## 4. Chaotic/Disruption Phase
### Triggered by:
- major workload shift
- dictionary mutation
- extreme entropy spike
- cold-cache shock
- SSM falls out of equilibrium temporarily.
## 5. Axioms of Steady State Machines
>Axiom 1 — Convergence
Every sustained workload induces the SSM to converge toward a stable attractor unless external entropy forces divergence.
>Axiom 2 — Locality of Heat
Execution heat reflects both locality and temporal relevance. Locality produces stability.
>Axiom 3 — Adaptive Reordering
Reordering is not symbolic mutation; it is thermodynamic self-optimization.
>Axiom 4 — Stability Maximizes Throughput
The steady state is always faster than the cold state, and usually faster than the naive baseline VM.
>Axiom 5 — Perturbation Response
When disrupted, the SSM seeks a new steady state; it does not thrash indefinitely.
## 6. Relationship to Other Computational Models
>FSM vs SSM
FSM: discrete symbolic states
SSM: emergent attractors in thermal space
>Turing Machine vs SSM
TM: behavior defined by tape symbols and transitions
SSM: behavior defined by adaptation toward equilibrium
>Markov Chain vs SSM
MC: probabilistic transitions between explicit states
SSM: deterministic drift toward stable basins, driven by observation
>Neural Network vs SSM
NN: learns weights
SSM: reorganizes runtime structures, not parameters
## 7. Why This Breaks New Ground
### A Steady State Machine is the first:
- runtime-thermodynamic computing model
- adaptive VM with convergence properties
- load-dependent reordering machine
- self-optimizing dictionary machine
- phase-shifting execution system
- state-machine whose state is emergent, not explicit
- This is a fundamental new category of computation.
## 8. Canonical Example: StarForth
### StarForth implements the first real SSM with:
- execution_heat
- rolling diversity windows
- phase-based lookup strategies
- adaptive dictionary ordering
- pipeline activation
- DOE-calibrated inference loop
- background physics monitor
- It is, quite literally, the reference implementation.
## 9. Formal Summary Statement
>A Steady State Machine (SSM) is a computational system in which the operational state is defined by attractor convergence in thermodynamic runtime space rather than by discrete symbolic transitions. An SSM continuously measures workload characteristics, adjusts internal structures, and stabilizes into performance-optimized steady states. When perturbed, it transitions through transient or chaotic phases before finding a new equilibrium. This model defines a new class of adaptive computational systems with properties distinct from finite state machines, pushdown automata, and Turing Machines.