rajames/LithosAnanke
1
1
Fork 0
Code Issues 83 Pull Requests 1 Actions 1 Packages Projects 1 Releases 6 Activity

docs: reorganise -- current reference gathered, finished work archived, superseded and dead ideas marked as such

Browse Source 
docs/CLAUDE.md Phase 2, 2026-10-14.  All moves by git mv; every archived
file carries a header saying what it is, why, and where to look instead.

- docs/reference/ (new): roadmap and versioning policy, M7.1, word-ACL
  design, kernel args, hosted acceptance test, dictionary, primitives,
  ontology, L8 table -- previously spread over docs/, docs/lithosananke/
  and docs/03-architecture/.
- docs/working/archive/historical/: finished work (session plan, test-gap
  audit, birthing plan, fix write-ups, the January architecture draft).
- docs/working/archive/superseded/: the Hermes-VM Tripod docs and July
  Hermes designs (-> FABRIC-3.5/3.6), the StarForth roadmap, every working
  doc already flagged superseded or obsolete.
- docs/working/archive/dead/: the hal_* target contract (never adopted),
  L4Re, metrics-and-knobs.
- Papers and proposals into docs/working/papers/.
- References rewritten across the repo (not logs/, not docs/patent/);
  CLAUDE.md's documentation section rewritten.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
rajames
2026-10-14 16:03:56 -04:00
co-authored by Claude Opus 5.5
parent 1a4b163861
commit 7aed588f78
116 changed files with 305 additions and 175 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/working/archive/dead/hal-target-contract/README.md
+246
View File
@@ -0,0 +1,246 @@
> **DEAD (docs triage 2026-10-14).** The proposed `hal_*` interface contract and its migration plan were never adopted; the kernel grew its own HAL (kernel/src/hal/, kernel/include/starkernel/hal/).
> Instead: kernel/src/hal/README.md. Moved here from `docs/working/architecture/03-architecture/hal/README.md`.
<!-- Moved from docs/03-architecture/hal/README.md to docs/working/archive/dead/hal-target-contract/README.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ TARGET HAL — NOT THE BUILT INTERFACE (flagged 2026-10-09).** This document family
> describes a proposed `hal_*`-named interface and a `src/platform/<platform>/` layout that
> was never adopted as written. The implemented kernel HAL lives in `src/starkernel/hal/`
> with different (time-backend-style) naming. Treat all five documents here as design intent,
> not the current API. For current architecture see `.claude/CLAUDE.md`.
# Hardware Abstraction Layer (HAL) Documentation
This directory contains comprehensive documentation for StarForth's Hardware Abstraction Layer (HAL), which enables the evolution from hosted VM to StarKernel to StarshipOS.
> **HISTORICAL — L4Re/Fiasco.OC**: L4Re references throughout this directory's docs describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build.
---
## Document Organization
Read in this order for full understanding:
### 1. **[overview.md](overview.md)** - Start Here
**What:** High-level HAL architecture, design principles, and rationale
**Who:** Everyone (developers, architects, contributors)
**Why read:** Understand the problem HAL solves and how it fits into StarForth → StarKernel → StarshipOS
**Key sections:**
- The Problem (why HAL is necessary)
- Architecture Overview (3-layer model)
- Design Principles (VM purity, contract-first, testability)
- HAL Subsystems (time, interrupts, memory, console, CPU)
- HAL and Physics Subsystems
- Success Criteria
---
### 2. **[interfaces.md](interfaces.md)** - The Contract
**What:** Detailed HAL interface specifications (function signatures, semantics, guarantees)
**Who:** VM developers, platform implementers
**Why read:** Understand exactly what each HAL function does and requires
**Key sections:**
- Time & Timers interface (`hal_time.h`)
- Interrupts interface (`hal_interrupt.h`)
- Memory interface (`hal_memory.h`)
- Console interface (`hal_console.h`)
- CPU interface (`hal_cpu.h`)
- Panic/Error interface (`hal_panic.h`)
- Interface safety summary (ISR-safe, thread-safe)
**Critical details:**
- Function semantics (what it does, guaranteed behavior)
- Error handling (when it can fail, how to handle)
- Performance expectations (latency, overhead)
- Concurrency model (thread-safe? ISR-safe?)
---
### 3. **[platform-implementations.md](platform-implementations.md)** - How to Implement
**What:** Platform-specific implementation examples and guidance
**Who:** Platform developers adding HAL support for new targets
**Why read:** Learn how to implement HAL for a specific platform
**Key sections:**
- Platform directory structure
- Linux HAL implementation (reference platform)
- Kernel HAL implementation (StarKernel)
- Platform testing strategy
- Common implementation pitfalls
**Includes complete code examples for:**
- Linux (POSIX, hosted)
- Kernel (freestanding, bare metal)
---
### 4. **[migration-plan.md](migration-plan.md)** - Refactoring Guide
**What:** Step-by-step plan for refactoring existing StarForth to use HAL
**Who:** Core developers doing the migration work
**Why read:** Execute the HAL migration safely with zero regressions
**Key sections:**
- 7-phase migration strategy
- Incremental refactoring approach
- Test validation after each phase
- Rollback strategy
- Timeline estimates (2-4 weeks)
**Phases:**
1. Define HAL interfaces (headers only)
2. Implement HAL for Linux
3. Migrate VM core to use HAL
4. Migrate physics subsystems to use HAL
5. Migrate REPL and word implementations
6. Implement HAL for L4Re (optional validation)
7. Validate deterministic behavior
---
### 5. **[starkernel-integration.md](starkernel-integration.md)** - Kernel Details
**What:** StarKernel-specific HAL implementation and boot sequence
**Who:** Kernel developers building StarKernel
**Why read:** Understand UEFI boot, freestanding C, hardware initialization
**Key sections:**
- StarKernel architecture (boot sequence)
- Directory structure (`src/platform/kernel/`)
- Boot sequence detail (UEFI → StarKernel → VM)
- HAL implementation details (time, interrupts, memory, console, CPU)
- Build system (freestanding toolchain, linker script)
- Testing on QEMU/OVMF
- Debugging techniques
- Roadmap to `ok` prompt
**Includes:**
- Complete UEFI loader implementation
- Kernel entry point
- Hardware initialization sequence
- QEMU testing guide
- Debugging strategies
---
## Quick Reference
### For Different Roles
**I'm a VM developer:**
- Read: `overview.md`, `interfaces.md`
- Use HAL functions in VM code
- Never call platform-specific APIs directly
**I'm adding a new platform:**
- Read: `overview.md`, `interfaces.md`, `platform-implementations.md`
- Implement all HAL interfaces for your platform
- Test against VM test suite (936+ tests must pass)
**I'm migrating existing code to HAL:**
- Read: `migration-plan.md`
- Follow 7-phase incremental approach
- Test after each phase
**I'm building StarKernel:**
- Read: All documents, especially `starkernel-integration.md`
- Start with UEFI boot + serial output
- Build incrementally to `ok` prompt
---
## Key Concepts
### HAL Principles
1. **VM Purity:** VM code is platform-agnostic, never knows which platform it's on
2. **Contract-First:** HAL interfaces are contracts with precise semantics
3. **Testability:** Develop/test on Linux, deploy on kernel
4. **Zero Overhead:** HAL inlines to direct hardware access when optimized
5. **Fail-Fast:** Validate platform assumptions at init, not during execution
### Success Criteria
The HAL is successful if:
- ✅ VM core has zero platform-specific code
- ✅ All 936+ tests pass on all platforms
- ✅ 0% algorithmic variance maintained
- ✅ No measurable performance regression
- ✅ StarKernel boots to `ok` prompt
- ✅ Physics subsystems work identically on all platforms
---
## Document Status
| Document | Status | Last Updated |
|----------|--------|--------------|
| overview.md | ✅ Complete | 2025-12-14 |
| interfaces.md | ✅ Complete | 2025-12-14 |
| platform-implementations.md | ✅ Complete | 2025-12-14 |
| migration-plan.md | ✅ Complete | 2025-12-14 |
| starkernel-integration.md | ✅ Complete | 2025-12-14 |
---
## Next Steps
### Immediate (HAL Migration)
1. Review and approve HAL interface specifications
2. Execute migration plan Phase 1 (define interfaces)
3. Execute migration plan Phase 2 (implement Linux HAL)
4. Continue through Phase 7 (validation)
### Near-Term (StarKernel)
1. Implement UEFI boot loader
2. Implement kernel HAL subsystems
3. Boot to `ok` prompt on QEMU
4. Validate physics subsystems on bare metal
### Long-Term (StarshipOS)
1. Add storage drivers (AHCI, NVMe)
2. Add networking stack (TCP/IP)
3. Implement process model (Forth tasks)
4. Build device model (block/net/char)
---
## References
### StarForth Documentation
- `docs/CLAUDE.md` - Project overview and build instructions
- `docs/03-architecture/heartbeat-system/` - Heartbeat subsystem details
- `docs/FEEDBACK_LOOPS.md` - Physics feedback loops
- `README.md` - Project quick start
### External Resources
- [UEFI Specification](https://uefi.org/specifications)
- [GNU-EFI](https://sourceforge.net/projects/gnu-efi/)
- [OSDev Wiki](https://wiki.osdev.org/)
- [Intel Software Developer Manual](https://www.intel.com/sdm)
- [ACPI Specification](https://uefi.org/acpi)
---
## Contributing
When updating HAL documentation:
1. **Keep documents synchronized** - Changes to interfaces require updates to implementation guides
2. **Maintain code examples** - Examples must compile and work
3. **Update this README** - Reflect new sections or document reorganization
4. **Version documentation** - Note "Last Updated" dates
5. **Test instructions** - Verify all commands and code snippets work
---
## Questions?
For questions about HAL architecture or StarKernel implementation, see:
- GitHub Issues: https://github.com/anthropics/starforth/issues (if public)
- Project maintainer: (contact info)
- Documentation feedback: (preferred method)
---
*The HAL is not just a kernel bootstrapping tool—it's the architectural foundation for StarForth → StarKernel → StarshipOS.*
docs/working/archive/dead/hal-target-contract/interfaces.md
+713
View File
File diff suppressed because it is too large Load Diff
docs/working/archive/dead/hal-target-contract/migration-plan.md
+660
View File
File diff suppressed because it is too large Load Diff
docs/working/archive/dead/hal-target-contract/overview.md
+349
View File
File diff suppressed because it is too large Load Diff
docs/working/archive/dead/hal-target-contract/platform-implementations.md
+741
View File
File diff suppressed because it is too large Load Diff
docs/working/archive/dead/hal-target-contract/starkernel-integration.md
+773
View File
File diff suppressed because it is too large Load Diff
docs/working/archive/dead/l4re/L4RE_DICTIONARY_ALLOCATION.adoc
+203
View File
@@ -0,0 +1,203 @@
// DEAD (docs triage 2026-10-14). L4Re/Fiasco.OC was removed as a platform target in mid-2026. Instead: .claude/CLAUDE.md (Project Overview). Moved from docs/working/hardware/platform-integration/L4RE_DICTIONARY_ALLOCATION.adoc.
// Moved from docs/src/platform-integration/L4RE_DICTIONARY_ALLOCATION.adoc to docs/working/scratch/src/platform-integration/L4RE_DICTIONARY_ALLOCATION.adoc on 2026-06-16 (docs reorg Phase 2)
== Dictionary Allocation Strategy for L4Re
:toc: left
:toc-title: Contents
:toclevels: 3
xref:../README.adoc[← Back to Documentation Index]
=== Current Implementation
StarForth currently uses `+malloc()+` and `+free()+` from the host C
library for dictionary entry allocation:
* *Location*: `+src/dictionary_management.c:228+`
* *Function*: `+vm_create_word()+` uses `+malloc(total)+` to allocate
DictEntry structures
* *Cleanup*: `+FORGET+` uses `+free(e)+` to deallocate entries (line 554
in defining_words.c)
This works fine for hosted environments but creates a dependency on libc
heap allocator.
=== Why This Matters for L4Re/StarshipOS
==== Issues with Current Approach:
[arabic]
. *Heap Dependency*: Requires full malloc implementation in L4Re
environment
. *Non-Deterministic*: Heap fragmentation and unpredictable allocation
behavior
. *Separate Address Space*: Dictionary entries live outside VM memory
model
. *Hard to Introspect*: Can’t dump entire VM state as single memory
region
==== Benefits of VM-Based Allocation:
[arabic]
. *Zero External Dependencies*: No libc malloc needed
. *Deterministic*: Simple bump allocator, predictable behavior
. *Unified Address Space*: Everything in one 5MB VM memory region
. *Easy Snapshots*: Can serialize entire VM state
. *Better for Microkernel*: Aligns with L4Re dataspace model
=== Recommended Approach for L4Re
==== Strategy: Conditional Compilation
Add a compile-time flag to switch between malloc and VM-based
allocation:
[source,c]
----
// In dictionary_management.c
#ifdef STARFORTH_VM_DICT_ALLOC
/* VM-based allocation (for L4Re/embedded) */
DictEntry *entry = dict_alloc_from_vm(vm, total);
#else
/* Standard malloc (for hosted environments) */
DictEntry *entry = (DictEntry *) malloc(total);
#endif
----
==== Memory Layout (VM-Based Mode):
....
vm->memory[0...VM_MEMORY_SIZE] (5MB):
┌──────────────────────────────────────────┐
│ Dictionary Headers (256KB) │ ← dict_headers_used grows up
│ - DictEntry structures │
│ - Allocated via dict_alloc() │
├──────────────────────────────────────────┤
│ User Data & Code (variable) │ ← HERE grows up
│ - Compiled definitions │
│ - Variables, constants │
│ - String literals │
├──────────────────────────────────────────┤
│ Block Storage (remainder) │
│ - 1KB blocks for Forth block system │
└──────────────────────────────────────────┘
....
==== Implementation Steps:
[arabic]
. *Add Dictionary Allocator* (256KB bump allocator at `+memory[0]+`):
[source,c]
----
static DictEntry *dict_alloc_from_vm(VM *vm, size_t total_bytes) {
/* Align to cell_t boundary */
size_t align = sizeof(cell_t);
total_bytes = (total_bytes + align - 1) & ~(align - 1);
/* Check space */
if (vm->dict_headers_used + total_bytes > DICT_HEADERS_MAX) {
vm->error = 1;
return NULL;
}
/* Allocate from VM memory */
DictEntry *entry = (DictEntry *)(vm->memory + vm->dict_headers_used);
vm->dict_headers_used += total_bytes;
return entry;
}
----
[arabic, start=2]
. *Update FORGET* (no individual frees, just rewind):
[source,c]
----
/* Instead of free(e) loop, rewind allocator */
size_t target_offset = (uint8_t*)target - vm->memory;
vm->dict_headers_used = target_offset;
----
[arabic, start=3]
. *Adjust VM Initialization*:
[source,c]
----
vm->dict_headers_used = 0; /* Headers start at memory[0] */
vm->here = DICT_HEADERS_MAX; /* User data starts at 256KB */
----
[arabic, start=4]
. *No Changes to Fast-Lookup Cache*: Cache buckets still use malloc for
the pointer arrays, but entries themselves are in VM memory.
==== Testing Strategy:
[source,bash]
----
# Test with standard malloc (current):
make clean && make test
# Test with VM-based allocation:
make clean
CFLAGS="-DSTARFORTH_VM_DICT_ALLOC" make test
# Test L4Re minimal build:
make clean
CFLAGS="-DSTARFORTH_MINIMAL -DSTARFORTH_VM_DICT_ALLOC" make rpi4-cross
----
=== Current Status
*Decision*: Keep using `+malloc()+` for now, document the abstraction
point clearly.
*Rationale*:
* Current implementation is stable and well-tested
* L4Re port can implement the switch when needed
* Abstraction layer is simple and well-defined
* No performance difference between approaches
*Action Items for L4Re Port*:
[arabic]
. Define `+STARFORTH_VM_DICT_ALLOC+` in L4Re build
. Implement `+dict_alloc_from_vm()+` function
. Update `+FORGET+` to use rewind instead of free loop
. Test with full test suite
. Verify no memory leaks in microkernel environment
=== Minimal Code Changes Needed
The switch to VM-based allocation requires changes to only 3 functions:
[arabic]
. *`+vm_create_word()+`* - Use `+dict_alloc_from_vm()+` instead of
`+malloc()+`
. *`+dictionary_word_forget()+`* - Rewind `+dict_headers_used+` instead
of `+free()+` loop
. *`+vm_init()+`* - Initialize `+dict_headers_used+` and offset `+HERE+`
Add to `+VM+` struct in `+include/vm.h+`:
[source,c]
----
size_t dict_headers_used; /* Bytes used for dictionary headers (VM mode only) */
----
*Estimated effort*: 4 hours to implement + 2 hours testing = *6 hours
total*
=== References
* Current malloc usage: `+src/dictionary_management.c:228+`
* Current free usage: `+src/word_source/defining_words.c:554+`
* Fast-lookup cache: `+src/dictionary_management.c:43-67+` (unchanged)
* L4Re dataspace docs: `+docs/L4RE_INTEGRATION.md+`
'''''
*Author*: R. A. James *Date*: 2025-10-01 *Status*: Documented,
implementation deferred to L4Re port
docs/working/archive/dead/l4re/L4RE_INTEGRATION.adoc
+1215
View File
File diff suppressed because it is too large Load Diff
docs/working/archive/dead/l4re/L4RE_INTEGRATION_TODO.adoc
+349
View File
File diff suppressed because it is too large Load Diff
docs/working/archive/dead/l4re/l_4_re_blkio_endpoints.adoc
+193
View File
@@ -0,0 +1,193 @@
// DEAD (docs triage 2026-10-14). L4Re/Fiasco.OC was removed as a platform target in mid-2026. Instead: .claude/CLAUDE.md (Project Overview). Moved from docs/working/hardware/platform-integration/l_4_re_blkio_endpoints.adoc.
// Moved from docs/src/platform-integration/l_4_re_blkio_endpoints.adoc to docs/working/scratch/src/platform-integration/l_4_re_blkio_endpoints.adoc on 2026-06-16 (docs reorg Phase 2)
== StarForth Block I/O — L4Re Porting Endpoints
:toc: left
:toc-title: Contents
:toclevels: 3
xref:../README.adoc[← Back to Documentation Index]
This document explains how the existing `+blkio+` API maps cleanly onto
L4Re, what you need to implement, and where your VM and Forth block
words connect. The idea is: the *northbound API never changes*; you only
swap in different backends (dataspace or IPC service) when porting to
L4Re.
'''''
=== 1. Northbound API (unchanged)
Your VM and block words always call:
[source,c]
----
int blkio_open (blkio_dev_t *dev, const blkio_vtable_t *vt, const blkio_params_t *p);
int blkio_close(blkio_dev_t *dev);
int blkio_read (blkio_dev_t *dev, uint32_t fblock, void *dst);
int blkio_write(blkio_dev_t *dev, uint32_t fblock, const void *src);
int blkio_flush(blkio_dev_t *dev);
int blkio_info (blkio_dev_t *dev, blkio_info_t *out);
----
No matter what backend you use, these remain stable. That’s your ABI.
'''''
=== 2. In-process L4Re Dataspace Backend
Use when your VM can directly attach a dataspace. This gives you the
fastest path: no IPC.
*Symbols you implement:*
[source,c]
----
size_t blkio_l4ds_state_size(void);
int blkio_l4ds_init_state(void *state_mem, size_t state_len,
l4_cap_idx_t ds_cap,
l4_addr_t attach_addr,
l4_size_t attach_size,
uint32_t total_blocks,
uint32_t fbs,
uint8_t read_only,
void **out_opaque);
const blkio_vtable_t *blkio_l4ds_vtable(void);
----
*State struct:*
[source,c]
----
typedef struct {
l4_cap_idx_t ds; /* dataspace cap */
void *base; /* attached VA */
l4_size_t size; /* mapped length */
uint32_t fbs; /* block size */
uint32_t blocks; /* #blocks */
uint8_t ro;
} blkio_l4ds_state_t;
----
*Ops:*
* `+open+`: `+l4re_rm_attach()+` dataspace, derive block count.
* `+read/write+`: `+memcpy+` from attached region.
* `+flush+`: NOP (or `+l4re_ds_op+` if needed).
* `+close+`: `+l4re_rm_detach()+`.
'''''
=== 3. Out-of-process Block Server (IPC Backend)
Use when storage is owned by a different task (e.g., AHCI, NVMe, eMMC
server).
*Client-side symbols:*
[source,c]
----
size_t blkio_l4svc_state_size(void);
int blkio_l4svc_init_state(void *state_mem, size_t state_len,
l4_cap_idx_t server_cap,
l4_cap_idx_t bounce_ds,
uint32_t fbs,
uint8_t read_only,
void **out_opaque);
const blkio_vtable_t *blkio_l4svc_vtable(void);
----
*IPC protocol (example):*
[source,c]
----
enum bsvc_op {
BSVC_INFO = 1,
BSVC_READ = 2,
BSVC_WRITE = 3,
BSVC_FLUSH = 4,
BSVC_PING = 5
};
----
Transport choices:
* Shared dataspace (preferred): server + client map the same page(s).
* Inline IPC (simpler, less efficient): block payload in IPC message.
*Ops mapping:*
* `+read+`: send op + fblock, server fills shared buffer.
* `+write+`: copy into shared buffer, send op + fblock.
* `+info+`: query block size, count, flags.
'''''
=== 4. Factory Helpers on L4Re
You can add overloads to `+blkio_factory+` for clarity:
[source,c]
----
int blkio_factory_open_l4ds (blkio_dev_t *dev, l4_cap_idx_t ds_cap, uint32_t fbs, uint8_t ro);
int blkio_factory_open_file (blkio_dev_t *dev, const char *path, uint32_t fbs, uint8_t ro);
int blkio_factory_open_l4svc(blkio_dev_t *dev, l4_cap_idx_t server_cap, l4_cap_idx_t bounce_ds, uint32_t fbs, uint8_t ro);
----
This way `+main.c+` can pick a backend cleanly depending on CLI args and
environment.
'''''
=== 5. CLI Semantics on L4Re
* `+--disk-img=<path>+` → Either:
** VFS+file server that returns a dataspace → *l4ds backend*
** Or connect to a block server → *l4svc backend*
* `+--ram-disk=<MB>+` → Allocate RAM dataspace, use *l4ds backend*.
Fallback: if no `+--disk-img+`, create RAM disk from allocator.
'''''
=== 6. Composite Backends (RAM + Disk, 4 KiB packing)
Your design needs:
* Blocks 0–1023 → RAM backend.
* Blocks ≥1024 → Disk backend.
* All wrapped in a composite vtable that dispatches based on `+fblock+`.
Later: add an outer wrapper that packs 3×1 KiB + 1 KiB metadata into 4
KiB pages for device alignment. Still the same five ops.
'''''
== TL;DR — Endpoints to Implement for L4Re
*New backends:*
[source,c]
----
/* In-proc dataspace */
blkio_l4ds_state_size();
blkio_l4ds_init_state(...);
blkio_l4ds_vtable();
/* IPC client/server */
blkio_l4svc_state_size();
blkio_l4svc_init_state(...);
blkio_l4svc_vtable();
----
*Optional factory helpers:*
[source,c]
----
blkio_factory_open_l4ds(...);
blkio_factory_open_l4svc(...);
----
*Everything else stays the same*: VM calls only `+blkio_read+`,
`+blkio_write+`, etc. You swap backends, not call sites.
docs/working/archive/dead/metrics-and-knobs.md
+148
View File
@@ -0,0 +1,148 @@
> **DEAD (docs triage 2026-10-14).** Already marked dead in its own header.
> Instead: its own header. Moved here from `docs/working/architecture/03-architecture/physics-engine/metrics-and-knobs.md`.
<!-- Moved from docs/03-architecture/physics-engine/metrics-and-knobs.md to docs/working/archive/dead/metrics-and-knobs.md on 2026-06-16 (docs reorg Phase 2) -->
# StarForth Metrics & Knobs: GAPS ONLY
## What's NOT Wired (Things to Add)
### DEAD CODE / NEVER CALLED
| What | Location | Status | Comment |
|------|----------|--------|---------|
| Bucket reordering | `physics_hotwords_cache.c:201-215` | Defined but NEVER CALLED | Bubble sort exists but no automatic trigger. Is this dead code or forgotten? |
---
### STACK METRICS (ZERO COLLECTION)
| Metric | File | Missing Since | Impact | Knob Needed |
|--------|------|---|---|---|
| Data stack peak depth | `stack_management.c` | Day 1 | Can't tell if workload needs larger stack | STACK_SIZE tuning blind |
| Return stack peak depth | `stack_management.c` | Day 1 | Can't measure colon definition nesting | Tail-call optimization impossible |
| Underflow attempts | `arithmetic_words.c:59-62` | Day 1 | Can't detect fragile code paths | Could optimize checks if zero underflows observed |
| Overflow attempts | `arithmetic_words.c` | Day 1 | Can't detect stack exhaustion | Early warning system missing |
---
### DICTIONARY METRICS (ZERO VISIBILITY)
| Metric | File | Missing Since | Impact | Why Matters |
|--------|------|---|---|---|
| Bucket load histogram | `vm.c: dict lookup` | Day 1 | First-char bucketing (26 buckets) may be unbalanced | Affects lookup latency, collision clustering |
| Dictionary growth rate | `memory_management.c` | Day 1 | No visibility into `here` pointer velocity | Can't predict "dictionary full" events |
| FORGET usage frequency | `dictionary_words.c` | Day 1 | No trace of word deletion patterns | Fragmentation unknown |
| Lookup failure rate | `vm.c` | Day 1 | How often is a word not found? | Error path frequency unknown |
| Word name length distribution | `vm.c` | Day 1 | Average name length unknown | Affects memory efficiency |
---
### BLOCK I/O METRICS (COMPLETELY DARK)
| Metric | File | Missing Since | Impact | Phase |
|--------|------|---|---|---|
| Block read frequency | `block_subsystem.c` | Day 1 | How often is BLOCK called? | DoE phase 1 |
| Block write frequency | `block_subsystem.c` | Day 1 | How often is UPDATE/FLUSH called? | DoE phase 1 |
| Writeback frequency | `block_subsystem.c:198-200` | Day 1 | When are dirty blocks flushed? | DoE phase 1 |
| Cache hit rate (disk) | `block_subsystem.c` | Day 1 | Blocks served from RAM cache % | DoE phase 1 |
| Dirty block ratio | `block_subsystem.c` | Day 1 | Blocks pending flush at time T | DoE phase 1 |
| I/O latency | `block_subsystem.c` | Day 1 | Time per read/write operation | DoE phase 2 |
| Fragmentation ratio | `block_subsystem.c` | Day 1 | Gaps in allocated blocks | DoE phase 2 |
---
### PER-WORD EXECUTION METRICS
| Metric | File | Missing Since | Impact | Use Case |
|--------|------|---|---|---|
| Per-word execution frequency | `vm.c: inner loop` | Day 1 | Which words are actually hot? | Profiling, hotspot detection |
| Per-word latency distribution | `vm.c` | Day 1 | Median/P99 time per word | Performance regression detection |
| Word category breakdown | `word_source/*.c` | Day 1 | Arithmetic vs. Stack vs. Control dominance | Workload characterization |
| Colon vs. native ratio | `vm.c` | Day 1 | User-defined word execution frequency | Code complexity analysis |
---
### HEAT DYNAMICS (PHASE 2 NOT ACTIVE)
| Metric | File | Status | Comment |
|--------|------|--------|---------|
| Heat decay trace | `physics_metadata.c: TODO` | Phase 2 placeholder | Need to measure if decay causes pattern loss |
| Heat percentile distribution | `physics_metadata.c` | Not updating dynamic | Stored but never dynamically recalculated |
| Heat concentration ratio | `physics_metadata.c` | Not tracked | (Top 10 words heat) / (total heat) |
| Heat velocity | `physics_metadata.c` | Not tracked | Rate of heat change per unit time |
| Heat plateau detection | `physics_metadata.c` | Not tracked | When does word heat level off? |
---
### PIPELINING METRICS (PHASE 1 INSTRUMENTATION, NOT USED)
| Metric | File | Status | Phase 2 Use |
|--------|------|--------|---|
| Prediction accuracy by depth | `physics_pipelining_metrics.c` | Not measured | Binary chop TRANSITION_WINDOW_SIZE |
| Context transition frequency | `physics_pipelining_metrics.c` | Counted but not reported | Validate context patterns |
| Speculation ROI histogram | `physics_pipelining_metrics.c` | Not aggregated | Tune SPECULATION_THRESHOLD |
| Misprediction cost (actual) | `physics_pipelining_metrics.c` | Measured but not validated | Calibrate cost model |
| Pipeline stall reduction | `physics_pipelining_metrics.c` | Not measured | Throughput impact unknown |
---
### CACHE POLLUTION & CYCLES
| Metric | File | Missing Since | Impact |
|--------|------|---|---|
| Cache eviction/re-promotion cycles | `physics_hotwords_cache.c` | Day 1 | Are we thrashing? |
| False negatives (words evicted before promotion) | `physics_hotwords_cache.c` | Day 1 | Should promotion threshold be lower? |
| Cache eviction candidate value | `physics_hotwords_cache.c` | Day 1 | Was LRU the right choice? |
| Bucket scan depth (position of hit) | `physics_hotwords_cache.c` | Day 1 | Is bucket order actually helping? |
| Promotion wait time | `physics_hotwords_cache.c` | Day 1 | How long does word stay in bucket before cache? |
---
## Priority for "Twisting the Dragon's Tail"
### 🔴 **CRITICAL (Do First)**
1. **Stack peak depth** - 5 minutes to add, massive insight
2. **Block I/O metrics** - Writeback behavior unknown (could be performance killer)
3. **Dictionary bucket load** - Unbalanced bucketing = latency tail
### 🟠 **HIGH (Phase 1 DoE)**
4. **Per-word execution frequency** - Validation for cache/window decisions
5. **Heat decay trace** - Validate Phase 2 feature before enabling
6. **Block I/O hit rate** - Complete the I/O picture
### 🟡 **MEDIUM (Phase 2 DoE)**
7. **Pipelining accuracy by depth** - Input to window size binary chop
8. **Colon definition nesting depth** - Tail-call optimization opportunities
9. **Dictionary growth velocity** - Predict "full" events
### 🟢 **NICE-TO-HAVE (Phase 3)**
10. Heat concentration ratio
11. Word category breakdown
12. Speculation ROI histogram
---
## Implementation Effort Estimate
| Category | Files | LOC | Time | Priority |
|----------|-------|-----|------|----------|
| Stack metrics | stack_management.c, vm.h | ~30 | 30 min | 🔴 |
| Dictionary metrics | vm.c, memory_management.c, vm.h | ~50 | 45 min | 🔴 |
| Block I/O metrics | block_subsystem.c, vm.h | ~80 | 1 hr | 🔴 |
| Per-word metrics | vm.c, physics_metadata.c, vm.h | ~100 | 1.5 hr | 🟠 |
| Heat dynamics | physics_metadata.c, vm.h | ~60 | 1 hr | 🟠 |
| Pipelining accuracy | physics_pipelining_metrics.c, vm.h | ~40 | 45 min | 🟡 |
---
## Bucket Reordering Mystery
**File**: `physics_hotwords_cache.c:201-215`
**Code**: `hotwords_bucket_reorder()` - bubble sort by execution_heat
**Status**: Function defined but **NEVER CALLED** from lookup path
**Question**: Is this dead code or forgot to hook it?
**If enabled, metrics needed:**
- Reorder frequency
- Cost (bubble sort latency)
- Benefit (does reordering actually improve hit latency?)