docs: reorganise -- current reference gathered, finished work archived, superseded and dead ideas marked as such
docs/CLAUDE.md Phase 2, 2026-10-14. All moves by git mv; every archived file carries a header saying what it is, why, and where to look instead. - docs/reference/ (new): roadmap and versioning policy, M7.1, word-ACL design, kernel args, hosted acceptance test, dictionary, primitives, ontology, L8 table -- previously spread over docs/, docs/lithosananke/ and docs/03-architecture/. - docs/working/archive/historical/: finished work (session plan, test-gap audit, birthing plan, fix write-ups, the January architecture draft). - docs/working/archive/superseded/: the Hermes-VM Tripod docs and July Hermes designs (-> FABRIC-3.5/3.6), the StarForth roadmap, every working doc already flagged superseded or obsolete. - docs/working/archive/dead/: the hal_* target contract (never adopted), L4Re, metrics-and-knobs. - Papers and proposals into docs/working/papers/. - References rewritten across the repo (not logs/, not docs/patent/); CLAUDE.md's documentation section rewritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
rajames
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
1a4b163861
commit
7aed588f78
@@ -0,0 +1,265 @@
|
||||
> **SUPERSEDED (docs triage 2026-10-14).** The 2^6 factorial DoE guide, flagged obsolete 2026-10-09.
|
||||
> Instead: experiments/bare_metal/README.md for the DoE as it runs now. Moved here from `docs/working/experiments/02-experiments/factorial-doe/guide.md`.
|
||||
|
||||
<!-- Moved from docs/02-experiments/factorial-doe/guide.md to docs/working/archive/superseded/factorial-doe/guide.md on 2026-06-16 (docs reorg Phase 2) -->
|
||||
|
||||
> **⚠ OBSOLETE (flagged 2026-10-09).** Describes the retired per-loop Makefile-toggle
|
||||
> architecture, replaced by the L8 Jacquard runtime mode selector. Kept for historical record.
|
||||
|
||||
# StarForth Complete 2^6 Factorial Design of Experiments
|
||||
|
||||
## Overview
|
||||
|
||||
**One-and-done comprehensive factorial testing** - collect all data systematically, analyze separately.
|
||||
|
||||
This guide describes how to run a **complete 2^6 factorial design of experiments** across all 6 feedback loops, yielding:
|
||||
- **64 unique configurations** (all binary combinations)
|
||||
- **1,920+ runs** (30 samples per configuration)
|
||||
- **Single unified dataset** for factorial analysis
|
||||
|
||||
## The Why: Factorial DoE
|
||||
|
||||
Instead of incrementally tuning parameters, we collect **comprehensive experimental data** in one go:
|
||||
|
||||
### Traditional Approach (Old)
|
||||
- Run baseline
|
||||
- Run with Loop #1
|
||||
- Run with Loops #1-2
|
||||
- Run with Loops #1-3
|
||||
- ... (sequential, incremental)
|
||||
- **Problem:** Can't see interactions, trade-offs, or surprises
|
||||
|
||||
### Factorial Approach (New)
|
||||
- Run ALL 2^6 = 64 combinations in randomized order
|
||||
- Collect all data at once
|
||||
- Analyze factorial effects, interactions, and main effects
|
||||
- **Advantage:** Complete picture of how loops interact
|
||||
|
||||
## The 6 Feedback Loops
|
||||
|
||||
Each loop is a binary toggle in the Makefile:
|
||||
|
||||
| Loop | Makefile Flag | Purpose |
|
||||
|------|---|---|
|
||||
| **#1** | `ENABLE_LOOP_1_HEAT_TRACKING` | Count word execution frequency |
|
||||
| **#2** | `ENABLE_LOOP_2_ROLLING_WINDOW` | Capture execution history (circular buffer) |
|
||||
| **#3** | `ENABLE_LOOP_3_LINEAR_DECAY` | Age words by exponential decay over time |
|
||||
| **#4** | `ENABLE_LOOP_4_PIPELINING_METRICS` | Track word-to-word transitions |
|
||||
| **#5** | `ENABLE_LOOP_5_WINDOW_INFERENCE` | Infer optimal window width (Levene's test) |
|
||||
| **#6** | `ENABLE_LOOP_6_DECAY_INFERENCE` | Infer decay slope via linear regression |
|
||||
|
||||
## Configuration Naming
|
||||
|
||||
Each of 64 configurations is named by its loop status:
|
||||
|
||||
```
|
||||
L1_L2_L3_L4_L5_L6
|
||||
|
||||
Where each digit is 0 (OFF) or 1 (ON)
|
||||
|
||||
Examples:
|
||||
000000 = Baseline (all loops OFF - pure FORTH-79)
|
||||
000001 = Only heat tracking
|
||||
111110 = All except decay inference
|
||||
111111 = All loops ON (likely optimal)
|
||||
101010 = Alternating pattern (for research)
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Minimal Test (validate script)
|
||||
```bash
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 1 TEST_RUN
|
||||
# 64 builds × 1 run = 64 total runs
|
||||
# Runtime: ~30-45 minutes
|
||||
# Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/TEST_RUN/
|
||||
```
|
||||
|
||||
### Standard Comprehensive Run
|
||||
```bash
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FACTORIAL
|
||||
# 64 builds × 30 runs = 1,920 total runs
|
||||
# Runtime: ~2-4 hours
|
||||
# Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FACTORIAL/
|
||||
```
|
||||
|
||||
### High-Precision Run
|
||||
```bash
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 100 HIGH_PRECISION_DOE
|
||||
# 64 builds × 100 runs = 6,400 total runs
|
||||
# Runtime: ~6-12 hours
|
||||
# Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/HIGH_PRECISION_DOE/
|
||||
```
|
||||
|
||||
## Output Structure
|
||||
|
||||
```
|
||||
/home/rajames/CLionProjects/StarForth-DoE/experiments/<EXPERIMENT_LABEL>/
|
||||
├── experiment_results.csv # All measurements (1920+ rows)
|
||||
├── experiment_summary.txt # Timing, metadata, statistics
|
||||
├── configuration_manifest.txt # All 64 configs documented
|
||||
├── test_matrix.txt # Randomized execution order
|
||||
└── run_logs/
|
||||
├── 000000_run_1.log
|
||||
├── 000000_run_2.log
|
||||
├── 000001_run_1.log
|
||||
├── ...
|
||||
└── 111111_run_30.log # 1,920 total log files
|
||||
```
|
||||
|
||||
## CSV Schema
|
||||
|
||||
Each row in `experiment_results.csv` contains:
|
||||
|
||||
```
|
||||
timestamp,configuration,run_number,
|
||||
<35+ metrics from test harness>,
|
||||
enable_loop_1_heat_tracking,
|
||||
enable_loop_2_rolling_window,
|
||||
enable_loop_3_linear_decay,
|
||||
enable_loop_4_pipelining_metrics,
|
||||
enable_loop_5_window_inference,
|
||||
enable_loop_6_decay_inference
|
||||
```
|
||||
|
||||
This lets you correlate every metric against every loop combination.
|
||||
|
||||
## Analysis Strategy
|
||||
|
||||
After collection, analyze in the separate `StarForth-DoE` repository:
|
||||
|
||||
### Step 1: Load Data
|
||||
```python
|
||||
import pandas as pd
|
||||
df = pd.read_csv("experiment_results.csv")
|
||||
print(df.shape) # Should be (1920, 38+)
|
||||
print(df.columns)
|
||||
```
|
||||
|
||||
### Step 2: Separate Metrics from Factors
|
||||
```python
|
||||
factor_cols = ['enable_loop_1_heat_tracking', ..., 'enable_loop_6_decay_inference']
|
||||
metric_cols = ['total_lookups', 'cache_hit_percent', ..., 'vm_workload_duration_ns_q48']
|
||||
|
||||
factors = df[factor_cols]
|
||||
metrics = df[metric_cols]
|
||||
```
|
||||
|
||||
### Step 3: Factorial Analysis
|
||||
```python
|
||||
# Main effects: What's the average change when Loop X goes from 0→1?
|
||||
main_effects = {}
|
||||
for loop in factor_cols:
|
||||
main_effects[loop] = metrics[factors[loop]==1].mean() - metrics[factors[loop]==0].mean()
|
||||
|
||||
# Interactions: Do loops work better/worse together?
|
||||
# Can use statsmodels.formula.api for ANOVA/regression
|
||||
|
||||
# Optimal subset: Which combinations of loops give best throughput?
|
||||
best_configs = df.nlargest(10, 'vm_workload_duration_ns_q48')
|
||||
```
|
||||
|
||||
## Key Design Choices
|
||||
|
||||
### 1. **Rebuild Every Pass** ✓
|
||||
- Each configuration requires a `make clean && make`
|
||||
- Ensures clean, isolated state
|
||||
- No cross-contamination
|
||||
- True ground truth for each variant
|
||||
|
||||
### 2. **Randomized Execution**
|
||||
- All 1,920 runs randomized into single matrix (not 64 separate runs)
|
||||
- Prevents thermal ramp, temporal bias, or systematic trends
|
||||
- DoE principle: randomization eliminates confounds
|
||||
|
||||
### 3. **Full Factorial** (not fractional)
|
||||
- Testing ALL 2^6 = 64 combinations
|
||||
- Can detect unexpected interactions
|
||||
- Most expensive, most informative
|
||||
|
||||
### 4. **One-and-Done Collection**
|
||||
- Single experiment, all data
|
||||
- No progressive tuning during collection
|
||||
- Separates measurement from analysis (good practice)
|
||||
|
||||
## Expected Results
|
||||
|
||||
### Hypothetical Output
|
||||
Assuming typical performance distribution:
|
||||
|
||||
```
|
||||
Configuration | Avg Time (ns) | Speed vs Baseline
|
||||
─────────────────────────────────────────────────
|
||||
000000 (base) | 10,000,000 | 1.0x
|
||||
000001 (L1) | 9,950,000 | 1.005x
|
||||
111111 (all) | 8,500,000 | 1.176x ← Likely best
|
||||
```
|
||||
|
||||
### Expected Patterns
|
||||
1. **Loop #1 (heat tracking)**: ~2-5% improvement alone
|
||||
2. **Loop #2 (rolling window)**: ~1-3% improvement (depends on L1)
|
||||
3. **Loops #3-6**: Increasingly synergistic (work better together)
|
||||
4. **Interactions**: Loop #5 + #6 likely synergistic (inference loops)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Build Fails for a Configuration
|
||||
```bash
|
||||
# Check specific config build log
|
||||
cat /home/rajames/CLionProjects/StarForth-DoE/experiments/<LABEL>/run_logs/build_<CONFIG>.log
|
||||
```
|
||||
|
||||
### Test Harness Crashes on Some Configs
|
||||
- Check if loop combination is invalid (e.g., Loop #5 without Loop #2)
|
||||
- Verify Makefile flags are correctly parsed
|
||||
- See: `include/vm.h` for interdependencies
|
||||
|
||||
### Results Look Suspicious
|
||||
- Check `test_matrix.txt` - ensure randomization is present
|
||||
- Verify CSV row counts match expected total
|
||||
- Check for missing runs in log directory
|
||||
|
||||
## Performance Notes
|
||||
|
||||
### Timing Estimates
|
||||
- **Minimal (1 run/config)**: 30-45 minutes
|
||||
- **Standard (30 runs/config)**: 2-4 hours
|
||||
- **High-precision (100 runs/config)**: 6-12 hours
|
||||
- **Depends on:** CPU speed, disk I/O, test harness overhead
|
||||
|
||||
### Optimization
|
||||
- Run overnight for high-precision experiments
|
||||
- Use `screen` or `tmux` to keep session alive
|
||||
- Monitor disk space (~100 MB for results + logs)
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Run the experiment:**
|
||||
```bash
|
||||
./scripts/run_factorial_doe.sh 2025_11_19_COMPLETE_FACTORIAL
|
||||
```
|
||||
|
||||
2. **Monitor progress:**
|
||||
```bash
|
||||
tail -f /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_COMPLETE_FACTORIAL/experiment_summary.txt
|
||||
```
|
||||
|
||||
3. **Move to analysis repo** (once complete):
|
||||
```bash
|
||||
cp -r /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_COMPLETE_FACTORIAL \
|
||||
/home/rajames/CLionProjects/StarForth-DoE-Analysis/data/
|
||||
cd /home/rajames/CLionProjects/StarForth-DoE-Analysis
|
||||
python3 analyze_factorial.py data/2025_11_19_COMPLETE_FACTORIAL/experiment_results.csv
|
||||
```
|
||||
|
||||
4. **Generate reports:**
|
||||
- Main effects plots
|
||||
- Interaction matrices
|
||||
- Pareto optimal configurations
|
||||
- Recommendations for production builds
|
||||
|
||||
---
|
||||
|
||||
**Version:** 1.0
|
||||
**Last Updated:** 2025-11-19
|
||||
**Author:** Claude Code (with Robert A. James)
|
||||
@@ -0,0 +1,404 @@
|
||||
> **SUPERSEDED (docs triage 2026-10-14).** The 2^6 factorial DoE guide, flagged obsolete 2026-10-09.
|
||||
> Instead: experiments/bare_metal/README.md for the DoE as it runs now. Moved here from `docs/working/experiments/02-experiments/factorial-doe/incompatible-configs.md`.
|
||||
|
||||
<!-- Moved from docs/02-experiments/factorial-doe/incompatible-configs.md to docs/working/archive/superseded/factorial-doe/incompatible-configs.md on 2026-06-16 (docs reorg Phase 2) -->
|
||||
|
||||
> **⚠ OBSOLETE (flagged 2026-10-09).** Build-time incompatibility handling for the retired 2^6
|
||||
> toggle experiment, replaced by the L8 Jacquard mode selector. Kept for historical record.
|
||||
|
||||
# Handling Incompatible Configurations in Factorial DoE
|
||||
|
||||
## Overview
|
||||
|
||||
Your intuition was correct: **some loop combinations can't coexist**. Rather than failing the entire experiment, the updated script treats incompatibility as valuable data.
|
||||
|
||||
This document explains how the script detects and records incompatible configurations during the factorial experiment.
|
||||
|
||||
---
|
||||
|
||||
## What is an Incompatible Configuration?
|
||||
|
||||
An incompatible configuration is a combination of feedback loop toggles that:
|
||||
|
||||
1. **Fails to compile** - Makefile target produces build errors
|
||||
2. **Causes StarForth to crash** - Binary exits abnormally (segfault, signal, etc.)
|
||||
3. **Produces invalid output** - Test harness runs but produces no metrics
|
||||
|
||||
Examples (hypothetical):
|
||||
- Loop 5 (window inference) without Loop 2 (rolling window) = no data to infer from
|
||||
- Loop 6 (decay inference) without Loop 3 (linear decay) = no slope to measure
|
||||
- Certain combinations trigger memory conflicts or uninitialized states
|
||||
|
||||
---
|
||||
|
||||
## How the Script Handles It
|
||||
|
||||
### Detection Strategy
|
||||
|
||||
**Build Phase:**
|
||||
```
|
||||
Configuration X attempted
|
||||
├─ make TARGET=fastest ENABLE_LOOP_1=1 ...ENABLE_LOOP_6=0
|
||||
├─ Check exit code
|
||||
├─ If success: mark as "OK", proceed to testing
|
||||
└─ If failure: mark as "BUILD_FAILED", skip all runs for this config
|
||||
```
|
||||
|
||||
**Execution Phase:**
|
||||
```
|
||||
Run configuration X, iteration N
|
||||
├─ Execute: ./starforth --doe-experiment
|
||||
├─ Check exit code
|
||||
├─ Try to extract metrics from output
|
||||
├─ If success: mark as "OK", record metrics
|
||||
└─ If failure: mark as "CRASH", record incompatibility
|
||||
```
|
||||
|
||||
### CSV Recording
|
||||
|
||||
Incompatible configurations are recorded with status flags:
|
||||
|
||||
**Successful run (viable config):**
|
||||
```
|
||||
timestamp,configuration,run_number,build_status,run_status,
|
||||
2025-11-19T10:30:45,1_1_0_1_0_0,5,OK,OK,
|
||||
<35+ metrics>,1,1,0,1,0,0
|
||||
```
|
||||
|
||||
**Build failure (incompatible):**
|
||||
```
|
||||
timestamp,configuration,run_number,build_status,run_status,
|
||||
2025-11-19T10:35:20,0_1_1_1_1_0,1,BUILD_FAILED,<skipped>,
|
||||
<empty>,0,1,1,1,1,0
|
||||
```
|
||||
|
||||
**Runtime crash (incompatible):**
|
||||
```
|
||||
timestamp,configuration,run_number,build_status,run_status,
|
||||
2025-11-19T10:40:15,1_0_0_1_1_1,3,OK,CRASH,
|
||||
<empty>,1,0,0,1,1,1
|
||||
```
|
||||
|
||||
### Experiment Flow
|
||||
|
||||
The experiment **continues** when incompatibility is detected:
|
||||
|
||||
```
|
||||
Run 1: Config A (OK) → Test completes, metrics recorded
|
||||
Run 2: Config B (BUILD_FAILED) → Build fails, skip all runs for B, continue
|
||||
Run 3: Config A (OK) → Test completes, metrics recorded
|
||||
Run 4: Config C (CRASH) → Crashes, record as incompatible, continue
|
||||
...
|
||||
Run N: Config D (OK) → Test completes, metrics recorded
|
||||
|
||||
SUMMARY:
|
||||
✓ Successful runs: 847/1920
|
||||
✗ Failed configs: 17 (incompatible combinations detected)
|
||||
✓ Experiment completed despite incompatibilities
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CSV Schema
|
||||
|
||||
### Two New Columns
|
||||
|
||||
| Column | Values | Meaning |
|
||||
|--------|--------|---------|
|
||||
| `build_status` | OK, BUILD_FAILED | Did Makefile compilation succeed? |
|
||||
| `run_status` | OK, CRASH | Did StarForth execute without crashing? |
|
||||
|
||||
### Analysis Filter Rules
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
df = pd.read_csv('experiment_results.csv')
|
||||
|
||||
# Only viable configurations
|
||||
viable = df[(df['build_status'] == 'OK') & (df['run_status'] == 'OK')]
|
||||
|
||||
# Incompatible configs (at compile time)
|
||||
build_failures = df[df['build_status'] == 'BUILD_FAILED']
|
||||
|
||||
# Incompatible configs (at runtime)
|
||||
runtime_crashes = df[df['run_status'] == 'CRASH']
|
||||
|
||||
# All incompatible
|
||||
incompatible = df[(df['build_status'] != 'OK') | (df['run_status'] != 'OK')]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Interpreting Results
|
||||
|
||||
### Experiment Summary
|
||||
|
||||
After completion, you'll see:
|
||||
|
||||
```
|
||||
═══════════════════════════════════════════════════════════
|
||||
EXPERIMENT COMPLETE (with incompatibility detection)
|
||||
═══════════════════════════════════════════════════════════
|
||||
✓ Successful runs: 1847/1920
|
||||
✓ Failed configurations (incompatible): 17
|
||||
|
||||
→ Incompatible configurations detected:
|
||||
✗ 0_1_1_1_1_0 (BUILD_FAILED)
|
||||
✗ 1_0_0_1_1_1 (CRASH)
|
||||
✗ 0_1_0_1_1_1 (CRASH)
|
||||
... (total 17)
|
||||
|
||||
→ These configs are marked in CSV with build_status or run_status
|
||||
→ They represent configurations where loops cannot coexist without conflict
|
||||
```
|
||||
|
||||
### What This Means
|
||||
|
||||
- **1847 successful runs** = viable, measurable configurations
|
||||
- **17 incompatible** = design space constraints discovered
|
||||
- **Experiment completed** = data collection succeeded despite incompatibilities
|
||||
- **CSV includes both** = full picture of parameter space
|
||||
|
||||
---
|
||||
|
||||
## Analysis Examples
|
||||
|
||||
### Example 1: Summary by Viability
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
df = pd.read_csv('experiment_results.csv')
|
||||
|
||||
# Count by status
|
||||
viable = len(df[(df['build_status'] == 'OK') & (df['run_status'] == 'OK')])
|
||||
build_fail = len(df[df['build_status'] == 'BUILD_FAILED'].drop_duplicates('configuration'))
|
||||
runtime_crash = len(df[df['run_status'] == 'CRASH'].drop_duplicates('configuration'))
|
||||
|
||||
print(f"Viable configurations: {viable} runs")
|
||||
print(f"Build failures: {build_fail} configurations")
|
||||
print(f"Runtime crashes: {runtime_crash} configurations")
|
||||
print(f"Total tested: {64} configurations, {64*30} runs")
|
||||
print(f"Success rate: {viable}/{len(df)} = {100*viable/len(df):.1f}%")
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
Viable configurations: 1847 runs
|
||||
Build failures: 5 configurations
|
||||
Runtime crashes: 12 configurations
|
||||
Total tested: 64 configurations, 1920 runs
|
||||
Success rate: 1847/1920 = 96.2%
|
||||
```
|
||||
|
||||
### Example 2: Identify Problematic Loop Combinations
|
||||
|
||||
```python
|
||||
# Find configs that always crash
|
||||
crashes = df[df['run_status'] == 'CRASH'].drop_duplicates('configuration')
|
||||
|
||||
print("Configurations that crash:")
|
||||
for config in crashes['configuration']:
|
||||
parts = config.split('_')
|
||||
loops_on = [f"Loop{i+1}" for i, bit in enumerate(parts) if bit == '1']
|
||||
print(f" {config}: {', '.join(loops_on) or 'None'}")
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
Configurations that crash:
|
||||
0_1_1_1_1_0: Loop2, Loop3, Loop4, Loop5
|
||||
1_0_0_1_1_1: Loop1, Loop4, Loop5, Loop6
|
||||
0_1_0_1_1_1: Loop2, Loop4, Loop5, Loop6
|
||||
... etc
|
||||
```
|
||||
|
||||
**Insight:** Loop 5 (window inference) appears in many crashes → suggests window inference has prerequisites or conflicts.
|
||||
|
||||
### Example 3: Feasibility Matrix
|
||||
|
||||
```python
|
||||
# Which loop combinations are feasible?
|
||||
df_viable = df[(df['build_status'] == 'OK') & (df['run_status'] == 'OK')].copy()
|
||||
|
||||
# For each config, mark as viable (1) or not (0)
|
||||
feasible = {}
|
||||
for config in df['configuration'].unique():
|
||||
is_viable = len(df_viable[df_viable['configuration'] == config]) > 0
|
||||
feasible[config] = 1 if is_viable else 0
|
||||
|
||||
# Show as binary matrix
|
||||
print("Feasibility (1=viable, 0=incompatible):")
|
||||
for i in range(64):
|
||||
config = f"{i%2}_{(i//2)%2}_{(i//4)%2}_{(i//8)%2}_{(i//16)%2}_{(i//32)%2}"
|
||||
print(f"{config}: {'✓' if feasible[config] else '✗'}")
|
||||
```
|
||||
|
||||
Result: Feasibility map of 64 configurations for maintenance and design decisions.
|
||||
|
||||
---
|
||||
|
||||
## Maintenance Insights
|
||||
|
||||
### Why Some Combinations Are Incompatible
|
||||
|
||||
Common reasons:
|
||||
|
||||
1. **Missing Prerequisites:**
|
||||
- Loop 5 (window inference) needs Loop 2 (rolling window) data
|
||||
- Loop 6 (decay inference) needs Loop 3 (linear decay) to measure
|
||||
|
||||
2. **Memory/Resource Conflicts:**
|
||||
- Multiple loops allocating same memory region
|
||||
- Circular dependencies in initialization
|
||||
|
||||
3. **Semantic Contradictions:**
|
||||
- Loop X disables feature that Loop Y requires
|
||||
- Incompatible macro definitions
|
||||
|
||||
4. **Bugs in Specific Combinations:**
|
||||
- Race conditions appearing only with certain feature combinations
|
||||
- Uninitialized state when paths not taken
|
||||
|
||||
### Using This Information
|
||||
|
||||
**For Future Development:**
|
||||
```
|
||||
Findings: Loop 5 incompatible without Loop 2
|
||||
→ Check: Does window inference code assume rolling window exists?
|
||||
→ Action: Either add guard (if !LOOP_5 without LOOP_2) or document prerequisite
|
||||
→ Test: Add compile-time or runtime check
|
||||
```
|
||||
|
||||
**For Optimization:**
|
||||
```
|
||||
Result: All configs with Loop 6 crash if Loop 1 is OFF
|
||||
→ Issue: Decay inference needs heat tracking for slope calculation
|
||||
→ Fix: Add LOOP_1_REQUIRED_BY_LOOP_6 guard in code
|
||||
→ Verify: Re-run factorial (expect fewer crashes)
|
||||
```
|
||||
|
||||
**For Production:**
|
||||
```
|
||||
Feasible subset: {0_0_0_0_0_0, 1_1_0_0_0_0, 1_1_1_0_0_0, 1_1_1_1_1_1}
|
||||
→ Recommendation: Use 1_1_1_1_1_1 (all compatible, best performance)
|
||||
→ Fallback: Use 1_1_0_0_0_0 (compatible, simpler)
|
||||
→ Baseline: Use 0_0_0_0_0_0 (no feedback, minimal overhead)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Run the experiment** to completion (incompatibilities are discovered organically)
|
||||
2. **Analyze the CSV** to identify which configs are viable
|
||||
3. **Examine incompatible configs** to understand why they fail
|
||||
4. **Document prerequisites** (which loops require others)
|
||||
5. **Fix or guard** the code to eliminate incompatibilities
|
||||
6. **Re-run** experiment to validate fixes
|
||||
|
||||
---
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Build Failure Detection
|
||||
|
||||
```bash
|
||||
if make TARGET=fastest ENABLE_LOOP_1=1 ... > build_log 2>&1; then
|
||||
# Build successful
|
||||
BUILD_STATUS="OK"
|
||||
else
|
||||
# Build failed - mark incompatible
|
||||
BUILD_STATUS="BUILD_FAILED"
|
||||
# Skip all runs for this config
|
||||
continue
|
||||
fi
|
||||
```
|
||||
|
||||
### Runtime Crash Detection
|
||||
|
||||
```bash
|
||||
if "${BINARY}" --doe-experiment > run_log 2>&1; then
|
||||
# Binary executed (exit code 0)
|
||||
if extract_csv_row(run_log) succeeds:
|
||||
RUN_STATUS="OK"
|
||||
else:
|
||||
RUN_STATUS="CRASH" # Binary ran but no output
|
||||
else
|
||||
# Binary crashed (non-zero exit)
|
||||
RUN_STATUS="CRASH"
|
||||
fi
|
||||
```
|
||||
|
||||
### CSV Recording
|
||||
|
||||
```bash
|
||||
# Viable run
|
||||
printf '%s,%s,%s,%s,%s,%s,%s\n' \
|
||||
"${timestamp}" "${config}" "${run_num}" \
|
||||
"OK" "OK" \
|
||||
"${csv_row}" "${flags}" >> results_run_01_2025_12_08.csv
|
||||
|
||||
# Incompatible run
|
||||
printf '%s,%s,%s,%s,%s,%s\n' \
|
||||
"${timestamp}" "${config}" "${run_num}" \
|
||||
"${build_status}" "${run_status}" \
|
||||
"${flags}" >> results_run_01_2025_12_08.csv
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example: Real Incompatibility
|
||||
|
||||
**Scenario:** Loop 5 (window inference) without Loop 2 (rolling window)
|
||||
|
||||
```c
|
||||
// Loop 5: Window Inference Code
|
||||
void vm_tick_inference_engine(VM *vm) {
|
||||
// Try to infer window width from rolling window
|
||||
RollingWindowOfTruth *window = &vm->rolling_window; // LOOP_2 data
|
||||
|
||||
// BUG: If LOOP_2 is OFF, window is uninitialized!
|
||||
uint32_t window_pos = window->window_pos; // CRASH: uninitialized read
|
||||
}
|
||||
```
|
||||
|
||||
**Symptom in CSV:**
|
||||
```
|
||||
Configuration 0_0_0_0_1_0 marked as CRASH
|
||||
Loop1=0, Loop2=0, Loop3=0, Loop4=0, Loop5=1, Loop6=0
|
||||
↑ ↑
|
||||
Missing prerequisite Problematic loop
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
```c
|
||||
// Add guard in code
|
||||
#if ENABLE_LOOP_5_WINDOW_INFERENCE
|
||||
#if !ENABLE_LOOP_2_ROLLING_WINDOW
|
||||
#error "Loop 5 requires Loop 2: window inference needs rolling window data"
|
||||
#endif
|
||||
#endif
|
||||
```
|
||||
|
||||
**Result after fix:** Configuration 0_0_0_0_1_0 removed from viable set (prevented at compile time), reduces crash count.
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Aspect | Approach |
|
||||
|--------|----------|
|
||||
| **Detection** | Build exit codes + runtime crash detection |
|
||||
| **Recording** | New `build_status` and `run_status` columns |
|
||||
| **Handling** | Skip incompatible config, continue experiment |
|
||||
| **Impact** | Partial dataset (viable configs) + incompatibility map |
|
||||
| **Analysis** | Filter by status, identify patterns |
|
||||
| **Action** | Fix code, re-run, measure improvement |
|
||||
|
||||
This design enables discovering incompatibilities without sacrificing experimental continuity.
|
||||
|
||||
---
|
||||
|
||||
**Version:** 1.0
|
||||
**Last Updated:** 2025-11-19
|
||||
**Status:** Integrated into run_factorial_doe.sh
|
||||
@@ -0,0 +1,271 @@
|
||||
> **SUPERSEDED (docs triage 2026-10-14).** The 2^6 factorial DoE guide, flagged obsolete 2026-10-09.
|
||||
> Instead: experiments/bare_metal/README.md for the DoE as it runs now. Moved here from `docs/working/experiments/02-experiments/factorial-doe/index.md`.
|
||||
|
||||
<!-- Moved from docs/02-experiments/factorial-doe/index.md to docs/working/archive/superseded/factorial-doe/index.md on 2026-06-16 (docs reorg Phase 2) -->
|
||||
|
||||
> **⚠ OBSOLETE (flagged 2026-10-09).** Index for the retired 2^6 factorial toggle
|
||||
> architecture, replaced by the L8 Jacquard mode selector. Kept for historical record.
|
||||
|
||||
# StarForth Complete 2^6 Factorial DoE - Documentation Index
|
||||
|
||||
## Overview
|
||||
|
||||
This is a **one-and-done comprehensive factorial design of experiments** to systematically test all 64 possible combinations of 6 feedback loops.
|
||||
|
||||
**Key insight:** Don't toggle back-and-forth. Instead, systematically test ALL 2^6 combinations in a single experiment, then analyze the results afterward.
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
### 1. **START HERE** 📋
|
||||
- **File:** `docs/QUICK_START_FACTORIAL_DoE.md`
|
||||
- **Purpose:** Entry point for quick reference
|
||||
- **Read time:** 5 minutes
|
||||
- **Contains:**
|
||||
- 3 command options (quick/standard/high-precision)
|
||||
- Time estimates
|
||||
- What to expect during execution
|
||||
- Basic troubleshooting
|
||||
|
||||
### 2. **Complete Guide** 📚
|
||||
- **File:** `docs/FACTORIAL_DoE_GUIDE.md`
|
||||
- **Purpose:** Full design documentation and rationale
|
||||
- **Read time:** 20-30 minutes
|
||||
- **Contains:**
|
||||
- Why factorial DoE (vs. incremental tuning)
|
||||
- 6 feedback loop definitions
|
||||
- Configuration naming scheme (0_0_0_0_0_0 → 1_1_1_1_1_1)
|
||||
- Design choices explained
|
||||
- Expected results and patterns
|
||||
- Troubleshooting guide
|
||||
- Performance notes
|
||||
|
||||
### 3. **Analysis Workflow** 🔍
|
||||
- **File:** `docs/FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
|
||||
- **Purpose:** End-to-end pipeline from collection to insights
|
||||
- **Read time:** 15-20 minutes (or 1-2 hours if coding along)
|
||||
- **Contains:**
|
||||
- Phase 1: Data collection (run script)
|
||||
- Phase 2: Data transfer (to analysis repo)
|
||||
- Phase 3: Statistical analysis (main effects, interactions)
|
||||
- Phase 4: Reporting (generating insights)
|
||||
- Python code examples:
|
||||
* Load data and validate
|
||||
* Compute main effects (individual loop impacts)
|
||||
* Detect interactions (loop synergies)
|
||||
* Optimize configuration subset
|
||||
* Visualization code
|
||||
- Example report structure
|
||||
- Checklist for completion
|
||||
|
||||
### 4. **Main Script** ⚙️
|
||||
- **File:** `scripts/run_factorial_doe.sh`
|
||||
- **Purpose:** Executable for running the entire experiment
|
||||
- **Read time:** 10 minutes (code + comments)
|
||||
- **Contains:**
|
||||
- Generates all 64 configurations
|
||||
- Rebuilds for each unique configuration
|
||||
- Randomizes execution order (1,920+ runs)
|
||||
- Collects metrics to CSV
|
||||
- Full instrumentation and progress reporting
|
||||
|
||||
---
|
||||
|
||||
## Reading Guide by Use Case
|
||||
|
||||
### "I just want to run the experiment"
|
||||
1. Read: `QUICK_START_FACTORIAL_DoE.md` (5 min)
|
||||
2. Execute: Choose command option (quick/standard/high-precision)
|
||||
3. Wait: 30 min - 12 hours depending on option
|
||||
4. Done!
|
||||
|
||||
### "I want to understand the design before running"
|
||||
1. Read: `QUICK_START_FACTORIAL_DoE.md` (5 min)
|
||||
2. Read: `FACTORIAL_DoE_GUIDE.md` (30 min)
|
||||
3. Execute: Run with confidence
|
||||
4. Analyze: Follow `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
|
||||
|
||||
### "I need to analyze the results"
|
||||
1. Read: `QUICK_START_FACTORIAL_DoE.md` → understand naming
|
||||
2. Read: `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md` → statistical methods
|
||||
3. Code along: Use Python examples in Analysis Workflow
|
||||
4. Interpret: Reference `FACTORIAL_DoE_GUIDE.md` for expected patterns
|
||||
|
||||
### "I want complete understanding"
|
||||
Read in order:
|
||||
1. `QUICK_START_FACTORIAL_DoE.md` (overview)
|
||||
2. `FACTORIAL_DoE_GUIDE.md` (design rationale)
|
||||
3. `scripts/run_factorial_doe.sh` (implementation)
|
||||
4. `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md` (analysis methods)
|
||||
|
||||
---
|
||||
|
||||
## Quick Command Reference
|
||||
|
||||
### Run Experiment
|
||||
```bash
|
||||
# Option A: Quick validation (30 min)
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 1 TEST_RUN
|
||||
|
||||
# Option B: Standard production (2-4 hours)
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FULL_FACTORIAL
|
||||
|
||||
# Option C: High-precision overnight (6-12 hours)
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 100 HIGH_PRECISION_FACTORIAL
|
||||
```
|
||||
|
||||
### Monitor Progress
|
||||
```bash
|
||||
# Check results_run_01_2025_12_08 CSV growth
|
||||
watch -n 5 'wc -l /path/to/experiment_results.csv'
|
||||
|
||||
# View summary
|
||||
tail /path/to/experiment_summary.txt
|
||||
|
||||
# Count completed configurations
|
||||
ls /path/to/run_logs/build_*.log | wc -l
|
||||
```
|
||||
|
||||
### Analyze Results
|
||||
```bash
|
||||
# Load and analyze in Python
|
||||
cd /path/to/StarForth-DoE-Analysis
|
||||
python3 << 'EOF'
|
||||
import pandas as pd
|
||||
df = pd.read_csv('data/2025_11_19_FULL_FACTORIAL/experiment_results.csv')
|
||||
print(f"Loaded {len(df)} runs")
|
||||
# See FACTORIAL_DoE_ANALYSIS_WORKFLOW.md for full analysis code
|
||||
EOF
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Concepts
|
||||
|
||||
### 64 Configurations
|
||||
Each configuration is a unique combination of 6 binary factors:
|
||||
- **0_0_0_0_0_0** = Baseline (all loops OFF)
|
||||
- **1_0_0_0_0_0** = Only heat tracking
|
||||
- **1_1_0_0_0_0** = Heat + rolling window
|
||||
- ...
|
||||
- **1_1_1_1_1_1** = All loops ON (current optimal)
|
||||
|
||||
### Randomized Execution
|
||||
All 1,920+ runs (64 configs × 30 samples) are randomized into a single execution matrix.
|
||||
- Eliminates temporal bias
|
||||
- Prevents thermal ramp (if CPU heats during night)
|
||||
- Proper DoE methodology
|
||||
|
||||
### One-and-Done Collection
|
||||
Collect all data in single experiment, analyze separately.
|
||||
- No tuning during collection (avoid bias)
|
||||
- Single pass through parameter space
|
||||
- Separates measurement from analysis
|
||||
|
||||
### Main Effects
|
||||
Individual impact of each loop:
|
||||
```
|
||||
Loop #1: Loop turns ON, average metric changes by X%
|
||||
Loop #2: Loop turns ON, average metric changes by Y%
|
||||
... etc
|
||||
```
|
||||
|
||||
### Interactions
|
||||
How loops work together (amplify or suppress each other):
|
||||
```
|
||||
Loop #1 + Loop #2: Do they work better together than independently?
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Files at a Glance
|
||||
|
||||
| File | Size | Purpose |
|
||||
|------|------|---------|
|
||||
| `scripts/run_factorial_doe.sh` | 19 KB | Main executable script |
|
||||
| `docs/QUICK_START_FACTORIAL_DoE.md` | 4.9 KB | Quick reference |
|
||||
| `docs/FACTORIAL_DoE_GUIDE.md` | 8.0 KB | Design documentation |
|
||||
| `docs/FACTORIAL_DoE_ANALYSIS_WORKFLOW.md` | 9.6 KB | Analysis pipeline |
|
||||
| `docs/FACTORIAL_DoE_INDEX.md` | This file | Documentation index |
|
||||
|
||||
---
|
||||
|
||||
## Execution Timeline
|
||||
|
||||
### Phase 1: Preparation (5-10 min)
|
||||
- Read `QUICK_START_FACTORIAL_DoE.md`
|
||||
- Choose execution option
|
||||
- Run command
|
||||
|
||||
### Phase 2: Collection (30 min - 12 hours)
|
||||
- Script builds all 64 configs (as needed)
|
||||
- Randomized execution of 1,920+ runs
|
||||
- Metrics collected to CSV
|
||||
- Progress shown every run
|
||||
|
||||
### Phase 3: Transfer (5 min)
|
||||
- Copy results to analysis repo
|
||||
- Verify file integrity
|
||||
|
||||
### Phase 4: Analysis (1-2 hours)
|
||||
- Load CSV in Python
|
||||
- Compute main effects
|
||||
- Detect interactions
|
||||
- Find optimal configurations
|
||||
- Generate visualizations
|
||||
|
||||
### Phase 5: Reporting (1-2 hours)
|
||||
- Summarize findings
|
||||
- Create report
|
||||
- Recommendations for production
|
||||
|
||||
**Total time:** Collection (2-4 hrs) + Analysis (2-4 hrs) = **4-8 hours**
|
||||
|
||||
---
|
||||
|
||||
## The Design Philosophy
|
||||
|
||||
### Problem Statement
|
||||
How do we systematically understand the performance contribution and interactions of 6 interdependent feedback loops?
|
||||
|
||||
### Solution
|
||||
**Complete 2^6 factorial design of experiments**
|
||||
|
||||
### Why This Approach
|
||||
1. **Complete:** Tests all 64 combinations (no gaps)
|
||||
2. **Systematic:** Proper experimental design methodology
|
||||
3. **Separable:** Collection ≠ Analysis (reduces bias)
|
||||
4. **Reproducible:** Randomized execution, full logging
|
||||
5. **Scalable:** Single script, extensible to more factors
|
||||
|
||||
### Alternatives Rejected
|
||||
- ❌ Incremental tuning ("baseline → +loop1 → +loop1+2"): Misses interactions
|
||||
- ❌ Dynamic toggle without rebuild: Introduces state contamination
|
||||
- ❌ Partial factorial: Misses key interactions
|
||||
- ❌ One-at-a-time testing: Expensive and confounded
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Read** `QUICK_START_FACTORIAL_DoE.md` (5 minutes)
|
||||
2. **Decide** which execution option (quick/standard/precise)
|
||||
3. **Run** the command and let it collect data
|
||||
4. **Monitor** progress in another terminal
|
||||
5. **Analyze** results using `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
|
||||
6. **Report** findings to stakeholders
|
||||
|
||||
---
|
||||
|
||||
## Questions?
|
||||
|
||||
- **"How long will this take?"** → See `QUICK_START_FACTORIAL_DoE.md`
|
||||
- **"What does configuration X mean?"** → See `FACTORIAL_DoE_GUIDE.md`
|
||||
- **"How do I analyze the results?"** → See `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
|
||||
- **"Why this approach?"** → See `FACTORIAL_DoE_GUIDE.md` - "The Why"
|
||||
- **"How do I run it?"** → `./scripts/run_factorial_doe.sh`
|
||||
|
||||
---
|
||||
|
||||
**Version:** 1.0
|
||||
**Created:** 2025-11-19
|
||||
**Status:** Ready for production use
|
||||
@@ -0,0 +1,158 @@
|
||||
> **SUPERSEDED (docs triage 2026-10-14).** Quick start for the same 2^6 factorial DoE, flagged obsolete 2026-10-09.
|
||||
> Instead: experiments/bare_metal/README.md. Moved here from `docs/working/architecture/getting-started/quick-start/factorial-doe.md`.
|
||||
|
||||
<!-- Moved from docs/01-getting-started/quick-start/factorial-doe.md to docs/working/archive/superseded/factorial-doe/quick-start.md on 2026-06-16 (docs reorg Phase 2) -->
|
||||
|
||||
> **⚠ OBSOLETE (flagged 2026-10-09).** Describes the retired per-loop `ENABLE_LOOP_1..6`
|
||||
> compile-time toggle architecture and hardcoded local paths; the physics adaptation is now
|
||||
> driven by the L8 Jacquard runtime mode selector. Kept for historical record.
|
||||
|
||||
# Quick Start: Complete 2^6 Factorial DoE
|
||||
|
||||
## TL;DR
|
||||
|
||||
Run a **complete, exhaustive 2^6 = 64 configuration experiment** in one go, collecting all data for later analysis.
|
||||
|
||||
## Three Command Options
|
||||
|
||||
### Option 1: Quick Validation (30 min)
|
||||
Test the script and infrastructure:
|
||||
```bash
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 1 QUICK_TEST
|
||||
```
|
||||
- 64 builds × 1 run = 64 total runs
|
||||
- Fast feedback on whether setup works
|
||||
- Output: `/home/rajames/CLionProjects/StarForth-DoE/experiments/QUICK_TEST/`
|
||||
|
||||
### Option 2: Standard Full DoE (2-4 hours)
|
||||
Comprehensive factorial collection:
|
||||
```bash
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FULL_FACTORIAL
|
||||
```
|
||||
- 64 builds × 30 runs = 1,920 total runs
|
||||
- Good statistical power for main effects
|
||||
- Standard production-grade experiment
|
||||
- Output: `/home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/`
|
||||
|
||||
### Option 3: High-Precision Run (6-12 hours)
|
||||
Maximum data for deep analysis:
|
||||
```bash
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 100 HIGH_PRECISION_FACTORIAL
|
||||
```
|
||||
- 64 builds × 100 runs = 6,400 total runs
|
||||
- Best for detecting subtle interactions
|
||||
- Run overnight for best results
|
||||
- Output: `/home/rajames/CLionProjects/StarForth-DoE/experiments/HIGH_PRECISION_FACTORIAL/`
|
||||
|
||||
## What Gets Generated
|
||||
|
||||
```
|
||||
/home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/
|
||||
├── experiment_results.csv ← All 1,920 rows of measurements
|
||||
├── experiment_summary.txt ← Timing and metadata
|
||||
├── configuration_manifest.txt ← All 64 configs listed
|
||||
├── test_matrix.txt ← Randomized execution order
|
||||
└── run_logs/
|
||||
├── 0_0_0_0_0_0_run_1.log
|
||||
├── 0_0_0_0_0_0_run_2.log
|
||||
├── ...
|
||||
└── 1_1_1_1_1_1_run_30.log
|
||||
```
|
||||
|
||||
## What's Being Tested
|
||||
|
||||
6 **independent binary feedback loops**:
|
||||
|
||||
| Loop | Toggle | What It Does |
|
||||
|------|--------|---|
|
||||
| 1 | `ENABLE_LOOP_1_HEAT_TRACKING` | Count execution frequency |
|
||||
| 2 | `ENABLE_LOOP_2_ROLLING_WINDOW` | Capture execution history |
|
||||
| 3 | `ENABLE_LOOP_3_LINEAR_DECAY` | Age words over time |
|
||||
| 4 | `ENABLE_LOOP_4_PIPELINING_METRICS` | Track word transitions |
|
||||
| 5 | `ENABLE_LOOP_5_WINDOW_INFERENCE` | Infer window width |
|
||||
| 6 | `ENABLE_LOOP_6_DECAY_INFERENCE` | Infer decay slope |
|
||||
|
||||
**All 2^6 = 64 combinations** tested:
|
||||
- `0_0_0_0_0_0` = Baseline (all OFF)
|
||||
- `1_0_0_0_0_0` = Only heat tracking
|
||||
- `0_1_0_0_0_0` = Only rolling window
|
||||
- ...
|
||||
- `1_1_1_1_1_1` = All loops ON (current optimal)
|
||||
|
||||
## During Execution
|
||||
|
||||
You'll see progress like:
|
||||
```
|
||||
═══════════════════════════════════════════════════════════
|
||||
STARFORTH COMPLETE 2^6 FACTORIAL DoE
|
||||
═══════════════════════════════════════════════════════════
|
||||
→ Factors: 6 feedback loops (LOOP_1 through LOOP_6)
|
||||
→ Configurations: 2^6 = 64
|
||||
→ Runs per config: 30
|
||||
→ TOTAL RUNS: 1,920
|
||||
→ Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL
|
||||
|
||||
>>> Generating randomized test matrix...
|
||||
✓ Test matrix generated
|
||||
|
||||
→ Run 1/1920 - config 0_1_0_1_1_0 #14...
|
||||
✓ Run 1/1920 completed (3s)
|
||||
→ Run 2/1920 - config 1_1_1_0_0_1 #2...
|
||||
✓ Run 2/1920 completed (2s)
|
||||
...
|
||||
```
|
||||
|
||||
The script:
|
||||
1. **Builds** each of 64 configurations (rebuilds when config changes)
|
||||
2. **Runs** tests in randomized order (no systematic bias)
|
||||
3. **Collects** metrics to CSV
|
||||
4. **Outputs** progress every run
|
||||
|
||||
## After Completion
|
||||
|
||||
Move to analysis repo to understand results:
|
||||
```bash
|
||||
cd /home/rajames/CLionProjects/StarForth-DoE-Analysis
|
||||
python3 analyze_factorial.py \
|
||||
data/2025_11_19_FULL_FACTORIAL/experiment_results.csv
|
||||
```
|
||||
|
||||
Expected analysis outputs:
|
||||
- **Main effects:** Which individual loops help most?
|
||||
- **Interactions:** Do loops work better/worse together?
|
||||
- **Optimal subset:** Best combination of loops?
|
||||
- **Trade-offs:** Performance vs complexity?
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
✅ **Rebuild every pass** - ensures clean, isolated testing
|
||||
✅ **Randomized execution** - eliminates temporal/thermal bias
|
||||
✅ **Complete factorial** - captures all interactions
|
||||
✅ **One-and-done** - collect all data, analyze later
|
||||
|
||||
## Estimated Time
|
||||
|
||||
| Runs/Config | Total | Time | When |
|
||||
|---|---|---|---|
|
||||
| 1 | 64 | 30-45 min | Quick test |
|
||||
| 30 | 1,920 | 2-4 hours | Standard |
|
||||
| 100 | 6,400 | 6-12 hours | Overnight |
|
||||
|
||||
Times depend on CPU, disk speed, and test harness overhead.
|
||||
|
||||
## Example Command
|
||||
|
||||
**Run it now:**
|
||||
```bash
|
||||
cd /home/rajames/CLionProjects/StarForth
|
||||
./scripts/run_factorial_doe.sh 2025_11_19_FULL_FACTORIAL
|
||||
```
|
||||
|
||||
**Monitor progress:**
|
||||
```bash
|
||||
tail -f /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_summary.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Ready?** Run the command above and let the experiment collect comprehensive data for downstream analysis.
|
||||
@@ -0,0 +1,300 @@
|
||||
> **SUPERSEDED (docs triage 2026-10-14).** The 2^6 factorial DoE guide, flagged obsolete 2026-10-09.
|
||||
> Instead: experiments/bare_metal/README.md for the DoE as it runs now. Moved here from `docs/working/experiments/02-experiments/factorial-doe/workflow.md`.
|
||||
|
||||
<!-- Moved from docs/02-experiments/factorial-doe/workflow.md to docs/working/archive/superseded/factorial-doe/workflow.md on 2026-06-16 (docs reorg Phase 2) -->
|
||||
|
||||
> **⚠ HISTORICAL (flagged 2026-10-09).** Documents the completed 2025-11-19 factorial
|
||||
> campaign whose data informed the current L8 design; the toggle architecture it runs on is
|
||||
> retired. Kept as a record.
|
||||
|
||||
# Factorial DoE Analysis Workflow
|
||||
|
||||
## End-to-End Process
|
||||
|
||||
```
|
||||
StarForth Repo
|
||||
└─ ./scripts/run_factorial_doe.sh
|
||||
│
|
||||
├─ Builds all 64 configurations
|
||||
├─ Runs 1,920+ tests (randomized)
|
||||
└─ Outputs: experiment_results.csv + logs
|
||||
│
|
||||
└─→ Copy to StarForth-DoE-Analysis repo
|
||||
│
|
||||
├─ Load CSV into Python/R
|
||||
├─ Separate factors from metrics
|
||||
├─ Compute main effects & interactions
|
||||
├─ Identify optimal subsets
|
||||
└─ Generate reports & visualizations
|
||||
```
|
||||
|
||||
## Phase 1: Data Collection (StarForth Repo)
|
||||
|
||||
### Step 1: Run Experiment
|
||||
|
||||
```bash
|
||||
cd /home/rajames/CLionProjects/StarForth
|
||||
|
||||
# Quick validation (optional)
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 1 TEST_01
|
||||
|
||||
# Full production run
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FULL_FACTORIAL
|
||||
```
|
||||
|
||||
### Step 2: Wait for Completion
|
||||
Monitor in another terminal:
|
||||
```bash
|
||||
watch -n 5 'wc -l /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_results.csv'
|
||||
```
|
||||
|
||||
Expected progression:
|
||||
- 0 min: 1 line (header)
|
||||
- 30 min: ~500 lines
|
||||
- 2 hours: ~1,500 lines
|
||||
- 2.5 hours: 1,921 lines (complete)
|
||||
|
||||
### Step 3: Verify Completion
|
||||
```bash
|
||||
tail /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_results.csv
|
||||
echo "---"
|
||||
cat /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_summary.txt
|
||||
```
|
||||
|
||||
## Phase 2: Data Transfer & Setup
|
||||
|
||||
### Step 1: Copy to Analysis Repo
|
||||
```bash
|
||||
RESULTS_DIR="/home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL"
|
||||
ANALYSIS_REPO="/home/rajames/CLionProjects/StarForth-DoE-Analysis"
|
||||
|
||||
mkdir -p "$ANALYSIS_REPO/data"
|
||||
cp -r "$RESULTS_DIR" "$ANALYSIS_REPO/data/2025_11_19_FULL_FACTORIAL"
|
||||
```
|
||||
|
||||
### Step 2: Verify Transfer
|
||||
```bash
|
||||
ls -lah "$ANALYSIS_REPO/data/2025_11_19_FULL_FACTORIAL/"
|
||||
wc -l "$ANALYSIS_REPO/data/2025_11_19_FULL_FACTORIAL/experiment_results.csv"
|
||||
```
|
||||
|
||||
## Phase 3: Analysis (StarForth-DoE-Analysis Repo)
|
||||
|
||||
### Load Data (Python)
|
||||
```python
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from scipy import stats
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
# Load results_run_01_2025_12_08
|
||||
df = pd.read_csv("data/2025_11_19_FULL_FACTORIAL/experiment_results.csv")
|
||||
print(f"Loaded {len(df)} runs across {len(df.groupby('configuration'))} configurations")
|
||||
|
||||
# Separate factors and metrics
|
||||
factors = df[['enable_loop_1_heat_tracking',
|
||||
'enable_loop_2_rolling_window',
|
||||
'enable_loop_3_linear_decay',
|
||||
'enable_loop_4_pipelining_metrics',
|
||||
'enable_loop_5_window_inference',
|
||||
'enable_loop_6_decay_inference']]
|
||||
|
||||
metrics = df[['total_lookups', 'cache_hit_percent', 'vm_workload_duration_ns_q48', ...]]
|
||||
|
||||
print(f"Shape: {len(df)} rows × {len(df.columns)} columns")
|
||||
print(f"Factors: {len(factors.columns)}")
|
||||
print(f"Metrics: {len(metrics.columns)}")
|
||||
```
|
||||
|
||||
### Main Effects Analysis
|
||||
```python
|
||||
# For each loop, compute average metric change when loop goes 0→1
|
||||
main_effects = {}
|
||||
for loop in factors.columns:
|
||||
loop_name = loop.replace('enable_', '').replace('_', ' ').title()
|
||||
metric_when_on = df[df[loop] == 1]['vm_workload_duration_ns_q48'].mean()
|
||||
metric_when_off = df[df[loop] == 0]['vm_workload_duration_ns_q48'].mean()
|
||||
|
||||
effect = metric_when_on - metric_when_off
|
||||
pct_change = 100 * (effect / metric_when_off)
|
||||
|
||||
main_effects[loop_name] = {
|
||||
'effect': effect,
|
||||
'pct_change': pct_change,
|
||||
'on_mean': metric_when_on,
|
||||
'off_mean': metric_when_off,
|
||||
}
|
||||
|
||||
# Print sorted by impact
|
||||
for loop, effect in sorted(main_effects.items(), key=lambda x: abs(x[1]['pct_change']), reverse=True):
|
||||
print(f"{loop:30} {effect['pct_change']:+7.2f}% (ON: {effect['on_mean']:.0f}, OFF: {effect['off_mean']:.0f})")
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
Loop 6 Decay Inference +12.3% (ON: 8,800,000, OFF: 7,840,000)
|
||||
Loop 5 Window Inference +8.5% (ON: 8,600,000, OFF: 7,920,000)
|
||||
Loop 4 Pipelining Metrics +6.2% (ON: 8,400,000, OFF: 7,900,000)
|
||||
Loop 3 Linear Decay +4.1% (ON: 8,200,000, OFF: 7,870,000)
|
||||
Loop 2 Rolling Window +3.0% (ON: 8,100,000, OFF: 7,850,000)
|
||||
Loop 1 Heat Tracking +2.1% (ON: 8,010,000, OFF: 7,850,000)
|
||||
```
|
||||
|
||||
### Interaction Analysis
|
||||
```python
|
||||
# Two-way interactions: Do loops amplify/suppress each other?
|
||||
interactions = {}
|
||||
|
||||
for i, loop_a in enumerate(factors.columns):
|
||||
for j, loop_b in enumerate(factors.columns):
|
||||
if i >= j:
|
||||
continue # Only upper triangle
|
||||
|
||||
# Four cases: (0,0), (0,1), (1,0), (1,1)
|
||||
case_00 = df[(df[loop_a] == 0) & (df[loop_b] == 0)]['vm_workload_duration_ns_q48'].mean()
|
||||
case_01 = df[(df[loop_a] == 0) & (df[loop_b] == 1)]['vm_workload_duration_ns_q48'].mean()
|
||||
case_10 = df[(df[loop_a] == 1) & (df[loop_b] == 0)]['vm_workload_duration_ns_q48'].mean()
|
||||
case_11 = df[(df[loop_a] == 1) & (df[loop_b] == 1)]['vm_workload_duration_ns_q48'].mean()
|
||||
|
||||
# Interaction effect: (1,1) - (1,0) - (0,1) + (0,0)
|
||||
interaction = (case_11 - case_10 - case_01 + case_00)
|
||||
|
||||
loop_a_name = loop_a.replace('enable_', '').replace('_', ' ')
|
||||
loop_b_name = loop_b.replace('enable_', '').replace('_', ' ')
|
||||
|
||||
if abs(interaction) > 100000: # Only report large interactions
|
||||
interactions[f"{loop_a_name} × {loop_b_name}"] = interaction
|
||||
|
||||
# Print sorted interactions
|
||||
for pair, interaction in sorted(interactions.items(), key=lambda x: abs(x[1]), reverse=True):
|
||||
print(f"{pair:40} {interaction:+12,.0f}")
|
||||
```
|
||||
|
||||
### Optimal Configuration Search
|
||||
```python
|
||||
# Find best k configurations by performance metric
|
||||
k = 10
|
||||
best_configs = df.nlargest(k, 'vm_workload_duration_ns_q48')[
|
||||
['configuration', 'vm_workload_duration_ns_q48', 'enable_loop_1_heat_tracking',
|
||||
'enable_loop_2_rolling_window', 'enable_loop_3_linear_decay',
|
||||
'enable_loop_4_pipelining_metrics', 'enable_loop_5_window_inference',
|
||||
'enable_loop_6_decay_inference']
|
||||
].drop_duplicates(subset=['configuration'])
|
||||
|
||||
print("Top 10 Configurations by Performance:")
|
||||
print("─" * 100)
|
||||
for idx, row in best_configs.iterrows():
|
||||
loops = (
|
||||
f"L1:{row['enable_loop_1_heat_tracking']} " +
|
||||
f"L2:{row['enable_loop_2_rolling_window']} " +
|
||||
f"L3:{row['enable_loop_3_linear_decay']} " +
|
||||
f"L4:{row['enable_loop_4_pipelining_metrics']} " +
|
||||
f"L5:{row['enable_loop_5_window_inference']} " +
|
||||
f"L6:{row['enable_loop_6_decay_inference']}"
|
||||
)
|
||||
print(f"{row['configuration']:20} | {row['vm_workload_duration_ns_q48']:12,.0f} ns | {loops}")
|
||||
```
|
||||
|
||||
### Visualization Example
|
||||
```python
|
||||
# Main effects plot
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
fig, axes = plt.subplots(2, 3, figsize=(15, 10))
|
||||
fig.suptitle('2^6 Factorial DoE Main Effects', fontsize=16)
|
||||
|
||||
loop_names = [f"Loop {i+1}" for i in range(6)]
|
||||
metric_key = 'vm_workload_duration_ns_q48'
|
||||
|
||||
for idx, loop in enumerate(factors.columns):
|
||||
ax = axes[idx // 3, idx % 3]
|
||||
|
||||
off_mean = df[df[loop] == 0][metric_key].mean()
|
||||
on_mean = df[df[loop] == 1][metric_key].mean()
|
||||
|
||||
bars = ax.bar(['OFF', 'ON'], [off_mean, on_mean], color=['lightcoral', 'lightgreen'])
|
||||
ax.set_ylabel('Workload Duration (ns)')
|
||||
ax.set_title(f'{loop_names[idx]}')
|
||||
ax.set_ylim([7000000, 10000000])
|
||||
|
||||
# Add value labels
|
||||
for bar in bars:
|
||||
height = bar.get_height()
|
||||
ax.text(bar.get_x() + bar.get_width()/2., height,
|
||||
f'{height/1e6:.1f}M', ha='center', va='bottom')
|
||||
|
||||
plt.tight_layout()
|
||||
plt.savefig('main_effects.png', dpi=150)
|
||||
print("Saved: main_effects.png")
|
||||
```
|
||||
|
||||
## Phase 4: Reporting
|
||||
|
||||
### Key Outputs to Generate
|
||||
|
||||
1. **Main Effects Summary** (which loops matter most?)
|
||||
- Table of individual loop impacts
|
||||
- Sorted by effect size
|
||||
|
||||
2. **Interaction Matrix** (do loops amplify/suppress?)
|
||||
- 6×6 table of two-way interactions
|
||||
- Highlights synergies/conflicts
|
||||
|
||||
3. **Optimal Configuration** (best 10-20 subsets)
|
||||
- Recommend top 3-5 production configs
|
||||
- Show Pareto frontier (performance vs complexity)
|
||||
|
||||
4. **Statistical Summary**
|
||||
- Variance by configuration
|
||||
- Confidence intervals on estimates
|
||||
- Regression model R²
|
||||
|
||||
### Example Report Structure
|
||||
|
||||
```markdown
|
||||
# Factorial DoE Analysis Report - 2025-11-19
|
||||
|
||||
## Executive Summary
|
||||
- Tested all 64 feedback loop combinations
|
||||
- 1,920 experimental runs collected
|
||||
- Key finding: Loops 5+6 show strong synergy (+18% combined)
|
||||
|
||||
## Main Effects
|
||||
| Loop | Impact | Confidence | Recommendation |
|
||||
|------|--------|-----------|---|
|
||||
| Loop 6 (Decay Inference) | +12.3% | p<0.001 | **Enabled** |
|
||||
| Loop 5 (Window Inference) | +8.5% | p<0.001 | **Enabled** |
|
||||
| Loop 4 (Pipelining) | +6.2% | p<0.01 | Enabled |
|
||||
| Loop 3 (Decay) | +4.1% | p<0.05 | Enabled |
|
||||
| Loop 2 (Window) | +3.0% | p<0.05 | Enabled |
|
||||
| Loop 1 (Heat) | +2.1% | p>0.05 | Optional |
|
||||
|
||||
## Recommended Configurations
|
||||
1. **111111** (All loops): +30% vs baseline
|
||||
2. **111110** (Except heat): +28% vs baseline, simpler
|
||||
3. **011111** (Except heat, decay): +24% vs baseline
|
||||
|
||||
## Trade-off Analysis
|
||||
- Full stack (111111): Best performance, highest code complexity
|
||||
- Recommended (111110): 28% improvement, 95% of benefit
|
||||
- Minimal set (001110): 18% improvement, 50% code overhead
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] Run factorial DoE script to completion
|
||||
- [ ] Verify 1,920 rows in experiment_results.csv
|
||||
- [ ] Copy results to analysis repo
|
||||
- [ ] Load data in Python/R
|
||||
- [ ] Compute main effects (6 loops)
|
||||
- [ ] Compute interactions (2-way or higher)
|
||||
- [ ] Identify top 5-10 configurations
|
||||
- [ ] Generate visualizations
|
||||
- [ ] Write summary report
|
||||
- [ ] Decide on production configuration
|
||||
|
||||
---
|
||||
|
||||
**Timeline:** Collection (2-4 hrs) + Analysis (2-4 hrs) = **6-8 hours total** for comprehensive factorial DoE.
|
||||
Reference in New Issue
Block a user