docs: reorganise -- current reference gathered, finished work archived, superseded and dead ideas marked as such
docs/CLAUDE.md Phase 2, 2026-10-14. All moves by git mv; every archived file carries a header saying what it is, why, and where to look instead. - docs/reference/ (new): roadmap and versioning policy, M7.1, word-ACL design, kernel args, hosted acceptance test, dictionary, primitives, ontology, L8 table -- previously spread over docs/, docs/lithosananke/ and docs/03-architecture/. - docs/working/archive/historical/: finished work (session plan, test-gap audit, birthing plan, fix write-ups, the January architecture draft). - docs/working/archive/superseded/: the Hermes-VM Tripod docs and July Hermes designs (-> FABRIC-3.5/3.6), the StarForth roadmap, every working doc already flagged superseded or obsolete. - docs/working/archive/dead/: the hal_* target contract (never adopted), L4Re, metrics-and-knobs. - Papers and proposals into docs/working/papers/. - References rewritten across the repo (not logs/, not docs/patent/); CLAUDE.md's documentation section rewritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
rajames
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
1a4b163861
commit
7aed588f78
File diff suppressed because it is too large
Load Diff
@@ -1,262 +0,0 @@
|
||||
<!-- Moved from docs/02-experiments/factorial-doe/guide.md to docs/working/experiments/02-experiments/factorial-doe/guide.md on 2026-06-16 (docs reorg Phase 2) -->
|
||||
|
||||
> **⚠ OBSOLETE (flagged 2026-10-09).** Describes the retired per-loop Makefile-toggle
|
||||
> architecture, replaced by the L8 Jacquard runtime mode selector. Kept for historical record.
|
||||
|
||||
# StarForth Complete 2^6 Factorial Design of Experiments
|
||||
|
||||
## Overview
|
||||
|
||||
**One-and-done comprehensive factorial testing** - collect all data systematically, analyze separately.
|
||||
|
||||
This guide describes how to run a **complete 2^6 factorial design of experiments** across all 6 feedback loops, yielding:
|
||||
- **64 unique configurations** (all binary combinations)
|
||||
- **1,920+ runs** (30 samples per configuration)
|
||||
- **Single unified dataset** for factorial analysis
|
||||
|
||||
## The Why: Factorial DoE
|
||||
|
||||
Instead of incrementally tuning parameters, we collect **comprehensive experimental data** in one go:
|
||||
|
||||
### Traditional Approach (Old)
|
||||
- Run baseline
|
||||
- Run with Loop #1
|
||||
- Run with Loops #1-2
|
||||
- Run with Loops #1-3
|
||||
- ... (sequential, incremental)
|
||||
- **Problem:** Can't see interactions, trade-offs, or surprises
|
||||
|
||||
### Factorial Approach (New)
|
||||
- Run ALL 2^6 = 64 combinations in randomized order
|
||||
- Collect all data at once
|
||||
- Analyze factorial effects, interactions, and main effects
|
||||
- **Advantage:** Complete picture of how loops interact
|
||||
|
||||
## The 6 Feedback Loops
|
||||
|
||||
Each loop is a binary toggle in the Makefile:
|
||||
|
||||
| Loop | Makefile Flag | Purpose |
|
||||
|------|---|---|
|
||||
| **#1** | `ENABLE_LOOP_1_HEAT_TRACKING` | Count word execution frequency |
|
||||
| **#2** | `ENABLE_LOOP_2_ROLLING_WINDOW` | Capture execution history (circular buffer) |
|
||||
| **#3** | `ENABLE_LOOP_3_LINEAR_DECAY` | Age words by exponential decay over time |
|
||||
| **#4** | `ENABLE_LOOP_4_PIPELINING_METRICS` | Track word-to-word transitions |
|
||||
| **#5** | `ENABLE_LOOP_5_WINDOW_INFERENCE` | Infer optimal window width (Levene's test) |
|
||||
| **#6** | `ENABLE_LOOP_6_DECAY_INFERENCE` | Infer decay slope via linear regression |
|
||||
|
||||
## Configuration Naming
|
||||
|
||||
Each of 64 configurations is named by its loop status:
|
||||
|
||||
```
|
||||
L1_L2_L3_L4_L5_L6
|
||||
|
||||
Where each digit is 0 (OFF) or 1 (ON)
|
||||
|
||||
Examples:
|
||||
000000 = Baseline (all loops OFF - pure FORTH-79)
|
||||
000001 = Only heat tracking
|
||||
111110 = All except decay inference
|
||||
111111 = All loops ON (likely optimal)
|
||||
101010 = Alternating pattern (for research)
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Minimal Test (validate script)
|
||||
```bash
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 1 TEST_RUN
|
||||
# 64 builds × 1 run = 64 total runs
|
||||
# Runtime: ~30-45 minutes
|
||||
# Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/TEST_RUN/
|
||||
```
|
||||
|
||||
### Standard Comprehensive Run
|
||||
```bash
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FACTORIAL
|
||||
# 64 builds × 30 runs = 1,920 total runs
|
||||
# Runtime: ~2-4 hours
|
||||
# Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FACTORIAL/
|
||||
```
|
||||
|
||||
### High-Precision Run
|
||||
```bash
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 100 HIGH_PRECISION_DOE
|
||||
# 64 builds × 100 runs = 6,400 total runs
|
||||
# Runtime: ~6-12 hours
|
||||
# Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/HIGH_PRECISION_DOE/
|
||||
```
|
||||
|
||||
## Output Structure
|
||||
|
||||
```
|
||||
/home/rajames/CLionProjects/StarForth-DoE/experiments/<EXPERIMENT_LABEL>/
|
||||
├── experiment_results.csv # All measurements (1920+ rows)
|
||||
├── experiment_summary.txt # Timing, metadata, statistics
|
||||
├── configuration_manifest.txt # All 64 configs documented
|
||||
├── test_matrix.txt # Randomized execution order
|
||||
└── run_logs/
|
||||
├── 000000_run_1.log
|
||||
├── 000000_run_2.log
|
||||
├── 000001_run_1.log
|
||||
├── ...
|
||||
└── 111111_run_30.log # 1,920 total log files
|
||||
```
|
||||
|
||||
## CSV Schema
|
||||
|
||||
Each row in `experiment_results.csv` contains:
|
||||
|
||||
```
|
||||
timestamp,configuration,run_number,
|
||||
<35+ metrics from test harness>,
|
||||
enable_loop_1_heat_tracking,
|
||||
enable_loop_2_rolling_window,
|
||||
enable_loop_3_linear_decay,
|
||||
enable_loop_4_pipelining_metrics,
|
||||
enable_loop_5_window_inference,
|
||||
enable_loop_6_decay_inference
|
||||
```
|
||||
|
||||
This lets you correlate every metric against every loop combination.
|
||||
|
||||
## Analysis Strategy
|
||||
|
||||
After collection, analyze in the separate `StarForth-DoE` repository:
|
||||
|
||||
### Step 1: Load Data
|
||||
```python
|
||||
import pandas as pd
|
||||
df = pd.read_csv("experiment_results.csv")
|
||||
print(df.shape) # Should be (1920, 38+)
|
||||
print(df.columns)
|
||||
```
|
||||
|
||||
### Step 2: Separate Metrics from Factors
|
||||
```python
|
||||
factor_cols = ['enable_loop_1_heat_tracking', ..., 'enable_loop_6_decay_inference']
|
||||
metric_cols = ['total_lookups', 'cache_hit_percent', ..., 'vm_workload_duration_ns_q48']
|
||||
|
||||
factors = df[factor_cols]
|
||||
metrics = df[metric_cols]
|
||||
```
|
||||
|
||||
### Step 3: Factorial Analysis
|
||||
```python
|
||||
# Main effects: What's the average change when Loop X goes from 0→1?
|
||||
main_effects = {}
|
||||
for loop in factor_cols:
|
||||
main_effects[loop] = metrics[factors[loop]==1].mean() - metrics[factors[loop]==0].mean()
|
||||
|
||||
# Interactions: Do loops work better/worse together?
|
||||
# Can use statsmodels.formula.api for ANOVA/regression
|
||||
|
||||
# Optimal subset: Which combinations of loops give best throughput?
|
||||
best_configs = df.nlargest(10, 'vm_workload_duration_ns_q48')
|
||||
```
|
||||
|
||||
## Key Design Choices
|
||||
|
||||
### 1. **Rebuild Every Pass** ✓
|
||||
- Each configuration requires a `make clean && make`
|
||||
- Ensures clean, isolated state
|
||||
- No cross-contamination
|
||||
- True ground truth for each variant
|
||||
|
||||
### 2. **Randomized Execution**
|
||||
- All 1,920 runs randomized into single matrix (not 64 separate runs)
|
||||
- Prevents thermal ramp, temporal bias, or systematic trends
|
||||
- DoE principle: randomization eliminates confounds
|
||||
|
||||
### 3. **Full Factorial** (not fractional)
|
||||
- Testing ALL 2^6 = 64 combinations
|
||||
- Can detect unexpected interactions
|
||||
- Most expensive, most informative
|
||||
|
||||
### 4. **One-and-Done Collection**
|
||||
- Single experiment, all data
|
||||
- No progressive tuning during collection
|
||||
- Separates measurement from analysis (good practice)
|
||||
|
||||
## Expected Results
|
||||
|
||||
### Hypothetical Output
|
||||
Assuming typical performance distribution:
|
||||
|
||||
```
|
||||
Configuration | Avg Time (ns) | Speed vs Baseline
|
||||
─────────────────────────────────────────────────
|
||||
000000 (base) | 10,000,000 | 1.0x
|
||||
000001 (L1) | 9,950,000 | 1.005x
|
||||
111111 (all) | 8,500,000 | 1.176x ← Likely best
|
||||
```
|
||||
|
||||
### Expected Patterns
|
||||
1. **Loop #1 (heat tracking)**: ~2-5% improvement alone
|
||||
2. **Loop #2 (rolling window)**: ~1-3% improvement (depends on L1)
|
||||
3. **Loops #3-6**: Increasingly synergistic (work better together)
|
||||
4. **Interactions**: Loop #5 + #6 likely synergistic (inference loops)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Build Fails for a Configuration
|
||||
```bash
|
||||
# Check specific config build log
|
||||
cat /home/rajames/CLionProjects/StarForth-DoE/experiments/<LABEL>/run_logs/build_<CONFIG>.log
|
||||
```
|
||||
|
||||
### Test Harness Crashes on Some Configs
|
||||
- Check if loop combination is invalid (e.g., Loop #5 without Loop #2)
|
||||
- Verify Makefile flags are correctly parsed
|
||||
- See: `include/vm.h` for interdependencies
|
||||
|
||||
### Results Look Suspicious
|
||||
- Check `test_matrix.txt` - ensure randomization is present
|
||||
- Verify CSV row counts match expected total
|
||||
- Check for missing runs in log directory
|
||||
|
||||
## Performance Notes
|
||||
|
||||
### Timing Estimates
|
||||
- **Minimal (1 run/config)**: 30-45 minutes
|
||||
- **Standard (30 runs/config)**: 2-4 hours
|
||||
- **High-precision (100 runs/config)**: 6-12 hours
|
||||
- **Depends on:** CPU speed, disk I/O, test harness overhead
|
||||
|
||||
### Optimization
|
||||
- Run overnight for high-precision experiments
|
||||
- Use `screen` or `tmux` to keep session alive
|
||||
- Monitor disk space (~100 MB for results + logs)
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Run the experiment:**
|
||||
```bash
|
||||
./scripts/run_factorial_doe.sh 2025_11_19_COMPLETE_FACTORIAL
|
||||
```
|
||||
|
||||
2. **Monitor progress:**
|
||||
```bash
|
||||
tail -f /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_COMPLETE_FACTORIAL/experiment_summary.txt
|
||||
```
|
||||
|
||||
3. **Move to analysis repo** (once complete):
|
||||
```bash
|
||||
cp -r /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_COMPLETE_FACTORIAL \
|
||||
/home/rajames/CLionProjects/StarForth-DoE-Analysis/data/
|
||||
cd /home/rajames/CLionProjects/StarForth-DoE-Analysis
|
||||
python3 analyze_factorial.py data/2025_11_19_COMPLETE_FACTORIAL/experiment_results.csv
|
||||
```
|
||||
|
||||
4. **Generate reports:**
|
||||
- Main effects plots
|
||||
- Interaction matrices
|
||||
- Pareto optimal configurations
|
||||
- Recommendations for production builds
|
||||
|
||||
---
|
||||
|
||||
**Version:** 1.0
|
||||
**Last Updated:** 2025-11-19
|
||||
**Author:** Claude Code (with Robert A. James)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,268 +0,0 @@
|
||||
<!-- Moved from docs/02-experiments/factorial-doe/index.md to docs/working/experiments/02-experiments/factorial-doe/index.md on 2026-06-16 (docs reorg Phase 2) -->
|
||||
|
||||
> **⚠ OBSOLETE (flagged 2026-10-09).** Index for the retired 2^6 factorial toggle
|
||||
> architecture, replaced by the L8 Jacquard mode selector. Kept for historical record.
|
||||
|
||||
# StarForth Complete 2^6 Factorial DoE - Documentation Index
|
||||
|
||||
## Overview
|
||||
|
||||
This is a **one-and-done comprehensive factorial design of experiments** to systematically test all 64 possible combinations of 6 feedback loops.
|
||||
|
||||
**Key insight:** Don't toggle back-and-forth. Instead, systematically test ALL 2^6 combinations in a single experiment, then analyze the results afterward.
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
### 1. **START HERE** 📋
|
||||
- **File:** `docs/QUICK_START_FACTORIAL_DoE.md`
|
||||
- **Purpose:** Entry point for quick reference
|
||||
- **Read time:** 5 minutes
|
||||
- **Contains:**
|
||||
- 3 command options (quick/standard/high-precision)
|
||||
- Time estimates
|
||||
- What to expect during execution
|
||||
- Basic troubleshooting
|
||||
|
||||
### 2. **Complete Guide** 📚
|
||||
- **File:** `docs/FACTORIAL_DoE_GUIDE.md`
|
||||
- **Purpose:** Full design documentation and rationale
|
||||
- **Read time:** 20-30 minutes
|
||||
- **Contains:**
|
||||
- Why factorial DoE (vs. incremental tuning)
|
||||
- 6 feedback loop definitions
|
||||
- Configuration naming scheme (0_0_0_0_0_0 → 1_1_1_1_1_1)
|
||||
- Design choices explained
|
||||
- Expected results and patterns
|
||||
- Troubleshooting guide
|
||||
- Performance notes
|
||||
|
||||
### 3. **Analysis Workflow** 🔍
|
||||
- **File:** `docs/FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
|
||||
- **Purpose:** End-to-end pipeline from collection to insights
|
||||
- **Read time:** 15-20 minutes (or 1-2 hours if coding along)
|
||||
- **Contains:**
|
||||
- Phase 1: Data collection (run script)
|
||||
- Phase 2: Data transfer (to analysis repo)
|
||||
- Phase 3: Statistical analysis (main effects, interactions)
|
||||
- Phase 4: Reporting (generating insights)
|
||||
- Python code examples:
|
||||
* Load data and validate
|
||||
* Compute main effects (individual loop impacts)
|
||||
* Detect interactions (loop synergies)
|
||||
* Optimize configuration subset
|
||||
* Visualization code
|
||||
- Example report structure
|
||||
- Checklist for completion
|
||||
|
||||
### 4. **Main Script** ⚙️
|
||||
- **File:** `scripts/run_factorial_doe.sh`
|
||||
- **Purpose:** Executable for running the entire experiment
|
||||
- **Read time:** 10 minutes (code + comments)
|
||||
- **Contains:**
|
||||
- Generates all 64 configurations
|
||||
- Rebuilds for each unique configuration
|
||||
- Randomizes execution order (1,920+ runs)
|
||||
- Collects metrics to CSV
|
||||
- Full instrumentation and progress reporting
|
||||
|
||||
---
|
||||
|
||||
## Reading Guide by Use Case
|
||||
|
||||
### "I just want to run the experiment"
|
||||
1. Read: `QUICK_START_FACTORIAL_DoE.md` (5 min)
|
||||
2. Execute: Choose command option (quick/standard/high-precision)
|
||||
3. Wait: 30 min - 12 hours depending on option
|
||||
4. Done!
|
||||
|
||||
### "I want to understand the design before running"
|
||||
1. Read: `QUICK_START_FACTORIAL_DoE.md` (5 min)
|
||||
2. Read: `FACTORIAL_DoE_GUIDE.md` (30 min)
|
||||
3. Execute: Run with confidence
|
||||
4. Analyze: Follow `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
|
||||
|
||||
### "I need to analyze the results"
|
||||
1. Read: `QUICK_START_FACTORIAL_DoE.md` → understand naming
|
||||
2. Read: `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md` → statistical methods
|
||||
3. Code along: Use Python examples in Analysis Workflow
|
||||
4. Interpret: Reference `FACTORIAL_DoE_GUIDE.md` for expected patterns
|
||||
|
||||
### "I want complete understanding"
|
||||
Read in order:
|
||||
1. `QUICK_START_FACTORIAL_DoE.md` (overview)
|
||||
2. `FACTORIAL_DoE_GUIDE.md` (design rationale)
|
||||
3. `scripts/run_factorial_doe.sh` (implementation)
|
||||
4. `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md` (analysis methods)
|
||||
|
||||
---
|
||||
|
||||
## Quick Command Reference
|
||||
|
||||
### Run Experiment
|
||||
```bash
|
||||
# Option A: Quick validation (30 min)
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 1 TEST_RUN
|
||||
|
||||
# Option B: Standard production (2-4 hours)
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FULL_FACTORIAL
|
||||
|
||||
# Option C: High-precision overnight (6-12 hours)
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 100 HIGH_PRECISION_FACTORIAL
|
||||
```
|
||||
|
||||
### Monitor Progress
|
||||
```bash
|
||||
# Check results_run_01_2025_12_08 CSV growth
|
||||
watch -n 5 'wc -l /path/to/experiment_results.csv'
|
||||
|
||||
# View summary
|
||||
tail /path/to/experiment_summary.txt
|
||||
|
||||
# Count completed configurations
|
||||
ls /path/to/run_logs/build_*.log | wc -l
|
||||
```
|
||||
|
||||
### Analyze Results
|
||||
```bash
|
||||
# Load and analyze in Python
|
||||
cd /path/to/StarForth-DoE-Analysis
|
||||
python3 << 'EOF'
|
||||
import pandas as pd
|
||||
df = pd.read_csv('data/2025_11_19_FULL_FACTORIAL/experiment_results.csv')
|
||||
print(f"Loaded {len(df)} runs")
|
||||
# See FACTORIAL_DoE_ANALYSIS_WORKFLOW.md for full analysis code
|
||||
EOF
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Concepts
|
||||
|
||||
### 64 Configurations
|
||||
Each configuration is a unique combination of 6 binary factors:
|
||||
- **0_0_0_0_0_0** = Baseline (all loops OFF)
|
||||
- **1_0_0_0_0_0** = Only heat tracking
|
||||
- **1_1_0_0_0_0** = Heat + rolling window
|
||||
- ...
|
||||
- **1_1_1_1_1_1** = All loops ON (current optimal)
|
||||
|
||||
### Randomized Execution
|
||||
All 1,920+ runs (64 configs × 30 samples) are randomized into a single execution matrix.
|
||||
- Eliminates temporal bias
|
||||
- Prevents thermal ramp (if CPU heats during night)
|
||||
- Proper DoE methodology
|
||||
|
||||
### One-and-Done Collection
|
||||
Collect all data in single experiment, analyze separately.
|
||||
- No tuning during collection (avoid bias)
|
||||
- Single pass through parameter space
|
||||
- Separates measurement from analysis
|
||||
|
||||
### Main Effects
|
||||
Individual impact of each loop:
|
||||
```
|
||||
Loop #1: Loop turns ON, average metric changes by X%
|
||||
Loop #2: Loop turns ON, average metric changes by Y%
|
||||
... etc
|
||||
```
|
||||
|
||||
### Interactions
|
||||
How loops work together (amplify or suppress each other):
|
||||
```
|
||||
Loop #1 + Loop #2: Do they work better together than independently?
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Files at a Glance
|
||||
|
||||
| File | Size | Purpose |
|
||||
|------|------|---------|
|
||||
| `scripts/run_factorial_doe.sh` | 19 KB | Main executable script |
|
||||
| `docs/QUICK_START_FACTORIAL_DoE.md` | 4.9 KB | Quick reference |
|
||||
| `docs/FACTORIAL_DoE_GUIDE.md` | 8.0 KB | Design documentation |
|
||||
| `docs/FACTORIAL_DoE_ANALYSIS_WORKFLOW.md` | 9.6 KB | Analysis pipeline |
|
||||
| `docs/FACTORIAL_DoE_INDEX.md` | This file | Documentation index |
|
||||
|
||||
---
|
||||
|
||||
## Execution Timeline
|
||||
|
||||
### Phase 1: Preparation (5-10 min)
|
||||
- Read `QUICK_START_FACTORIAL_DoE.md`
|
||||
- Choose execution option
|
||||
- Run command
|
||||
|
||||
### Phase 2: Collection (30 min - 12 hours)
|
||||
- Script builds all 64 configs (as needed)
|
||||
- Randomized execution of 1,920+ runs
|
||||
- Metrics collected to CSV
|
||||
- Progress shown every run
|
||||
|
||||
### Phase 3: Transfer (5 min)
|
||||
- Copy results to analysis repo
|
||||
- Verify file integrity
|
||||
|
||||
### Phase 4: Analysis (1-2 hours)
|
||||
- Load CSV in Python
|
||||
- Compute main effects
|
||||
- Detect interactions
|
||||
- Find optimal configurations
|
||||
- Generate visualizations
|
||||
|
||||
### Phase 5: Reporting (1-2 hours)
|
||||
- Summarize findings
|
||||
- Create report
|
||||
- Recommendations for production
|
||||
|
||||
**Total time:** Collection (2-4 hrs) + Analysis (2-4 hrs) = **4-8 hours**
|
||||
|
||||
---
|
||||
|
||||
## The Design Philosophy
|
||||
|
||||
### Problem Statement
|
||||
How do we systematically understand the performance contribution and interactions of 6 interdependent feedback loops?
|
||||
|
||||
### Solution
|
||||
**Complete 2^6 factorial design of experiments**
|
||||
|
||||
### Why This Approach
|
||||
1. **Complete:** Tests all 64 combinations (no gaps)
|
||||
2. **Systematic:** Proper experimental design methodology
|
||||
3. **Separable:** Collection ≠ Analysis (reduces bias)
|
||||
4. **Reproducible:** Randomized execution, full logging
|
||||
5. **Scalable:** Single script, extensible to more factors
|
||||
|
||||
### Alternatives Rejected
|
||||
- ❌ Incremental tuning ("baseline → +loop1 → +loop1+2"): Misses interactions
|
||||
- ❌ Dynamic toggle without rebuild: Introduces state contamination
|
||||
- ❌ Partial factorial: Misses key interactions
|
||||
- ❌ One-at-a-time testing: Expensive and confounded
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Read** `QUICK_START_FACTORIAL_DoE.md` (5 minutes)
|
||||
2. **Decide** which execution option (quick/standard/precise)
|
||||
3. **Run** the command and let it collect data
|
||||
4. **Monitor** progress in another terminal
|
||||
5. **Analyze** results using `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
|
||||
6. **Report** findings to stakeholders
|
||||
|
||||
---
|
||||
|
||||
## Questions?
|
||||
|
||||
- **"How long will this take?"** → See `QUICK_START_FACTORIAL_DoE.md`
|
||||
- **"What does configuration X mean?"** → See `FACTORIAL_DoE_GUIDE.md`
|
||||
- **"How do I analyze the results?"** → See `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
|
||||
- **"Why this approach?"** → See `FACTORIAL_DoE_GUIDE.md` - "The Why"
|
||||
- **"How do I run it?"** → `./scripts/run_factorial_doe.sh`
|
||||
|
||||
---
|
||||
|
||||
**Version:** 1.0
|
||||
**Created:** 2025-11-19
|
||||
**Status:** Ready for production use
|
||||
@@ -1,297 +0,0 @@
|
||||
<!-- Moved from docs/02-experiments/factorial-doe/workflow.md to docs/working/experiments/02-experiments/factorial-doe/workflow.md on 2026-06-16 (docs reorg Phase 2) -->
|
||||
|
||||
> **⚠ HISTORICAL (flagged 2026-10-09).** Documents the completed 2025-11-19 factorial
|
||||
> campaign whose data informed the current L8 design; the toggle architecture it runs on is
|
||||
> retired. Kept as a record.
|
||||
|
||||
# Factorial DoE Analysis Workflow
|
||||
|
||||
## End-to-End Process
|
||||
|
||||
```
|
||||
StarForth Repo
|
||||
└─ ./scripts/run_factorial_doe.sh
|
||||
│
|
||||
├─ Builds all 64 configurations
|
||||
├─ Runs 1,920+ tests (randomized)
|
||||
└─ Outputs: experiment_results.csv + logs
|
||||
│
|
||||
└─→ Copy to StarForth-DoE-Analysis repo
|
||||
│
|
||||
├─ Load CSV into Python/R
|
||||
├─ Separate factors from metrics
|
||||
├─ Compute main effects & interactions
|
||||
├─ Identify optimal subsets
|
||||
└─ Generate reports & visualizations
|
||||
```
|
||||
|
||||
## Phase 1: Data Collection (StarForth Repo)
|
||||
|
||||
### Step 1: Run Experiment
|
||||
|
||||
```bash
|
||||
cd /home/rajames/CLionProjects/StarForth
|
||||
|
||||
# Quick validation (optional)
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 1 TEST_01
|
||||
|
||||
# Full production run
|
||||
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FULL_FACTORIAL
|
||||
```
|
||||
|
||||
### Step 2: Wait for Completion
|
||||
Monitor in another terminal:
|
||||
```bash
|
||||
watch -n 5 'wc -l /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_results.csv'
|
||||
```
|
||||
|
||||
Expected progression:
|
||||
- 0 min: 1 line (header)
|
||||
- 30 min: ~500 lines
|
||||
- 2 hours: ~1,500 lines
|
||||
- 2.5 hours: 1,921 lines (complete)
|
||||
|
||||
### Step 3: Verify Completion
|
||||
```bash
|
||||
tail /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_results.csv
|
||||
echo "---"
|
||||
cat /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_summary.txt
|
||||
```
|
||||
|
||||
## Phase 2: Data Transfer & Setup
|
||||
|
||||
### Step 1: Copy to Analysis Repo
|
||||
```bash
|
||||
RESULTS_DIR="/home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL"
|
||||
ANALYSIS_REPO="/home/rajames/CLionProjects/StarForth-DoE-Analysis"
|
||||
|
||||
mkdir -p "$ANALYSIS_REPO/data"
|
||||
cp -r "$RESULTS_DIR" "$ANALYSIS_REPO/data/2025_11_19_FULL_FACTORIAL"
|
||||
```
|
||||
|
||||
### Step 2: Verify Transfer
|
||||
```bash
|
||||
ls -lah "$ANALYSIS_REPO/data/2025_11_19_FULL_FACTORIAL/"
|
||||
wc -l "$ANALYSIS_REPO/data/2025_11_19_FULL_FACTORIAL/experiment_results.csv"
|
||||
```
|
||||
|
||||
## Phase 3: Analysis (StarForth-DoE-Analysis Repo)
|
||||
|
||||
### Load Data (Python)
|
||||
```python
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from scipy import stats
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
# Load results_run_01_2025_12_08
|
||||
df = pd.read_csv("data/2025_11_19_FULL_FACTORIAL/experiment_results.csv")
|
||||
print(f"Loaded {len(df)} runs across {len(df.groupby('configuration'))} configurations")
|
||||
|
||||
# Separate factors and metrics
|
||||
factors = df[['enable_loop_1_heat_tracking',
|
||||
'enable_loop_2_rolling_window',
|
||||
'enable_loop_3_linear_decay',
|
||||
'enable_loop_4_pipelining_metrics',
|
||||
'enable_loop_5_window_inference',
|
||||
'enable_loop_6_decay_inference']]
|
||||
|
||||
metrics = df[['total_lookups', 'cache_hit_percent', 'vm_workload_duration_ns_q48', ...]]
|
||||
|
||||
print(f"Shape: {len(df)} rows × {len(df.columns)} columns")
|
||||
print(f"Factors: {len(factors.columns)}")
|
||||
print(f"Metrics: {len(metrics.columns)}")
|
||||
```
|
||||
|
||||
### Main Effects Analysis
|
||||
```python
|
||||
# For each loop, compute average metric change when loop goes 0→1
|
||||
main_effects = {}
|
||||
for loop in factors.columns:
|
||||
loop_name = loop.replace('enable_', '').replace('_', ' ').title()
|
||||
metric_when_on = df[df[loop] == 1]['vm_workload_duration_ns_q48'].mean()
|
||||
metric_when_off = df[df[loop] == 0]['vm_workload_duration_ns_q48'].mean()
|
||||
|
||||
effect = metric_when_on - metric_when_off
|
||||
pct_change = 100 * (effect / metric_when_off)
|
||||
|
||||
main_effects[loop_name] = {
|
||||
'effect': effect,
|
||||
'pct_change': pct_change,
|
||||
'on_mean': metric_when_on,
|
||||
'off_mean': metric_when_off,
|
||||
}
|
||||
|
||||
# Print sorted by impact
|
||||
for loop, effect in sorted(main_effects.items(), key=lambda x: abs(x[1]['pct_change']), reverse=True):
|
||||
print(f"{loop:30} {effect['pct_change']:+7.2f}% (ON: {effect['on_mean']:.0f}, OFF: {effect['off_mean']:.0f})")
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
Loop 6 Decay Inference +12.3% (ON: 8,800,000, OFF: 7,840,000)
|
||||
Loop 5 Window Inference +8.5% (ON: 8,600,000, OFF: 7,920,000)
|
||||
Loop 4 Pipelining Metrics +6.2% (ON: 8,400,000, OFF: 7,900,000)
|
||||
Loop 3 Linear Decay +4.1% (ON: 8,200,000, OFF: 7,870,000)
|
||||
Loop 2 Rolling Window +3.0% (ON: 8,100,000, OFF: 7,850,000)
|
||||
Loop 1 Heat Tracking +2.1% (ON: 8,010,000, OFF: 7,850,000)
|
||||
```
|
||||
|
||||
### Interaction Analysis
|
||||
```python
|
||||
# Two-way interactions: Do loops amplify/suppress each other?
|
||||
interactions = {}
|
||||
|
||||
for i, loop_a in enumerate(factors.columns):
|
||||
for j, loop_b in enumerate(factors.columns):
|
||||
if i >= j:
|
||||
continue # Only upper triangle
|
||||
|
||||
# Four cases: (0,0), (0,1), (1,0), (1,1)
|
||||
case_00 = df[(df[loop_a] == 0) & (df[loop_b] == 0)]['vm_workload_duration_ns_q48'].mean()
|
||||
case_01 = df[(df[loop_a] == 0) & (df[loop_b] == 1)]['vm_workload_duration_ns_q48'].mean()
|
||||
case_10 = df[(df[loop_a] == 1) & (df[loop_b] == 0)]['vm_workload_duration_ns_q48'].mean()
|
||||
case_11 = df[(df[loop_a] == 1) & (df[loop_b] == 1)]['vm_workload_duration_ns_q48'].mean()
|
||||
|
||||
# Interaction effect: (1,1) - (1,0) - (0,1) + (0,0)
|
||||
interaction = (case_11 - case_10 - case_01 + case_00)
|
||||
|
||||
loop_a_name = loop_a.replace('enable_', '').replace('_', ' ')
|
||||
loop_b_name = loop_b.replace('enable_', '').replace('_', ' ')
|
||||
|
||||
if abs(interaction) > 100000: # Only report large interactions
|
||||
interactions[f"{loop_a_name} × {loop_b_name}"] = interaction
|
||||
|
||||
# Print sorted interactions
|
||||
for pair, interaction in sorted(interactions.items(), key=lambda x: abs(x[1]), reverse=True):
|
||||
print(f"{pair:40} {interaction:+12,.0f}")
|
||||
```
|
||||
|
||||
### Optimal Configuration Search
|
||||
```python
|
||||
# Find best k configurations by performance metric
|
||||
k = 10
|
||||
best_configs = df.nlargest(k, 'vm_workload_duration_ns_q48')[
|
||||
['configuration', 'vm_workload_duration_ns_q48', 'enable_loop_1_heat_tracking',
|
||||
'enable_loop_2_rolling_window', 'enable_loop_3_linear_decay',
|
||||
'enable_loop_4_pipelining_metrics', 'enable_loop_5_window_inference',
|
||||
'enable_loop_6_decay_inference']
|
||||
].drop_duplicates(subset=['configuration'])
|
||||
|
||||
print("Top 10 Configurations by Performance:")
|
||||
print("─" * 100)
|
||||
for idx, row in best_configs.iterrows():
|
||||
loops = (
|
||||
f"L1:{row['enable_loop_1_heat_tracking']} " +
|
||||
f"L2:{row['enable_loop_2_rolling_window']} " +
|
||||
f"L3:{row['enable_loop_3_linear_decay']} " +
|
||||
f"L4:{row['enable_loop_4_pipelining_metrics']} " +
|
||||
f"L5:{row['enable_loop_5_window_inference']} " +
|
||||
f"L6:{row['enable_loop_6_decay_inference']}"
|
||||
)
|
||||
print(f"{row['configuration']:20} | {row['vm_workload_duration_ns_q48']:12,.0f} ns | {loops}")
|
||||
```
|
||||
|
||||
### Visualization Example
|
||||
```python
|
||||
# Main effects plot
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
fig, axes = plt.subplots(2, 3, figsize=(15, 10))
|
||||
fig.suptitle('2^6 Factorial DoE Main Effects', fontsize=16)
|
||||
|
||||
loop_names = [f"Loop {i+1}" for i in range(6)]
|
||||
metric_key = 'vm_workload_duration_ns_q48'
|
||||
|
||||
for idx, loop in enumerate(factors.columns):
|
||||
ax = axes[idx // 3, idx % 3]
|
||||
|
||||
off_mean = df[df[loop] == 0][metric_key].mean()
|
||||
on_mean = df[df[loop] == 1][metric_key].mean()
|
||||
|
||||
bars = ax.bar(['OFF', 'ON'], [off_mean, on_mean], color=['lightcoral', 'lightgreen'])
|
||||
ax.set_ylabel('Workload Duration (ns)')
|
||||
ax.set_title(f'{loop_names[idx]}')
|
||||
ax.set_ylim([7000000, 10000000])
|
||||
|
||||
# Add value labels
|
||||
for bar in bars:
|
||||
height = bar.get_height()
|
||||
ax.text(bar.get_x() + bar.get_width()/2., height,
|
||||
f'{height/1e6:.1f}M', ha='center', va='bottom')
|
||||
|
||||
plt.tight_layout()
|
||||
plt.savefig('main_effects.png', dpi=150)
|
||||
print("Saved: main_effects.png")
|
||||
```
|
||||
|
||||
## Phase 4: Reporting
|
||||
|
||||
### Key Outputs to Generate
|
||||
|
||||
1. **Main Effects Summary** (which loops matter most?)
|
||||
- Table of individual loop impacts
|
||||
- Sorted by effect size
|
||||
|
||||
2. **Interaction Matrix** (do loops amplify/suppress?)
|
||||
- 6×6 table of two-way interactions
|
||||
- Highlights synergies/conflicts
|
||||
|
||||
3. **Optimal Configuration** (best 10-20 subsets)
|
||||
- Recommend top 3-5 production configs
|
||||
- Show Pareto frontier (performance vs complexity)
|
||||
|
||||
4. **Statistical Summary**
|
||||
- Variance by configuration
|
||||
- Confidence intervals on estimates
|
||||
- Regression model R²
|
||||
|
||||
### Example Report Structure
|
||||
|
||||
```markdown
|
||||
# Factorial DoE Analysis Report - 2025-11-19
|
||||
|
||||
## Executive Summary
|
||||
- Tested all 64 feedback loop combinations
|
||||
- 1,920 experimental runs collected
|
||||
- Key finding: Loops 5+6 show strong synergy (+18% combined)
|
||||
|
||||
## Main Effects
|
||||
| Loop | Impact | Confidence | Recommendation |
|
||||
|------|--------|-----------|---|
|
||||
| Loop 6 (Decay Inference) | +12.3% | p<0.001 | **Enabled** |
|
||||
| Loop 5 (Window Inference) | +8.5% | p<0.001 | **Enabled** |
|
||||
| Loop 4 (Pipelining) | +6.2% | p<0.01 | Enabled |
|
||||
| Loop 3 (Decay) | +4.1% | p<0.05 | Enabled |
|
||||
| Loop 2 (Window) | +3.0% | p<0.05 | Enabled |
|
||||
| Loop 1 (Heat) | +2.1% | p>0.05 | Optional |
|
||||
|
||||
## Recommended Configurations
|
||||
1. **111111** (All loops): +30% vs baseline
|
||||
2. **111110** (Except heat): +28% vs baseline, simpler
|
||||
3. **011111** (Except heat, decay): +24% vs baseline
|
||||
|
||||
## Trade-off Analysis
|
||||
- Full stack (111111): Best performance, highest code complexity
|
||||
- Recommended (111110): 28% improvement, 95% of benefit
|
||||
- Minimal set (001110): 18% improvement, 50% code overhead
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] Run factorial DoE script to completion
|
||||
- [ ] Verify 1,920 rows in experiment_results.csv
|
||||
- [ ] Copy results to analysis repo
|
||||
- [ ] Load data in Python/R
|
||||
- [ ] Compute main effects (6 loops)
|
||||
- [ ] Compute interactions (2-way or higher)
|
||||
- [ ] Identify top 5-10 configurations
|
||||
- [ ] Generate visualizations
|
||||
- [ ] Write summary report
|
||||
- [ ] Decide on production configuration
|
||||
|
||||
---
|
||||
|
||||
**Timeline:** Collection (2-4 hrs) + Analysis (2-4 hrs) = **6-8 hours total** for comprehensive factorial DoE.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user