rajames/LithosAnanke
1
1
Fork 0
Code Issues 41 Pull Requests Actions Packages Projects Releases Activity

Reorganize source tree: kernel/, v3/, v4/ split and board infrastructure

Browse Source 
Source tree reorganization:
- Move StarForth v3 engine to v3/ (src/, include/, Makefile)
- Move kernel to kernel/ (src/, include/, linker/, Makefile)
- Create v4/ skeleton for F18-ISA golden model (DECOMPOSITION.md, JUSTIFICATION.md)
- Move FABRIC-0..4.md to docs/fabric/
- Move ONTOLOGY.md and ROADMAP.md to docs/

Board infrastructure:
- Add boards/ser5/, boards/raspi/, boards/milkv/, boards/zynq7020/
- Each board has board.mk (ISA, CPU flags, boot recipe) and README.md
- Root Makefile becomes thin dispatcher: boot_image, all, clean, docs take TARGET
- make boot_image TARGET=SER5|RASPI|MILKV builds one GPT/MBR image per board
- ZYNQ7020 target exists but stops with clear error (ARMv7 port not built yet)
- scripts/mkdiskimage.sh builds disk images for all boards

Docs pipeline:
- docs/book/ with LaTeX master (main.tex) and Makefile
- pandoc converts Markdown to LaTeX at build time
- Two Lua filters: table-widths.lua (wide tables wrap), code-breaks.lua (inline code breaks)
- make docs builds single PDF (754 pages, 0 missing characters)
- make docs TARGET=<board> adds board appendix
- build/docs/<book|board>/meta.tex stamps git commit into PDF

Bug fixes:
- 42 include paths that only worked by accident now use correct relative paths
- clang-18 hardcode replaced with configurable CC variable (fixed aarch64 build)
- Pi 5: kernel_2712.img linked at 0x80000, .bss zeroed, memory reserved
- Doxyfile, .clang-tidy, README.md, Kconfig paths updated

Verified:
- Hosted v3 build passes 1012 tests, 0 failures
- SER5 image boots in QEMU (OVMF), POST passes, K exact (65536 = Q48_ONE)
- Milk-V image boots in QEMU (OpenSBI + U-Boot + bootefi), POST passes
- make clean TARGET=<board> removes only that board and its ISA objects
- make all builds all boards, hosted v3, and docs in one run

Co-authored-by: Junie <junie@jetbrains.com>
This commit is contained in:
rajames
2026-10-01 15:40:09 -04:00
co-authored by Junie
parent 3c709c115b
commit a8b70e88d3
630 changed files with 2646 additions and 2036 deletions
Download Patch File Download Diff File Expand all files Collapse all files
kernel/src/arch/aarch64/README.md
+17
View File
@@ -0,0 +1,17 @@
# src/starkernel/arch/aarch64/
AArch64 architecture support for LithosAnanke.
- `arch.c` — architecture init entry point.
- `apic.c` — GIC (Generic Interrupt Controller) stub, named `apic.c` to
match the shared per-arch file-naming convention across
amd64/aarch64/riscv64. Satisfies the common `apic_init()`/`apic_eoi()`
call sites without real GIC bring-up yet; deferred to a later milestone.
- `timer.c` — architected timer setup (heartbeat tick source).
- `interrupts.c` — exception vector table setup and interrupt dispatch.
- `boot.S` — early boot assembly.
- `isr.S` — interrupt service routine stubs.
No `kernel_entry.S` — aarch64 uses a different UEFI loader entry path
(`clang`/`lld-link` COFF target, since GCC cannot produce an aarch64 PE
image; see `Makefile.starkernel`).
kernel/src/arch/aarch64/apic.c
+556
View File
@@ -0,0 +1,556 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James. All rights reserved.
Licensed under the StarForth License, Version 1.0.
*/
/**
* apic.c (aarch64) - Minimal GICv2 driver + ARM Generic Timer (items 0.6/0.7/
* 4.3.5d).
*
* "Minimal" is deliberate and stays deliberate: item 0.6 enabled exactly one
* PPI (the timer); item 4.3.5d adds the ability to enable exactly one SPI
* (for virtio-keyboard-pci, item 4.3.5e), still by explicit per-INTID call,
* not a general GIC driver -- no SGI support, no GICv3/ITS, no SPI beyond
* the one 4.3.5e will actually enable.
*
* GICD/GICC base addresses -- DTB-discovered when a DTB is present, QEMU
* virt-machine constants otherwise (FABRIC-3.md §IV.3 item 7, 2026-09-04
* fix; originally recorded as a deliberate hardcode, FABRIC-0.md item 0.6
* GAP-B1 follow-up, because `fdt_valid(boot_info->dtb)` fails on this
* system's QEMU aarch64 firmware -- qemu-efi-aarch64 2025.11-3ubuntu7 does
* not forward a devicetree to the guest, confirmed live rather than
* assumed). That reasoning covered the QEMU/UEFI path correctly but never
* covered the native Pi 5 boot path (`rpi5_native_boot.c`), which receives
* a real DTB directly and calls this same, unmodified `apic_init()` --
* audited 2026-09-04, found still hardcoding the QEMU addresses
* unconditionally, which blocks reaching `ok>` on real BCM2712 hardware
* (its real GIC-400 sits at `0x10_7fff9000`, nowhere near QEMU's
* `0x08000000`). `gic_bases_from_dtb()` below closes that gap: tries
* DTB discovery first, falls back to the QEMU constants only when no DTB
* is present or the GIC node isn't found there -- so the QEMU/UEFI path's
* existing, working behaviour is unchanged.
*
* QEMU's own values were not recalled from memory either -- they were read
* out of QEMU 10.2.1's own internal devicetree
* (`qemu-system-aarch64 -machine virt,dumpdtb=...`, decoded with this
* tree's own fdt.c reader) and are therefore correct for this exact QEMU
* version, though not guaranteed stable across others. Register *offsets*
* within each block (GICD_CTLR, GICC_IAR, etc.) are fixed by the GICv2
* architecture, not the board, and were cross-checked against Linux's own
* driver header (linux/irqchip/arm-gic.h, from the linux-headers package
* installed on the build host) rather than recalled either.
*/
#include "apic.h"
#include "uefi.h"
#include "console.h"
#include "timer.h"
#include "starkernel/fdt.h"
#include <stdint.h>
/* Defined in arch.c (item 0.4). Same extern-in-place convention as
* interrupts.c: "exception level" has no cross-ISA meaning. */
extern int aarch64_current_el(void);
/* Defined in arch.c (FABRIC-3.md §IV.3 item 7 fix, 2026-09-04) --
* discovers the real PSCI conduit for arch_cold_reset() from the DTB.
* Collocated with its consumer there, not defined here, even though this
* is the one point in boot with boot_info->dtb already in hand for GIC
* discovery -- see arch.c's own doc comment on this function for why. */
extern void aarch64_psci_conduit_init(const void *dtb);
/* ---- QEMU virt machine constants (fallback when no DTB GIC node is
* found -- see file header) ----------------------------------------------- */
#define GICD_BASE_PA_QEMU_DEFAULT 0x08000000UL
#define GICC_BASE_PA_QEMU_DEFAULT 0x08010000UL
/* Runtime GIC base addresses -- set once by apic_init(), defaulting to the
* QEMU constants above until/unless gic_bases_from_dtb() overrides them.
* uintptr_t, not a #define, precisely because these are no longer
* compile-time constants on this path. */
static uintptr_t s_gicd_base = GICD_BASE_PA_QEMU_DEFAULT;
static uintptr_t s_gicc_base = GICC_BASE_PA_QEMU_DEFAULT;
/* Non-secure EL1 physical timer PPI = 30 (INTID 16+14); EL2 hypervisor
* timer PPI = 26 (INTID 16+10). Both read directly out of QEMU's own DT
* dump for the `timer` node's four-entry `interrupts` property (secure,
* non-secure, virtual, hypervisor, in that architected order) -- not the
* architecture-generic PPI numbers assumed from memory. */
#define TIMER_PPI_EL1 30u
#define TIMER_PPI_EL2 26u
/* GICv2 Distributor register byte offsets (ARM IHI 0048B; confirmed against
* Linux's arm-gic.h, see file header). */
#define GICD_CTLR 0x000
#define GICD_ISENABLER0 0x100
#define GICD_IPRIORITYR 0x400
#define GICD_ITARGETSR 0x800 /* item 4.3.5d -- SPI target-CPU byte array */
#define GICD_ICFGR 0xC00
#define GICD_ISPENDR 0x200 /* item 4.3.5d -- software set-pending, SPIs only */
/* GICv2 CPU Interface register byte offsets. */
#define GICC_CTLR 0x00
#define GICC_PMR 0x04
#define GICC_IAR 0x0C
#define GICC_EOIR 0x10
#define GICD_CTLR_ENABLE_GRP0 (1u << 0)
#define GICC_CTLR_ENABLE_GRP0 (1u << 0)
/* Priority: GICv2 compares as "interrupt allowed if priority < PMR", lower
* value = more urgent. One interrupt is configured, so the specific value
* only has to be numerically below the PMR mask set in apic_init(). */
#define TIMER_PRIORITY 0x80u
#define PMR_ALLOW_ALL 0xFFu
/* MMIO accessed as identity/physical addresses: arch_mmu_init() for aarch64
* is still the M3 stub (page-table bring-up deferred), so whatever mapping
* EDK2 left active at ExitBootServices governs -- the same assumption
* amd64/timer.c already documents and relies on for its own MMIO (HPET). */
static inline void mmio_write32(uintptr_t base, uint32_t offset, uint32_t val)
{
*(volatile uint32_t *)(base + offset) = val;
}
static inline uint32_t mmio_read32(uintptr_t base, uint32_t offset)
{
return *(volatile uint32_t *)(base + offset);
}
/* Set once by apic_init(), read by aarch64_irq_handler() to know which PPI
* to expect -- computed once (EL cannot change post-boot, same rationale
* as item 0.5's el2_mode_flag) rather than re-derived per interrupt. */
static uint32_t s_timer_ppi = TIMER_PPI_EL1;
static uint64_t s_timer_period_tsc = 0;
static uint64_t s_counter_hz_apic = 0; /* CNTFRQ_EL0 (item 0.8/§26) */
/**
* @brief Read one big-endian 32-bit cell from a devicetree property blob.
*
* `fdt.c` keeps its own `be32()` helper file-local (freestanding, no shared
* byte-swap utility to reuse) -- duplicated here rather than exposing it,
* same "a few lines is simpler than a new shared dependency" precedent
* `rpi5_dtb.c`'s own `reg_first_cell_be32()` already set for this exact
* situation.
*/
static uint32_t be32_cell(const unsigned char *b)
{
return ((uint32_t) b[0] << 24) | ((uint32_t) b[1] << 16) |
((uint32_t) b[2] << 8) | (uint32_t) b[3];
}
/**
* @brief Read one two-cell (64-bit) big-endian address from a `reg` entry.
*
* BCM2712's `gicv2` node (see file header) uses two address cells and two
* size cells per `reg` entry -- confirmed directly against `bcm2712.dtsi`'s
* `axi`/`gicv2` nodes (`#address-cells = <2>; #size-cells = <2>;`), unlike
* `rpi5_dtb.c`'s UART/mailbox lookups which read a single 32-bit cell
* because their `soc` parent node declares `#address-cells = <1>`. Reads
* only the two address cells; the caller advances past the two size cells
* itself via @c GIC_REG_ENTRY_BYTES to reach the next entry.
*/
static uint64_t be64_addr_cell_pair(const unsigned char *b)
{
return ((uint64_t) be32_cell(b) << 32) | (uint64_t) be32_cell(b + 4);
}
/* One gicv2 `reg` entry is 4 cells (2 address + 2 size) * 4 bytes = 16
* bytes. Entry 0 = GICD (distributor), entry 1 = GICC (CPU interface 0) --
* confirmed against bcm2712.dtsi's own `reg` list order (GICD, GICC, GICH,
* GICV, the standard arm,gic-400 binding order), not assumed. */
#define GIC_REG_ENTRY_BYTES 16u
/**
* @brief Try to discover the real GICD/GICC base addresses from the DTB.
*
* Looks up the `"arm,gic-400"` node and reads its `reg` property's first
* two entries. Returns 0 (leaving @p gicd_out / @p gicc_out untouched) if
* @p dtb is not a valid FDT, no matching node exists, or the `reg`
* property is shorter than two entries -- callers must keep their own
* QEMU-constant default in that case, exactly as before this function
* existed (FABRIC-3.md §IV.3 item 7 fix, 2026-09-04; see file header for
* why this is needed on the native Pi 5 path specifically).
*
* @param dtb Candidate devicetree blob (@c BootInfo->dtb); NULL-safe.
* @param gicd_out Receives the distributor base address on success.
* @param gicc_out Receives the CPU interface 0 base address on success.
* @return 1 on success, 0 if discovery failed for any reason.
*/
static int gic_bases_from_dtb(const void *dtb, uintptr_t *gicd_out, uintptr_t *gicc_out)
{
const void *node;
const unsigned char *reg;
uint32_t len;
if (!fdt_valid(dtb)) return 0;
node = fdt_find_node_by_compatible(dtb, "arm,gic-400");
if (!node) return 0;
reg = (const unsigned char *) fdt_find_prop_in_node(dtb, node, "reg", &len);
if (!reg || len < 2u * GIC_REG_ENTRY_BYTES) return 0;
*gicd_out = (uintptr_t) be64_addr_cell_pair(reg);
*gicc_out = (uintptr_t) be64_addr_cell_pair(reg + GIC_REG_ENTRY_BYTES);
return 1;
}
/**
* @brief Initialise the GICv2 distributor and CPU interface (M4 milestone).
*
* Enables exactly one interrupt source: the physical timer PPI for the
* exception level detected in item 0.4 (30 at EL1, 26 at EL2 — see
* @c TIMER_PPI_EL1 / @c TIMER_PPI_EL2 above). Sequence:
*
* 0. Try @c gic_bases_from_dtb() against @p boot_info->dtb; on success,
* @c s_gicd_base / @c s_gicc_base are overridden from the real
* hardware addresses. On failure (no DTB, e.g. this system's QEMU/UEFI
* firmware — see file header) they keep the QEMU constant defaults,
* exactly the behaviour this function had before this step existed.
* 1. Set this interrupt's priority (@c GICD_IPRIORITYR) below the CPU
* interface's priority mask, so it is never itself masked out.
* 2. Enable it in @c GICD_ISENABLER0 (PPIs 16–31 live in the first word).
* 3. Enable the distributor, Group 0 only (@c GICD_CTLR) — this deployment
* has no GICv2 security-extension split in use.
* 4. Set the CPU interface priority mask to allow everything through
* (@c GICC_PMR = 0xFF) and enable the CPU interface (@c GICC_CTLR).
*
* Deliberately not configured: @c GICD_ICFGR (trigger sensitivity). The
* ARM generic timer's PPI is architecturally level-sensitive; this is not
* a general GIC driver and item 0.6's own scope does not call for
* reconfiguring it.
*
* @param boot_info Kernel @c BootInfo; @c dtb is consulted for real-hardware
* GIC base discovery as of the 2026-09-04 fix (previously
* unused — see file header), and, as of the same fix, for
* @c aarch64_psci_conduit_init()'s PSCI conduit discovery
* (arch.c) — bundled into this call for the same reason
* GIC discovery is: this is the one point in boot with
* @c boot_info->dtb already in hand.
* @return 0 always.
*/
int apic_init(BootInfo *boot_info)
{
aarch64_psci_conduit_init(boot_info ? boot_info->dtb : (void *) 0);
if (boot_info && gic_bases_from_dtb(boot_info->dtb, &s_gicd_base, &s_gicc_base)) {
console_puts("GICv2: base addresses discovered from DTB (GICD=0x");
for (int s = 60; s >= 0; s -= 4)
console_putc("0123456789abcdef"[(s_gicd_base >> s) & 0xF]);
console_puts(", GICC=0x");
for (int s = 60; s >= 0; s -= 4)
console_putc("0123456789abcdef"[(s_gicc_base >> s) & 0xF]);
console_println(")");
} else {
console_println("GICv2: no DTB GIC node -- using QEMU virt-machine defaults");
}
s_timer_ppi = (aarch64_current_el() == 2) ? TIMER_PPI_EL2 : TIMER_PPI_EL1;
/* IPRIORITYR is byte-indexed, one byte per INTID. */
{
uint32_t reg_off = GICD_IPRIORITYR + (s_timer_ppi & ~3u);
uint32_t shift = (s_timer_ppi & 3u) * 8u;
uint32_t val = mmio_read32(s_gicd_base, reg_off);
val = (val & ~(0xFFu << shift)) | (TIMER_PRIORITY << shift);
mmio_write32(s_gicd_base, reg_off, val);
}
mmio_write32(s_gicd_base, GICD_ISENABLER0, 1u << s_timer_ppi);
mmio_write32(s_gicd_base, GICD_CTLR, GICD_CTLR_ENABLE_GRP0);
mmio_write32(s_gicc_base, GICC_PMR, PMR_ALLOW_ALL);
mmio_write32(s_gicc_base, GICC_CTLR, GICC_CTLR_ENABLE_GRP0);
console_puts("GICv2: distributor+CPU interface enabled, PPI ");
console_putc((char)('0' + s_timer_ppi / 10));
console_putc((char)('0' + s_timer_ppi % 10));
console_println("");
return 0;
}
/**
* @brief Enable and route one SPI (item 4.3.5d).
*
* Extends item 0.6's PPI-only init just far enough to bring up one Shared
* Peripheral Interrupt -- still not a general GIC driver, still one
* explicit INTID per call, same bounded-scope posture 0.6 used.
*
* Programs, for @p intid (expected range: SPIs, INTID >= 32):
* 1. @c GICD_IPRIORITYR (byte-indexed, same pattern @c apic_init() already
* uses for the timer PPI) -- @c TIMER_PRIORITY, below @c PMR_ALLOW_ALL.
* 2. @c GICD_ISENABLER at @c 0x100 + 4*(intid/32) -- PPIs/SGIs (0-31) live
* in word 0 (@c GICD_ISENABLER0); SPIs beyond use word @c intid/32, same
* architecture layout, not board-specific.
* 3. @c GICD_ITARGETSR (byte-indexed like IPRIORITYR) -- routes to CPU 0
* only; this target has no @c -smp, so no other bit is ever valid.
* 4. @c GICD_ICFGR is read back, not written, unless the readback disagrees
* with the expected level-triggered configuration -- FABRIC-0.md §27.5.1's
* decoded QEMU `interrupt-map` says PCI legacy INTx on this board already
* is level-triggered by default, so this keeps item 0.6's "don't touch
* ICFGR unless forced to" posture rather than writing it unconditionally.
*
* Does **not** touch @c GICD_CTLR or @c GICC_CTLR/@c PMR -- @c apic_init()
* already enabled the distributor and CPU interface for the timer PPI, and
* those are global, not per-INTID; nothing here needs to repeat that.
*
* @param intid Target SPI's INTID (32 + GIC SPI number).
*/
void apic_spi_enable(uint32_t intid)
{
{
uint32_t reg_off = GICD_IPRIORITYR + (intid & ~3u);
uint32_t shift = (intid & 3u) * 8u;
uint32_t val = mmio_read32(s_gicd_base, reg_off);
val = (val & ~(0xFFu << shift)) | (TIMER_PRIORITY << shift);
mmio_write32(s_gicd_base, reg_off, val);
}
{
uint32_t reg_off = GICD_ISENABLER0 + 4u * (intid / 32u);
uint32_t bit = intid % 32u;
mmio_write32(s_gicd_base, reg_off, 1u << bit);
}
{
uint32_t reg_off = GICD_ITARGETSR + (intid & ~3u);
uint32_t shift = (intid & 3u) * 8u;
uint32_t val = mmio_read32(s_gicd_base, reg_off);
val = (val & ~(0xFFu << shift)) | (0x01u << shift); /* CPU 0 only */
mmio_write32(s_gicd_base, reg_off, val);
}
{
uint32_t reg_off = GICD_ICFGR + 4u * (intid / 16u);
uint32_t bitpos = (intid % 16u) * 2u;
uint32_t val = mmio_read32(s_gicd_base, reg_off);
uint32_t level_expected = 0u; /* level-triggered = the "trigger mode"
* bit (bit 1 of the 2-bit field) clear */
if (((val >> bitpos) & 0x2u) != level_expected) {
val = (val & ~(0x3u << bitpos)) | (level_expected << bitpos);
mmio_write32(s_gicd_base, reg_off, val);
}
}
}
/**
* @brief Read the GIC CPU interface's Interrupt Acknowledge Register.
*
* Reading @c GICC_IAR returns the INTID of the highest-priority pending
* interrupt and simultaneously moves it to the active state — this is the
* GICv2 acknowledgement step. Called from @c aarch64_irq_handler() in
* @c interrupts.c to identify which interrupt fired; the matching
* @c apic_eoi_intid() call below completes it.
*
* @return INTID (10 bits used by GICv2).
*/
uint32_t apic_read_iar(void)
{
return mmio_read32(s_gicc_base, GICC_IAR);
}
/**
* @brief Return the timer PPI this build is watching for.
*
* Set once in @c apic_init() from @c aarch64_current_el() (30 at EL1, 26 at
* EL2). @c aarch64_irq_handler() compares @c apic_read_iar()'s result
* against this rather than a hardcoded constant.
*/
uint32_t apic_timer_ppi(void)
{
return s_timer_ppi;
}
/**
* @brief Signal End of Interrupt to the GIC CPU interface.
*
* Writes @p intid back to @c GICC_EOIR, matching the value @c GICC_IAR
* returned in @c aarch64_irq_handler(). Required after every acknowledged
* interrupt or the GIC considers it permanently active and will not
* re-signal it.
*
* @param intid INTID as read from @c GICC_IAR (10 bits used by GICv2).
*/
void apic_eoi_intid(uint32_t intid)
{
mmio_write32(s_gicc_base, GICC_EOIR, intid);
}
/**
* @brief Signal End of Interrupt (common cross-ISA signature — see below).
*
* The shared @c apic.h interface has no way to pass which interrupt is
* being completed, but GICv2's @c GICC_EOIR needs the exact INTID @c GICC_IAR
* returned, not just "an" EOI. @c aarch64_irq_handler() calls
* @c apic_eoi_intid() directly with that value instead; this form exists
* only to satisfy the common call site and is not used on the interrupt
* path GIC brings up.
*/
void apic_eoi(void)
{
}
/**
* @brief Compute the expected system-counter ticks per heartbeat period.
*
* Computes @c s_timer_period_tsc (the expected number of @c CNTPCT_EL0
* ticks between heartbeats), used both by @c heartbeat_init() to seed the
* rolling-window @c expected_delta and by @c apic_timer_start() /
* @c apic_timer_rearm() (item 0.7) as the @c TVAL step written on every
* arm and re-arm.
*
* If either @p tsc_hz or @p tick_hz is zero, a fallback of 10,000,000
* ticks is stored (approximately 160 ms at 62.5 MHz, a safe non-zero
* sentinel that prevents division-by-zero in the heartbeat path).
*
* @param tsc_hz System counter frequency from @c timer_init() (Hz).
* @param tick_hz Desired heartbeat rate (Hz); 0 → uses fallback.
* @return 0 always.
*/
int apic_timer_init(uint64_t tsc_hz, uint32_t tick_hz)
{
if (tick_hz > 0 && tsc_hz > 0) {
s_timer_period_tsc = tsc_hz / tick_hz;
} else {
s_timer_period_tsc = 10000000;
}
s_counter_hz_apic = tsc_hz;
return 0;
}
/**
* @brief Arm the ARM Generic Timer and enable its interrupt (item 0.7).
*
* Writes @c CNTP_TVAL_EL0 (or @c CNTHP_TVAL_EL2 at EL2 — @c
* aarch64_current_el(), same EL-aware discipline as items 0.4–0.6) to
* @c s_timer_period_tsc, then sets @c CTL.ENABLE=1, @c CTL.IMASK=0.
*
* @c TVAL is a signed 32-bit *relative* countdown — writing it sets
* "fire @c s_timer_period_tsc ticks from this instant," unlike riscv64's
* SBI interface (item 0.3), which takes an absolute deadline the caller
* must compute. No "now" reading is needed here at all.
*
* The @c ISB after the @c CTL write is required, not decorative: per the
* architecture, and confirmed against Linux's own
* @c arch_timer_reg_write_cp15() (arch/arm64/include/asm/arch_timer.h),
* only the *control* register write needs synchronising before the
* enable/mask state is guaranteed visible to the interrupt pipeline —
* @c TVAL/@c CVAL writes do not carry the same requirement, which is why
* @c apic_timer_rearm() below omits it.
*
* @c s_timer_period_tsc must already be set by @c apic_timer_init(). If
* it is still 0, @c TVAL=0 fires immediately and continuously until
* @c apic_timer_rearm() runs — a correctly-functioning if oddly-paced
* timer, not a hang.
*/
void apic_timer_start(void)
{
/* CNTx_TVAL_EL is architecturally 32-bit (bits[63:32] RES0 on write),
* but MSR to a system register is always a 64-bit (Xn) instruction
* form -- there is no 32-bit encoding to match a narrower operand.
* Truncate to 32 bits ourselves, then zero-extend back to 64 for the
* operand: this supplies explicit, provably-correct zeros in the RES0
* field rather than depending on how the hardware treats whatever
* upper bits a narrower operand happened to leave there -- a
* distinction we could not confirm against a local authoritative
* source (unlike the GICv2 offsets and CTL bit layout above,
* cross-checked against Linux headers on this system; TVAL is not,
* since Linux's own driver only ever uses the 64-bit CVAL form and
* never exercises this path). */
uint64_t tval = (uint64_t)(uint32_t)s_timer_period_tsc;
if (aarch64_current_el() == 2) {
__asm__ volatile ("msr cnthp_tval_el2, %0" :: "r"(tval));
__asm__ volatile ("msr cnthp_ctl_el2, %0" :: "r"((uint64_t)1) : "memory");
} else {
__asm__ volatile ("msr cntp_tval_el0, %0" :: "r"(tval));
__asm__ volatile ("msr cntp_ctl_el0, %0" :: "r"((uint64_t)1) : "memory");
}
__asm__ volatile ("isb" ::: "memory");
}
/**
* @brief Re-arm the ARM Generic Timer for the next period (item 0.7).
*
* Writes a fresh @c TVAL — the entirety of a periodic re-arm, since @c TVAL
* is relative to "now" rather than an absolute deadline. Unlike riscv64's
* SBI re-arm (item 0.3), there is no drift-correction bookkeeping: each
* write means "@c s_timer_period_tsc ticks from this instant," so time
* spent servicing the interrupt before this call simply lengthens that one
* period rather than accumulating error across ticks.
*
* **This is the one call that must never be skipped.** The ARM Generic
* Timer does not auto-reload: once @c TVAL crosses zero, @c CTL.ISTATUS
* latches and stays asserted — a level condition — until software writes
* a new @c TVAL or clears @c CTL.ENABLE. A return path that skips this
* leaves the interrupt permanently pending, which the GIC would deliver
* again the instant it is EOI'd: a real interrupt storm, not a
* hypothetical one (this is exactly the failure this driver was checked
* against during item 0.6's own verification, and confirmed absent there
* only because nothing armed the timer yet).
*
* Called from @c aarch64_irq_handler() (interrupts.c) before
* @c heartbeat_tick() — same ordering discipline as
* @c riscv64_timer_rearm() (item 0.3): re-arm first, so a fault in the
* rest of the handler cannot also cost the next tick.
*
* The step written is @c heartbeat_next_period_ns() (item 0.8, §26)
* converted to @c CNTPCT_EL0 ticks via @c s_counter_hz_apic, not the fixed
* @c s_timer_period_tsc @c apic_timer_start() used for the very first arm
* -- that value remains @c apic_timer_period_tsc()'s return for
* @c heartbeat_init()'s initial @c expected_delta, unchanged.
*/
void apic_timer_rearm(void)
{
uint64_t period_ticks = (s_counter_hz_apic > 0)
? (heartbeat_next_period_ns() * s_counter_hz_apic) / 1000000000ULL
: s_timer_period_tsc;
if (period_ticks == 0) {
period_ticks = 1;
}
/* uint64_t, zero-extended -- see apic_timer_start() for why. */
uint64_t tval = (uint64_t)(uint32_t)period_ticks;
if (aarch64_current_el() == 2) {
__asm__ volatile ("msr cnthp_tval_el2, %0" :: "r"(tval));
} else {
__asm__ volatile ("msr cntp_tval_el0, %0" :: "r"(tval));
}
}
/**
* @brief Stop timer delivery by clearing CTL.ENABLE (item 0.7).
*
* Per the architecture, @c ENABLE=0 unconditionally stops the count and
* de-asserts the interrupt condition regardless of @c IMASK — a single
* write is sufficient, no separate mask step needed.
*/
void apic_timer_stop(void)
{
if (aarch64_current_el() == 2) {
__asm__ volatile ("msr cnthp_ctl_el2, %0" :: "r"((uint64_t)0) : "memory");
} else {
__asm__ volatile ("msr cntp_ctl_el0, %0" :: "r"((uint64_t)0) : "memory");
}
__asm__ volatile ("isb" ::: "memory");
}
/**
* @brief Return the expected system-counter ticks per heartbeat period.
*
* Returns @c s_timer_period_tsc, set by @c apic_timer_init() as
* @c tsc_hz / @c tick_hz. This value is consumed by @c heartbeat_init()
* to seed the rolling-window @c expected_delta so that inter-tick
* deviation measurements are relative to the correct nominal interval.
*
* @return Expected @c CNTPCT_EL0 ticks per heartbeat period; 10,000,000
* if frequencies were not available at init time.
*/
uint64_t apic_timer_period_tsc(void)
{
return s_timer_period_tsc;
}
kernel/src/arch/aarch64/arch.c
+270
View File
@@ -0,0 +1,270 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James. All rights reserved.
Licensed under the StarForth License, Version 1.0.
*/
/**
* arch.c (aarch64) - Architecture abstractions for StarKernel on AArch64
*/
#include "arch.h"
#include "console.h"
#include "starkernel/fdt.h"
#include <stdint.h>
/* Installed by isr.S — installs VBAR_EL2 at EL2, VBAR_EL1 at EL1 (EL-aware
* since before this comment was corrected, 2026-09-04 — see
* interrupts.c's arch_interrupts_init() doc comment for the fuller
* correction; this one-word staleness was caught in passing while fixing
* that one). */
extern void aarch64_install_vectors(void);
/* -1 = not yet read. Cached rather than re-read on every call: CurrentEL
* cannot change during the kernel's lifetime post-ExitBootServices, and a
* cached value gives every consumer (vector install, GIC timer register
* selection) the same answer without repeating the MRS. */
static int s_current_el = -1;
/**
* @brief Return the AArch64 exception level the kernel is running at.
*
* Reads @c CurrentEL (bits [3:2] hold the EL number, [1:0] are RES0) on
* first call and caches the result; every later call returns the cached
* value. Safe to call at any point in boot, including before
* @c console_init() — the read itself has no dependency, only the
* printing of it does (see @c arch_interrupts_init() in @c interrupts.c,
* the earliest point at which both this and a working console are
* available).
*
* EDK2 on QEMU's aarch64 @c virt machine has been observed to leave the
* kernel at either EL1 or EL2 depending on firmware build; nothing in this
* tree may assume one over the other (FABRIC-0.md §25.7.1 GAP-B3). Every
* EL-dependent choice — @c VBAR_EL1 vs @c VBAR_EL2, the @c ELR_ELx /
* @c SPSR_ELx saved-state pair, and @c CNTP_*_EL0 vs @c CNTHP_*_EL2 — must
* read this accessor rather than hardcode a level.
*
* @return 1 or 2. Never 0 or 3: a UEFI application that survived far enough
* to reach this call cannot be running at EL0 (unprivileged) or EL3
* (secure monitor — QEMU's virt firmware runs the kernel at EL1/EL2).
*/
int aarch64_current_el(void)
{
if (s_current_el < 0) {
uint64_t current_el;
__asm__ volatile ("mrs %0, CurrentEL" : "=r"(current_el));
s_current_el = (int)((current_el >> 2) & 0x3);
}
return s_current_el;
}
/**
* @brief Perform the earliest AArch64 architecture initialisation (no-op).
*
* On x86-64 this function loads the GDT; AArch64 has no equivalent because
* privilege separation is provided by the exception-level architecture
* (EL0/EL1/EL2/EL3) rather than segment descriptors. The exception vector
* table is installed later in @c arch_interrupts_init() by calling
* @c aarch64_install_vectors() which writes @c VBAR_EL1.
*
* This stub exists to satisfy the common @c arch_early_init() call site in
* @c kernel_main() so that the same milestone sequence compiles across all
* three supported ISAs (amd64 / aarch64 / riscv64).
*/
void arch_early_init(void)
{
/* No GDT on AArch64; privilege separation via exception levels.
* Interrupt vectors installed later in arch_interrupts_init(). */
}
/**
* @brief Enable IRQ delivery on the current core (AArch64 DAIFCLR).
*
* Issues @c MSR DAIFCLR, #2 which clears the @c I bit (IRQ mask) in the
* @c DAIF (Debug / Abort / IRQ / FIQ) flags of the current exception level.
* After this instruction the core will take IRQ exceptions from the GIC.
* The @c memory clobber prevents the compiler from reordering stores across
* the barrier.
*
* Must be called after @c arch_interrupts_init() has installed the vector
* table in @c VBAR_EL1 and the GIC timer driver has been configured.
*/
void arch_enable_interrupts(void)
{
__asm__ volatile ("msr daifclr, #2" ::: "memory");
}
/**
* @brief Disable IRQ delivery on the current core (AArch64 DAIFSET).
*
* Issues @c MSR DAIFSET, #2 which sets the @c I bit (IRQ mask) in the
* @c DAIF flags, masking IRQ exceptions. FIQ, SError, and Debug exceptions
* are unaffected. Used around critical sections in the single-threaded
* kernel where interrupt delivery would corrupt shared state.
*
* The @c memory clobber acts as a compiler-level barrier; for a full
* hardware memory barrier use @c DSB before calling this function.
*/
void arch_disable_interrupts(void)
{
__asm__ volatile ("msr daifset, #2" ::: "memory");
}
/**
* @brief Halt the core until the next event (AArch64 WFI).
*
* Issues the @c WFI (Wait For Interrupt) instruction. The core enters a
* low-power standby state and resumes execution when an IRQ, FIQ, SError,
* debug event, or reset is detected. In the kernel panic path this is
* called inside an infinite loop with interrupts disabled (@c DAIFSET)
* to stop the processor permanently. In the idle path it reduces power
* consumption between heartbeat ticks.
*
* The @c memory clobber prevents the compiler from sinking stores across
* the @c WFI.
*/
void arch_halt(void)
{
__asm__ volatile ("wfi" ::: "memory");
}
/* PSCI conduit selection (FABRIC-3.md §IV.3 item 7 fix, 2026-09-04).
* Defaults to 0 (HVC) so a boot with no DTB -- this system's QEMU/UEFI
* firmware, same `fdt_valid()` failure mode already documented in
* apic.c's file header for GIC discovery -- keeps arch_cold_reset()'s
* existing, working HVC behaviour unchanged. Set to 1 (SMC) by
* aarch64_psci_conduit_init() below when the real DTB's own `/psci` node
* says so. */
static int s_psci_use_smc = 0;
/**
* @brief Discover the real PSCI conduit ("smc" vs "hvc") from the DTB.
*
* Looks up the devicetree's PSCI firmware node and reads its `method`
* property. BCM2712's own `/psci` node declares
* `compatible = "arm,psci-1.0", "arm,psci-0.2", "arm,psci"` and
* `method = "smc"` -- confirmed directly against `bcm2712.dtsi`, not
* assumed. `fdt_find_node_by_compatible()` matches any entry in a node's
* compatible list, so searching for just the first (most specific) string
* is sufficient; the other two exist for older PSCI-spec-version
* consumers, not because this node might be tagged differently.
*
* Leaves @c s_psci_use_smc at its default (0, HVC) if @p dtb is not a
* valid FDT, no PSCI node is found, or `method` is anything other than
* exactly `"smc"` -- including the explicit `"hvc"` case, and the
* "property absent" case, both of which should keep the existing HVC
* behaviour rather than guess.
*
* Called once from @c apic_init() (`arch/aarch64/apic.c`), the one point
* in boot that already has @c boot_info->dtb in hand at the right time
* for this kind of discovery (same rationale as that file's own GIC-base
* lookup) -- collocated here in @c arch.c, not there, because
* @c arch_cold_reset() is this function's actual consumer and has no
* @c boot_info of its own to probe with by the time it runs (called from
* deep in VM execution, via `BYE`). Same extern-in-place convention this
* file and apic.c already use for @c aarch64_current_el().
*
* @param dtb Candidate devicetree blob (@c BootInfo->dtb); NULL-safe.
*/
void aarch64_psci_conduit_init(const void *dtb)
{
const void *node;
const char *method;
uint32_t len;
if (!fdt_valid(dtb)) {
console_println("PSCI: no DTB -- using HVC (QEMU virt-machine default)");
return;
}
node = fdt_find_node_by_compatible(dtb, "arm,psci-1.0");
if (!node) {
console_println("PSCI: no DTB PSCI node -- using HVC (QEMU virt-machine default)");
return;
}
method = (const char *) fdt_find_prop_in_node(dtb, node, "method", &len);
if (method && len >= 3 && method[0] == 's' && method[1] == 'm' && method[2] == 'c') {
s_psci_use_smc = 1;
console_println("PSCI: conduit discovered from DTB (method=\"smc\")");
} else {
console_println("PSCI: DTB method != \"smc\" -- using HVC");
}
}
void arch_cold_reset(void)
{
arch_disable_interrupts();
/* PSCI SYSTEM_RESET (function 0x84000009). SYSTEM_RESET takes no
* arguments and has no SMC64 variant defined by the PSCI spec -- only
* the SMC32 encoding is valid (fixed 2026-08-18, was 0xC4000009).
*
* Conduit is @c s_psci_use_smc, set by @c aarch64_psci_conduit_init()
* above from the real DTB when one is present (real Pi 5 hardware's
* own `/psci` node declares `method = "smc"`, confirmed against
* `bcm2712.dtsi`).
*
* FABRIC-1.md Section I, 2026-08-18: live gdb tracing (using the real
* UEFI-relocated runtime address, not the standalone kernel.elf's
* link-time address -- see that section for why those differ) proved
* the SMC call itself traps on *this system's* QEMU aarch64 boot
* (AAVMF UEFI firmware, no genuine EL3/TrustZone secure monitor, no
* DTB forwarded to the guest either): PC does not fall through to the
* wfi loop below, it jumps straight into this kernel's own exception
* vector. PSCI there is served via HVC (hypervisor call, EL2)
* instead -- exactly the @c s_psci_use_smc == 0 default case. The
* function ID and calling convention are unchanged between conduits,
* only the trapping instruction differs. */
if (s_psci_use_smc) {
__asm__ volatile (
"mov x0, #0x84000000\n"
"movk x0, #0x0009\n"
"smc #0\n"
::: "x0", "memory"
);
} else {
__asm__ volatile (
"mov x0, #0x84000000\n"
"movk x0, #0x0009\n"
"hvc #0\n"
::: "x0", "memory"
);
}
for (;;) __asm__ volatile ("wfi" ::: "memory");
}
/**
* @brief Read the AArch64 system counter (CNTPCT_EL0).
*
* Issues @c MRS @c %0, @c cntpct_el0 to read the 64-bit physical system
* counter. On AArch64 the system counter is the architectural equivalent
* of the x86-64 TSC: a monotonically incrementing clock whose frequency
* is reported by @c CNTFRQ_EL0 and set by firmware at boot.
*
* Unlike @c RDTSC on x86-64, @c CNTPCT_EL0 is fully synchronised across
* cores (no per-core skew), invariant across power-state changes, and
* does not require serialisation for most measurement use cases. It is
* used as the time base for @c timer_now_ns() and @c heartbeat_tick().
*
* @return Current 64-bit system counter value in counter ticks
* (frequency given by @c CNTFRQ_EL0, typically 10–62.5 MHz).
*/
uint64_t arch_read_timestamp(void)
{
uint64_t val;
__asm__ volatile ("mrs %0, cntpct_el0" : "=r"(val));
return val;
}
/**
* @brief Initialise the AArch64 MMU (stub — deferred to a later milestone).
*
* Full page-table setup (TTBR0/TTBR1, MAIR, TCR, SCTLR.M) is deferred to a
* later milestone. This stub exists to satisfy the common @c arch_mmu_init()
* call site shared across all three ISAs. The VMM subsystem (@c vmm.c)
* handles page-table management independently of this hook.
*/
void arch_mmu_init(void)
{
/* Stub: MMU bring-up deferred for later milestone */
}
kernel/src/arch/aarch64/boot.S
+25
View File
@@ -0,0 +1,25 @@
/*
* boot.S (aarch64) - Entry stub and linker-script symbol substitutes.
*
* With lld-link (PE/COFF), GNU ld linker-script symbols are unavailable.
* Define __text_start, __text_end, __kernel_start, __kernel_end here so
* hal.c and pmm.c resolve. They all point to the image base, which is
* sufficient for the monolithic loader build (UEFI manages page ownership).
*
* The .reloc section (PE base-relocation table) is generated automatically
* by lld-link from the COFF relocations in the object files.
*/
.text
.global __text_start
.global __kernel_start
__text_start:
__kernel_start:
.global arch_bootstrap
arch_bootstrap:
ret
.global __text_end
.global __kernel_end
__text_end:
__kernel_end:
kernel/src/arch/aarch64/interrupts.c
+223
View File
@@ -0,0 +1,223 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James. All rights reserved.
Licensed under the StarForth License, Version 1.0.
*/
/**
* interrupts.c (aarch64) - Exception vector setup and common handler
*/
#include <stdint.h>
#include "arch.h"
#include "console.h"
#include "starkernel/timer.h"
volatile const char *g_sk_fault_word = (void *)0;
/* Defined in isr.S */
extern void aarch64_install_vectors(void);
/* Defined in arch.c. Extern-declared here rather than in the shared arch.h:
* "exception level" has no amd64/riscv64 equivalent, so it does not belong
* in the cross-ISA header — same convention as riscv64_timer_rearm() in
* arch/riscv64/interrupts.c. */
extern int aarch64_current_el(void);
/* Defined in apic.c (items 0.6/0.7). Same extern-in-place convention: GICv2
* and generic-timer register-level details belong to the file that owns
* the MMIO base addresses / EL-aware register selection, not the shared
* apic.h. */
extern uint32_t apic_read_iar(void);
extern uint32_t apic_timer_ppi(void);
extern void apic_eoi_intid(uint32_t intid);
extern void apic_timer_rearm(void);
/* Defined in virtio_input.c (item 4.3.5e). g_virtio_input_gic_intid is 0
* (not a valid SPI INTID -- SPIs start at 32) until
* virtio_input_find_keyboard() finds a real device and computes its routed
* INTID, so this branch is inert on any boot where the device isn't
* present. Same extern-in-place convention as the rest of this file. */
extern uint32_t g_virtio_input_gic_intid;
extern void virtio_input_isr(void);
/* INTID 1023 = "spurious" (GICv2 spec): the CPU interface has nothing
* pending, typically because another interrupt at the same or higher
* priority raced this one to acknowledgement. Architecturally defined,
* not this board's invention, so it is named rather than left as a magic
* number in the comparison below. */
#define GIC_INTID_SPURIOUS 1023u
/**
* @brief Dispatch an IRQ taken at current-EL with SPx (punch-list item 0.7).
*
* Called from @c irq_spx_trampoline in @c isr.S, which has already saved the
* full caller-saved integer and FP/SIMD register sets plus @c ELR_ELx /
* @c SPSR_ELx, and will restore all of it and @c ERET after this function
* returns — unlike @c aarch64_exception_handler(), this path is designed to
* return normally.
*
* Reads @c GICC_IAR once (the GICv2 acknowledgement step — this also moves
* the interrupt to the active state), dispatches on the returned INTID, and
* always completes with the matching @c GICC_EOIR write, spurious or not:
* a spurious read must still not be treated as "nothing to clean up," and
* an unrecognised real INTID still needs the CPU interface unblocked for
* the next one.
*
* For the timer PPI (@c apic_timer_ppi() — 30 at EL1, 26 at EL2, decided
* once in @c apic_init()): @c apic_timer_rearm() runs **first**, before
* @c heartbeat_tick() — the ARM Generic Timer does not auto-reload, so
* skipping this leaves the interrupt condition permanently latched (see
* @c apic_timer_rearm()'s own doc comment). Ordering the re-arm ahead of
* the rest of the dispatch means a fault in @c heartbeat_tick() cannot
* also cost the next tick — same discipline as @c riscv64_timer_rearm()
* (item 0.3).
*
* virtio-input (item 4.3.5e, @c g_virtio_input_gic_intid): dispatched
* **before** the @c GICC_EOIR write below, not after — the ISR itself is
* what deasserts the device's (level-triggered) interrupt line by reading
* its ISR-status capability; EOI'ing first would let the GIC immediately
* re-signal a still-asserted line. Same claim-dispatch-complete ordering
* @c riscv64_interrupt_handler() uses for its own PLIC-routed devices.
*/
void aarch64_irq_handler(void)
{
uint32_t intid = apic_read_iar();
if (intid == apic_timer_ppi()) {
apic_timer_rearm();
heartbeat_tick();
}
if (intid == g_virtio_input_gic_intid) {
virtio_input_isr();
}
if (intid != GIC_INTID_SPURIOUS) {
apic_eoi_intid(intid);
}
}
/**
* @brief Print a 64-bit value as "0xNNNNNNNNNNNNNNNN" to the kernel console.
*
* Formats @p val as a 16-nibble lowercase hex string preceded by "0x" into
* a 19-byte stack buffer (including null terminator) then emits it via
* @c console_puts(). Used by @c aarch64_exception_handler() to print
* @c ESR_EL1, @c ELR_EL1, @c FAR_EL1, and @c SPSR_EL1 in the exception
* diagnostic dump. No heap allocation or libc dependency.
*
* @param val 64-bit value to format.
*/
static void print_hex(uint64_t val)
{
char buf[19];
buf[0] = '0'; buf[1] = 'x';
for (int i = 15; i >= 0; i--) {
int nibble = (int)(val & 0xF);
buf[2 + i] = (char)(nibble < 10 ? '0' + nibble : 'a' + nibble - 10);
val >>= 4;
}
buf[18] = '\0';
console_puts(buf);
}
/**
* @brief AArch64 common exception handler — print diagnostic and spin.
*
* Called from the @c isr.S exception vector table stubs for all exception
* types (synchronous, IRQ, FIQ, SError) that are not handled by a specific
* driver. Reads the four key EL1 exception system registers directly:
*
* - @c ESR_EL1 — Exception Syndrome Register: encodes the exception class
* (EC, bits [31:26]) and instruction-specific syndrome (ISS, bits [24:0]).
* The EC identifies the cause (e.g., 0x21 = data abort from EL1, 0x25 =
* data abort from EL0, 0x15 = SVC64).
* - @c ELR_EL1 — Exception Link Register: the PC of the instruction that
* caused the exception (or the instruction after, for IRQ/FIQ/SError).
* - @c FAR_EL1 — Fault Address Register: the virtual address that triggered
* a Data Abort or Instruction Abort; undefined for other exception types.
* - @c SPSR_EL1 — Saved Program Status Register: the PSTATE at the time of
* exception, including the mode field and interrupt-mask bits.
*
* Prints all four values to the kernel console then enters an infinite
* @c WFE loop to halt the affected core permanently. All exceptions that
* reach this handler are fatal — there is no recovery path.
*
* The global @c g_sk_fault_word is set by the VM FORTH dispatcher just
* before calling each word's function pointer; it names the word executing
* at the time of the fault, aiding post-mortem analysis from the serial log.
* (Currently logged only on the amd64 #GP path; future AArch64 diagnostic
* expansion can include it here.)
*/
void aarch64_exception_handler(void)
{
uint64_t esr, elr, far_val, spsr;
__asm__ volatile ("mrs %0, esr_el1" : "=r"(esr));
__asm__ volatile ("mrs %0, elr_el1" : "=r"(elr));
__asm__ volatile ("mrs %0, far_el1" : "=r"(far_val));
__asm__ volatile ("mrs %0, spsr_el1" : "=r"(spsr));
console_println("*** EXCEPTION (aarch64) ***");
console_puts(" ESR_EL1 = "); print_hex(esr); console_puts("\n");
console_puts(" ELR_EL1 = "); print_hex(elr); console_puts("\n");
console_puts(" FAR_EL1 = "); print_hex(far_val); console_puts("\n");
console_puts(" SPSR_EL1= "); print_hex(spsr); console_puts("\n");
for (;;) {
__asm__ volatile ("wfe" ::: "memory");
}
}
/**
* @brief Install the AArch64 exception vector table (M4 milestone).
*
* First reads and reports the exception level via @c aarch64_current_el()
* (punch-list item 0.4) — this is the earliest point in boot with both a
* working console (up since M1) and a genuine need for the answer (vector
* installation is the first EL-dependent operation). The read itself is
* cheap and safe far earlier than this; only the console print is gated on
* this call site.
*
* Then calls @c aarch64_install_vectors() (defined in @c isr.S) which
* writes the address of the vector table base into @c VBAR_EL1 or
* @c VBAR_EL2 (Vector Base Address Register), whichever matches the
* exception level @c aarch64_current_el() just reported. After this
* instruction all exceptions at that level (synchronous, IRQ, FIQ,
* SError) dispatch through the 512-byte-aligned vector table assembled
* in @c isr.S.
*
* **Correction, 2026-09-04 (FABRIC-3.md §IV.3 item 7's code audit):**
* this doc comment and the runtime console message below previously
* claimed EL2 was "a known gap... @c VBAR_EL1 is written unconditionally
* regardless of the detected level," citing punch-list item 0.5 as the
* still-open work to fix it. Checked against @c isr.S's own
* @c aarch64_install_vectors() (the actual implementation, not this
* comment) and found that claim stale: the assembly already branches on
* @c aarch64_current_el() and writes @c vbar_el2 at EL2 /
* @c vbar_el1 at EL1, threading the same answer through
* @c el2_mode_flag for the IRQ trampoline's @c ELR_ELx/@c SPSR_ELx
* selection too (see @c isr.S's own "EL selection" header comment).
* There never was a gap here to close — item 0.5's EL2 vector work was
* already done by the time this comment was written describing it as
* open; only the comment was wrong, not the code.
*
* On AArch64 there is no IDT and no PIC to disable; the GIC replaces
* both. The @c arch_interrupts_init() name is kept identical across all
* ISAs so that @c kernel_main() can call the same symbol regardless of
* the target architecture.
*
* Must be called after the kernel stack is established (so that the
* correct-EL SP is valid) and before @c arch_enable_interrupts().
*/
void arch_interrupts_init(void)
{
int el = aarch64_current_el();
console_puts("AArch64: running at EL");
console_putc((char)('0' + el));
console_println(el == 2
? " (VBAR_EL2/CNTHP_*_EL2 -- both already EL-aware, see isr.S/apic.c)"
: "");
aarch64_install_vectors();
}
kernel/src/arch/aarch64/isr.S
+334
View File
@@ -0,0 +1,334 @@
/*
* isr.S (aarch64) - Exception vector table stub.
*
* Fourteen of the sixteen vectors still route to the shared fatal handler in
* interrupts.c (unchanged by punch-list item 0.5). irq_spx -- the vector that
* matters, since the kernel runs at current-EL with SPx (SP_EL1/SP_EL2, per
* the detected EL) -- branches to a real trampoline: save state, call a C
* handler, restore, ERET. Everything else keeps spinning on a fatal WFE/WFI;
* only the interrupt path becomes returnable.
*/
.section .text
.balign 2048
/* Hidden for consistency with the amd64 isr.S convention. Neither symbol
* here is ever address-read from C (interrupts.c only *calls*
* aarch64_install_vectors(); the vector table itself is entirely
* self-contained in this file), so this isn't fixing a live bug the way
* the amd64 .hidden directives do -- it's defensive hardening against the
* same GOT-indirection class of issue should that ever change.
*
* .hidden is an ELF visibility directive with no COFF equivalent; this
* file is also assembled for the aarch64 loader, which clang targets at
* aarch64-pc-windows-msvc (PE/COFF, since GCC cannot produce aarch64 PE)
* -- guard on _WIN32 (defined only by that target) so the ELF-only
* kernel compile gets the directive and the PE loader compile doesn't. */
#ifndef _WIN32
.hidden aarch64_vector_table
.hidden aarch64_install_vectors
#endif
.global aarch64_vector_table
aarch64_vector_table:
/* Current EL with SP0 */
.balign 128
synchronous_sp0:
b aarch64_exception_handler
.balign 128
irq_sp0:
b aarch64_exception_handler
.balign 128
fiq_sp0:
b aarch64_exception_handler
.balign 128
serror_sp0:
b aarch64_exception_handler
/* Current EL with SPx */
.balign 128
synchronous_spx:
b aarch64_exception_handler
.balign 128
irq_spx:
/* 128-byte slot limit (32 instructions) -- the real save/restore
* sequence does not fit inline. One branch out, well within budget. */
b irq_spx_trampoline
.balign 128
fiq_spx:
b aarch64_exception_handler
.balign 128
serror_spx:
b aarch64_exception_handler
/* Lower EL using AArch64 */
.balign 128
synchronous_lower_el64:
b aarch64_exception_handler
.balign 128
irq_lower_el64:
b aarch64_exception_handler
.balign 128
fiq_lower_el64:
b aarch64_exception_handler
.balign 128
serror_lower_el64:
b aarch64_exception_handler
/* Lower EL using AArch32 */
.balign 128
synchronous_lower_el32:
b aarch64_exception_handler
.balign 128
irq_lower_el32:
b aarch64_exception_handler
.balign 128
fiq_lower_el32:
b aarch64_exception_handler
.balign 128
serror_lower_el32:
b aarch64_exception_handler
/* ---------------------------------------------------------------------------
* EL selection.
*
* The kernel runs at exactly one exception level for its entire lifetime
* (item 0.4's own caching rationale: CurrentEL cannot change post-boot), so
* EL is decided **once here**, not per interrupt. aarch64_current_el()
* (arch.c) is consulted to choose which VBAR_ELx gets the table's address,
* and el2_mode_flag records the same answer for irq_spx_trampoline below to
* pick ELR_EL1/SPSR_EL1 vs ELR_EL2/SPSR_EL2 without re-deriving it on every
* interrupt.
* ------------------------------------------------------------------------- */
.section .bss
.balign 1
el2_mode_flag:
.byte 0
.text
.global aarch64_install_vectors
aarch64_install_vectors:
stp x29, x30, [sp, #-16]!
bl aarch64_current_el /* w0 = 1 or 2 */
adrp x1, el2_mode_flag
add x1, x1, :lo12:el2_mode_flag
cmp w0, #2
b.ne 1f
mov w2, #1
strb w2, [x1]
adr x0, aarch64_vector_table
msr vbar_el2, x0
isb
b 2f
1: mov w2, #0
strb w2, [x1]
adr x0, aarch64_vector_table
msr vbar_el1, x0
isb
2: ldp x29, x30, [sp], #16
ret
/* ---------------------------------------------------------------------------
* irq_spx_trampoline — save / call C handler / restore / ERET.
*
* Frame layout (800 bytes, 16-byte aligned; SP is guaranteed 16-aligned at
* any point AAPCS64 code can be asynchronously interrupted, so no extra
* alignment handling is needed around the initial SUB):
*
* 0..239 x0-x29 (15 STP pairs)
* 240 x30
* 248 ELR_ELx
* 256 SPSR_ELx
* 264 FPSR
* 272 FPCR
* 280 (8 bytes unused -- keeps the V-register block 16-aligned)
* 288..415 v0-v7 (4 STP Q-pairs) -- AAPCS64 caller-saved
* 416..543 v8-v15 (4 STP Q-pairs) -- AAPCS64 callee-saved
* 544..799 v16-v31 (8 STP Q-pairs) -- AAPCS64 caller-saved
*
* FABRIC-3.md SXXVIII (Stage 0, preemptive-context-switch trap-frame parity,
* 2026-09-13): v8-v15 now saved too. They are AAPCS64 callee-saved, so
* ordinary compiled code (the C handler below, or anything else called
* through a normal ABI boundary) already preserves them across a *call* --
* but an interrupt is not a call the interrupted code's own compiled
* instructions know about. A value the interrupted function is genuinely
* live-using in v8-v15 at the moment of interrupt has no ABI guarantee of
* surviving an async trap at all; only a frame that saves everything makes
* "resume execution anywhere it was taken" actually true. This trap frame
* is not yet copied into any per-VM saved context (that's Stage 2) -- this
* stage only makes the frame itself complete.
* ------------------------------------------------------------------------- */
.equ TF_X0, 0
.equ TF_X2, 16
.equ TF_X4, 32
.equ TF_X6, 48
.equ TF_X8, 64
.equ TF_X10, 80
.equ TF_X12, 96
.equ TF_X14, 112
.equ TF_X16, 128
.equ TF_X18, 144
.equ TF_X20, 160
.equ TF_X22, 176
.equ TF_X24, 192
.equ TF_X26, 208
.equ TF_X28, 224
.equ TF_X30, 240
.equ TF_ELR, 248
.equ TF_SPSR, 256
.equ TF_FPSR, 264
.equ TF_FPCR, 272
.equ TF_V0, 288
.equ TF_V2, 320
.equ TF_V4, 352
.equ TF_V6, 384
.equ TF_V8, 416
.equ TF_V10, 448
.equ TF_V12, 480
.equ TF_V14, 512
.equ TF_V16, 544
.equ TF_V18, 576
.equ TF_V20, 608
.equ TF_V22, 640
.equ TF_V24, 672
.equ TF_V26, 704
.equ TF_V28, 736
.equ TF_V30, 768
.equ TF_SIZE, 800
.balign 4
irq_spx_trampoline:
sub sp, sp, #TF_SIZE
stp x0, x1, [sp, #TF_X0]
stp x2, x3, [sp, #TF_X2]
stp x4, x5, [sp, #TF_X4]
stp x6, x7, [sp, #TF_X6]
stp x8, x9, [sp, #TF_X8]
stp x10, x11, [sp, #TF_X10]
stp x12, x13, [sp, #TF_X12]
stp x14, x15, [sp, #TF_X14]
stp x16, x17, [sp, #TF_X16]
stp x18, x19, [sp, #TF_X18]
stp x20, x21, [sp, #TF_X20]
stp x22, x23, [sp, #TF_X22]
stp x24, x25, [sp, #TF_X24]
stp x26, x27, [sp, #TF_X26]
stp x28, x29, [sp, #TF_X28]
str x30, [sp, #TF_X30]
/* Original x0-x30 are safe in memory now; free to use x0-x2 as scratch
* for the rest of the save. EL-dependent register bank (B3): the two
* forms are genuinely different MRS encodings, chosen from the flag
* install_vectors set once, not re-derived per interrupt. */
adrp x0, el2_mode_flag
add x0, x0, :lo12:el2_mode_flag
ldrb w0, [x0]
cbnz w0, 1f
mrs x1, elr_el1
mrs x2, spsr_el1
b 2f
1: mrs x1, elr_el2
mrs x2, spsr_el2
2: stp x1, x2, [sp, #TF_ELR]
/* FP/SIMD (B2): built without -mgeneral-regs-only, so the C handler may
* use any of these. */
mrs x1, fpsr
mrs x2, fpcr
stp x1, x2, [sp, #TF_FPSR]
stp q0, q1, [sp, #TF_V0]
stp q2, q3, [sp, #TF_V2]
stp q4, q5, [sp, #TF_V4]
stp q6, q7, [sp, #TF_V6]
stp q8, q9, [sp, #TF_V8]
stp q10, q11, [sp, #TF_V10]
stp q12, q13, [sp, #TF_V12]
stp q14, q15, [sp, #TF_V14]
stp q16, q17, [sp, #TF_V16]
stp q18, q19, [sp, #TF_V18]
stp q20, q21, [sp, #TF_V20]
stp q22, q23, [sp, #TF_V22]
stp q24, q25, [sp, #TF_V24]
stp q26, q27, [sp, #TF_V26]
stp q28, q29, [sp, #TF_V28]
stp q30, q31, [sp, #TF_V30]
bl aarch64_irq_handler
ldp q0, q1, [sp, #TF_V0]
ldp q2, q3, [sp, #TF_V2]
ldp q4, q5, [sp, #TF_V4]
ldp q6, q7, [sp, #TF_V6]
ldp q8, q9, [sp, #TF_V8]
ldp q10, q11, [sp, #TF_V10]
ldp q12, q13, [sp, #TF_V12]
ldp q14, q15, [sp, #TF_V14]
ldp q16, q17, [sp, #TF_V16]
ldp q18, q19, [sp, #TF_V18]
ldp q20, q21, [sp, #TF_V20]
ldp q22, q23, [sp, #TF_V22]
ldp q24, q25, [sp, #TF_V24]
ldp q26, q27, [sp, #TF_V26]
ldp q28, q29, [sp, #TF_V28]
ldp q30, q31, [sp, #TF_V30]
ldp x0, x1, [sp, #TF_FPSR]
msr fpsr, x0
msr fpcr, x1
/* x0-x30 are still their post-interrupt-entry values (untouched since
* the initial save) except x0-x2, which the save sequence above used as
* scratch -- all three are about to be overwritten by the final GP
* restore below regardless, so clobbering them here is harmless. */
ldp x0, x1, [sp, #TF_ELR] /* x0=ELR, x1=SPSR */
adrp x2, el2_mode_flag
add x2, x2, :lo12:el2_mode_flag
ldrb w2, [x2]
cbnz w2, 3f
msr elr_el1, x0
msr spsr_el1, x1
b 4f
3: msr elr_el2, x0
msr spsr_el2, x1
4:
ldp x0, x1, [sp, #TF_X0]
ldp x2, x3, [sp, #TF_X2]
ldp x4, x5, [sp, #TF_X4]
ldp x6, x7, [sp, #TF_X6]
ldp x8, x9, [sp, #TF_X8]
ldp x10, x11, [sp, #TF_X10]
ldp x12, x13, [sp, #TF_X12]
ldp x14, x15, [sp, #TF_X14]
ldp x16, x17, [sp, #TF_X16]
ldp x18, x19, [sp, #TF_X18]
ldp x20, x21, [sp, #TF_X20]
ldp x22, x23, [sp, #TF_X22]
ldp x24, x25, [sp, #TF_X24]
ldp x26, x27, [sp, #TF_X26]
ldp x28, x29, [sp, #TF_X28]
ldr x30, [sp, #TF_X30]
add sp, sp, #TF_SIZE
eret
kernel/src/arch/aarch64/native_rpi5_entry.S
+97
View File
@@ -0,0 +1,97 @@
/*
* native_rpi5_entry.S (aarch64) - Raspberry Pi 5 native (non-UEFI) entry
* stub (FABRIC-3.md §IV.3 item 1, 2026-09-04).
*
* **Never run on real hardware** -- the Pi 5 isn't in hand until
* 2026-09-17 (FABRIC-3.md §II). Compile-only-verified.
*
* Entered directly by Pi 5 firmware at the classic bare-metal load address
* 0x80000 (item 6, `config.txt`'s `os_check=0`, is still open -- that's
* what actually gets a real image loaded there; this file is just the
* code that would run once it is). No UEFI, no ACPI -- none of
* `boot/uefi_loader.c`'s PE-loader shape applies on this path. Per
* FABRIC-3.md §IV.1's own researched entry protocol:
*
* x0 = 32-bit DTB pointer, upper 32 bits of the 64-bit register
* UNSPECIFIED -- must mask before use.
* x1-x3 = reserved / zero, unused here.
*
* This stub's own job: mask and capture that DTB pointer into
* `g_rpi5_dtb_ptr` (declared in `include/starkernel/rpi5_native_entry.h`),
* establish a dedicated stack -- this path has no EDK2 boot stack to
* inherit, there is no EDK2 at all here -- then tail-call
* `rpi5_native_boot()` (`rpi5_native_boot.c`, §IV.3 item 2), which builds
* `BootInfo` from the devicetree and calls the existing `kernel_main()`.
* `rpi5_native_boot()` never returns; the `wfe`/`b` loop after the `bl`
* exists only so this file's own control flow has somewhere defined to
* go if that contract is ever violated.
*
* Build-system note: `kernel/Makefile`'s `KERNEL_ASM` wildcards every
* `*.S` file in this directory into the kernel ELF build (not the PE/COFF
* loader build -- that list is explicit, boot.S/isr.S/switch.S only), so
* this file links into the ARCH=aarch64 QEMU/UEFI kernel ELF as dead
* code, and is the entry point of the separately-linked Pi 5 native
* image (`make boot_image TARGET=RASPI`):
* `kernel/linker/starkernel-native-rpi5.ld` places `.text.entry` first
* at 0x80000 and the image is flattened to `kernel_2712.img`.
*
* Pi 5 firmware loads only the file's bytes; `.bss` (NOLOAD) is whatever
* was in RAM. It is zeroed here, before anything in it -- including
* `g_rpi5_dtb_ptr` and this stub's own stack -- is used. x0 (the DTB
* pointer) is preserved across the loop; only x4-x6 are touched.
*/
.section .bss
.align 8
.global g_rpi5_dtb_ptr
g_rpi5_dtb_ptr:
.space 8 /* uint64_t -- masked DTB pointer */
.align 4
g_rpi5_native_stack:
.space 0x200000 /* 2 MiB, same convention as the
* amd64/riscv64 kernel_entry.S
* BSS stacks */
g_rpi5_native_stack_top:
.section .text.entry, "ax"
.global rpi5_native_start
rpi5_native_start:
/* Mask x0 to its documented 32-bit DTB-pointer range (§IV.1: the
* upper 32 bits of the 64-bit register are unspecified by the
* firmware's own entry protocol). */
mov x1, #0xffffffff
and x0, x0, x1
/* Zero .bss: [__bss_start, __bss_end), both 16-byte aligned by the
* native linker script. */
adrp x4, __bss_start
add x4, x4, :lo12:__bss_start
adrp x5, __bss_end
add x5, x5, :lo12:__bss_end
.Lbss_zero:
cmp x4, x5
b.hs .Lbss_done
stp xzr, xzr, [x4], #16
b .Lbss_zero
.Lbss_done:
adrp x2, g_rpi5_dtb_ptr
add x2, x2, :lo12:g_rpi5_dtb_ptr
str x0, [x2]
/* Establish this path's own dedicated stack. */
adrp x0, g_rpi5_native_stack_top
add x0, x0, :lo12:g_rpi5_native_stack_top
mov sp, x0
/* x0 was clobbered above for the stack-top address -- reload the
* masked DTB pointer from g_rpi5_dtb_ptr before calling in. */
adrp x0, g_rpi5_dtb_ptr
add x0, x0, :lo12:g_rpi5_dtb_ptr
ldr x0, [x0]
bl rpi5_native_boot
.Lhalt:
wfe
b .Lhalt
kernel/src/arch/aarch64/rpi5_dtb.c
+93
View File
@@ -0,0 +1,93 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James. All rights reserved.
Licensed under the StarForth License, Version 1.0.
*/
/**
* rpi5_dtb.c - Raspberry Pi 5 (BCM2712) devicetree-based peripheral
* discovery (FABRIC-3.md §IV.3, 2026-09-04).
*
* Confirmed directly against `bcm2712.dtsi` (raspberrypi/linux,
* rpi-6.12.y) before writing this, not assumed:
*
* soc: soc@107c000000 {
* compatible = "simple-bus";
* ranges = <0x0 0x10 0x0 0x80000000>;
* #address-cells = <1>;
* #size-cells = <1>;
* ...
* uart10: serial@7d001000 {
* compatible = "arm,pl011", "arm,primecell";
* reg = <0x7d001000 0x200>;
* };
* mailbox: mailbox@7c013880 {
* compatible = "brcm,bcm2835-mbox";
* reg = <0x7c013880 0x40>;
* };
* };
*
* Every peripheral this file cares about lives under this one "soc"
* node, which applies exactly one translation to every child `reg`
* value: child address 0 maps to parent (CPU-physical) address
* 0x10_0000_0000 (`ranges`'s own `<0x0 0x10 0x0 ...>` — child cell 0,
* parent cells `0x10 0x0`, i.e. `0x10 << 32`). The child side's own
* `#address-cells = <1>` confirms each `reg` value's first cell is the
* whole child address (32-bit, no cell-splitting needed) before adding
* the offset. `fdt.c`'s reader deliberately does not parse `ranges`
* generally (it is "not a general devicetree library," per its own
* header comment) — this one, fixed, SoC-wide offset is applied here
* by name instead, since it is a single known hardware fact, not a
* general mechanism this codebase needs elsewhere yet.
*/
#include "starkernel/rpi5_dtb.h"
#include "starkernel/fdt.h"
/* soc@107c000000's own `ranges` offset -- see this file's own doc
* comment above for where it was confirmed. */
#define BCM2712_SOC_RANGES_OFFSET 0x1000000000ULL
/**
* @brief Read a devicetree `reg` property's first cell as big-endian.
*
* `fdt.c` keeps its own `be32()` helper file-local (freestanding, no
* shared byte-swap utility to reuse) -- duplicated here rather than
* exposing it, same "a few lines is simpler than a new shared
* dependency" precedent `tools/pkcs8_ed25519.c` already set in this
* codebase.
*/
static uint32_t reg_first_cell_be32(const unsigned char* b)
{
return ((uint32_t) b[0] << 24) | ((uint32_t) b[1] << 16) |
((uint32_t) b[2] << 8) | (uint32_t) b[3];
}
/**
* @brief Find @p compatible's node, read its `reg` base address, and
* apply the one fixed BCM2712 `soc`-node translation offset.
*/
static uint64_t rpi5_peripheral_base(const void* dtb, const char* compatible)
{
const void* node;
const void* val;
uint32_t len;
node = fdt_find_node_by_compatible(dtb, compatible);
if (!node) return 0;
val = fdt_find_prop_in_node(dtb, node, "reg", &len);
if (!val || len < 4) return 0;
return BCM2712_SOC_RANGES_OFFSET + reg_first_cell_be32((const unsigned char*) val);
}
uint64_t rpi5_uart_base(const void* dtb)
{
return rpi5_peripheral_base(dtb, "arm,pl011");
}
uint64_t rpi5_mailbox_base(const void* dtb)
{
return rpi5_peripheral_base(dtb, "brcm,bcm2835-mbox");
}
kernel/src/arch/aarch64/rpi5_mailbox.c
+237
View File
@@ -0,0 +1,237 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James. All rights reserved.
Licensed under the StarForth License, Version 1.0.
*/
/**
* rpi5_mailbox.c - VideoCore mailbox property-interface framebuffer
* setup (FABRIC-3.md §IV.3 item 3, 2026-09-04).
*
* **Never run on real hardware** — the Pi 5 isn't in hand until
* 2026-09-17 (FABRIC-3.md §II). Everything below is built from real,
* cited sources rather than assumed, but is compile-only-verified.
*
* Register layout (offsets from `rpi5_mailbox_base()`, i.e. the
* `mailbox` devicetree node's own translated base address —
* `rpi5_dtb.c` already confirmed `0x107C013880` on real Pi 5
* hardware, matching a Pi-5-specific bare-metal reference
* (main.lv/writeup/raspberry5_baremetal_framebuffer.md) independently
* confirming the same `0x107C000000` SoC base + `0x13880` mailbox
* offset this codebase's own devicetree-driven lookup already
* produces — two independent sources agreeing is why this offset is
* trusted, not just cited once):
*
* +0x00 MBOX0_READ (receive: VC -> ARM)
* +0x18 MBOX0_STATUS (bit30 MBOX_EMPTY: nothing to read)
* +0x20 MBOX1_WRITE (send: ARM -> VC)
* +0x38 MBOX1_STATUS (bit31 MBOX_FULL: cannot write yet)
*
* (`+0x20/+0x38` per the same Pi-5-specific reference above, which
* treats MBOX0/MBOX1 as two structurally identical 0x20-byte blocks
* back to back — a real difference from older-chip tutorials, which
* often describe a single shared status register at `+0x18` for both
* directions. Trusted here because it's the one source actually
* confirmed against Pi 5 hardware, not an older SoC.)
*
* Message encoding (official wiki:
* github.com/raspberrypi/firmware/wiki/Mailbox-property-interface,
* channel and buffer format; property tag IDs/lengths from the same
* page): write `(buffer_addr & ~0xF) | channel` to MBOX1_WRITE; read
* back the same encoded value from MBOX0_READ once available, confirming
* it before trusting the buffer (the response overwrites the request
* in place — same shared buffer, not a separate response buffer). No
* bus-address translation applied to the buffer pointer itself
* (confirmed against the same Pi-5-specific reference: it writes the
* plain masked pointer, no added GPU-bus-alias bit, unlike the
* classic 32-bit-Pi convention some older tutorials describe) —
* genuinely different per-SoC-generation behavior, not an oversight.
*
* Buffer/tag format (official wiki, same page): u32 total size, u32
* request(0)/response(0x80000000|len) code, tags..., u32 end-tag(0).
* Each tag: u32 id, u32 value-buffer size, u32 req-size/resp-flag,
* value bytes padded to 4-byte alignment. Buffer itself must be
* 16-byte aligned (only the upper 28 bits of its address travel
* through the mailbox).
*/
#include "starkernel/rpi5_mailbox.h"
#include "starkernel/rpi5_dtb.h"
#define MBOX_OFF_READ 0x00u
#define MBOX_OFF_STATUS0 0x18u
#define MBOX_OFF_WRITE 0x20u
#define MBOX_OFF_STATUS1 0x38u
#define MBOX_FULL 0x80000000u
#define MBOX_EMPTY 0x40000000u
#define MBOX_CH_PROPERTY 8u
#define MBOX_CODE_REQUEST 0x00000000u
#define MBOX_CODE_RESPONSE_OK 0x80000000u
#define TAG_SET_PHYS_SIZE 0x00048003u
#define TAG_SET_VIRT_SIZE 0x00048004u
#define TAG_SET_DEPTH 0x00048005u
#define TAG_SET_PIXEL_ORDER 0x00048006u
#define TAG_SET_VIRTUAL_OFFS 0x00048009u
#define TAG_ALLOCATE_BUFFER 0x00040001u
#define TAG_GET_PITCH 0x00040008u
#define TAG_END 0x00000000u
#define PIXEL_ORDER_RGB 1u /* per the wiki's own tag doc: 0=BGR, 1=RGB */
/* 16-byte-aligned static request/response buffer -- no allocator
* exists this early in boot (this driver runs before kmalloc's own
* init, same as every other pre-M6 boot step), and one buffer is
* enough: this driver only ever has one request in flight, never
* concurrent. Exact word count for the 7-tag sequence below, counted
* word-by-word (not estimated): 2 [header] + 5 [phys size] + 5 [virt
* size] + 4 [depth] + 4 [pixel order] + 5 [virtual offset] + 5
* [allocate] + 4 [get pitch] + 1 [end tag] = 35 words (140 bytes) --
* 40 gives real margin without the array itself getting large enough
* to matter. (An earlier version of this comment claimed 32 words was
* "generous headroom" for a 38-word estimate against a 32-word array
* -- that was wrong on its own terms before even being checked against
* the real 35-word count; recomputed properly this time, not
* re-estimated.) */
static uint32_t g_mbox_buf[40] __attribute__((aligned(16)));
static inline void mmio_write32(uint64_t addr, uint32_t val)
{
*(volatile uint32_t*) addr = val;
}
static inline uint32_t mmio_read32(uint64_t addr)
{
return *(volatile uint32_t*) addr;
}
/* Appends one tag (id, value-buffer size in bytes, request size in
* bytes, then `nvalues` u32 value words) to g_mbox_buf starting at
* word index *idx, advancing *idx past it (including alignment
* padding -- every value here is a whole number of u32 words already,
* so no padding math is actually needed, but the wiki's own format
* allows non-multiple-of-4 value sizes in general; this driver's own
* tags never do). */
static void append_tag(uint32_t* idx, uint32_t tag_id, uint32_t value_bytes,
const uint32_t* values, uint32_t nvalues)
{
uint32_t i;
g_mbox_buf[(*idx)++] = tag_id;
g_mbox_buf[(*idx)++] = value_bytes;
/* Request-size word: set equal to value_bytes (the value-buffer's
* own size, which for TAG_ALLOCATE_BUFFER is 8 -- the response's
* size, since the buffer must be big enough for whichever is
* larger -- not the 4-byte true request payload). Common practice
* across multiple reference implementations surveyed while writing
* this, not a single spec-quoted number; firmware is not observed
* to validate an exact match against the true request length.
* Unverified against real hardware either way. */
g_mbox_buf[(*idx)++] = value_bytes;
for (i = 0; i < nvalues; i++) g_mbox_buf[(*idx)++] = values[i];
}
int rpi5_mailbox_get_framebuffer(const void* dtb, uint32_t width, uint32_t height,
uint32_t bpp, Rpi5FramebufferInfo* out)
{
uint64_t base;
uint32_t idx;
uint32_t send_val;
uint32_t v2[2];
if (!out) return -1;
base = rpi5_mailbox_base(dtb);
if (!base) return -2; /* mailbox node not found/malformed in the DTB */
idx = 2; /* words 0/1 are the header, filled in after tags are built */
v2[0] = width;
v2[1] = height;
append_tag(&idx, TAG_SET_PHYS_SIZE, 8, v2, 2);
append_tag(&idx, TAG_SET_VIRT_SIZE, 8, v2, 2);
v2[0] = bpp;
append_tag(&idx, TAG_SET_DEPTH, 4, v2, 1);
v2[0] = PIXEL_ORDER_RGB;
append_tag(&idx, TAG_SET_PIXEL_ORDER, 4, v2, 1);
v2[0] = 0; v2[1] = 0; /* virtual offset x,y -- always (0,0), no scrolling support */
append_tag(&idx, TAG_SET_VIRTUAL_OFFS, 8, v2, 2);
v2[0] = 16; /* alignment request, bytes */
v2[1] = 0; /* response slot (base addr); request only sends v2[0] */
append_tag(&idx, TAG_ALLOCATE_BUFFER, 8, v2, 2);
v2[0] = 0;
append_tag(&idx, TAG_GET_PITCH, 4, v2, 1);
g_mbox_buf[idx++] = TAG_END;
g_mbox_buf[0] = idx * 4u; /* total buffer size, bytes */
g_mbox_buf[1] = MBOX_CODE_REQUEST;
send_val = ((uint32_t) (uintptr_t) g_mbox_buf & ~0xFu) | MBOX_CH_PROPERTY;
while ((mmio_read32(base + MBOX_OFF_STATUS1) & MBOX_FULL) != 0)
{ /* spin until MBOX1 (send) has room */
}
mmio_write32(base + MBOX_OFF_WRITE, send_val);
for (;;)
{
while ((mmio_read32(base + MBOX_OFF_STATUS0) & MBOX_EMPTY) != 0)
{ /* spin until MBOX0 (receive) has a message */
}
if (mmio_read32(base + MBOX_OFF_READ) == send_val) break;
/* A message arrived, but not the one we sent (another channel's
* traffic) -- discard and keep waiting, per the wiki's own
* documented channel-matching convention. */
}
if (g_mbox_buf[1] != MBOX_CODE_RESPONSE_OK) return -3; /* VC rejected the request */
/* Re-walk the tags to pull out the allocate-buffer and get-pitch
* responses -- same fixed order they were sent in, so fixed word
* offsets are safe to compute directly rather than re-parsing
* generically (this driver only ever sends this one sequence). */
{
uint32_t alloc_tag_value_off = 2 /* header */
+ (3 + 2) /* phys size */
+ (3 + 2) /* virt size */
+ (3 + 1) /* depth */
+ (3 + 1) /* pixel order */
+ (3 + 2) /* virtual offset */
+ 3; /* allocate-buffer tag header, then its value words */
uint32_t pitch_tag_value_off = alloc_tag_value_off + 2 /* alloc response: base+size */
+ 3; /* get-pitch tag header, then its value word */
uint32_t fb_addr = g_mbox_buf[alloc_tag_value_off];
uint32_t fb_size = g_mbox_buf[alloc_tag_value_off + 1];
uint32_t pitch_bytes = g_mbox_buf[pitch_tag_value_off];
if (fb_addr == 0 || fb_size == 0 || pitch_bytes == 0) return -4;
/* Allocate-buffer's own response address is a VC bus address,
* not necessarily the plain ARM-physical one -- masking the
* top bits to recover the ARM-physical alias is the documented
* convention on earlier Pi chips; kept here defensively even
* though the Pi-5-specific reference this file cites for the
* *send* side found no bus-alias bit in play there. Genuinely
* unverified which of these is correct for Pi 5's own
* allocate-buffer response specifically -- flagged, not
* guessed past this comment. */
out->base = (void*) (uintptr_t) (fb_addr & 0x3FFFFFFFu);
out->size = fb_size;
out->width = width;
out->height = height;
out->pixels_per_scanline = pitch_bytes / (bpp / 8u);
out->pixel_format = 0; /* PixelRedGreenBlueReserved8BitPerColor --
* matches PIXEL_ORDER_RGB requested above */
}
return 0;
}
kernel/src/arch/aarch64/rpi5_native_boot.c
+422
View File
@@ -0,0 +1,422 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James. All rights reserved.
Licensed under the StarForth License, Version 1.0.
*/
/**
* rpi5_native_boot.c - Raspberry Pi 5 DTB->BootInfo constructor
* (FABRIC-3.md §IV.3 item 2, 2026-09-04).
*
* **Never run on real hardware** -- the Pi 5 isn't in hand until
* 2026-09-17 (FABRIC-3.md §II). Compile-only-verified; 3-arch QEMU boot
* confirms this compiles and links, and does not disturb the existing
* UEFI/ACPI path -- it cannot exercise a single line of this function,
* since nothing in that path calls it (same caveat as
* `rpi5_dtb.c`/`rpi5_mailbox.c`/`native_rpi5_entry.S` before it).
*
* `rpi5_native_start` (`native_rpi5_entry.S`) tail-calls this function
* with the masked DTB pointer it captured. This builds the *existing*,
* unmodified `BootInfo` struct from the devicetree instead of UEFI
* protocols, then calls the *existing*, unmodified `kernel_main()` --
* this is the crux of why most of M1-M9 stays shared between the UEFI
* and native boot paths.
*
* `/memory`'s own `reg` (root `#address-cells`/`#size-cells`, confirmed
* against `bcm2712.dtsi`'s actual root node -- `<2>`/`<2>` -- not assumed)
* is turned into `EfiConventionalMemory` descriptors, with `/reserved-memory`
* children that carry a static `reg` (`brcm,bcm2712-ds.dtsi`'s real
* `reserved-memory` node has one, `atf@0`, confirmed by fetching that file
* rather than assumed) carved out as `EfiReservedMemoryType` descriptors
* instead of being silently left inside a conventional range. Children
* with no `reg` (`linux,cma`, which uses `size`/`alloc-ranges` -- a
* dynamic reservation with no fixed address, confirmed against the same
* file) have nothing fixed to carve and are skipped -- there is no
* allocator this early to service a dynamic reservation against anyway.
* `no-map`/`reusable` flags are not distinguished: every static
* reservation is excluded from `EfiConventionalMemory` regardless of
* which flag it carries, since this pass has no different treatment for
* either. `/reserved-memory` itself may declare its own
* `#address-cells`/`#size-cells` different from the root's (confirmed:
* `bcm2712-ds.dtsi`'s does -- `<2>`/`<1>`, not `<2>`/`<2>`) -- read from
* the node itself, falling back to root's only if absent, per DT spec
* §3.5.4's own inheritance rule.
*
* What this deliberately does NOT do yet, named honestly rather than
* silently skipped:
*
* - **The mailbox framebuffer request uses a fixed 1920x1080x32
* default**, not real display negotiation (no EDID query exists in
* this codebase). Flagged, not guessed past this comment -- revisit
* once real hardware and a real attached display are in hand.
* - **Root and `/reserved-memory` `#address-cells`/`#size-cells` must
* each be exactly 1 or 2** -- other widths halt rather than guess.
*/
#include "starkernel/rpi5_native_boot.h"
#include "starkernel/fdt.h"
#include "starkernel/rpi5_mailbox.h"
#include "starkernel/cmdline.h"
#include "starkernel/uefi.h"
#define RPI5_FB_DEFAULT_WIDTH 1920u
#define RPI5_FB_DEFAULT_HEIGHT 1080u
#define RPI5_FB_DEFAULT_BPP 32u
#define RPI5_MAX_RAM_REGIONS 4
#define RPI5_MAX_RESERVED_RANGES 16
/* Worst case per RAM region: one EfiConventionalMemory gap before, plus
* one EfiReservedMemoryType segment for, each reserved range that
* intersects it, plus one trailing conventional segment after the last
* one = 2*RESERVED_RANGES + 1. Total worst case across all regions:
* RAM_REGIONS * (2*RESERVED_RANGES + 1). Recomputed exactly, not
* estimated -- rpi5_mailbox.c's own buffer-size bug is the lesson here.
* Real DTBs need far fewer: bcm2712-ds.dtsi's own reserved-memory node
* has exactly one static reservation (atf@0) against one RAM region. */
#define RPI5_MAX_MEMMAP_ENTRIES (RPI5_MAX_RAM_REGIONS * (2 * RPI5_MAX_RESERVED_RANGES + 1))
extern void kernel_main(BootInfo *boot_info);
static BootInfo g_rpi5_boot_info = {0};
static EFI_MEMORY_DESCRIPTOR g_rpi5_memmap[RPI5_MAX_MEMMAP_ENTRIES];
typedef struct
{
uint64_t addr;
uint64_t size;
} rpi5_range_t;
static uint32_t be32_at(const unsigned char *p)
{
return ((uint32_t)p[0] << 24) | ((uint32_t)p[1] << 16) |
((uint32_t)p[2] << 8) | (uint32_t)p[3];
}
/**
* @brief Read one big-endian cell group (1 or 2 u32 cells) as a uint64_t.
*/
static uint64_t read_cells(const unsigned char *p, uint32_t ncells)
{
if (ncells == 1) return (uint64_t)be32_at(p);
return ((uint64_t)be32_at(p) << 32) | (uint64_t)be32_at(p + 4);
}
/**
* @brief Halt this core. No console exists yet (M1 hasn't run) and there
* is nowhere safe to report failure -- this is the same halt shape
* `native_rpi5_entry.S` itself uses before this function existed.
*/
static void rpi5_native_halt(void) __attribute__((noreturn));
static void rpi5_native_halt(void)
{
for (;;)
{
__asm__ volatile("wfe");
}
}
/**
* @brief Parse `/memory`'s `reg` into raw (addr, size) ranges, honoring
* the root's own `#address-cells`/`#size-cells` (NOT the `soc` node's --
* see this file's own doc comment; `/memory` is a direct child of root).
*
* @return Number of ranges written (0 on any failure).
*/
static uint32_t collect_ram_regions(const void *dtb, uint32_t acells, uint32_t scells,
rpi5_range_t *out, uint32_t max)
{
const void *mem_node;
const void *reg;
uint32_t reg_len, entry_bytes, count, i;
const unsigned char *p;
mem_node = fdt_find_node_by_device_type(dtb, "memory");
if (!mem_node) return 0;
reg = fdt_find_prop_in_node(dtb, mem_node, "reg", &reg_len);
if (!reg) return 0;
entry_bytes = (acells + scells) * 4u;
if (entry_bytes == 0) return 0;
count = reg_len / entry_bytes;
if (count > max) count = max;
p = (const unsigned char *)reg;
for (i = 0; i < count; i++)
{
out[i].addr = read_cells(p, acells);
out[i].size = read_cells(p + acells * 4u, scells);
p += entry_bytes;
}
return count;
}
/**
* @brief Read one node-scoped `#address-cells`/`#size-cells`-style u32
* property, falling back to @p fallback if the node doesn't declare its
* own (DT spec §3.5.4's own inheritance rule for `/reserved-memory`).
*/
static uint32_t node_cells_or_fallback(const void *dtb, const void *node,
const char *prop_name, uint32_t fallback)
{
uint32_t len;
const void *v = fdt_find_prop_in_node(dtb, node, prop_name, &len);
if (!v || len != 4) return fallback;
return be32_at((const unsigned char *)v);
}
/**
* @brief Collect every `/reserved-memory` child's static `reg` range(s).
* Children with no `reg` (dynamic reservations -- `size`/`alloc-ranges`
* only) are skipped: there is no allocator this early to service a
* dynamic reservation against, and nothing fixed to carve either way.
*
* @return Number of ranges written (0 if `/reserved-memory` is absent or
* empty of static reservations -- not an error, just nothing to carve).
*/
static uint32_t collect_reserved_ranges(const void *dtb, uint32_t root_acells,
uint32_t root_scells,
rpi5_range_t *out, uint32_t max)
{
const void *rsvmem;
const void *child;
uint32_t acells, scells, entry_bytes, count;
rsvmem = fdt_find_node_by_name(dtb, "reserved-memory");
if (!rsvmem) return 0;
acells = node_cells_or_fallback(dtb, rsvmem, "#address-cells", root_acells);
scells = node_cells_or_fallback(dtb, rsvmem, "#size-cells", root_scells);
if ((acells != 1 && acells != 2) || (scells != 1 && scells != 2)) return 0;
entry_bytes = (acells + scells) * 4u;
count = 0;
child = (const void *)0;
while (count < max &&
(child = fdt_next_child_node(dtb, rsvmem, child)) != (const void *)0)
{
uint32_t reg_len, n, i;
const unsigned char *p;
const void *reg = fdt_find_prop_in_node(dtb, child, "reg", &reg_len);
if (!reg) continue;
n = reg_len / entry_bytes;
p = (const unsigned char *)reg;
for (i = 0; i < n && count < max; i++)
{
out[count].addr = read_cells(p, acells);
out[count].size = read_cells(p + acells * 4u, scells);
count++;
p += entry_bytes;
}
}
return count;
}
/** @brief Insertion sort -- small, fixed N; no libc qsort in freestanding. */
static void sort_ranges_by_addr(rpi5_range_t *arr, uint32_t n)
{
uint32_t i, j;
for (i = 1; i < n; i++)
{
rpi5_range_t key = arr[i];
j = i;
while (j > 0 && arr[j - 1].addr > key.addr)
{
arr[j] = arr[j - 1];
j--;
}
arr[j] = key;
}
}
static void add_memmap_entry(uint32_t *idx, uint32_t max_entries, uint32_t type,
uint64_t addr, uint64_t size)
{
if (*idx >= max_entries) return;
g_rpi5_memmap[*idx].Type = type;
g_rpi5_memmap[*idx].PhysicalStart = addr;
g_rpi5_memmap[*idx].VirtualStart = addr;
/* Floor to whole pages either way: any leftover fractional page is
* simply covered by no descriptor at all, which is safe in both
* directions -- PMM's bitmap starts fully reserved and only clears
* pages an EfiConventionalMemory descriptor explicitly lists
* (memory/pmm.c's Pass 3), so an uncovered fractional page just
* stays reserved by default, same as if it were never mentioned. */
g_rpi5_memmap[*idx].NumberOfPages = size / 4096u;
g_rpi5_memmap[*idx].Attribute = 0;
(*idx)++;
}
/**
* @brief Emit one RAM region as alternating conventional/reserved
* descriptors, clipping @p reserved (already sorted by address) to the
* region's own bounds.
*/
static void emit_region_with_carveouts(uint64_t ram_addr, uint64_t ram_size,
const rpi5_range_t *reserved, uint32_t reserved_count,
uint32_t *idx, uint32_t max_entries)
{
uint64_t cursor = ram_addr;
uint64_t ram_end = ram_addr + ram_size;
uint32_t i;
for (i = 0; i < reserved_count && *idx < max_entries; i++)
{
uint64_t r_start = reserved[i].addr;
uint64_t r_end = r_start + reserved[i].size;
uint64_t clip_start, clip_end;
if (r_end <= cursor) continue; /* fully before the cursor already */
if (r_start >= ram_end) break; /* sorted -- no more overlaps possible */
if (r_start > cursor)
{
add_memmap_entry(idx, max_entries, EfiConventionalMemory,
cursor, r_start - cursor);
}
clip_start = r_start > cursor ? r_start : cursor;
clip_end = r_end < ram_end ? r_end : ram_end;
if (clip_end > clip_start)
{
add_memmap_entry(idx, max_entries, EfiReservedMemoryType,
clip_start, clip_end - clip_start);
cursor = clip_end;
}
}
if (cursor < ram_end)
{
add_memmap_entry(idx, max_entries, EfiConventionalMemory, cursor, ram_end - cursor);
}
}
/**
* @brief Build `g_rpi5_memmap` from `/memory` with `/reserved-memory`'s
* static children carved out. See this file's own top doc comment for
* exactly what is and is not handled.
*
* @return Number of entries written (0 on any failure -- absence of
* `/reserved-memory` itself is not a failure, just zero carve-outs).
*/
static uint32_t build_memory_map(const void *dtb)
{
uint32_t acells, scells;
rpi5_range_t ram_regions[RPI5_MAX_RAM_REGIONS];
rpi5_range_t reserved[RPI5_MAX_RESERVED_RANGES];
uint32_t ram_count, reserved_count, idx, i;
if (!fdt_prop_u32(dtb, "#address-cells", &acells) ||
!fdt_prop_u32(dtb, "#size-cells", &scells))
{
return 0;
}
if ((acells != 1 && acells != 2) || (scells != 1 && scells != 2))
{
return 0;
}
ram_count = collect_ram_regions(dtb, acells, scells, ram_regions, RPI5_MAX_RAM_REGIONS);
if (ram_count == 0) return 0;
reserved_count = collect_reserved_ranges(dtb, acells, scells,
reserved, RPI5_MAX_RESERVED_RANGES);
sort_ranges_by_addr(reserved, reserved_count);
idx = 0;
for (i = 0; i < ram_count; i++)
{
emit_region_with_carveouts(ram_regions[i].addr, ram_regions[i].size,
reserved, reserved_count,
&idx, RPI5_MAX_MEMMAP_ENTRIES);
}
return idx;
}
void rpi5_native_boot(uint64_t dtb)
{
const void *dtb_ptr = (const void *)(uintptr_t)dtb;
uint32_t memmap_count;
const void *bootargs;
uint32_t bootargs_len;
Rpi5FramebufferInfo fb;
if (!fdt_valid(dtb_ptr))
{
rpi5_native_halt();
}
memmap_count = build_memory_map(dtb_ptr);
if (memmap_count == 0)
{
rpi5_native_halt();
}
g_rpi5_boot_info.memory_map = g_rpi5_memmap;
g_rpi5_boot_info.memory_map_size =
(UINTN)memmap_count * sizeof(EFI_MEMORY_DESCRIPTOR);
g_rpi5_boot_info.memory_map_descriptor_size = sizeof(EFI_MEMORY_DESCRIPTOR);
g_rpi5_boot_info.runtime_services = (void *)0;
/* No ACPI at all on this path (§IV.1) -- NULL is the already-correct
* value every existing acpi_table consumer already degrades safely
* against. */
g_rpi5_boot_info.acpi_table = (void *)0;
g_rpi5_boot_info.dtb = (void *)dtb_ptr;
/* No UEFI boot services ever existed on this path -- there is no
* EDK2 here at all, so "exited" is the closest honest value (no
* downstream consumer branches on this field today; it is not a
* live behavioral gate). */
g_rpi5_boot_info.uefi_boot_services_exited = 1;
/* No dynamic allocator exists this early -- fall back to the 2 MiB
* BSS stack, same convention as every other boot path. Unlike
* amd64/riscv64, aarch64 has no kernel_entry.S trampoline to honor
* this field: `native_rpi5_entry.S`'s own BSS stack already *is* the
* stack kernel_main_impl runs on, so this is a documented fact about
* this path, not a live fallback switch. */
g_rpi5_boot_info.kernel_stack_base = (void *)0;
g_rpi5_boot_info.kernel_stack_size = 0;
bootargs = fdt_find_prop(dtb_ptr, "bootargs", &bootargs_len);
cmdline_parse_ascii(bootargs ? (const char *)bootargs : (const char *)0,
&g_rpi5_boot_info.args);
g_rpi5_boot_info.framebuffer.base = (void *)0;
g_rpi5_boot_info.framebuffer.size = 0;
g_rpi5_boot_info.framebuffer.width = 0;
g_rpi5_boot_info.framebuffer.height = 0;
g_rpi5_boot_info.framebuffer.pixels_per_scanline = 0;
g_rpi5_boot_info.framebuffer.pixel_format = 0;
if (rpi5_mailbox_get_framebuffer(dtb_ptr, RPI5_FB_DEFAULT_WIDTH,
RPI5_FB_DEFAULT_HEIGHT,
RPI5_FB_DEFAULT_BPP, &fb) == 0)
{
g_rpi5_boot_info.framebuffer.base = fb.base;
g_rpi5_boot_info.framebuffer.size = fb.size;
g_rpi5_boot_info.framebuffer.width = fb.width;
g_rpi5_boot_info.framebuffer.height = fb.height;
g_rpi5_boot_info.framebuffer.pixels_per_scanline = fb.pixels_per_scanline;
g_rpi5_boot_info.framebuffer.pixel_format = fb.pixel_format;
}
/* On failure, framebuffer stays zeroed -- the same degraded state the
* UEFI path already leaves it in when GOP is BltOnly/absent
* (`boot/uefi_loader.c`). */
kernel_main(&g_rpi5_boot_info);
/* kernel_main() never returns (REPL loop or panic) -- this is
* unreachable, kept only so the noreturn contract holds even if that
* ever changes. */
rpi5_native_halt();
}
kernel/src/arch/aarch64/switch.S
+105
View File
@@ -0,0 +1,105 @@
/*
* switch.S (aarch64) - Cooperative VM context switch primitive.
*
* FABRIC-3.md §XXVIII (Stage 2, preemptive-context-switch save/restore
* primitive, 2026-09-13). Deliberately NOT isr.S's interrupt trap frame
* (Stage 0) -- that saves everything because an interrupt has no call
* site. sk_vm_switch_to() is an ordinary `bl`, so AAPCS64 already makes
* every caller-saved register (x0-x18, v0-v7, v16-v31) the caller's own
* problem across the call. Only the callee-saved set needs explicit
* handling: x19-x28, x29 (fp), x30 (lr), and d8-d15 (only the low 64
* bits of v8-v15 are callee-saved per AAPCS64).
*
* void sk_vm_switch_to(uint64_t *save_sp_out, uint64_t new_sp);
* x0 = save_sp_out, x1 = new_sp
*
* For a VM that has never run before, new_sp must point at a synthetic
* 160-byte frame built by the C glue (switch.c) to this exact layout,
* with x30 (lr) set to the trampoline entry address and everything else
* zeroed, so the final `ret` below (which branches to lr) jumps there.
*/
.section .text
.balign 4
.equ SW_X19, 0
.equ SW_X21, 16
.equ SW_X23, 32
.equ SW_X25, 48
.equ SW_X27, 64
.equ SW_X29, 80 /* x29 (fp), x30 (lr) */
.equ SW_D8, 96
.equ SW_D10, 112
.equ SW_D12, 128
.equ SW_D14, 144
.equ SW_SIZE, 160
/* .hidden is an ELF-only visibility directive with no COFF equivalent;
* this file is also assembled for the aarch64 loader, which clang targets
* at aarch64-pc-windows-msvc (PE/COFF) -- guard on _WIN32 (defined only by
* that target), matching isr.S's own precedent exactly. */
#ifndef _WIN32
.hidden sk_vm_switch_to
#endif
.global sk_vm_switch_to
sk_vm_switch_to:
sub sp, sp, #SW_SIZE
stp x19, x20, [sp, #SW_X19]
stp x21, x22, [sp, #SW_X21]
stp x23, x24, [sp, #SW_X23]
stp x25, x26, [sp, #SW_X25]
stp x27, x28, [sp, #SW_X27]
stp x29, x30, [sp, #SW_X29]
stp d8, d9, [sp, #SW_D8]
stp d10, d11, [sp, #SW_D10]
stp d12, d13, [sp, #SW_D12]
stp d14, d15, [sp, #SW_D14]
mov x2, sp
str x2, [x0] /* *save_sp_out = sp (this context's parked SP) */
mov sp, x1 /* sp = new_sp (switch stacks) */
ldp x19, x20, [sp, #SW_X19]
ldp x21, x22, [sp, #SW_X21]
ldp x23, x24, [sp, #SW_X23]
ldp x25, x26, [sp, #SW_X25]
ldp x27, x28, [sp, #SW_X27]
ldp x29, x30, [sp, #SW_X29]
ldp d8, d9, [sp, #SW_D8]
ldp d10, d11, [sp, #SW_D10]
ldp d12, d13, [sp, #SW_D12]
ldp d14, d15, [sp, #SW_D14]
add sp, sp, #SW_SIZE
ret
/*
* uint64_t sk_vm_switch_prime(uint64_t stack_top, void *entry);
*
* Builds the synthetic frame above at the top of a fresh native stack, so
* the first ever sk_vm_switch_to() into it lands at `entry` (x30/lr) via
* its final `ret`. Kept in this file (not C) so the layout can never
* drift from sk_vm_switch_to() above -- one set of offsets, not two.
*/
#ifndef _WIN32
.hidden sk_vm_switch_prime
#endif
.global sk_vm_switch_prime
sk_vm_switch_prime:
sub x2, x0, #SW_SIZE
stp xzr, xzr, [x2, #SW_X19]
stp xzr, xzr, [x2, #SW_X21]
stp xzr, xzr, [x2, #SW_X23]
stp xzr, xzr, [x2, #SW_X25]
stp xzr, xzr, [x2, #SW_X27]
stp xzr, x1, [x2, #SW_X29] /* x29=0, x30(lr)=entry */
stp xzr, xzr, [x2, #SW_D8]
stp xzr, xzr, [x2, #SW_D10]
stp xzr, xzr, [x2, #SW_D12]
stp xzr, xzr, [x2, #SW_D14]
mov x0, x2
ret
kernel/src/arch/aarch64/timer.c
+275
View File
@@ -0,0 +1,275 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James. All rights reserved.
Licensed under the StarForth License, Version 1.0.
*/
/**
* timer.c (aarch64) - Timer subsystem using ARM Generic Timer (CNTPCT_EL0)
*
* Uses the 64-bit system counter available as CNTPCT_EL0. The counter
* frequency is read from CNTFRQ_EL0 (set by firmware).
*/
#include "timer.h"
#include "console.h"
#include "q48_16.h"
#include "uefi.h"
#include <stdint.h>
#include <string.h>
/* ─── ARM generic-timer helpers ─────────────────────────────────────── */
/* FABRIC-2.md §I.5, 2026-09-04: real hypervisor-vs-hardware detection.
* s_cal.vm_mode was hardcoded to 1 unconditionally below (comment:
* "QEMU SBSA always uses virtualised Generic Timer") -- true for the
* *timing policy* this file cares about, but wrong to reuse as a
* general "are we in QEMU" signal elsewhere (contrib-capsule trust-tier
* enforcement, §I.5), which is exactly what happened before this was
* caught: it would report "always QEMU" on real hardware too.
*
* ARM has no single CPUID-equivalent hypervisor-present bit the way
* amd64's CPUID.1:ECX[31] is (arch/amd64/timer.c's own
* running_under_hypervisor()). Checks the ACPI RSDP's OEM ID instead --
* "BOCHS " is the SeaBIOS-heritage signature QEMU's ACPI table
* generation has used since long before this project, on every machine
* type it emulates, aarch64 virt included. Real hardware vendors set
* their own OEMID, never this string. Mirrors pci.c's own private Rsdp2
* struct layout (signature[8], checksum, oem_id[6]) -- duplicated
* rather than shared, same "a few TLV-walking lines is simpler than a
* new header" precedent tools/pkcs8_ed25519.c already set. */
static int running_under_hypervisor(BootInfo *boot_info) {
if (!boot_info || !boot_info->acpi_table) return 0;
const uint8_t *rsdp = (const uint8_t *) boot_info->acpi_table;
if (memcmp(rsdp, "RSD PTR ", 8) != 0) return 0;
return memcmp(rsdp + 9, "BOCHS ", 6) == 0 ? 1 : 0;
}
/**
* @brief Read the AArch64 Physical System Counter with ISB serialisation.
*
* Issues an @c ISB (Instruction Synchronisation Barrier) before reading
* @c CNTPCT_EL0 to ensure all preceding instructions have completed before
* the counter value is sampled. This prevents the counter read from being
* speculated ahead of a preceding store or flag update, which would produce
* an erroneously early timestamp.
*
* The ISB is heavier than strictly necessary for most timing uses but
* provides the same ordering guarantee as @c RDTSCP on x86-64. For
* interval measurements where start and end reads are in the same path,
* the overhead (1–3 cycles) is negligible.
*
* @return Current 64-bit @c CNTPCT_EL0 counter value in system-counter ticks.
*/
static inline uint64_t cntpct_read(void)
{
uint64_t val;
__asm__ volatile ("isb; mrs %0, cntpct_el0" : "=r"(val));
return val;
}
/**
* @brief Read the AArch64 Counter Frequency Register (@c CNTFRQ_EL0).
*
* @c CNTFRQ_EL0 is a read-only EL0-accessible register whose value is
* written by EL3/EL2 firmware at boot to advertise the system counter
* frequency in Hz. It is the architectural way to discover @c CNTPCT_EL0's
* tick rate without out-of-band knowledge of the board.
*
* QEMU's SBSA/virt machine models typically set this to 62,500,000 Hz
* (62.5 MHz) for Cortex-A57 compatibility. Real hardware varies widely
* (10 MHz on Raspberry Pi 4, 24 MHz on Cortex-A53 dev boards, etc.).
* @c timer_init() falls back to 62,500,000 if the register reads zero.
*
* @return Counter frequency in Hz as set by firmware; 0 if firmware did
* not initialise the register (uncommon but possible in bare-metal
* bring-up without a proper firmware stack).
*/
static inline uint64_t cntfrq_read(void)
{
uint64_t val;
__asm__ volatile ("mrs %0, cntfrq_el0" : "=r"(val));
return val;
}
/* ─── Module state ───────────────────────────────────────────────────── */
static uint64_t s_counter_hz = 0; /* CNTFRQ_EL0 (ticks/second) */
static uint64_t s_ns_per_tick = 0; /* nanoseconds per counter tick (scaled) */
static uint64_t s_base_count = 0; /* counter value at timer_init() */
static uint64_t s_base_ns = 0; /* ns offset at timer_init() */
static timer_calibration_record_t s_cal;
/* ─── timer_init ─────────────────────────────────────────────────────── */
/*
* @brief Initialise the AArch64 timer subsystem from @c CNTFRQ_EL0 (M5).
*
* Reads the system counter frequency from @c CNTFRQ_EL0. If firmware left
* this register zero (a non-conforming but possible situation in bare-metal
* bring-up) the function falls back to 62,500,000 Hz — the canonical
* Cortex-A57 / QEMU SBSA frequency.
*
* Derived values stored in module state:
* - @c s_counter_hz — raw frequency in Hz.
* - @c s_base_count — @c CNTPCT_EL0 sample at init time; all subsequent
* @c timer_now_ns() calls compute elapsed ticks as
* @c (cntpct_read() - s_base_count).
* - @c s_base_ns — 0 at init; the ns origin for @c timer_now_ns().
* - @c s_ns_per_tick — Q16.16 fixed-point nanoseconds per counter tick,
* computed as @c (1e9 << 16) / @c freq to avoid floating-point in the
* freestanding build. @c timer_now_ns() right-shifts by 16 after the
* multiply to recover the integer nanosecond count.
*
* The calibration record @c s_cal is populated with the firmware-reported
* frequency, @c converged = 1, and @c trust = @c TIMER_TRUST_ABSOLUTE
* because the ARM Generic Timer is invariant and synchronised by
* architecture — no drift-detection loop is needed.
*
* @param boot_info Kernel @c BootInfo -- GIC base address will be used in a
* later milestone for the interrupt controller; as of
* 2026-09-04 also consulted here for @c acpi_table, so
* @c s_cal.vm_mode can be a real hypervisor-vs-hardware
* detection instead of the hardcoded-1 timing-policy
* shortcut this field used to double as (see
* running_under_hypervisor()'s own doc comment above).
* @return 0 always.
*/
int timer_init(BootInfo *boot_info)
{
uint64_t freq = cntfrq_read();
if (freq == 0) {
/* Firmware did not set CNTFRQ; assume 62.5 MHz (Cortex-A57 default) */
freq = 62500000ULL;
}
s_counter_hz = freq;
s_base_count = cntpct_read();
s_base_ns = 0;
/* ns_per_tick = 1e9 / freq — store as (1e9 << 16) / freq for fixed-point */
s_ns_per_tick = (1000000000ULL << 16) / freq;
s_cal.tsc_hz_mean = freq;
s_cal.hpet_hz = 0;
s_cal.pit_hz_mean = 0;
s_cal.converged = 1;
s_cal.vm_mode = running_under_hypervisor(boot_info) ? 1 : 0;
s_cal.trust = TIMER_TRUST_ABSOLUTE;
console_println("Timer: AArch64 generic timer initialised.");
return 0;
}
/* ─── timer_tsc_hz ───────────────────────────────────────────────────── */
/**
* @brief Return the system counter frequency in Hz.
*
* Returns @c s_counter_hz as set from @c CNTFRQ_EL0 by @c timer_init().
* If @c timer_init() has not been called yet (or firmware reported 0 and
* the fallback was not applied), returns 62,500,000 Hz as a safe default.
*
* Used by @c apic_timer_init() to compute @c s_timer_period_tsc and by
* the shim time backend to convert counter ticks to nanoseconds before
* the full timer subsystem is available.
*
* @return System counter frequency in Hz; at least 62,500,000.
*/
uint64_t timer_tsc_hz(void)
{
return s_counter_hz ? s_counter_hz : 62500000ULL;
}
/* ─── timer_now_ns ───────────────────────────────────────────────────── */
/**
* @brief Return a monotonic nanosecond timestamp.
*
* Reads @c CNTPCT_EL0, computes the delta from @c s_base_count (captured
* at @c timer_init()), and converts to nanoseconds using the Q16.16
* fixed-point @c s_ns_per_tick:
*
* @code
* ns = s_base_ns + ((delta * s_ns_per_tick) >> 16)
* @endcode
*
* The multiplication uses 64-bit arithmetic; at typical counter frequencies
* (≤ 100 MHz) and reasonable uptime intervals (days to weeks) there is no
* overflow risk in the delta-to-ns conversion.
*
* @return Monotonic nanosecond count since @c timer_init(), starting at 0.
*/
uint64_t timer_now_ns(void)
{
uint64_t delta = cntpct_read() - s_base_count;
/* delta * (1e9 / freq) = (delta * ns_per_tick_q16) >> 16 */
return s_base_ns + ((delta * s_ns_per_tick) >> 16);
}
/* ─── timer_check_drift_now ──────────────────────────────────────────── */
/**
* @brief Check for timer drift (AArch64 stub — always returns 0).
*
* On x86-64 this function cross-checks the TSC against the HPET or PM
* Timer to detect frequency drift. The AArch64 Generic Timer is
* architecturally invariant and synchronised across all cores in the
* system; it cannot drift against itself. This stub always returns 0
* (no drift detected) to satisfy the common @c timer_check_drift_now()
* call site.
*
* @return 0 always.
*/
int timer_check_drift_now(void)
{
return 0; /* Stub: ARM counter has no drift against itself */
}
/* ─── timer_calibration_record ───────────────────────────────────────── */
/**
* @brief Return a pointer to the timer calibration record.
*
* Returns @c &s_cal which is populated by @c timer_init() with:
* - @c tsc_hz_mean = @c CNTFRQ_EL0 (or 62.5 MHz fallback)
* - @c hpet_hz = 0 (no HPET on AArch64)
* - @c pit_hz_mean = 0 (no PIT on AArch64)
* - @c converged = 1 (Generic Timer is already calibrated by firmware)
* - @c vm_mode = real hypervisor-vs-hardware detection as of 2026-09-04
* (ACPI RSDP OEM ID check, running_under_hypervisor() above) -- the
* Generic Timer itself works identically either way, so this field's
* *timing* meaning here is purely informational, unlike amd64 where
* vm_mode gates a real calibration-path choice
* - @c trust = @c TIMER_TRUST_ABSOLUTE
*
* The record is exposed to @c kernel_main() and @c timer.h consumers for
* boot-log reporting and for the heartbeat subsystem's trust initialisation.
*
* @return Pointer to the module-static calibration record; valid for the
* lifetime of the kernel.
*/
const timer_calibration_record_t *timer_calibration_record(void)
{
return &s_cal;
}
/* ─── Heartbeat ──────────────────────────────────────────────────────── */
/**
* @brief Read the raw counter the aarch64 heartbeat is paced against.
*
* Item 0.8 (FABRIC-0.md §25.1): the shared heartbeat.c now owns
* heartbeat_init()/heartbeat_tick()/heartbeat_service()/heartbeat_ticks()/
* heartbeat_trust()/heartbeat_state() and the per-arch @c g_heartbeat
* state that used to live in this file. This is the one piece that stays
* per-architecture -- the same @c CNTPCT_EL0 the rest of this file's
* calibration already reads via @c cntpct_read().
*
* @return Current @c CNTPCT_EL0 value.
*/
uint64_t heartbeat_read_counter(void)
{
return cntpct_read();
}
kernel/src/arch/amd64/README.md
+11
View File
@@ -0,0 +1,11 @@
# src/starkernel/arch/amd64/
AMD64 architecture support for LithosAnanke.
- `arch.c` — architecture init entry point.
- `apic.c` — Local APIC interrupt controller driver.
- `timer.c` — TSC/HPET/APIC timer setup (100 Hz heartbeat tick).
- `interrupts.c` — IDT setup and interrupt dispatch.
- `boot.S` — early boot assembly.
- `kernel_entry.S` — kernel entry trampoline invoked from the loader.
- `isr.S` — interrupt service routine stubs.
kernel/src/arch/amd64/apic.c
+662
View File
@@ -0,0 +1,662 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James
All rights reserved.
This file is part of the StarForth project.
Licensed under the StarForth License, Version 1.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at:
https://github.com/star.4th@proton.me/StarForth/LICENSE.txt
This software is provided "AS IS", WITHOUT WARRANTY OF ANY KIND,
express or implied, including but not limited to the warranties of
merchantability, fitness for a particular purpose, and noninfringement.
See the License for the specific language governing permissions and
limitations under the License.
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James
All rights reserved.
This file is part of the StarForth project.
Licensed under the StarForth License, Version 1.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at:
https://github.com/star.4th@proton.me/StarForth/LICENSE.txt
This software is provided "AS IS", WITHOUT WARRANTY OF ANY KIND,
express or implied, including but not limited to the warranties of
merchantability, fitness for a particular purpose, and noninfringement.
See the License for the specific language governing permissions and
limitations under the License.
*/
/**
* apic.c - Local APIC and Timer (amd64)
*
* Provides:
* - APIC initialization
* - APIC timer for periodic heartbeat interrupts
*/
#include <stdint.h>
#include "apic.h"
#include "console.h"
#include "vmm.h"
#include "arch.h"
#include "starkernel/timer.h"
/* ============================================================================
* APIC Register Offsets
* ============================================================================ */
#define LAPIC_DEFAULT_PHYS 0xFEE00000u
#define LAPIC_VIRT_BASE 0xFEE00000u /* identity-mapped */
/* IA32_APIC_BASE MSR */
#define IA32_APIC_BASE_MSR 0x1Bu
#define IA32_APIC_BASE_GLOBAL_EN (1ull << 11) /* Global APIC hardware enable */
#define IA32_APIC_BASE_X2APIC_EN (1ull << 10) /* x2APIC mode enable */
/* Core registers */
#define APIC_REG_ID 0x020 /* APIC ID */
#define APIC_REG_TPR 0x080 /* Task Priority Register */
#define APIC_REG_EOI 0x0B0 /* End of Interrupt */
#define APIC_REG_SIVR 0x0F0 /* Spurious Interrupt Vector Register */
/* Timer registers */
#define APIC_REG_LVT_TIMER 0x320 /* LVT Timer Register */
#define APIC_REG_TIMER_ICR 0x380 /* Timer Initial Count Register */
#define APIC_REG_TIMER_CCR 0x390 /* Timer Current Count Register */
#define APIC_REG_TIMER_DCR 0x3E0 /* Timer Divide Configuration Register */
/* LVT Timer bits */
#define LVT_MASKED (1u << 16) /* Interrupt masked */
#define LVT_MODE_PERIODIC (1u << 17) /* Periodic mode */
#define LVT_MODE_ONESHOT 0 /* One-shot mode (default) */
/* Divide configuration values (for APIC_REG_TIMER_DCR) */
#define TIMER_DIV_1 0x0B /* Divide by 1 */
#define TIMER_DIV_2 0x00 /* Divide by 2 */
#define TIMER_DIV_4 0x01 /* Divide by 4 */
#define TIMER_DIV_8 0x02 /* Divide by 8 */
#define TIMER_DIV_16 0x03 /* Divide by 16 */
#define TIMER_DIV_32 0x08 /* Divide by 32 */
#define TIMER_DIV_64 0x09 /* Divide by 64 */
#define TIMER_DIV_128 0x0A /* Divide by 128 */
/* ============================================================================
* State
* ============================================================================ */
static volatile uint32_t *lapic_base = (volatile uint32_t *)(uintptr_t)LAPIC_VIRT_BASE;
static uint64_t lapic_phys_base = LAPIC_DEFAULT_PHYS;
static uint32_t timer_initial_count = 0;
static uint64_t timer_period_tsc_ticks = 0;
static uint32_t timer_tick_hz = 0;
static uint32_t s_apic_hz = 0; /* Calibrated APIC timer frequency (item 0.8/§26) */
/* ============================================================================
* MSR access (freestanding — no libgcc, no libc)
* ============================================================================ */
/**
* @brief Read a 64-bit Model Specific Register via @c RDMSR.
*
* Executes the @c RDMSR instruction with @p msr in @c ECX. The processor
* returns the high 32 bits in @c EDX and the low 32 bits in @c EAX; this
* function combines them into a single 64-bit value. Must run at CPL 0
* (ring 0); a #GP fault results if executed in user mode or if @p msr
* denotes a reserved or non-existent register.
*
* @param msr 32-bit MSR index (e.g., @c IA32_APIC_BASE_MSR = 0x1B).
* @return Current 64-bit value of the specified MSR.
*/
static uint64_t rdmsr64(uint32_t msr)
{
uint32_t lo, hi;
__asm__ volatile ("rdmsr" : "=a"(lo), "=d"(hi) : "c"(msr));
return ((uint64_t)hi << 32) | lo;
}
/**
* @brief Write a 64-bit value to a Model Specific Register via @c WRMSR.
*
* Executes the @c WRMSR instruction with @p msr in @c ECX, the high 32 bits
* of @p val in @c EDX, and the low 32 bits in @c EAX. Must run at CPL 0;
* a #GP results for reserved, non-existent, or read-only MSRs, or for writes
* that set reserved bits. Used here exclusively to manipulate
* @c IA32_APIC_BASE_MSR to switch from x2APIC to xAPIC mode and to assert
* the global hardware-enable bit.
*
* @param msr 32-bit MSR index.
* @param val 64-bit value to write.
*/
static void wrmsr64(uint32_t msr, uint64_t val)
{
__asm__ volatile ("wrmsr"
: : "a"((uint32_t)(val & 0xFFFFFFFFu)),
"d"((uint32_t)(val >> 32)),
"c"(msr));
}
/* ============================================================================
* Low-level APIC access
* ============================================================================ */
/**
* @brief Write a 32-bit value to a Local APIC MMIO register.
*
* All xAPIC registers are 32-bit wide and 16-byte aligned within the
* 4 KB MMIO page at @c lapic_base. The register offset @p reg is a
* byte offset (e.g., @c APIC_REG_EOI = 0x0B0); dividing by 4 converts
* it to a @c uint32_t array index. Writes must be 32-bit aligned and
* must not be merged or reordered by the compiler, hence the
* @c volatile qualifier on @c lapic_base.
*
* @param reg Byte offset of the APIC register (e.g., @c APIC_REG_EOI).
* @param val 32-bit value to write.
*/
static void lapic_write(uint32_t reg, uint32_t val) {
lapic_base[reg / 4] = val;
}
/**
* @brief Read a 32-bit value from a Local APIC MMIO register.
*
* Performs a @c volatile 32-bit MMIO load from the register at byte
* offset @p reg within the LAPIC page. The @c volatile qualifier on
* @c lapic_base prevents the compiler from caching the value across
* reads, which is essential for registers that change asynchronously
* (e.g., @c APIC_REG_TIMER_CCR which counts down independently of the
* CPU instruction stream).
*
* @param reg Byte offset of the APIC register (e.g., @c APIC_REG_TIMER_CCR).
* @return Current 32-bit value of the register.
*/
static uint32_t lapic_read(uint32_t reg) {
return lapic_base[reg / 4];
}
/* ============================================================================
* APIC Initialization
* ============================================================================ */
/**
* @brief Initialise the Local APIC in xAPIC MMIO mode.
*
* Called during M4 (interrupt controller init) before the IDT is loaded.
* Performs the following steps:
*
* 1. Reads @c IA32_APIC_BASE_MSR (0x1B) to determine the current APIC mode.
* 2. If x2APIC mode (bit 10) is active, clears it to switch to xAPIC MMIO mode
* while keeping the global-enable bit (bit 11) set, as required by Intel
* SDM Vol.3 §10.12.5 (never clear both bits simultaneously).
* 3. If the global-enable bit (bit 11) was clear, sets it.
* 4. Software-enables the APIC via @c APIC_REG_SIVR (Spurious Interrupt Vector
* Register) and assigns spurious vector 0xFF.
* 5. Clears the Task Priority Register (TPR=0) so all interrupt priorities
* are delivered; a non-zero TPR left by UEFI would silently suppress all
* APIC interrupts at or below that priority level.
* 6. Issues a spurious EOI write to clear any stale in-service interrupt
* that UEFI may have left pending (specifically the APIC timer vector
* that fires just before @c ExitBootServices).
*
* The @p boot_info parameter is reserved for future use (e.g., reading the
* ACPI MADT for the physical LAPIC base address); it is currently unused and
* the default identity-mapped address @c LAPIC_DEFAULT_PHYS = 0xFEE00000 is
* always used.
*
* @param boot_info Pointer to kernel @c BootInfo (ACPI/memory-map data);
* currently unused, reserved for MADT-based LAPIC relocation.
* @return 0 on success (always; the APIC is assumed to be present on x86-64).
*/
int apic_init(BootInfo *boot_info) {
(void)boot_info;
lapic_phys_base = LAPIC_DEFAULT_PHYS;
lapic_base = (volatile uint32_t *)(uintptr_t)LAPIC_VIRT_BASE;
/*
* Read IA32_APIC_BASE MSR. UEFI (OVMF) may have left the APIC in x2APIC
* mode (bit 10) or may have left the global hardware enable (bit 11) in an
* indeterminate state. We need xAPIC MMIO mode with global enable set.
*/
uint64_t apic_base_msr = rdmsr64(IA32_APIC_BASE_MSR);
console_puts("APIC: IA32_APIC_BASE MSR=0x");
{
/* print upper 32 bits then lower 32 bits */
uint32_t hi = (uint32_t)(apic_base_msr >> 32);
uint32_t lo = (uint32_t)(apic_base_msr & 0xFFFFFFFFu);
for (int s = 28; s >= 0; s -= 4)
console_putc("0123456789abcdef"[(hi >> s) & 0xF]);
for (int s = 28; s >= 0; s -= 4)
console_putc("0123456789abcdef"[(lo >> s) & 0xF]);
}
console_putc('\n');
if (apic_base_msr & IA32_APIC_BASE_X2APIC_EN) {
/*
* x2APIC is active. Switching back to xAPIC requires: disable x2APIC
* (clear bit 10) while keeping global enable (bit 11) set.
* Intel SDM Vol.3 §10.12.5: software must not clear both bits at once.
*/
console_puts("APIC: x2APIC active — switching to xAPIC MMIO mode\n");
apic_base_msr &= ~IA32_APIC_BASE_X2APIC_EN;
wrmsr64(IA32_APIC_BASE_MSR, apic_base_msr);
}
if (!(apic_base_msr & IA32_APIC_BASE_GLOBAL_EN)) {
console_puts("APIC: global enable was clear — asserting bit 11\n");
apic_base_msr |= IA32_APIC_BASE_GLOBAL_EN;
wrmsr64(IA32_APIC_BASE_MSR, apic_base_msr);
}
/* Software-enable APIC and set spurious vector */
const uint8_t SPURIOUS_VECTOR = 0xFF;
const uint32_t APIC_ENABLE = (1u << 8);
lapic_write(APIC_REG_SIVR, APIC_ENABLE | SPURIOUS_VECTOR);
/*
* Clear Task Priority Register. If UEFI left TPR non-zero, all APIC
* interrupts at or below that priority are silently suppressed — the timer
* ISR never fires. TPR=0 allows all priorities through.
*/
lapic_write(APIC_REG_TPR, 0);
/*
* Drain ALL stale in-service bits left by UEFI firmware, not just one.
* UEFI's own interrupt handling commonly uses the same 0x20-based
* legacy-IRQ vector remap convention this kernel does, and can leave
* MORE THAN ONE vector "in service" when ExitBootServices() cuts it
* off mid-handling (its own timer tick AND, per item 4.3.5's finding,
* its own keyboard IRQ1 handling -- both land in the same 0x20-0x2F
* range). A single EOI clears only the highest-priority stale bit;
* firmware's other stale bit then sits permanently in-service and
* silently demotes every future same-or-lower-class interrupt to the
* spurious vector instead of its real one. EOI always clears whichever
* ISR bit is currently highest-priority, so looping while any of the
* 8 ISR registers (0x100-0x170, vectors 0-255) is nonzero drains all
* of them. Bounded to 8 iterations -- one firmware is not expected to
* leave more stale bits than that; if it does, stop rather than spin
* forever on real hardware.
*/
{
#define APIC_REG_ISR_BASE 0x100
int drained;
for (drained = 0; drained < 8; drained++) {
int any_set = 0;
for (int r = 0; r < 8; r++) {
if (lapic_read(APIC_REG_ISR_BASE + (uint32_t)r * 0x10) != 0) {
any_set = 1;
break;
}
}
if (!any_set) break;
lapic_write(APIC_REG_EOI, 0);
}
console_puts("APIC: stale-ISR drain: ");
{
char buf[8]; int i = 0; uint32_t v = (uint32_t)drained;
if (v == 0) buf[i++] = '0';
else { char tmp[8]; int j = 0; while (v > 0) { tmp[j++] = (char)('0' + (v % 10)); v /= 10; } while (j > 0) buf[i++] = tmp[--j]; }
buf[i] = '\0';
console_puts(buf);
}
console_puts(" EOI(s) issued\n");
}
console_puts("APIC: initialized (xAPIC MMIO, TPR=0, SIVR=0x1FF, EOI-clear)\n");
return 0;
}
/**
* @brief Signal End of Interrupt to the Local APIC.
*
* Writes zero to @c APIC_REG_EOI (offset 0x0B0). This informs the APIC
* that the currently serviced interrupt has been handled so that the next
* interrupt in the vector's priority class can be delivered. Must be called
* at the end of every interrupt service routine before @c IRET; omitting
* the EOI stalls all future interrupt delivery at the same or lower priority.
*
* For the APIC timer heartbeat path this is called inside
* @c isr_common_handler() immediately after @c heartbeat_tick().
*/
void apic_eoi(void) {
lapic_write(APIC_REG_EOI, 0);
}
/**
* @brief Return this CPU's xAPIC ID.
*
* Reads @c APIC_REG_ID and extracts bits 31:24, the xAPIC ID field. Used by
* the I/O APIC driver (item 4.3.5) as the destination field when programming
* a redirection entry — interrupts routed to this ID are delivered to the
* current CPU's Local APIC.
*
* @return This CPU's 8-bit xAPIC ID.
*/
uint8_t apic_id(void) {
return (uint8_t)(lapic_read(APIC_REG_ID) >> 24);
}
/* ============================================================================
* APIC Timer
* ============================================================================ */
/**
* @brief Calibrate the APIC timer against the TSC to determine its frequency.
*
* The APIC timer runs from an internal bus-clock-derived source that has no
* fixed architectural relationship to the TSC. This function measures the
* APIC timer frequency empirically using a TSC-timed spin loop:
*
* 1. Configures the APIC timer DCR (Divide Configuration Register) to
* divisor 1 for maximum resolution.
* 2. Masks the LVT timer entry so no interrupt fires during calibration.
* 3. Loads the maximum initial count (0xFFFFFFFF) so the counter does not
* wrap during the measurement window.
* 4. Spins for a calibration window of @c tsc_hz/100 TSC ticks (≈ 10 ms)
* using @c arch_read_timestamp(). Falls back to 10,000,000 cycles if
* @p tsc_hz is not yet known.
* 5. Reads @c APIC_REG_TIMER_CCR and subtracts from 0xFFFFFFFF to get the
* number of APIC ticks that elapsed in the window.
* 6. Scales by @c tsc_hz / calibration_tsc_ticks to obtain APIC Hz.
*
* If @p tsc_hz is 0 (TSC not yet calibrated), the APIC count is multiplied
* by 100 as a rough fallback (assumes the 10M-cycle window was ≈ 10 ms on
* a 1 GHz core — good enough for initial bringup on QEMU TCG).
*
* @param tsc_hz TSC frequency in Hz from @c timer_init(); 0 if not yet known.
* @return Measured APIC timer frequency in ticks per second (APIC Hz),
* or 0 if calibration could not produce a meaningful result.
*/
static uint32_t calibrate_apic_timer(uint64_t tsc_hz) {
/* Use divisor 1 for maximum resolution */
lapic_write(APIC_REG_TIMER_DCR, TIMER_DIV_1);
/* Mask timer (disable interrupts during calibration) */
lapic_write(APIC_REG_LVT_TIMER, LVT_MASKED);
/* Set initial count to max */
lapic_write(APIC_REG_TIMER_ICR, 0xFFFFFFFFu);
/* Measure TSC for ~10ms (or less if tsc_hz is unknown) */
uint64_t calibration_tsc_ticks;
if (tsc_hz > 0) {
calibration_tsc_ticks = tsc_hz / 100; /* 10ms */
} else {
calibration_tsc_ticks = 10000000; /* Fallback: ~10M cycles */
}
uint64_t tsc_start = arch_read_timestamp();
uint64_t tsc_end = tsc_start + calibration_tsc_ticks;
/* Spin until TSC reaches target */
while (arch_read_timestamp() < tsc_end) {
arch_relax();
}
/* Read how many APIC ticks elapsed */
uint32_t apic_elapsed = 0xFFFFFFFFu - lapic_read(APIC_REG_TIMER_CCR);
/* Stop timer */
lapic_write(APIC_REG_TIMER_ICR, 0);
/* Compute APIC ticks per second */
/* apic_hz = apic_elapsed * (1 / calibration_time_seconds) */
/* apic_hz = apic_elapsed * (tsc_hz / calibration_tsc_ticks) */
uint64_t apic_hz;
if (tsc_hz > 0 && calibration_tsc_ticks > 0) {
apic_hz = ((uint64_t)apic_elapsed * tsc_hz) / calibration_tsc_ticks;
} else {
/* Rough fallback: assume APIC runs at ~1 GHz */
apic_hz = (uint64_t)apic_elapsed * 100;
}
return (uint32_t)(apic_hz & 0xFFFFFFFFu);
}
/**
* @brief Initialise and configure the APIC timer for periodic heartbeat delivery.
*
* Calibrates the APIC timer frequency via @c calibrate_apic_timer() then
* programs the LVT Timer Register and Initial Count Register to deliver
* @c APIC_TIMER_VECTOR periodically at @p tick_hz interrupts per second.
* The timer is left @b masked after initialisation; call @c apic_timer_start()
* after the IDT is loaded and interrupts are enabled to begin delivery.
*
* Derived quantities stored for later use:
* - @c timer_initial_count — APIC ticks per heartbeat period
* (@c apic_hz / @p tick_hz), written to @c APIC_REG_TIMER_ICR.
* - @c timer_tick_hz — requested tick rate (100 Hz default).
* - @c timer_period_tsc_ticks — expected TSC ticks per heartbeat
* (@c tsc_hz / @p tick_hz), used by @c heartbeat_tick() to compute
* inter-tick variance for the TIME-TRUST Q48.16 metric.
*
* If @p tick_hz is 0 it defaults to 100 Hz (10 ms period). If calibration
* returns 0 (no APIC timer detected or counter stuck) the function prints
* a diagnostic and returns -1 without programming the timer.
*
* @param tsc_hz TSC frequency in Hz (from @c timer_init()); used to derive
* the 10 ms calibration window and @c timer_period_tsc_ticks.
* Pass 0 if TSC is not yet calibrated.
* @param tick_hz Desired interrupt rate in Hz; 0 → defaults to 100 Hz.
* @return 0 on success, -1 if APIC timer calibration failed.
*/
int apic_timer_init(uint64_t tsc_hz, uint32_t tick_hz) {
if (tick_hz == 0) {
tick_hz = 100; /* Default: 100 Hz (10ms period) */
}
console_puts("APIC Timer: calibrating...\r\n");
/* Calibrate to find APIC timer frequency */
uint32_t apic_hz = calibrate_apic_timer(tsc_hz);
if (apic_hz == 0) {
console_puts("APIC Timer: calibration failed!\r\n");
return -1;
}
/* Compute initial count for desired tick rate */
timer_initial_count = apic_hz / tick_hz;
timer_tick_hz = tick_hz;
s_apic_hz = apic_hz;
/* Compute expected TSC ticks per heartbeat (for TIME-TRUST variance) */
if (tsc_hz > 0) {
timer_period_tsc_ticks = tsc_hz / tick_hz;
} else {
timer_period_tsc_ticks = 0; /* Unknown */
}
console_puts("APIC Timer: apic_hz=");
/* Simple decimal print for debugging */
{
char buf[32];
uint64_t v = apic_hz;
int i = 0;
if (v == 0) buf[i++] = '0';
else {
char tmp[32];
int j = 0;
while (v > 0) { tmp[j++] = '0' + (v % 10); v /= 10; }
while (j > 0) buf[i++] = tmp[--j];
}
buf[i] = '\0';
console_puts(buf);
}
console_puts(", tick_hz=");
{
char buf[32];
uint64_t v = tick_hz;
int i = 0;
if (v == 0) buf[i++] = '0';
else {
char tmp[32];
int j = 0;
while (v > 0) { tmp[j++] = '0' + (v % 10); v /= 10; }
while (j > 0) buf[i++] = tmp[--j];
}
buf[i] = '\0';
console_puts(buf);
}
console_puts(", initial_count=");
{
char buf[32];
uint64_t v = timer_initial_count;
int i = 0;
if (v == 0) buf[i++] = '0';
else {
char tmp[32];
int j = 0;
while (v > 0) { tmp[j++] = '0' + (v % 10); v /= 10; }
while (j > 0) buf[i++] = tmp[--j];
}
buf[i] = '\0';
console_puts(buf);
}
console_println("");
/* Configure timer: periodic mode, our vector, initially masked */
lapic_write(APIC_REG_TIMER_DCR, TIMER_DIV_1);
lapic_write(APIC_REG_LVT_TIMER, LVT_MASKED | LVT_MODE_PERIODIC | APIC_TIMER_VECTOR);
lapic_write(APIC_REG_TIMER_ICR, timer_initial_count);
console_puts("APIC Timer: configured (masked, ready to start)\r\n");
return 0;
}
/**
* @brief Unmask and re-arm the APIC timer to begin periodic interrupt delivery.
*
* Called after @c arch_interrupts_init() and @c arch_enable_interrupts()
* to start the 100 Hz heartbeat that drives @c heartbeat_tick(). The
* sequence is:
*
* 1. Read the current LVT Timer Register.
* 2. Clear the @c LVT_MASKED bit (bit 16) to unmask the interrupt.
* 3. Write the LVT register back.
* 4. Write @c timer_initial_count to @c APIC_REG_TIMER_ICR to (re-)arm the
* countdown. QEMU TCG requires an explicit ICR write after unmask — it
* does not auto-restart a previously masked periodic timer. On real
* hardware this is harmless and equally correct.
* 5. Reads back LVT and ICR values and prints them for boot-log verification.
*
* Must be called exactly once per boot after the IDT is loaded. Calling it
* again restarts the countdown from the initial value without otherwise
* reconfiguring the timer.
*/
void apic_timer_start(void) {
/* Unmask the timer first */
uint32_t lvt = lapic_read(APIC_REG_LVT_TIMER);
lvt &= ~LVT_MASKED;
lapic_write(APIC_REG_LVT_TIMER, lvt);
/*
* Re-arm by writing ICR after unmask. QEMU TCG's APIC emulation arms its
* internal timer only when ICR is written; unmasking alone may not trigger
* a new countdown cycle in the emulator. On real hardware this is also
* correct — ICR write restarts the countdown from the programmed value.
*/
lapic_write(APIC_REG_TIMER_ICR, timer_initial_count);
/* Readback verification — confirm writes reached the LAPIC */
uint32_t lvt_rb = lapic_read(APIC_REG_LVT_TIMER);
uint32_t icr_rb = lapic_read(APIC_REG_TIMER_ICR);
console_puts("APIC Timer: started (LVT=0x");
{
for (int s = 28; s >= 0; s -= 4)
console_putc("0123456789abcdef"[(lvt_rb >> s) & 0xF]);
}
console_puts(" ICR=");
{
char buf[16]; int i = 0;
uint32_t v = icr_rb;
if (v == 0) buf[i++] = '0';
else { char tmp[16]; int j = 0;
while (v > 0) { tmp[j++] = '0' + (v % 10); v /= 10; }
while (j > 0) buf[i++] = tmp[--j]; }
buf[i] = '\0';
console_puts(buf);
}
console_puts(")\r\n");
}
/**
* @brief Mask the APIC timer to suppress further periodic interrupts.
*
* Sets the @c LVT_MASKED bit (bit 16) in the LVT Timer Register by
* performing a read-modify-write on @c APIC_REG_LVT_TIMER. The existing
* periodic-mode and vector configuration is preserved; only the mask bit
* changes. The APIC timer's internal countdown continues running but
* interrupt delivery is suppressed. Calling @c apic_timer_start() again
* after this function will unmask and re-arm the timer.
*
* Used in the kernel panic path and during any critical section where
* heartbeat interrupts would corrupt shared state.
*/
void apic_timer_stop(void) {
/* Mask the timer to stop interrupts */
uint32_t lvt = lapic_read(APIC_REG_LVT_TIMER);
lvt |= LVT_MASKED;
lapic_write(APIC_REG_LVT_TIMER, lvt);
}
/**
* @brief Re-arm the APIC timer at the current adaptive period (item 0.8, §26).
*
* amd64's APIC timer runs in periodic mode: hardware auto-reloads
* @c APIC_REG_TIMER_ICR on every expiry with no software intervention, unlike
* riscv64's one-shot SBI deadline or aarch64's one-shot @c CNTP_TVAL_EL0 --
* neither of which needed a rearm function before this item for the same
* reason this one now exists. To make the period adaptive, this function
* recomputes @c timer_initial_count from @c heartbeat_next_period_ns() (§26)
* and @c s_apic_hz, then writes it to @c APIC_REG_TIMER_ICR. A periodic-mode
* ICR write takes effect immediately and restarts the countdown at the new
* value -- the same mechanism @c apic_timer_start() already relies on to
* force QEMU TCG's emulated APIC to begin counting.
*
* Called from @c isr_common_handler() on every @c APIC_TIMER_VECTOR
* interrupt, before @c heartbeat_tick() -- same ordering discipline as
* riscv64/aarch64's rearm-before-heartbeat_tick(), so a fault in
* @c heartbeat_tick() cannot also cost the next tick.
*/
void apic_timer_rearm(void) {
uint64_t period_ns = heartbeat_next_period_ns();
uint64_t new_count = ((uint64_t)s_apic_hz * period_ns) / 1000000000ULL;
if (new_count == 0) {
new_count = 1;
}
timer_initial_count = (uint32_t)new_count;
lapic_write(APIC_REG_TIMER_ICR, timer_initial_count);
}
/**
* @brief Return the expected TSC-tick count per APIC heartbeat period.
*
* Returns @c timer_period_tsc_ticks, computed by @c apic_timer_init() as
* @c tsc_hz / @c tick_hz. The value represents the ideal number of TSC
* ticks that should elapse between consecutive @c APIC_TIMER_VECTOR
* interrupts when the TSC runs at @c tsc_hz Hz and the heartbeat runs at
* @c tick_hz Hz.
*
* Used by @c timer.c to seed the heartbeat rolling window with the
* nominal inter-tick interval so that the first few ticks produce
* meaningful variance estimates rather than comparing against zero.
* Returns 0 if @c tsc_hz was 0 at initialisation time (TSC unknown).
*
* @return Expected TSC ticks per heartbeat period, or 0 if unavailable.
*/
uint64_t apic_timer_period_tsc(void) {
return timer_period_tsc_ticks;
}
kernel/src/arch/amd64/arch.c
+249
View File
@@ -0,0 +1,249 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James
All rights reserved.
This file is part of the StarForth project.
Licensed under the StarForth License, Version 1.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at:
https://github.com/star.4th@proton.me/StarForth/LICENSE.txt
This software is provided "AS IS", WITHOUT WARRANTY OF ANY KIND,
express or implied, including but not limited to the warranties of
merchantability, fitness for a particular purpose, and noninfringement.
See the License for the specific language governing permissions and
limitations under the License.
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James
All rights reserved.
This file is part of the StarForth project.
Licensed under the StarForth License, Version 1.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at:
https://github.com/star.4th@proton.me/StarForth/LICENSE.txt
This software is provided "AS IS", WITHOUT WARRANTY OF ANY KIND,
express or implied, including but not limited to the warranties of
merchantability, fitness for a particular purpose, and noninfringement.
See the License for the specific language governing permissions and
limitations under the License.
*/
/**
* arch.c (amd64) - Architecture abstractions for StarKernel on x86_64
*/
#include "arch.h"
#include <stdint.h>
/*
* GDT entries for 64-bit mode.
* In long mode, the base and limit fields of code/data segments are ignored,
* but the access and flags bytes must be correct.
*/
struct gdt_entry {
uint16_t limit_low;
uint16_t base_low;
uint8_t base_mid;
uint8_t access;
uint8_t granularity; /* includes high nibble of limit */
uint8_t base_high;
} __attribute__((packed));
struct gdtr {
uint16_t limit;
uint64_t base;
} __attribute__((packed));
/*
* GDT layout:
* 0x00: Null descriptor
* 0x08: 64-bit kernel code segment (selector used in IDT)
* 0x10: Kernel data segment
*/
static struct gdt_entry gdt[3] __attribute__((aligned(16)));
/**
* @brief Populate a GDT entry with limit 0xFFFF and a zero base.
*
* In 64-bit long mode the base and limit fields of code/data segment
* descriptors are ignored by the processor; only the @p access and
* @p granularity flag bytes are architecturally significant. This helper
* writes the canonical long-mode descriptor layout: limit = 0xFFFF, all base
* bytes = 0, then stores the provided access and granularity values.
*
* @param index Index into the @c gdt[] array (0 = null, 1 = code, 2 = data).
* @param access Access byte (e.g., 0x9A for kernel code, 0x92 for data).
* @param granularity Granularity/flags byte (e.g., 0x20 sets the L-bit for 64-bit code).
*/
static void gdt_set_entry(int index, uint8_t access, uint8_t granularity)
{
gdt[index].limit_low = 0xFFFF;
gdt[index].base_low = 0;
gdt[index].base_mid = 0;
gdt[index].access = access;
gdt[index].granularity = granularity;
gdt[index].base_high = 0;
}
/**
* @brief Initialise and load the minimal 3-entry GDT required for long mode.
*
* Builds three GDT entries in the @c gdt[] array and loads the descriptor via
* @c LGDT followed by a far-return sequence to reload @c CS to selector 0x08
* and all other segment registers to 0x10:
*
* - Slot 0 (0x00): Null descriptor — all fields zero.
* - Slot 1 (0x08): 64-bit kernel code (access=0x9A, granularity=0x20).
* - Slot 2 (0x10): Kernel data (access=0x92, granularity=0x00).
*
* Must be called before @c arch_interrupts_init() because the IDT uses
* selector 0x08 for all gate descriptors.
*/
static void gdt_init(void)
{
/* Null descriptor */
gdt[0].limit_low = 0;
gdt[0].base_low = 0;
gdt[0].base_mid = 0;
gdt[0].access = 0;
gdt[0].granularity = 0;
gdt[0].base_high = 0;
/*
* 64-bit kernel code segment (selector 0x08)
* Access: 0x9A = Present (1), DPL 0 (00), S=1 (code/data), Exec (1), Conforming (0), Readable (1), Accessed (0)
* Granularity: 0x20 = Long mode (L=1), Default operand size (D=0, must be 0 for long mode)
*/
gdt_set_entry(1, 0x9A, 0x20);
/*
* Kernel data segment (selector 0x10)
* Access: 0x92 = Present (1), DPL 0 (00), S=1 (code/data), Exec (0), Direction (0), Writable (1), Accessed (0)
* Granularity: 0x00 (no special flags for data segment)
*/
gdt_set_entry(2, 0x92, 0x00);
struct gdtr gdtr_val;
gdtr_val.limit = sizeof(gdt) - 1;
gdtr_val.base = (uint64_t)&gdt[0];
__asm__ volatile (
"lgdt %0\n\t"
"pushq $0x08\n\t" /* Push new CS */
"leaq 1f(%%rip), %%rax\n\t" /* Load address of label 1 */
"pushq %%rax\n\t" /* Push return address */
"lretq\n\t" /* Far return to reload CS */
"1:\n\t"
"mov $0x10, %%ax\n\t" /* Load new DS/ES/SS/FS/GS */
"mov %%ax, %%ds\n\t"
"mov %%ax, %%es\n\t"
"mov %%ax, %%ss\n\t"
"mov %%ax, %%fs\n\t"
"mov %%ax, %%gs\n\t"
:
: "m"(gdtr_val)
: "rax", "memory"
);
}
/**
* @brief Perform the earliest x86-64 architecture initialisation.
*
* Called immediately after the UEFI boot stub transfers control to the
* kernel, before the console or interrupt controller are available.
* Loads the minimal three-entry GDT via @c gdt_init() so that subsequent
* code has valid segment selectors. IDT setup is deferred until after the
* console is live (called separately via @c arch_interrupts_init()).
*/
void arch_early_init(void)
{
gdt_init(); /* Must be done before IDT, since IDT uses selector 0x08 */
/* IDT setup is done later via arch_interrupts_init() after console is live */
}
/**
* @brief Enable hardware interrupts on the current CPU core (x86-64 STI).
*
* Issues the @c STI instruction to set the Interrupt Flag. Must be called
* after the IDT is loaded and the APIC is configured, otherwise the first
* interrupt will triple-fault.
*/
void arch_enable_interrupts(void)
{
__asm__ volatile ("sti");
}
/**
* @brief Disable hardware interrupts on the current CPU core (x86-64 CLI).
*
* Issues the @c CLI instruction to clear the Interrupt Flag. Used around
* critical sections in the single-threaded kernel where interrupt delivery
* would corrupt shared state.
*/
void arch_disable_interrupts(void)
{
__asm__ volatile ("cli");
}
/**
* @brief Halt the CPU until the next interrupt (x86-64 HLT).
*
* Issues the @c HLT instruction. In the kernel panic path this is called
* in an infinite loop (interrupts disabled) to stop the processor permanently.
* In normal use it is called once per idle iteration to reduce power consumption.
*/
void arch_halt(void)
{
__asm__ volatile ("hlt");
}
void arch_cold_reset(void)
{
arch_disable_interrupts();
/* Pulse keyboard controller reset line — standard bare-metal amd64 reset */
__asm__ volatile ("outb %0, %1" : : "a"((uint8_t)0xFE), "Nd"((uint16_t)0x64));
/* Should not reach here; spin as fallback */
for (;;) __asm__ volatile ("cli; hlt");
}
/**
* @brief Read the x86-64 Time Stamp Counter (RDTSC).
*
* Executes @c RDTSC and combines @c EDX:EAX into a 64-bit value. Does not
* serialise instruction execution; use @c RDTSCP or a surrounding @c CPUID
* where ordering guarantees are required.
*
* @return Current TSC value in CPU cycles.
*/
uint64_t arch_read_timestamp(void)
{
uint32_t lo, hi;
__asm__ volatile ("rdtsc" : "=a"(lo), "=d"(hi));
return ((uint64_t)hi << 32) | lo;
}
/**
* @brief Initialise the x86-64 MMU (stub — deferred to a later milestone).
*
* Full four-level page table setup is deferred; the VMM subsystem
* (@c vmm.c) handles page table management independently. This function
* exists as an architectural hook for future per-CPU MMU configuration
* that must run after GDT/IDT setup but before the first memory access
* through a non-identity-mapped virtual address.
*/
void arch_mmu_init(void)
{
/* Stub: paging will be wired up in later milestones */
}
kernel/src/arch/amd64/boot.S
+10
View File
@@ -0,0 +1,10 @@
/*
* boot.S (amd64) - Placeholder entry for future low-level bring-up.
* The UEFI loader currently calls efi_main() directly; this stub exists
* to reserve the symbol for later early-boot work.
*/
.section .text.entry
.code64
.global arch_bootstrap
arch_bootstrap:
ret
kernel/src/arch/amd64/i8042.c
+150
View File
@@ -0,0 +1,150 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James
All rights reserved.
This file is part of the StarForth project.
Licensed under the StarForth License, Version 1.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at:
https://github.com/star.4th@proton.me/StarForth/LICENSE.txt
This software is provided "AS IS", WITHOUT WARRANTY OF ANY KIND,
express or implied, including but not limited to the warranties of
merchantability, fitness for a particular purpose, and noninfringement.
See the License for the specific language governing permissions and
limitations under the License.
*/
/**
* i8042.c - PS/2 keyboard controller driver (amd64)
*
* Item 4.3.5 (FABRIC-0.md §27.5).
*/
#ifndef __STARKERNEL__
#error "i8042.c is kernel-only"
#endif
#include <stdint.h>
#include "starkernel/i8042.h"
#include "console.h"
#define I8042_DATA_PORT 0x60
#define I8042_STATUS_PORT 0x64
#define I8042_CMD_PORT 0x64
#define I8042_STATUS_OUTPUT_FULL (1u << 0)
#define I8042_STATUS_INPUT_FULL (1u << 1)
#define I8042_CMD_READ_CONFIG 0x20
#define I8042_CMD_WRITE_CONFIG 0x60
#define I8042_CONFIG_IRQ1_ENABLE (1u << 0)
#define I8042_DEV_CMD_ENABLE_SCANNING 0xF4
#define I8042_DEV_ACK 0xFA
/* Bounded wait -- command handshakes during one-time init only, never in
* the interrupt path or a per-keystroke loop. */
#define I8042_INIT_WAIT_ITERS 100000
static inline void outb(uint16_t port, uint8_t val) {
__asm__ volatile ("outb %0, %1" : : "a"(val), "Nd"(port));
}
static inline uint8_t inb(uint16_t port) {
uint8_t val;
__asm__ volatile ("inb %1, %0" : "=a"(val) : "Nd"(port));
return val;
}
void i8042_init(void) {
/* Drain any stale output byte left by firmware before enabling IRQ1 —
* otherwise the first "interrupt" would really be this leftover byte
* read via polling, not a real IRQ1 delivery. */
while (inb(I8042_STATUS_PORT) & I8042_STATUS_OUTPUT_FULL) {
(void)inb(I8042_DATA_PORT);
}
outb(I8042_CMD_PORT, I8042_CMD_READ_CONFIG);
uint8_t config = inb(I8042_DATA_PORT);
config |= I8042_CONFIG_IRQ1_ENABLE;
outb(I8042_CMD_PORT, I8042_CMD_WRITE_CONFIG);
outb(I8042_DATA_PORT, config);
/* Tell the keyboard device itself to (re)enable scanning. OVMF's own
* PS/2 driver commonly sends 0xF5 (disable scanning) to the device as
* part of its own teardown before handoff, the same way it stops
* driving the PIT/PIC -- the controller's IRQ1-enable bit above only
* governs the controller, not whether the device generates scancodes
* at all. Bounded waits: one-time init handshake, not steady-state
* polling. */
{
int i;
for (i = 0; i < I8042_INIT_WAIT_ITERS; i++) {
if (!(inb(I8042_STATUS_PORT) & I8042_STATUS_INPUT_FULL)) break;
}
outb(I8042_DATA_PORT, I8042_DEV_CMD_ENABLE_SCANNING);
for (i = 0; i < I8042_INIT_WAIT_ITERS; i++) {
if (inb(I8042_STATUS_PORT) & I8042_STATUS_OUTPUT_FULL) {
uint8_t resp = inb(I8042_DATA_PORT);
if (resp == I8042_DEV_ACK) {
console_println("i8042: keyboard ACKed enable-scanning");
} else {
console_println("i8042: keyboard responded (not ACK) to enable-scanning");
}
break;
}
}
}
console_println("i8042: IRQ1 enabled in controller config byte");
}
void i8042_drain_stale(void) {
/* Edge-triggered IRQ1 + OBF already set at unmask time == no rising
* edge ever again (the textbook "fires once, then never" pattern).
* i8042_init()'s drain happens at M4; VM bootstrap runs for real wall
* time between then and unmask, so re-drain immediately before. */
while (inb(I8042_STATUS_PORT) & I8042_STATUS_OUTPUT_FULL) {
(void)inb(I8042_DATA_PORT);
}
}
/* Ring buffer: single producer (the ISR), single reader (i8042_pop_scancode(),
* called from ordinary non-interrupt context, e.g. a diagnostic FORTH word).
* 32 entries is far more than one QEMU test keypress burst can fill. */
#define RING_SIZE 32
static volatile uint8_t ring[RING_SIZE];
static volatile uint8_t ring_head = 0; /* next slot the ISR writes */
static volatile uint8_t ring_tail = 0; /* next slot the reader takes */
/* Count of real keyboard IRQs serviced since boot -- a plain counter, no
* I/O, is cheap enough to keep in the ISR itself and doubles as the
* quickest way to confirm the interrupt path is alive without touching
* the ring buffer at all. */
volatile uint32_t g_i8042_isr_count = 0;
void i8042_handle_irq(void) {
/* Must always read 0x60, even to discard — otherwise OBF never clears
* and the controller stops delivering further interrupts. */
uint8_t scancode = inb(I8042_DATA_PORT);
g_i8042_isr_count++;
uint8_t next_head = (uint8_t)((ring_head + 1) % RING_SIZE);
if (next_head != ring_tail) { /* drop the byte if the ring is full */
ring[ring_head] = scancode;
ring_head = next_head;
}
}
int i8042_pop_scancode(uint8_t *out) {
if (ring_tail == ring_head) return 0; /* empty */
*out = ring[ring_tail];
ring_tail = (uint8_t)((ring_tail + 1) % RING_SIZE);
return 1;
}
kernel/src/arch/amd64/interrupts.c
+476
View File
@@ -0,0 +1,476 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James
All rights reserved.
This file is part of the StarForth project.
Licensed under the StarForth License, Version 1.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at:
https://github.com/star.4th@proton.me/StarForth/LICENSE.txt
This software is provided "AS IS", WITHOUT WARRANTY OF ANY KIND,
express or implied, including but not limited to the warranties of
merchantability, fitness for a particular purpose, and noninfringement.
See the License for the specific language governing permissions and
limitations under the License.
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James
All rights reserved.
This file is part of the StarForth project.
Licensed under the StarForth License, Version 1.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at:
https://github.com/star.4th@proton.me/StarForth/LICENSE.txt
This software is provided "AS IS", WITHOUT WARRANTY OF ANY KIND,
express or implied, including but not limited to the warranties of
merchantability, fitness for a particular purpose, and noninfringement.
See the License for the specific language governing permissions and
limitations under the License.
*/
/**
* interrupts.c (amd64) - Interrupt setup and exception/IRQ handling
*/
#include <stdint.h>
#include "arch.h"
#include "console.h"
#include "apic.h"
#include "timer.h"
#include "starkernel/i8042.h"
/* Set by the FORTH dispatcher just before calling entry->func(vm).
* Printed on fault to identify which word was executing. */
volatile const char *g_sk_fault_word = (void *)0;
/* Count of LAPIC spurious-vector interrupts since boot (item 4.3.5). */
volatile uint32_t g_spurious_count = 0;
#define IDT_ENTRIES 256
#define INTERRUPT_GATE 0x8E
/**
* @brief Write a single byte to an x86 I/O port via @c OUT.
*
* Issues the @c OUT instruction to port @p port with data byte @p val.
* Used exclusively to program the legacy 8259 PIC's interrupt-mask
* registers during @c pic_disable(). In a kernel that uses the Local APIC
* for all interrupt delivery the PIC must be fully masked before enabling
* CPU interrupts; otherwise spurious IRQ0 from the PIC timer can corrupt
* the APIC timer ISR path.
*
* The @c "Nd" constraint allows the assembler to emit the short-form
* @c OUT @p imm8 encoding for constants in [0, 255] and the register form
* @c OUT @c DX otherwise.
*
* @param port 16-bit I/O port number (e.g., 0x21 = PIC1 data, 0xA1 = PIC2 data).
* @param val Byte value to write to the port.
*/
static inline void outb(uint16_t port, uint8_t val)
{
__asm__ volatile ("outb %0, %1" : : "a"(val), "Nd"(port));
}
struct idt_entry {
uint16_t offset_low;
uint16_t selector;
uint8_t ist;
uint8_t type_attr;
uint16_t offset_mid;
uint32_t offset_high;
uint32_t zero;
} __attribute__((packed));
struct idtr {
uint16_t limit;
uint64_t base;
} __attribute__((packed));
extern void *isr_stub_table[IDT_ENTRIES] __attribute__((visibility("hidden")));
/*
* Assembly hands us:
* - vector
* - error_code (normalized to 0 if not present)
* - rip, cs, rflags
* - cr2 (captured for convenience; meaningful primarily for #PF)
*/
extern void isr_common_handler(uint64_t vector,
uint64_t error_code,
uint64_t rip,
uint64_t cs,
uint64_t rflags,
uint64_t cr2,
uint64_t stack_frame_ptr);
static struct idt_entry idt[IDT_ENTRIES];
/**
* @brief Write a single 64-bit interrupt gate descriptor into the IDT.
*
* Fills @c idt[@p vector] with an x86-64 interrupt gate (type 0x8E):
* - @c offset_low / @c offset_mid / @c offset_high — three parts of the
* ISR handler address @p isr split across the 16-byte descriptor layout.
* - @c selector = 0x08 — kernel code segment (GDT slot 1).
* - @c ist = 0 — no interrupt stack table; uses the current RSP.
* - @c type_attr = @c INTERRUPT_GATE (0x8E) — P=1, DPL=0, 64-bit interrupt gate;
* the CPU automatically clears RFLAGS.IF on entry.
* - @c zero = 0 — upper reserved dword must be zero.
*
* Called once per vector inside @c arch_interrupts_init()'s loop after
* the relocation offset has been applied to the stub address from
* @c isr_stub_table.
*
* @param vector IDT vector index [0, 255].
* @param isr Runtime address of the per-vector assembly stub in @c isr.S.
*/
static void set_idt_entry(int vector, void *isr) {
uint64_t addr = (uint64_t)isr;
idt[vector].offset_low = (uint16_t)(addr & 0xFFFFu);
idt[vector].selector = 0x08; /* kernel code selector */
idt[vector].ist = 0;
idt[vector].type_attr = INTERRUPT_GATE;
idt[vector].offset_mid = (uint16_t)((addr >> 16) & 0xFFFFu);
idt[vector].offset_high = (uint32_t)((addr >> 32) & 0xFFFFFFFFu);
idt[vector].zero = 0;
}
/**
* @brief Load the Interrupt Descriptor Table register (@c LIDT).
*
* Executes the @c LIDT instruction with the descriptor at @p idtr_desc,
* which contains the 16-bit limit (byte size of the IDT minus 1) and the
* 64-bit linear base address of the IDT array. After this instruction the
* CPU uses the new IDT for all exception and interrupt dispatch.
*
* Must be called after all 256 entries in @c idt[] have been filled by
* @c set_idt_entry(). Called at the end of @c arch_interrupts_init().
*
* @param idtr_desc Pointer to an @c idtr struct containing limit and base.
*/
static void lidt(struct idtr *idtr_desc) {
__asm__ volatile ("lidt %0" :: "m"(*idtr_desc));
}
/**
* @brief Print a 64-bit value as 16 lowercase hex digits to the kernel console.
*
* Emits exactly 16 nibbles (no @c 0x prefix, no separators) by iterating
* from bit 60 down to bit 0 in steps of 4. Used exclusively by
* @c isr_common_handler() to print register values (RIP, CR2, RSP, RBP,
* return address) in the exception diagnostic dump. No allocation or libc
* dependency.
*
* @param value 64-bit value to format as lowercase hex.
*/
static void print_hex64(uint64_t value) {
for (int i = 60; i >= 0; i -= 4) {
uint8_t nib = (uint8_t)((value >> (uint64_t)i) & 0xFu);
console_putc((char)(nib < 10 ? ('0' + nib) : ('a' + (nib - 10))));
}
}
/**
* @brief Print a 64-bit unsigned integer in decimal to the kernel console.
*
* Converts @p v to a decimal ASCII string using a 32-byte stack buffer
* (digit reversal technique) and emits each character via @c console_putc().
* Prints "0" for a zero value. Used by @c isr_common_handler() to print
* the exception vector number in decimal alongside its hex representation.
* No allocation or libc dependency.
*
* @param v 64-bit unsigned integer to print.
*/
static void print_dec_u64(uint64_t v) {
/* tiny decimal printer, no malloc, no libc */
char buf[32];
int i = 0;
if (v == 0) {
console_putc('0');
return;
}
while (v > 0 && i < (int)sizeof(buf)) {
buf[i++] = (char)('0' + (v % 10u));
v /= 10u;
}
while (i-- > 0) {
console_putc(buf[i]);
}
}
/**
* @brief Fully mask the legacy 8259A PIC to suppress all legacy IRQs.
*
* Writes 0xFF to PIC1's interrupt-mask register (port 0x21) and PIC2's
* interrupt-mask register (port 0xA1). When all bits are set every IRQ
* line (IRQ0–IRQ15) is masked and the PIC will not assert INTR to the CPU.
*
* This is necessary because UEFI firmware initialises the 8259 PIC in
* compatibility mode with IRQ0 (timer) running at 18.2 Hz. After
* @c ExitBootServices() the firmware no longer handles these IRQs; if the
* PIC is not masked before @c STI, spurious PIC interrupts are delivered
* to the kernel's IDT at legacy vectors 0x20–0x2F, most of which have not
* been set up as ISRs and will triple-fault.
*
* Called from @c arch_interrupts_init() after the IDT is loaded but
* before @c arch_enable_interrupts().
*/
static void pic_disable(void)
{
/* Mask all IRQs on PIC1 and PIC2 to avoid spurious legacy interrupts */
outb(0x21, 0xFF);
outb(0xA1, 0xFF);
}
/**
* @brief Decode and print the x86-64 Page Fault error code to the kernel console.
*
* The processor pushes a 32-bit error code onto the stack for a #PF
* (vector 14). Each bit encodes information about the faulting access:
*
* - bit 0 (P): 0 = page not present; 1 = protection violation.
* - bit 1 (W/R): 0 = read access; 1 = write access.
* - bit 2 (U/S): 0 = supervisor mode access; 1 = user mode access.
* - bit 3 (RSVD): 1 = fault caused by a reserved bit set in a PTE.
* - bit 4 (I/D): 1 = instruction fetch (no-execute violation).
* - bit 5 (PK): 1 = protection-key violation (if PKRU is active).
* - bit 6 (SS): 1 = shadow-stack access violation (CET).
* - bit 15 (SGX): 1 = SGX enclave access violation.
*
* Prints each active flag as a short tag to the kernel console followed
* by a newline. The CR2 register (captured by the ISR stub) holds the
* faulting linear address and is printed separately by
* @c isr_common_handler().
*
* @param ec 64-bit page fault error code (upper 32 bits are always zero
* on current x86-64 hardware; tested up to bit 15 per SDM).
*/
static void decode_pf_error(uint64_t ec)
{
/*
* Page Fault Error Code bits:
* bit0 P: 0=non-present, 1=protection violation
* bit1 W/R: 0=read, 1=write
* bit2 U/S: 0=supervisor, 1=user
* bit3 Rsvd: 1=reserved bit violation
* bit4 I/D: 1=instruction fetch
* bit5 PK: 1=protection key violation (if supported)
* bit6 SS: 1=shadow stack (if supported)
* bit15 SGX: 1=SGX (if supported)
*/
console_puts("PF EC : ");
console_puts((ec & 0x1u) ? "PROT " : "NP ");
console_puts((ec & 0x2u) ? "W " : "R ");
console_puts((ec & 0x4u) ? "USR " : "SUP ");
if (ec & 0x8u) console_puts("RSVD ");
if (ec & 0x10u) console_puts("IFETCH ");
if (ec & 0x20u) console_puts("PK ");
if (ec & 0x40u) console_puts("SS ");
if (ec & 0x8000u) console_puts("SGX ");
console_println("");
}
/**
* @brief Unified C-level interrupt and exception handler.
*
* All 256 IDT entries funnel through a common assembly trampoline in
* @c isr.S which captures caller-saved registers, reads @c CR2 (the
* page-fault linear address), normalises the error code to 0 for vectors
* that do not push one, and then calls this function with six arguments
* reconstructed from the interrupt stack frame.
*
* **APIC timer path (vector == @c APIC_TIMER_VECTOR):**
* Calls @c heartbeat_tick() to update the rolling window of inter-tick TSC
* deviations, then issues @c apic_eoi() and returns. This is the hot path
* — it executes at 100 Hz during normal kernel operation and must be kept
* brief.
*
* **Exception path (all other vectors):**
* Prints a structured diagnostic to the kernel console:
* - Vector number (decimal and hex)
* - Error code (hex)
* - RIP (hex) — faulting instruction pointer
* - CR2 (hex) — page-fault linear address (meaningful only for vector 14)
* - RSP (derived from @p stack_frame_ptr + 160, covering the ISR stub's
* pushed register block)
* - RBP (read from the saved-register block at @p stack_frame_ptr + 64)
* - Return address (first quadword at the derived RSP)
* - Vector-specific decoding: @c #DE (0), @c #GP (13) with the active FORTH
* word name from @c g_sk_fault_word, @c #PF (14) with @c decode_pf_error().
* Then halts the CPU in an infinite @c HLT loop — exceptions are fatal.
*
* The @p cs and @p rflags parameters are currently unused (suppressed with
* @c (void) casts) but retained in the signature for future use (e.g.,
* printing privilege level on #GP).
*
* @param vector IDT vector that fired [0, 255].
* @param error_code Error code from the CPU stack (0 if not applicable).
* @param rip Faulting or interrupted instruction pointer.
* @param cs Code segment selector at the time of the interrupt.
* @param rflags RFLAGS at the time of the interrupt.
* @param cr2 Value of @c CR2 (page-fault linear address; only
* architecturally meaningful for vector 14).
* @param stack_frame_ptr Address of the ISR stub's saved-register block on
* the stack; used to reconstruct RSP and RBP for the
* diagnostic dump.
*/
void isr_common_handler(uint64_t vector,
uint64_t error_code,
uint64_t rip,
uint64_t cs,
uint64_t rflags,
uint64_t cr2,
uint64_t stack_frame_ptr)
{
/* Handle APIC timer interrupt (heartbeat) */
if (vector == APIC_TIMER_VECTOR) {
apic_timer_rearm();
heartbeat_tick();
/* Acknowledge interrupt and return (don't halt!) */
apic_eoi();
return;
}
/* Handle i8042 keyboard interrupt (item 4.3.5) */
if (vector == I8042_KEYBOARD_VECTOR) {
i8042_handle_irq();
apic_eoi();
return;
}
/* Spurious interrupt: normal occasional race per Intel SDM Vol.3 §10.9,
* not a fault. No EOI -- the SDM is explicit that spurious-vector
* interrupts must not be acknowledged. g_spurious_count is a cheap,
* permanent diagnostic (item 4.3.5 found a real keyboard IRQ silently
* misreported as spurious due to a missing dedicated ISR stub -- see
* isr.S -- so a way to notice "spurious firing when it shouldn't be"
* is worth keeping, not just scaffolding for that one investigation). */
if (vector == APIC_SPURIOUS_VECTOR) {
g_spurious_count++;
return;
}
/* All other vectors are exceptions - print diagnostic and halt */
console_println("\n=== INTERRUPT/EXCEPTION ===");
console_puts("Vector : ");
print_dec_u64(vector);
console_puts(" (0x");
print_hex64(vector);
console_println(")");
console_puts("Error : 0x"); print_hex64(error_code); console_println("");
(void)cs;
(void)rflags;
console_puts("RIP : 0x"); print_hex64(rip); console_println("");
console_puts("CR2 : 0x"); print_hex64(cr2); console_println("");
uint64_t fault_rsp = stack_frame_ptr + 160u;
console_puts("RSP : 0x"); print_hex64(fault_rsp); console_println("");
console_puts("RBP : 0x");
print_hex64(*((uint64_t *)(uintptr_t)(stack_frame_ptr + 64u)));
console_println("");
console_puts("RET : 0x");
print_hex64(*((uint64_t *)(uintptr_t)fault_rsp));
console_println("");
switch (vector) {
case 0:
console_println("Fault: Divide Error (#DE)");
break;
case 13:
console_println("Fault: General Protection (#GP)");
if (g_sk_fault_word) {
console_puts("Word : ");
console_println((const char *)g_sk_fault_word);
}
break;
case 14:
console_println("Fault: Page Fault (#PF)");
decode_pf_error(error_code);
break;
default:
console_println("Fault: Unhandled vector");
break;
}
console_println("Halting.");
while (1) {
arch_halt();
}
}
/*
* Compute the runtime load offset.
* The linker places isr_stub0 at a known link-time address. At runtime,
* we compare that to the actual address to determine the relocation delta.
* This is necessary because UEFI PE loading doesn't apply relocations to
* addresses stored in the .data section (isr_stub_table).
*/
extern void isr_stub0(void) __attribute__((visibility("hidden"))); /* Defined in isr.S */
/**
* @brief Initialise the x86-64 IDT and disable the legacy 8259 PIC (M4).
*
* This is the M4 milestone function called from @c kernel_main() after the
* GDT is live (@c arch_early_init()) and before @c arch_enable_interrupts().
* It performs three operations:
*
* 1. **Relocation fixup for the ISR stub table.**
* @c isr.S exports @c isr_stub_table[], a 256-entry table of link-time
* function pointers assembled at their expected virtual addresses. In a
* UEFI PE32+ image the UEFI loader places the image at an arbitrary base;
* pointer-sized relocations in @c .data are not applied by the PE loader
* (only code-relative relocations in @c .text are). The fixup computes the
* runtime delta between @c isr_stub0's runtime address and its link-time
* address stored in @c isr_stub_table[0], then applies that delta to every
* entry before installing it in the IDT.
*
* 2. **Fill all 256 IDT entries.**
* Calls @c set_idt_entry() for each vector with the corrected stub address,
* selector 0x08, IST 0, and type_attr @c INTERRUPT_GATE (0x8E).
*
* 3. **Load the IDT and mask the PIC.**
* Calls @c lidt() with the fully populated @c idt[] array, then calls
* @c pic_disable() to mask both PIC1 and PIC2 before @c STI is issued.
* This prevents spurious legacy PIC IRQs from reaching the IDT before the
* Local APIC takes over.
*
* Must be called after @c gdt_init() (for selector 0x08) and before
* @c apic_init() / @c apic_timer_init() / @c arch_enable_interrupts().
*/
void arch_interrupts_init(void)
{
uint64_t link_time_addr = (uint64_t)isr_stub_table[0];
uint64_t runtime_addr = (uint64_t)&isr_stub0;
int64_t reloc_offset = (int64_t)(runtime_addr - link_time_addr);
for (int i = 0; i < IDT_ENTRIES; ++i) {
void *isr_addr = (void *)((uint64_t)isr_stub_table[i] + (uint64_t)reloc_offset);
set_idt_entry(i, isr_addr);
}
struct idtr idtr_desc;
idtr_desc.limit = (uint16_t)(sizeof(idt) - 1u);
idtr_desc.base = (uint64_t)&idt[0];
lidt(&idtr_desc);
pic_disable();
}
kernel/src/arch/amd64/ioapic.c
+319
View File
@@ -0,0 +1,319 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James
All rights reserved.
This file is part of the StarForth project.
Licensed under the StarForth License, Version 1.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at:
https://github.com/star.4th@proton.me/StarForth/LICENSE.txt
This software is provided "AS IS", WITHOUT WARRANTY OF ANY KIND,
express or implied, including but not limited to the warranties of
merchantability, fitness for a particular purpose, and noninfringement.
See the License for the specific language governing permissions and
limitations under the License.
*/
/**
* ioapic.c - I/O APIC driver (amd64)
*
* Item 4.3.5 (FABRIC-0.md §27.5). MADT parsing mirrors pci.c's RSDP -> XSDT ->
* table-by-signature walk (the two files don't share a header for this —
* same duplication pci.c already has relative to a hypothetical shared
* acpi.c, not introduced fresh here).
*/
#ifndef __STARKERNEL__
#error "ioapic.c is kernel-only"
#endif
#include <stddef.h>
#include <stdint.h>
#include "starkernel/ioapic.h"
#include "console.h"
#include "vmm.h"
/* -------------------------------------------------------------------------
* ACPI structures for MADT discovery
* ------------------------------------------------------------------------- */
typedef struct {
uint8_t signature[8];
uint8_t checksum;
uint8_t oem_id[6];
uint8_t revision;
uint32_t rsdt_address;
uint32_t length;
uint64_t xsdt_address;
uint8_t extended_checksum;
uint8_t reserved[3];
} __attribute__((packed)) Rsdp2;
typedef struct {
uint8_t signature[4];
uint32_t length;
uint8_t revision;
uint8_t checksum;
uint8_t oem_id[6];
uint8_t oem_table_id[8];
uint32_t oem_revision;
uint32_t creator_id;
uint32_t creator_revision;
} __attribute__((packed)) AcpiHeader;
typedef struct {
AcpiHeader hdr;
uint32_t local_apic_address;
uint32_t flags;
/* variable-length entries follow */
} __attribute__((packed)) MadtHeader;
typedef struct {
uint8_t type; /* 1 = I/O APIC */
uint8_t length; /* 12 */
uint8_t io_apic_id;
uint8_t reserved;
uint32_t io_apic_address;
uint32_t gsi_base;
} __attribute__((packed)) MadtIoApic;
typedef struct {
uint8_t type; /* 2 = Interrupt Source Override */
uint8_t length; /* 10 */
uint8_t bus;
uint8_t source; /* legacy ISA IRQ */
uint32_t gsi;
uint16_t flags; /* bits[1:0] polarity, bits[3:2] trigger mode */
} __attribute__((packed)) MadtIntSrcOverride;
#define MADT_TYPE_IOAPIC 1
#define MADT_TYPE_INT_SRC 2
#define ISO_POLARITY_MASK 0x3u
#define ISO_POLARITY_HIGH 0x1u
#define ISO_POLARITY_LOW 0x3u
#define ISO_TRIGGER_MASK 0xCu
#define ISO_TRIGGER_EDGE 0x4u
#define ISO_TRIGGER_LEVEL 0xCu
/* -------------------------------------------------------------------------
* Module state
* ------------------------------------------------------------------------- */
#define MAX_OVERRIDES 16
static uint64_t g_ioapic_base = 0;
static uint32_t g_ioapic_gsi_base = 0;
static int g_ioapic_ready = 0;
static MadtIntSrcOverride g_overrides[MAX_OVERRIDES];
static int g_override_count = 0;
/* Cached low-dword redirection values, indexed by GSI - g_ioapic_gsi_base,
* so ioapic_unmask_legacy_irq() can clear just the mask bit without having
* to re-derive polarity/trigger from the override table again. */
#define MAX_REDIR_ENTRIES 24
static uint32_t g_redir_low[MAX_REDIR_ENTRIES];
static int g_redir_valid[MAX_REDIR_ENTRIES];
/* -------------------------------------------------------------------------
* I/O APIC MMIO access (IOREGSEL @ +0x00, IOWIN @ +0x10)
* ------------------------------------------------------------------------- */
static void ioapic_write(uint8_t reg, uint32_t val) {
volatile uint32_t *regsel = (volatile uint32_t *)(uintptr_t)(g_ioapic_base + 0x00);
volatile uint32_t *iowin = (volatile uint32_t *)(uintptr_t)(g_ioapic_base + 0x10);
*regsel = reg;
*iowin = val;
}
/* -------------------------------------------------------------------------
* ACPI MADT parsing
* ------------------------------------------------------------------------- */
static int sig4_eq(const uint8_t *p, const char *s) {
return p[0] == (uint8_t)s[0] && p[1] == (uint8_t)s[1] &&
p[2] == (uint8_t)s[2] && p[3] == (uint8_t)s[3];
}
static int sig8_eq(const uint8_t *p, const char *s) {
int i;
for (i = 0; i < 8; i++)
if (p[i] != (uint8_t)s[i]) return 0;
return 1;
}
static int parse_madt(void *rsdp) {
if (!rsdp) return -1;
Rsdp2 *r = (Rsdp2 *)rsdp;
if (!sig8_eq(r->signature, "RSD PTR ")) return -1;
if (r->revision < 2 || r->xsdt_address == 0) return -1;
AcpiHeader *xsdt = (AcpiHeader *)(uintptr_t)r->xsdt_address;
if (!sig4_eq(xsdt->signature, "XSDT")) return -1;
uint32_t hdr_len = xsdt->length;
if (hdr_len <= 36u) return -1;
uint32_t n_entries = (hdr_len - 36u) / 8u;
uint64_t *entries = (uint64_t *)((uint8_t *)xsdt + 36u);
uint32_t i;
for (i = 0; i < n_entries; i++) {
AcpiHeader *sdt = (AcpiHeader *)(uintptr_t)entries[i];
if (!sig4_eq(sdt->signature, "APIC")) continue;
MadtHeader *madt = (MadtHeader *)sdt;
uint8_t *p = (uint8_t *)madt + sizeof(MadtHeader);
uint8_t *end = (uint8_t *)madt + madt->hdr.length;
int found_ioapic = 0;
while (p + 2 <= end) {
uint8_t type = p[0];
uint8_t len = p[1];
if (len == 0 || p + len > end) break;
if (type == MADT_TYPE_IOAPIC && !found_ioapic) {
MadtIoApic *io = (MadtIoApic *)p;
g_ioapic_base = io->io_apic_address;
g_ioapic_gsi_base = io->gsi_base;
found_ioapic = 1;
} else if (type == MADT_TYPE_INT_SRC && g_override_count < MAX_OVERRIDES) {
MadtIntSrcOverride *iso = (MadtIntSrcOverride *)p;
g_overrides[g_override_count++] = *iso;
}
p += len;
}
return found_ioapic ? 0 : -1;
}
return -1;
}
/* -------------------------------------------------------------------------
* Public API
* ------------------------------------------------------------------------- */
int ioapic_init(void *acpi_rsdp) {
if (parse_madt(acpi_rsdp) != 0) {
console_println("I/O APIC: no MADT I/O APIC entry found");
return -1;
}
if (vmm_map_range(g_ioapic_base, g_ioapic_base, 0x1000u,
VMM_FLAG_WRITABLE | VMM_FLAG_CACHE_DISABLE) != 0) {
console_println("I/O APIC: failed to map MMIO page");
return -1;
}
g_ioapic_ready = 1;
console_puts("I/O APIC: base=0x");
for (int s = 28; s >= 0; s -= 4)
console_putc("0123456789abcdef"[((uint32_t)g_ioapic_base >> s) & 0xF]);
console_puts(", gsi_base=");
{
char buf[16]; int i = 0; uint32_t v = g_ioapic_gsi_base;
if (v == 0) buf[i++] = '0';
else { char tmp[16]; int j = 0; while (v > 0) { tmp[j++] = (char)('0' + (v % 10)); v /= 10; } while (j > 0) buf[i++] = tmp[--j]; }
buf[i] = '\0';
console_puts(buf);
}
console_puts(", overrides=");
{
char buf[8]; int i = 0; uint32_t v = (uint32_t)g_override_count;
if (v == 0) buf[i++] = '0';
else { char tmp[8]; int j = 0; while (v > 0) { tmp[j++] = (char)('0' + (v % 10)); v /= 10; } while (j > 0) buf[i++] = tmp[--j]; }
buf[i] = '\0';
console_puts(buf);
}
console_println("");
/* TEMP DEBUG (item 4.3.5 investigation) */
for (int oi = 0; oi < g_override_count; oi++) {
console_puts(" override: source=");
console_putc("0123456789abcdef"[(g_overrides[oi].source >> 4) & 0xF]);
console_putc("0123456789abcdef"[g_overrides[oi].source & 0xF]);
console_puts(" gsi=");
{
char buf[16]; int i = 0; uint32_t v = g_overrides[oi].gsi;
if (v == 0) buf[i++] = '0';
else { char tmp[16]; int j = 0; while (v > 0) { tmp[j++] = (char)('0' + (v % 10)); v /= 10; } while (j > 0) buf[i++] = tmp[--j]; }
buf[i] = '\0';
console_puts(buf);
}
console_puts(" flags=0x");
console_putc("0123456789abcdef"[(g_overrides[oi].flags >> 12) & 0xF]);
console_putc("0123456789abcdef"[(g_overrides[oi].flags >> 8) & 0xF]);
console_putc("0123456789abcdef"[(g_overrides[oi].flags >> 4) & 0xF]);
console_putc("0123456789abcdef"[g_overrides[oi].flags & 0xF]);
console_println("");
}
return 0;
}
int ioapic_route_legacy_irq(uint8_t isa_irq, uint8_t vector, uint8_t dest_apic_id) {
if (!g_ioapic_ready) return -1;
uint32_t gsi = isa_irq;
uint32_t polarity = 0; /* active-high (ISA default) */
uint32_t trigger = 0; /* edge (ISA default) */
int i;
for (i = 0; i < g_override_count; i++) {
if (g_overrides[i].source == isa_irq) {
gsi = g_overrides[i].gsi;
uint16_t flags = g_overrides[i].flags;
if ((flags & ISO_POLARITY_MASK) == ISO_POLARITY_LOW) polarity = 1;
if ((flags & ISO_TRIGGER_MASK) == ISO_TRIGGER_LEVEL) trigger = 1;
break;
}
}
if (gsi < g_ioapic_gsi_base) return -1;
uint32_t redir_index = gsi - g_ioapic_gsi_base;
if (redir_index >= MAX_REDIR_ENTRIES) return -1;
uint32_t low = (uint32_t)vector
| (polarity << 13)
| (trigger << 15)
| (1u << 16); /* masked until ioapic_unmask_legacy_irq() */
uint32_t high = (uint32_t)dest_apic_id << 24;
ioapic_write((uint8_t)(0x10 + 2 * redir_index + 1), high);
ioapic_write((uint8_t)(0x10 + 2 * redir_index), low);
g_redir_low[redir_index] = low;
g_redir_valid[redir_index] = 1;
return 0;
}
void ioapic_unmask_legacy_irq(uint8_t isa_irq) {
if (!g_ioapic_ready) return;
uint32_t gsi = isa_irq;
int i;
for (i = 0; i < g_override_count; i++) {
if (g_overrides[i].source == isa_irq) {
gsi = g_overrides[i].gsi;
break;
}
}
if (gsi < g_ioapic_gsi_base) return;
uint32_t redir_index = gsi - g_ioapic_gsi_base;
if (redir_index >= MAX_REDIR_ENTRIES || !g_redir_valid[redir_index]) return;
uint32_t low = g_redir_low[redir_index] & ~(1u << 16);
ioapic_write((uint8_t)(0x10 + 2 * redir_index), low);
g_redir_low[redir_index] = low;
}
kernel/src/arch/amd64/isr.S
+227
View File
@@ -0,0 +1,227 @@
/*
* isr.S - Interrupt/exception stubs for amd64
*
* Stack on entry to ISR stub (after CPU pushes):
* [rsp+0] = error code (if any, or pushed 0)
* [rsp+8] = vector (pushed by stub)
* [rsp+16] = RIP
* [rsp+24] = CS
* [rsp+32] = RFLAGS
* [rsp+40] = RSP (if privilege change)
* [rsp+48] = SS (if privilege change)
*
* For IRQs (vector >= 32), we return via IRETQ.
* For exceptions (vector < 32), we halt.
*/
.intel_syntax noprefix
.section .text
/* Mark symbols referenced from C as hidden so the C compiler uses direct
* RIP-relative addressing (LEA) rather than GOT indirection. Without this,
* -fPIC code collapses the GOT reference to a direct MOV that reads the
* symbol's *bytes* instead of its *address*, corrupting the IDT setup. */
.hidden isr_stub_table
.hidden isr_stub0
.hidden isr_stub32
.hidden isr_stub33
.hidden isr_stub_default
.hidden isr_common_entry
.macro ISR_NOERR num
.global isr_stub\num
isr_stub\num:
push 0 /* fake error code */
push \num /* vector number */
jmp isr_common_entry
.endm
.macro ISR_ERR num
.global isr_stub\num
isr_stub\num:
/* error code already on stack from CPU */
push \num /* vector number */
jmp isr_common_entry
.endm
.global isr_common_entry
isr_common_entry:
/* Save all general-purpose registers */
push rax
push rbx
push rcx
push rdx
push rsi
push rdi
push rbp
push r8
push r9
push r10
push r11
push r12
push r13
push r14
push r15
/* Stack layout now:
* [rsp+0] = r15
* [rsp+8] = r14
* ...
* [rsp+112] = rax
* [rsp+120] = vector
* [rsp+128] = error
* [rsp+136] = rip
* [rsp+144] = cs
* [rsp+152] = rflags
*/
/* FABRIC-3.md SXXVIII (Stage 0, preemptive-context-switch trap-frame
* parity, 2026-09-13): FXSAVE/FXRSTOR added here to close a real,
* pre-existing gap, not just a future-preemption nicety -- confirmed
* live double-precision FP code reachable from ordinary interpreter
* dispatch (vm_runtime.c's Loop #5/#6 physics inference, called from
* vm_tick() during word execution, not just at boot). The GPR pushes
* above only cover integer state; on x86-64 SysV all XMM registers are
* caller-saved, meaning the compiler is free to leave a live value in
* one between any two instructions of straight-line code -- an
* asynchronous interrupt is not a call site the interrupted code's own
* instructions know about, so nothing here previously guaranteed FP
* state actually survived a timer tick landing mid-computation.
*
* rbp is repurposed below as a fixed anchor pointing at "rsp right after
* the 15 GPR pushes" (rbp's own true pushed value already lives safely
* on the stack above and is restored by the ordinary `pop rbp` later) --
* FXSAVE requires a 16-byte-aligned address and rsp's alignment here is
* unpredictable (the interrupt could have landed anywhere), so the
* FXSAVE area is carved out and aligned separately, with rbp used to
* keep addressing everything else at its original, fixed offsets
* regardless of that adjustment. */
mov rbp, rsp
sub rsp, 528 /* 512-byte FXSAVE area + 16 slack for alignment */
and rsp, -16
fxsave [rsp]
/* Set up arguments for isr_common_handler(vector, error, rip, cs, rflags, cr2) */
mov rdi, [rbp + 120] /* vector */
mov rsi, [rbp + 128] /* error */
mov rdx, [rbp + 136] /* rip */
mov rcx, [rbp + 144] /* cs */
mov r8, [rbp + 152] /* rflags */
mov r9, cr2 /* cr2 */
/* Pass stack frame pointer (after pushes) as 7th argument */
push rbp
call isr_common_handler
add rsp, 8
fxrstor [rsp]
mov rsp, rbp /* collapse back to right after the GPR pushes */
/* Restore general-purpose registers */
pop r15
pop r14
pop r13
pop r12
pop r11
pop r10
pop r9
pop r8
pop rbp
pop rdi
pop rsi
pop rdx
pop rcx
pop rbx
pop rax
/* Remove vector and error code from stack */
add rsp, 16
/* Return from interrupt */
iretq
/* Exception/interrupt stubs (0-31 explicit) */
ISR_NOERR 0
ISR_NOERR 1
ISR_NOERR 2
ISR_NOERR 3
ISR_NOERR 4
ISR_NOERR 5
ISR_NOERR 6
ISR_NOERR 7
ISR_ERR 8
ISR_NOERR 9
ISR_ERR 10
ISR_ERR 11
ISR_ERR 12
ISR_ERR 13
ISR_ERR 14
ISR_NOERR 15
ISR_NOERR 16
ISR_ERR 17
ISR_NOERR 18
ISR_NOERR 19
ISR_NOERR 20
ISR_ERR 21
ISR_NOERR 22
ISR_NOERR 23
ISR_NOERR 24
ISR_NOERR 25
ISR_NOERR 26
ISR_NOERR 27
ISR_NOERR 28
ISR_NOERR 29
ISR_ERR 30
ISR_NOERR 31
/* Stub for APIC Timer (vector 0x20 = 32) */
/* Diagnostic: write raw 'I' to COM1 (0x3F8) when stub is entered.
* If 'I' never appears in the serial log, the stub is never reached
* (bad IDT entry or fault during CPU interrupt delivery). */
.global isr_stub32
isr_stub32:
push 0
push 0x20
jmp isr_common_entry
/* Stub for i8042 keyboard IRQ1, routed via I/O APIC to vector 0x21 = 33
* (item 4.3.5, FABRIC-0.md §27.5). Before this stub existed, vector 33 fell
* through to isr_stub_default below, which unconditionally reports "255"
* regardless of which IDT slot actually fired -- the CPU legitimately took
* IDT[33] (confirmed via the LAPIC's own ISR register), but isr_common_handler
* only ever saw vector 255 and treated a real keyboard IRQ as spurious,
* silently skipping EOI and leaving vector 33 permanently stuck in-service. */
.global isr_stub33
isr_stub33:
push 0
push 33
jmp isr_common_entry
/* Default stub for other vectors (34-255) */
.global isr_stub_default
isr_stub_default:
push 0 /* fake error */
push 255 /* unknown vector, will be logged */
jmp isr_common_entry
/* Use .data instead of .rodata so UEFI applies relocations */
.section .data
.align 8
.global isr_stub_table
isr_stub_table:
.quad isr_stub0, isr_stub1, isr_stub2, isr_stub3
.quad isr_stub4, isr_stub5, isr_stub6, isr_stub7
.quad isr_stub8, isr_stub9, isr_stub10, isr_stub11
.quad isr_stub12, isr_stub13, isr_stub14, isr_stub15
.quad isr_stub16, isr_stub17, isr_stub18, isr_stub19
.quad isr_stub20, isr_stub21, isr_stub22, isr_stub23
.quad isr_stub24, isr_stub25, isr_stub26, isr_stub27
.quad isr_stub28, isr_stub29, isr_stub30, isr_stub31
/* Vector 32 = APIC Timer */
.quad isr_stub32
/* Vector 33 = i8042 keyboard IRQ1 (item 4.3.5) */
.quad isr_stub33
/* Vectors 34-255 default to unknown stub */
.rept 222
.quad isr_stub_default
.endr
kernel/src/arch/amd64/kernel_entry.S
+63
View File
@@ -0,0 +1,63 @@
/*
* kernel_entry.S — kernel stack setup and entry trampoline (amd64)
*
* UEFI hands control to kernel_main() running on a small UEFI-provided stack
* (typically 128 KB–256 KB). The FORTH interpreter + DOE experiment loop can
* easily exceed that depth. This trampoline allocates a 2 MiB BSS stack and
* switches to it before calling the real implementation (kernel_main_impl).
*
* kernel_main is the symbol the UEFI loader calls (extern in uefi_loader.c).
* kernel_main_impl is defined in kernel_main.c and contains all the C code.
*
* Calling convention: SysV AMD64 — boot_info pointer arrives in %rdi and is
* passed unchanged to kernel_main_impl via the tail-call.
*/
.intel_syntax noprefix
#include "starkernel/boot_info_offsets.h"
/* 2 MiB kernel stack in BSS — fallback when no dynamic stack requested */
.section .bss
.align 16
.global g_kernel_stack
g_kernel_stack:
.space 0x200000
g_kernel_stack_top:
.global g_kernel_stack_top
.section .text
.extern kernel_main_impl
/*
* kernel_main — stack-switch trampoline, then tail-call kernel_main_impl.
*
* rdi = BootInfo* (SysV AMD64 ABI — first argument register)
*
* Stack selection (checked in order):
* 1. boot_info->kernel_stack_base != 0 → use loader-allocated stack
* (top = base + size, aligned down to 16, minus 8 for ABI)
* 2. Otherwise → fall back to g_kernel_stack (2 MiB BSS)
*
* Never returns; kernel_main_impl runs the REPL forever or panics.
*/
.global kernel_main
kernel_main:
mov rax, [rdi + BOOT_INFO_KERNEL_STACK_BASE_OFFSET]
test rax, rax
jnz .Ldynamic_stack
/* BSS fallback */
lea rax, [rip + g_kernel_stack_top]
jmp .Lstack_ready
.Ldynamic_stack:
mov rcx, [rdi + BOOT_INFO_KERNEL_STACK_SIZE_OFFSET]
add rax, rcx /* top = base + size (stack grows down) */
.Lstack_ready:
and rax, -16 /* 16-byte alignment */
sub rax, 8 /* SysV ABI: RSP ≡ 8 mod 16 at C call boundary */
mov rsp, rax
xor rbp, rbp /* terminate frame chain */
jmp kernel_main_impl /* rdi (boot_info) already in place */
kernel/src/arch/amd64/switch.S
+79
View File
@@ -0,0 +1,79 @@
/*
* switch.S (amd64) - Cooperative VM context switch primitive.
*
* FABRIC-3.md §XXVIII (Stage 2, preemptive-context-switch save/restore
* primitive, 2026-09-13). This is deliberately NOT the same mechanism as
* isr.S's interrupt trap frame (Stage 0) -- that saves every register
* because an interrupt is not a call site the interrupted code knows
* about. sk_vm_switch_to() IS an ordinary function call (SWITCH-TO's C
* handler calls it directly), so the standard System V AMD64 ABI already
* guarantees every caller-saved register (rax, rcx, rdx, rsi, rdi, r8-r11,
* all XMM) is the caller's own problem to preserve across the call, exactly
* like any other call. Only the callee-saved set -- rbx, rbp, r12-r15 --
* needs explicit handling here, because this function does not really
* "return" in the normal sense: it swaps to a completely different stack
* and (from the newly-resumed context's point of view) appears to return
* from whichever earlier call to this same function originally parked it.
* SysV has no callee-saved XMM registers at all, so there is no FP state
* to carry here.
*
* void sk_vm_switch_to(uint64_t *save_sp_out, uint64_t new_sp);
* rdi = save_sp_out, rsi = new_sp
*
* For a VM that has never run before, new_sp must point at a synthetic
* frame built to this exact layout by the C glue (switch.c): six zeroed
* quadwords (r15, r14, r13, r12, rbx, rbp -- values irrelevant, never a
* real prior context) followed by the trampoline entry address at
* new_sp+48, so the final `ret` below jumps there instead of into some
* prior caller.
*/
.intel_syntax noprefix
.section .text
.hidden sk_vm_switch_to
.global sk_vm_switch_to
sk_vm_switch_to:
push rbp
push rbx
push r12
push r13
push r14
push r15
mov [rdi], rsp /* *save_sp_out = rsp (this context's parked SP) */
mov rsp, rsi /* rsp = new_sp (switch stacks) */
pop r15
pop r14
pop r13
pop r12
pop rbx
pop rbp
ret
/*
* uint64_t sk_vm_switch_prime(uint64_t stack_top, void *entry);
*
* Builds the synthetic frame described above at the top of a fresh native
* stack, so the first ever sk_vm_switch_to() into it lands at `entry`
* instead of some real prior caller. Returns the resulting sp value to
* pass as sk_vm_switch_to()'s new_sp. Kept in this file (not C) so the
* layout can never drift from what sk_vm_switch_to() above actually
* expects -- one set of offsets, not two copies to keep in sync.
*/
.hidden sk_vm_switch_prime
.global sk_vm_switch_prime
sk_vm_switch_prime:
mov rax, rdi
sub rax, 56
mov qword ptr [rax+0], 0 /* r15 */
mov qword ptr [rax+8], 0 /* r14 */
mov qword ptr [rax+16], 0 /* r13 */
mov qword ptr [rax+24], 0 /* r12 */
mov qword ptr [rax+32], 0 /* rbx */
mov qword ptr [rax+40], 0 /* rbp */
mov [rax+48], rsi /* return address = entry */
ret
kernel/src/arch/amd64/timer.c
+1393
View File
File diff suppressed because it is too large Load Diff
kernel/src/arch/riscv64/README.md
+18
View File
@@ -0,0 +1,18 @@
# src/starkernel/arch/riscv64/
RISC-V 64-bit architecture support for LithosAnanke.
- `arch.c` — architecture init entry point.
- `apic.c` — PLIC (Platform-Level Interrupt Controller) stub, named
`apic.c` to match the shared per-arch file-naming convention across
amd64/aarch64/riscv64 (RISC-V has no Local APIC). Satisfies the common
`apic_init()`/EOI call sites without real PLIC bring-up yet; deferred to
a later milestone.
- `timer.c` — timer setup (heartbeat tick source).
- `interrupts.c` — trap vector setup and interrupt dispatch.
- `boot.S` — early boot assembly.
- `isr.S` — interrupt service routine stubs.
No `kernel_entry.S` — see `Makefile.starkernel` for how the riscv64 loader
image is produced (ELF → `scripts/gen_pe_reloc.py`-patched relocations →
`objcopy` → PE).
kernel/src/arch/riscv64/apic.c
+283
View File
@@ -0,0 +1,283 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James. All rights reserved.
Licensed under the StarForth License, Version 1.0.
*/
/**
* apic.c (riscv64) - APIC stub. No Local APIC on RISC-V; interrupt
* controller is PLIC (Platform-Level Interrupt Controller). apic_init()
* calls into plic.c for the real PLIC bring-up (item 4.3.5b); apic_eoi()
* stays a no-op since PLIC completion happens per-source in the trap
* handler (claim/complete), not via a single shared EOI register.
*/
#include "apic.h"
#include "uefi.h"
#include "console.h"
#include "timer.h"
#include "starkernel/plic.h"
#include <stdint.h>
static uint64_t s_timer_period_tsc = 0;
static uint64_t s_time_hz = 0; /* `time` CSR frequency (item 0.8/§26) */
/* ---------------------------------------------------------------------------
* SBI (Supervisor Binary Interface)
*
* RISC-V S-mode cannot program the timer directly: the CLINT's mtimecmp is an
* M-mode register. The timer is armed by asking the SEE (OpenSBI, running in
* M-mode beneath EDK2) via ECALL.
*
* Calling convention, SBI v0.2+ (SBI spec §3): a7 = EID, a6 = FID,
* a0.. = arguments; returns a0 = error, a1 = value.
* ------------------------------------------------------------------------- */
#define SBI_EXT_BASE 0x10UL
#define SBI_BASE_FID_PROBE_EXT 3UL
#define SBI_EXT_TIME 0x54494D45UL /* "TIME" */
#define SBI_TIME_FID_SET_TIMER 0UL
#define SBI_SUCCESS 0L
/* sie.STIE — supervisor timer interrupt enable (Privileged Spec §4.1.3) */
#define SIE_STIE (1UL << 5)
typedef struct
{
long error;
long value;
} sbiret_t;
/** @brief Issue an SBI ECALL with one argument. */
static sbiret_t sbi_call1(unsigned long eid, unsigned long fid,
unsigned long arg0)
{
register unsigned long r_a0 __asm__("a0") = arg0;
register unsigned long r_a1 __asm__("a1") = 0;
register unsigned long r_a6 __asm__("a6") = fid;
register unsigned long r_a7 __asm__("a7") = eid;
sbiret_t ret;
__asm__ volatile (
"ecall"
: "+r"(r_a0), "+r"(r_a1)
: "r"(r_a6), "r"(r_a7)
: "memory");
ret.error = (long)r_a0;
ret.value = (long)r_a1;
return ret;
}
/**
* @brief Read the RISC-V @c time CSR.
*
* Deadlines handed to @c sbi_set_timer() are absolute values on this counter.
* @c arch/riscv64/timer.c keeps its own copy of this accessor; duplicating
* four instructions is preferable to widening @c timer.h with an
* architecture-specific accessor that only these two files can use.
*/
static inline uint64_t rdtime(void)
{
uint64_t val;
__asm__ volatile (
"rdtime %0" : "=r"(val));
return val;
}
/* Set once in apic_timer_start(): 1 when the TIME extension probed present,
* 0 when the timer could not be armed at all. */
static int s_sbi_time_ok = 0;
/* Absolute `time` value of the next expected interrupt. Advanced by period
* rather than recomputed from "now" so that a late tick does not push the
* whole schedule out; see riscv64_timer_rearm(). */
static uint64_t s_next_deadline = 0;
/**
* @brief Arm the SBI timer for @p deadline.
* @return 1 on success, 0 if the SEE rejected the call.
*/
static int sbi_set_timer(uint64_t deadline)
{
sbiret_t r = sbi_call1(SBI_EXT_TIME, SBI_TIME_FID_SET_TIMER,
(unsigned long)deadline);
return r.error == SBI_SUCCESS;
}
/**
* @brief Re-arm the one-shot SBI timer and account the tick.
*
* **The SBI timer is one-shot by nature.** Servicing a timer interrupt without
* programming the next deadline leaves the heartbeat stopped permanently, with
* no error anywhere — the single most likely silent failure of this driver.
* Every path out of a timer interrupt must reach this function.
*
* The step is @c heartbeat_next_period_ns() (item 0.8, §26) converted to
* `time`-counter ticks via @c s_time_hz, not the fixed @c s_timer_period_tsc
* used to seed the very first deadline in @c apic_timer_start() -- the
* latter remains @c apic_timer_period_tsc()'s return value for
* @c heartbeat_init()'s initial @c expected_delta, unchanged.
*
* Called from @c riscv64_interrupt_handler() in @c interrupts.c on
* @c scause cause 5.
*/
void riscv64_timer_rearm(void)
{
uint64_t now;
uint64_t step;
if (!s_sbi_time_ok) return;
step = (s_time_hz > 0)
? (heartbeat_next_period_ns() * s_time_hz) / 1000000000ULL
: s_timer_period_tsc;
if (step == 0) {
step = 1;
}
s_next_deadline += step;
/* If servicing ran long enough that the next deadline is already behind
* us, resynchronise rather than burn through a backlog of instant
* interrupts. */
now = rdtime();
if (s_next_deadline <= now)
{
s_next_deadline = now + step;
}
sbi_set_timer(s_next_deadline);
}
/*
* @brief Initialise the interrupt controller: PLIC bring-up (item 4.3.5b).
*
* On x86-64 this function configures the Local APIC. On RISC-V the
* Platform-Level Interrupt Controller (PLIC) handles external interrupt
* routing -- @c plic_init() (arch/riscv64/plic.c) maps it, sets the S-mode
* context's threshold, and enables @c sie.SEIE. No source is enabled here;
* that is each future consumer's own job (4.3.5c for virtio-keyboard).
*
* @param boot_info Kernel boot information; @c dtb is passed through to
* @c plic_init() for real-hardware PLIC base discovery as
* of the 2026-09-05 fix (FABRIC-3.md §V.3 item 3) -- see
* plic.c's file header for the discovery mechanism.
* @return 0 always.
*/
int apic_init(BootInfo *boot_info)
{
plic_init(boot_info ? boot_info->dtb : (void *) 0);
return 0;
}
/**
* @brief Signal End of Interrupt to the interrupt controller (RISC-V stub).
*
* On x86-64 this writes to the Local APIC EOI register. On RISC-V EOI
* is handled by reading the PLIC claim/complete register; that path is
* not yet wired. This stub is a no-op that satisfies the common interface.
*/
void apic_eoi(void) { }
/*
* @brief Compute the expected `time`-counter ticks per heartbeat period.
*
* Unlike x86-64's APIC timer, this does not itself arm anything — RISC-V S-mode
* cannot program the timer directly, so arming happens via SBI in
* @c apic_timer_start() below. This function only computes
* @c s_timer_period_tsc, the expected number of @c time-counter ticks between
* heartbeats, which seeds both @c apic_timer_start()'s first deadline and
* @c heartbeat_init()'s @c expected_delta.
*
* Falls back to 10,000,000 ticks if either frequency is zero (a safe
* non-zero sentinel preventing division-by-zero in the heartbeat path).
*
* @param tsc_hz `time` counter frequency from @c timer_init() (Hz).
* @param tick_hz Desired heartbeat rate (Hz); 0 → uses fallback.
* @return 0 always.
*/
int apic_timer_init(uint64_t tsc_hz, uint32_t tick_hz)
{
if (tick_hz > 0 && tsc_hz > 0)
s_timer_period_tsc = tsc_hz / tick_hz;
else
s_timer_period_tsc = 10000000;
s_time_hz = tsc_hz;
return 0;
}
/**
* @brief Start timer delivery via the SBI TIME extension.
*
* Probes for the TIME extension first. If the SEE does not provide it the
* timer is **not** armed and the condition is reported loudly rather than
* papered over with the legacy EID 0x00 call: a heartbeat that silently never
* ticks is far worse to diagnose than one that says why at boot.
*
* On success: computes the first absolute deadline, arms it, and sets
* @c sie.STIE. Global delivery is gated separately by @c sstatus.SIE, which
* @c arch_enable_interrupts() sets.
*/
void apic_timer_start(void)
{
sbiret_t probe;
probe = sbi_call1(SBI_EXT_BASE, SBI_BASE_FID_PROBE_EXT, SBI_EXT_TIME);
if (probe.error != SBI_SUCCESS || probe.value == 0)
{
console_println("SBI: TIME extension ABSENT - timer NOT armed, "
"heartbeat will not tick");
s_sbi_time_ok = 0;
return;
}
s_sbi_time_ok = 1;
s_next_deadline = rdtime() + s_timer_period_tsc;
if (!sbi_set_timer(s_next_deadline))
{
console_println("SBI: set_timer REJECTED - timer NOT armed");
s_sbi_time_ok = 0;
return;
}
__asm__ volatile (
"csrs sie, %0"
::
"r"(SIE_STIE) : "memory");
console_println("SBI: timer armed (TIME extension)");
}
/**
* @brief Stop timer delivery by masking @c sie.STIE.
*
* The SBI timer cannot be cancelled outright — masking the enable bit is the
* supported way to stop delivery. Any deadline already programmed simply goes
* unserviced.
*/
void apic_timer_stop(void)
{
__asm__ volatile (
"csrc sie, %0"
::
"r"(SIE_STIE) : "memory");
s_sbi_time_ok = 0;
}
/**
* @brief Return the expected `time`-counter ticks per heartbeat period.
*
* Returns @c s_timer_period_tsc, set by @c apic_timer_init() as
* @c tsc_hz / @c tick_hz. Consumed by @c heartbeat_init() to seed the
* rolling-window @c expected_delta, and by @c apic_timer_start() /
* @c riscv64_timer_rearm() as the SBI deadline step.
*
* @return Expected `time`-counter ticks per heartbeat period; 10,000,000 if
* frequencies were not available at init time.
*/
uint64_t apic_timer_period_tsc(void)
{
return s_timer_period_tsc;
}
kernel/src/arch/riscv64/arch.c
+221
View File
@@ -0,0 +1,221 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James. All rights reserved.
Licensed under the StarForth License, Version 1.0.
*/
/**
* arch.c (riscv64) - Architecture abstractions for StarKernel on RISC-V 64-bit
*/
#include "arch.h"
#include "console.h"
#include <stdint.h>
extern void riscv64_install_vectors(void);
extern char __kernel_start[]; /* same pattern pmm.c:67 already uses */
/* Item 4.3.5a. Prints satp's MODE/PPN fields and a known running address
* as a permanent boot record (same diagnostic-verbosity convention as
* apic.c's "AArch64: running at EL..." on the other ISA). Confirmed live,
* 2026-08-08: firmware (EDK2 RISC-V) leaves satp.MODE=10 (Sv57) active at
* kernel entry -- not Sv39/Sv48 as this file's own arch_mmu_init() stub
* comment assumed; that comment is stale, corrected below. __kernel_start
* printing here (0xbdd56982 in the confirming boot) landed inside the
* UEFI-reported total physical RAM window (0x80000000 + 1020 MB), near the
* top past the PMM-free region -- consistent with the kernel's own running
* range being identity-mapped (VA==PA) under firmware's Sv57 table, not
* some high canonical virtual range. That is the basis for the Bare-mode
* switch below being safe *when firmware leaves a translation active at
* all* -- see arch_early_init()'s own comment (FABRIC-3.md §V.3 item 7
* fix, 2026-09-04) for why that no longer covers every boot path this
* kernel now has. */
static void print_hex64(uint64_t val)
{
char buf[19];
buf[0] = '0'; buf[1] = 'x';
for (int i = 15; i >= 0; i--) {
int nibble = (int)(val & 0xF);
buf[2 + i] = (char)(nibble < 10 ? '0' + nibble : 'a' + nibble - 10);
val >>= 4;
}
buf[18] = '\0';
console_puts(buf);
}
/**
* @brief Perform the earliest RISC-V architecture initialisation (no-op).
*
* RISC-V has no GDT or IDT; privilege separation is handled by the
* machine/supervisor/user execution modes (M/S/U), and the trap vector
* table is installed by writing @c stvec in @c arch_interrupts_init()
* via @c riscv64_install_vectors(). This stub satisfies the common
* @c arch_early_init() call site in @c kernel_main() without any
* RISC-V-specific action.
*
* **Correction, 2026-09-04 (FABRIC-3.md §V.3 item 7's code audit).** The
* unconditional Bare-mode switch this function used to perform was
* justified entirely by behaviour observed under this system's QEMU/EDK2
* RISC-V firmware (@c satp.MODE=10/Sv57, kernel identity-mapped within
* it — see the comment on @c print_hex64() above). That observation
* cannot apply to the native boot path (U-Boot+OpenSBI, no EDK2 at all,
* FABRIC-3.md §V.1) this kernel now also has, and OpenSBI's own S-mode
* handoff convention leaves @c satp already at 0 (Bare) in the common
* case — meaning the switch below was very likely running as a harmless
* no-op there, but on an unverified assumption rather than a checked one.
* Rather than build a general Sv39/Sv48/Sv57 page-table walker just to
* *prove* the running address is identity-mapped before switching (out of
* proportion to this item's real scope), the fix narrows what actually
* needs the QEMU-specific safety argument: **skip the switch entirely
* when @c satp is already 0.** That is the expected case on both a
* from-Bare native boot and any firmware that already left translation
* off, and it needs no safety argument at all — there is nothing to
* switch away from. The unconditional-switch path still exists for the
* confirmed QEMU/EDK2 case (@c mode != 0), where the identity-mapping
* argument above remains the actual justification, unchanged.
*/
void arch_early_init(void)
{
/* No GDT/IDT on RISC-V; privilege via exception levels */
uint64_t satp;
__asm__ volatile ("csrr %0, satp" : "=r"(satp));
uint64_t mode = satp >> 60;
uint64_t ppn = satp & 0xFFFFFFFFFFFULL;
console_println("riscv64 item 4.3.5a: satp state at kernel entry (before switch)");
console_puts(" satp.MODE = "); print_hex64(mode); console_puts("\n");
console_puts(" satp.PPN = "); print_hex64(ppn); console_puts("\n");
console_puts(" __kernel_start = "); print_hex64((uint64_t)__kernel_start); console_puts("\n");
if (mode == 0) {
/* Already Bare -- the expected case on a native (OpenSBI) boot.
* Nothing to switch away from, so no identity-mapping assumption
* is needed at all: skipping is strictly safer than an
* unconditional csrw+sfence that would just write back the same
* zero value. */
console_println("riscv64: satp already Bare -- no switch needed (item 4.3.5a)");
return;
}
/* Explicit switch to Bare mode (satp.MODE=0): this kernel builds no
* riscv64 page table of its own (vmm.c's is x86-64-shaped and never
* activated here, per its own load_cr3() no-op outside __x86_64__) and
* has no present use for virtual memory on this ISA, so there is no
* reason to inherit firmware's mapping -- which is confirmed, for the
* QEMU/EDK2 case this branch was written for, to have at least one
* hole (PLIC_THRESHOLD, FABRIC-0.md item 4.3.5a). ExitBootServices()
* has already completed several checkpoints before this function
* runs on that path (ConOut/GOP done, BootServices exited per the
* "[CKPT 008]" trace), so nothing downstream depends on firmware's
* table surviving there. On the native boot path, reaching this
* branch at all means OpenSBI's handoff left satp non-zero -- not
* the case this kernel has confirmed, so the identity-mapping safety
* argument is unverified here; flagged, not guessed past. Single asm
* block: the CSR write and SFENCE.VMA must not be separated by a
* compiler-scheduled memory access (RISC-V Privileged Spec §4.2.1,
* SFENCE.VMA ordering). */
__asm__ volatile (
"csrw satp, x0\n"
"sfence.vma\n"
::: "memory"
);
console_println("riscv64: satp cleared -- Bare mode, explicit (item 4.3.5a)");
}
/**
* @brief Enable supervisor-mode interrupts (RISC-V CSRSI sstatus).
*
* Issues @c CSRSI @c sstatus, @c 0x2 which sets bit 1 (@c SIE —
* Supervisor Interrupt Enable) in the @c sstatus CSR. While @c SIE = 1,
* pending supervisor-mode interrupts are taken. The @c memory clobber
* prevents the compiler from reordering stores across the barrier.
*
* Must be called after @c arch_interrupts_init() has installed the
* @c stvec trap handler, and after the PLIC driver has been configured
* to forward the desired interrupt sources.
*/
void arch_enable_interrupts(void)
{
__asm__ volatile ("csrsi sstatus, 0x2" ::: "memory");
}
/**
* @brief Disable supervisor-mode interrupts (RISC-V CSRCI sstatus).
*
* Issues @c CSRCI @c sstatus, @c 0x2 which clears bit 1 (@c SIE) in
* @c sstatus, masking all supervisor-mode interrupts. Machine-mode
* interrupts are unaffected. Used around critical sections in the
* single-threaded kernel. The @c memory clobber acts as a compiler
* barrier; pair with a @c FENCE instruction for a hardware barrier.
*/
void arch_disable_interrupts(void)
{
__asm__ volatile ("csrci sstatus, 0x2" ::: "memory");
}
/**
* @brief Halt the core until the next interrupt (RISC-V WFI).
*
* Issues the @c WFI (Wait For Interrupt) instruction in supervisor mode.
* The core enters a low-power state and resumes when a pending interrupt
* is detected (or after a platform-specific timeout). In the kernel panic
* path this is called inside an infinite loop with @c SIE cleared to stop
* the processor permanently. In the idle path it reduces power consumption
* between heartbeat ticks.
*
* The @c memory clobber prevents store-sinking across the @c WFI.
*/
void arch_halt(void)
{
__asm__ volatile ("wfi" ::: "memory");
}
void arch_cold_reset(void)
{
arch_disable_interrupts();
/* SBI SRST extension (0x53525354): sbi_system_reset, cold reboot = type 1 */
register unsigned long a7 __asm__("a7") = 0x53525354UL;
register unsigned long a6 __asm__("a6") = 0UL;
register unsigned long a0 __asm__("a0") = 1UL; /* cold reboot */
register unsigned long a1 __asm__("a1") = 0UL; /* no reason */
__asm__ volatile ("ecall" : "+r"(a0) : "r"(a1), "r"(a6), "r"(a7) : "memory");
for (;;) __asm__ volatile ("wfi" ::: "memory");
}
/**
* @brief Read the RISC-V CPU cycle counter (@c rdcycle).
*
* Issues the @c RDCYCLE pseudo-instruction (a @c CSRRS on @c cycle, CSR
* 0xC00) to read the 64-bit hardware performance counter that counts
* elapsed clock cycles since reset. On RISC-V the cycle counter is the
* architectural equivalent of the x86-64 TSC.
*
* Unlike the AArch64 @c CNTPCT_EL0, the RISC-V @c cycle register is a
* per-hart counter and is not architecturally guaranteed to be synchronised
* across harts. For single-core LithosAnanke this is not a concern.
* Frequency is not architecturally specified; @c timer_init() assumes
* 1 GHz for QEMU @c virt board compatibility.
*
* @return Current 64-bit cycle counter value; frequency platform-dependent.
*/
uint64_t arch_read_timestamp(void)
{
uint64_t val;
__asm__ volatile ("rdcycle %0" : "=r"(val));
return val;
}
/**
* @brief Initialise the RISC-V MMU (stub — deferred to a later milestone).
*
* Full Sv39/Sv48 page-table setup (SATP, PMP, etc.) is deferred to a
* later milestone. This stub satisfies the common @c arch_mmu_init()
* call site shared across all three supported ISAs. The VMM subsystem
* (@c vmm.c) handles page-table management independently of this hook.
*/
void arch_mmu_init(void)
{
/* Stub: Sv39/Sv48 bring-up deferred */
}
kernel/src/arch/riscv64/boot.S
+21
View File
@@ -0,0 +1,21 @@
/*
* boot.S (riscv64) - Entry stub and linker-script symbol substitutes.
*
* Defines __text_start, __text_end, __kernel_start, __kernel_end so
* hal.c and pmm.c resolve. They all point to the image base, which is
* sufficient for the monolithic loader build (UEFI manages page ownership).
*/
.text
.global __text_start
.global __kernel_start
__text_start:
__kernel_start:
.global arch_bootstrap
arch_bootstrap:
ret
.global __text_end
.global __kernel_end
__text_end:
__kernel_end:
kernel/src/arch/riscv64/interrupts.c
+181
View File
@@ -0,0 +1,181 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James. All rights reserved.
Licensed under the StarForth License, Version 1.0.
*/
/**
* interrupts.c (riscv64) - Trap vector setup and common exception handler
*/
#include <stdint.h>
#include "arch.h"
#include "console.h"
#include "starkernel/timer.h"
#include "starkernel/plic.h"
volatile const char *g_sk_fault_word = (void *)0;
extern void riscv64_install_vectors(void);
/* Defined in arch/riscv64/apic.c. Declared here rather than in the shared
* starkernel/apic.h because re-arming is specific to the one-shot SBI timer
* and has no meaning for the amd64 periodic APIC timer. Same extern-in-place
* convention as riscv64_install_vectors above. */
extern void riscv64_timer_rearm(void);
/* Defined in virtio_input.c (item 4.3.5c). g_virtio_input_plic_source is 0
* (plic_claim()'s own "nothing pending" sentinel) until
* virtio_input_find_keyboard() finds a real device and computes its
* routed source -- so this branch is inert on any boot where the device
* isn't present, same as the generic dispatch was inert before 4.3.5c. */
extern uint32_t g_virtio_input_plic_source;
extern void virtio_input_isr(void);
/* scause cause codes for supervisor-mode interrupts (RISC-V Privileged Spec
* §4.1.9, Table "Supervisor cause register values"). */
#define SCAUSE_INTERRUPT_BIT (1ULL << 63)
#define SCAUSE_S_TIMER 5ULL
#define SCAUSE_S_EXTERNAL 9ULL
/* Count of real PLIC claims serviced since boot (item 4.3.5b). Cheap
* standing diagnostic, same shape as amd64's g_i8042_isr_count/
* g_spurious_count -- confirms the claim/dispatch/complete path is alive
* without needing a live debugger. Nothing enables any PLIC source by
* default, so this stays 0 unless a future consumer (or a live test)
* explicitly calls plic_enable(). */
volatile uint32_t g_plic_claim_count = 0;
/**
* @brief Dispatch an asynchronous supervisor interrupt.
*
* Called from @c riscv64_trap_entry in @c isr.S when @c scause has bit 63 set.
* Unlike @c riscv64_exception_handler() this **returns** — the trap entry
* restores the caller-saved register set and issues @c SRET.
*
* Supervisor timer (cause 5) re-arms the one-shot SBI timer and then accounts
* the tick. **Re-arm comes first**: the SBI timer fires once per programmed
* deadline, so any return path that skips the re-arm stops the heartbeat
* permanently and silently. Ordering it ahead of @c heartbeat_tick() means a
* fault in the bookkeeping cannot also cost the next tick.
*
* Supervisor external (cause 9): claim/dispatch/complete through the PLIC
* (substrate, item 4.3.5b). Source 10 (UART) stays unclaimed by anything
* here -- 4.3.5b's own synthetic-interrupt self-test that used it was run
* once and reverted, not a standing consumer. virtio-input (item 4.3.5c)
* is the first real per-source dispatch, the same shape as i8042's
* keyboard branch in amd64's isr_common_handler().
*
* Any other cause is ignored rather than fatal: an unexpected-but-harmless
* asynchronous interrupt should not take the kernel down.
*
* @param scause Raw @c scause value, interrupt bit still set.
*/
void riscv64_interrupt_handler(uint64_t scause)
{
uint64_t cause = scause & ~SCAUSE_INTERRUPT_BIT;
if (cause == SCAUSE_S_TIMER) {
riscv64_timer_rearm();
heartbeat_tick();
} else if (cause == SCAUSE_S_EXTERNAL) {
uint32_t irq = plic_claim();
if (irq != 0) {
g_plic_claim_count++;
if (irq == g_virtio_input_plic_source) {
virtio_input_isr();
}
plic_complete(irq);
}
}
}
/**
* @brief Print a 64-bit value as "0xNNNNNNNNNNNNNNNN" to the kernel console.
*
* Formats @p val as a 16-nibble lowercase hex string preceded by "0x" into
* a 19-byte stack buffer (including null terminator) and emits it via
* @c console_puts(). Used by @c riscv64_exception_handler() to print
* @c scause, @c sepc, and @c stval in the exception diagnostic dump.
* No heap allocation or libc dependency.
*
* @param val 64-bit value to format.
*/
static void print_hex(uint64_t val)
{
char buf[19];
buf[0] = '0'; buf[1] = 'x';
for (int i = 15; i >= 0; i--) {
int nibble = (int)(val & 0xF);
buf[2 + i] = (char)(nibble < 10 ? '0' + nibble : 'a' + nibble - 10);
val >>= 4;
}
buf[18] = '\0';
console_puts(buf);
}
/**
* @brief RISC-V common trap handler — print diagnostic and spin.
*
* Called from the @c isr.S trap vector for all synchronous exceptions
* and asynchronous interrupts that are not handled by a specific driver.
* Reads three supervisor-mode CSRs directly:
*
* - @c scause — Supervisor Cause Register. Bit 63 distinguishes
* interrupts (1) from exceptions (0); bits [62:0] encode the cause
* code (e.g., 0 = instruction address misaligned, 2 = illegal
* instruction, 5 = load access fault, 13 = store page fault,
* 8 = ecall from U-mode). See RISC-V Privileged Spec §4.1.9.
* - @c sepc — Supervisor Exception Program Counter. The PC of the
* instruction that caused the trap (or the return address for
* interrupt handlers); execution resumes from @c sepc on SRET.
* - @c stval — Supervisor Trap Value. For load/store page faults this
* holds the faulting virtual address; for illegal instruction faults
* it holds the faulting instruction encoding; zero for other causes.
*
* Prints all three values to the kernel console then enters an infinite
* @c WFI loop to halt the core permanently. All exceptions that reach
* this handler are fatal in the current single-hart kernel.
*
* @c g_sk_fault_word is set by the VM FORTH dispatcher and names the
* word executing at the time of the trap, aiding post-mortem serial
* log analysis.
*/
void riscv64_exception_handler(void)
{
uint64_t scause, sepc, stval;
__asm__ volatile ("csrr %0, scause" : "=r"(scause));
__asm__ volatile ("csrr %0, sepc" : "=r"(sepc));
__asm__ volatile ("csrr %0, stval" : "=r"(stval));
console_println("*** EXCEPTION (riscv64) ***");
console_puts(" scause = "); print_hex(scause); console_puts("\n");
console_puts(" sepc = "); print_hex(sepc); console_puts("\n");
console_puts(" stval = "); print_hex(stval); console_puts("\n");
for (;;) {
__asm__ volatile ("wfi" ::: "memory");
}
}
/**
* @brief Install the RISC-V supervisor trap vector table (M4 milestone).
*
* Calls @c riscv64_install_vectors() (defined in @c isr.S) which writes
* the trap vector base address into @c stvec (Supervisor Trap Vector Base
* Address Register, CSR 0x105). The vector mode field (bits [1:0]) is
* set to @c 0 (Direct mode) so all traps dispatch to the single base
* address, or @c 1 (Vectored mode) so async interrupts dispatch to
* @c stvec + 4*cause.
*
* On RISC-V there is no IDT and no legacy PIC; the PLIC replaces both.
* The @c arch_interrupts_init() name is shared across all three ISAs so
* that @c kernel_main() calls the same symbol regardless of target.
*
* Must be called after the supervisor stack (@c sp) is established and
* before @c arch_enable_interrupts().
*/
void arch_interrupts_init(void)
{
riscv64_install_vectors();
}
kernel/src/arch/riscv64/isr.S
+256
View File
@@ -0,0 +1,256 @@
/*
* isr.S (riscv64) - Trap vector installation.
* Routes all S-mode exceptions to a common C handler.
*/
.section .text
.balign 4
/* Hidden for consistency with the amd64 isr.S convention. Neither symbol
* here is ever address-read from C (interrupts.c only *calls*
* riscv64_install_vectors(); the trap entry itself is entirely
* self-contained in this file), so this isn't fixing a live bug the way
* the amd64 .hidden directives do -- it's defensive hardening against the
* same GOT-indirection class of issue should that ever change. */
.hidden riscv64_trap_entry
.hidden riscv64_install_vectors
/* ---------------------------------------------------------------------------
* Trap frame layout (offsets from sp), 512 bytes, 16-byte aligned.
*
* FABRIC-3.md SXXVIII (Stage 0, preemptive-context-switch trap-frame parity,
* 2026-09-13): originally only the LP64D psABI's *caller-saved* registers
* were saved here, on the reasoning that a C handler preserves callee-saved
* registers itself. That reasoning holds for an ordinary function call, but
* not for an asynchronous trap: the interrupted code's own compiled
* instructions have no call site to justify losing state at, so a value
* genuinely live in a callee-saved register at the moment of interrupt has
* no ABI guarantee of surviving. s0-s11/fs0-fs11 are now saved too, so this
* frame is, in principle, sufficient to resume execution anywhere it was
* taken. Not yet copied into any per-VM saved context (that's Stage 2) --
* this stage only makes the frame itself complete.
*
* integer caller-saved (16): ra, t0-t6, a0-a7
* integer callee-saved (12): s0-s11
* FP caller-saved (20): ft0-ft11, fa0-fa7 + fcsr
* FP callee-saved (12): fs0-fs11
* supervisor CSRs (2): sepc, sstatus
*
* The FP half (caller- and callee-saved alike) is saved only when
* sstatus.FS != Off. Nothing in boot.S or kernel_entry.S programs FS, so its
* state is whatever firmware left; touching an f-register while FS == Off
* raises an illegal-instruction trap, which inside the trap handler would be
* unrecoverable. The kernel image does contain real FP code (fld/fmul.d/fcvt
* on the Q48.16 and statistics paths), so the save cannot simply be omitted
* -- it has to be conditional.
* ------------------------------------------------------------------------- */
.equ TF_RA, 0
.equ TF_T0, 8
.equ TF_T1, 16
.equ TF_T2, 24
.equ TF_T3, 32
.equ TF_T4, 40
.equ TF_T5, 48
.equ TF_T6, 56
.equ TF_A0, 64
.equ TF_A1, 72
.equ TF_A2, 80
.equ TF_A3, 88
.equ TF_A4, 96
.equ TF_A5, 104
.equ TF_A6, 112
.equ TF_A7, 120
.equ TF_S0, 128 /* s0-s11 : 128 .. 216 */
.equ TF_SEPC, 224
.equ TF_SSTATUS,232
.equ TF_FCSR, 240
.equ TF_F0, 248 /* ft0-ft7 : 248 .. 304 */
.equ TF_FA0, 312 /* fa0-fa7 : 312 .. 368 */
.equ TF_FT8, 376 /* ft8-ft11 : 376 .. 400 */
.equ TF_FS0, 408 /* fs0-fs11 : 408 .. 496 */
.equ TF_SIZE, 512
/* sstatus.FS occupies bits [14:13]; 0 = Off */
.equ SSTATUS_FS_SHIFT, 13
.equ SSTATUS_FS_MASK, 3
/* Direct-mode trap entry: stvec[1:0] = 0 means all traps branch here.
* stvec requires 4-byte alignment. */
.balign 4
.global riscv64_trap_entry
riscv64_trap_entry:
addi sp, sp, -TF_SIZE
sd ra, TF_RA(sp)
sd t0, TF_T0(sp)
sd t1, TF_T1(sp)
sd t2, TF_T2(sp)
sd t3, TF_T3(sp)
sd t4, TF_T4(sp)
sd t5, TF_T5(sp)
sd t6, TF_T6(sp)
sd a0, TF_A0(sp)
sd a1, TF_A1(sp)
sd a2, TF_A2(sp)
sd a3, TF_A3(sp)
sd a4, TF_A4(sp)
sd a5, TF_A5(sp)
sd a6, TF_A6(sp)
sd a7, TF_A7(sp)
sd s0, TF_S0+0(sp)
sd s1, TF_S0+8(sp)
sd s2, TF_S0+16(sp)
sd s3, TF_S0+24(sp)
sd s4, TF_S0+32(sp)
sd s5, TF_S0+40(sp)
sd s6, TF_S0+48(sp)
sd s7, TF_S0+56(sp)
sd s8, TF_S0+64(sp)
sd s9, TF_S0+72(sp)
sd s10, TF_S0+80(sp)
sd s11, TF_S0+88(sp)
csrr t0, sepc
sd t0, TF_SEPC(sp)
csrr t0, sstatus
sd t0, TF_SSTATUS(sp)
/* Save FP only if FS != Off */
srli t1, t0, SSTATUS_FS_SHIFT
andi t1, t1, SSTATUS_FS_MASK
beqz t1, 1f
frcsr t2
sd t2, TF_FCSR(sp)
fsd ft0, TF_F0+0(sp)
fsd ft1, TF_F0+8(sp)
fsd ft2, TF_F0+16(sp)
fsd ft3, TF_F0+24(sp)
fsd ft4, TF_F0+32(sp)
fsd ft5, TF_F0+40(sp)
fsd ft6, TF_F0+48(sp)
fsd ft7, TF_F0+56(sp)
fsd fa0, TF_FA0+0(sp)
fsd fa1, TF_FA0+8(sp)
fsd fa2, TF_FA0+16(sp)
fsd fa3, TF_FA0+24(sp)
fsd fa4, TF_FA0+32(sp)
fsd fa5, TF_FA0+40(sp)
fsd fa6, TF_FA0+48(sp)
fsd fa7, TF_FA0+56(sp)
fsd ft8, TF_FT8+0(sp)
fsd ft9, TF_FT8+8(sp)
fsd ft10, TF_FT8+16(sp)
fsd ft11, TF_FT8+24(sp)
fsd fs0, TF_FS0+0(sp)
fsd fs1, TF_FS0+8(sp)
fsd fs2, TF_FS0+16(sp)
fsd fs3, TF_FS0+24(sp)
fsd fs4, TF_FS0+32(sp)
fsd fs5, TF_FS0+40(sp)
fsd fs6, TF_FS0+48(sp)
fsd fs7, TF_FS0+56(sp)
fsd fs8, TF_FS0+64(sp)
fsd fs9, TF_FS0+72(sp)
fsd fs10, TF_FS0+80(sp)
fsd fs11, TF_FS0+88(sp)
1:
/* scause bit 63 set => interrupt (negative when read as signed). */
csrr a0, scause
bltz a0, 2f
/* Synchronous exception: fatal, does not return. */
call riscv64_exception_handler
9: wfi
j 9b
2: /* Interrupt: a0 already holds scause for the C dispatcher. */
call riscv64_interrupt_handler
/* Restore. Reload the saved sstatus first -- it decides whether the FP
* half was written, and it must be restored *after* the f-registers so
* that FS is back to the interrupted context's value on exit. */
ld t0, TF_SSTATUS(sp)
srli t1, t0, SSTATUS_FS_SHIFT
andi t1, t1, SSTATUS_FS_MASK
beqz t1, 3f
ld t2, TF_FCSR(sp)
fscsr t2
fld ft0, TF_F0+0(sp)
fld ft1, TF_F0+8(sp)
fld ft2, TF_F0+16(sp)
fld ft3, TF_F0+24(sp)
fld ft4, TF_F0+32(sp)
fld ft5, TF_F0+40(sp)
fld ft6, TF_F0+48(sp)
fld ft7, TF_F0+56(sp)
fld fa0, TF_FA0+0(sp)
fld fa1, TF_FA0+8(sp)
fld fa2, TF_FA0+16(sp)
fld fa3, TF_FA0+24(sp)
fld fa4, TF_FA0+32(sp)
fld fa5, TF_FA0+40(sp)
fld fa6, TF_FA0+48(sp)
fld fa7, TF_FA0+56(sp)
fld ft8, TF_FT8+0(sp)
fld ft9, TF_FT8+8(sp)
fld ft10, TF_FT8+16(sp)
fld ft11, TF_FT8+24(sp)
fld fs0, TF_FS0+0(sp)
fld fs1, TF_FS0+8(sp)
fld fs2, TF_FS0+16(sp)
fld fs3, TF_FS0+24(sp)
fld fs4, TF_FS0+32(sp)
fld fs5, TF_FS0+40(sp)
fld fs6, TF_FS0+48(sp)
fld fs7, TF_FS0+56(sp)
fld fs8, TF_FS0+64(sp)
fld fs9, TF_FS0+72(sp)
fld fs10, TF_FS0+80(sp)
fld fs11, TF_FS0+88(sp)
3:
csrw sstatus, t0
ld t0, TF_SEPC(sp)
csrw sepc, t0
ld ra, TF_RA(sp)
ld t0, TF_T0(sp)
ld t1, TF_T1(sp)
ld t2, TF_T2(sp)
ld t3, TF_T3(sp)
ld t4, TF_T4(sp)
ld t5, TF_T5(sp)
ld t6, TF_T6(sp)
ld a0, TF_A0(sp)
ld a1, TF_A1(sp)
ld a2, TF_A2(sp)
ld a3, TF_A3(sp)
ld a4, TF_A4(sp)
ld a5, TF_A5(sp)
ld a6, TF_A6(sp)
ld a7, TF_A7(sp)
ld s0, TF_S0+0(sp)
ld s1, TF_S0+8(sp)
ld s2, TF_S0+16(sp)
ld s3, TF_S0+24(sp)
ld s4, TF_S0+32(sp)
ld s5, TF_S0+40(sp)
ld s6, TF_S0+48(sp)
ld s7, TF_S0+56(sp)
ld s8, TF_S0+64(sp)
ld s9, TF_S0+72(sp)
ld s10, TF_S0+80(sp)
ld s11, TF_S0+88(sp)
addi sp, sp, TF_SIZE
sret
.global riscv64_install_vectors
riscv64_install_vectors:
la t0, riscv64_trap_entry
csrw stvec, t0
ret
kernel/src/arch/riscv64/kernel_entry.S
+69
View File
@@ -0,0 +1,69 @@
/*
* kernel_entry.S — kernel stack setup and entry trampoline (riscv64)
*
* UEFI hands control to kernel_main() running on EDK2's own boot-time stack.
* Unlike amd64 (see arch/amd64/kernel_entry.S, which has carried this same
* trampoline since early on), riscv64 previously called straight into
* kernel_main_impl with no stack switch at all. The VM bootstrap's call
* depth (27 chained word-registration modules, physics/SSM init, capsule
* birth, Tripod fleet spawn) overflowed EDK2's riscv64 boot stack partway
* through vm_init_with_host()'s return, corrupting the return address and
* producing a wild jump shortly after — observed as a load/instruction
* page fault (scause 0xd or 0xc) whose exact address drifted between
* builds while the crash always landed in this same call region. amd64
* never had this problem because kernel_entry.S already switches to a
* dedicated 2 MiB BSS stack before anything deep runs; aarch64 apparently
* gets away with EDK2's default boot stack being large enough, but that's
* incidental, not a guarantee.
*
* kernel_main is the symbol the UEFI loader calls (extern in uefi_loader.c).
* kernel_main_impl is defined in kernel_main.c and contains all the C code.
*
* Calling convention: RISC-V LP64D — boot_info pointer arrives in a0 and is
* passed unchanged to kernel_main_impl via the tail-call.
*/
#include "starkernel/boot_info_offsets.h"
/* 2 MiB kernel stack in BSS — fallback when no dynamic stack requested */
.section .bss
.align 4
.global g_kernel_stack
g_kernel_stack:
.space 0x200000
g_kernel_stack_top:
.global g_kernel_stack_top
.section .text
.extern kernel_main_impl
/*
* kernel_main — stack-switch trampoline, then tail-call kernel_main_impl.
*
* a0 = BootInfo* (RISC-V LP64D ABI — first argument register)
*
* Stack selection (checked in order):
* 1. boot_info->kernel_stack_base != 0 → use loader-allocated stack
* (top = base + size, aligned down to 16)
* 2. Otherwise → fall back to g_kernel_stack (2 MiB BSS)
*
* Never returns; kernel_main_impl runs the REPL forever or panics.
*/
.global kernel_main
kernel_main:
ld t0, BOOT_INFO_KERNEL_STACK_BASE_OFFSET(a0)
bnez t0, .Ldynamic_stack
/* BSS fallback */
la t0, g_kernel_stack_top
j .Lstack_ready
.Ldynamic_stack:
ld t1, BOOT_INFO_KERNEL_STACK_SIZE_OFFSET(a0)
add t0, t0, t1 /* top = base + size (stack grows down) */
.Lstack_ready:
andi t0, t0, -16 /* 16-byte alignment (RISC-V psABI requirement) */
mv sp, t0
mv s0, zero /* terminate frame-pointer chain */
tail kernel_main_impl /* a0 (boot_info) already in place */
kernel/src/arch/riscv64/plic.c
+167
View File
@@ -0,0 +1,167 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James. All rights reserved.
Licensed under the StarForth License, Version 1.0.
*/
/**
* plic.c (riscv64) - Platform-Level Interrupt Controller driver
*
* Item 4.3.5b. Register layout (priority/pending/enable/threshold/claim-
* complete) is the standard SiFive PLIC-1.0.0 layout QEMU implements --
* architectural, not board-specific, same status as the GICD/GICC register
* *offsets* item 0.6 cross-checked against the Linux arm-gic header.
*
* PLIC_BASE_QEMU_DEFAULT and PLIC_CONTEXT_S are QEMU-virt-specific and were
* NOT hardcoded from memory: verified 2026-08-08 via
* `qemu-system-riscv64 -machine virt,dumpdtb=...`, decoded with a script
* using this tree's own fdt.c parsing logic (not dtc, which is not
* installed). The real devicetree shows:
* soc/plic@c000000: compatible = "sifive,plic-1.0.0","riscv,plic0",
* reg = <0x0 0xc000000 0x0 0x600000>, riscv,ndev = 95
* soc/plic@c000000: interrupts-extended = <&cpu0-intc 11 &cpu0-intc 9>
* -- two contexts for hart 0: context 0 feeds cause 11 (MEIE, M-mode
* external), context 1 feeds cause 9 (SEIE, S-mode external). Context
* index is position in this list, confirmed rather than assumed.
* soc/serial@10000000 (ns16550a): interrupts = <10>, interrupt-parent =
* &plic -- PLIC source 10, matching console.c's existing NS16550_BASE.
* Confirmed for this exact QEMU 10.2.1 build; not assumed stable across
* versions, same caveat item 0.6 recorded for its own dumped values.
*
* DTB-discovered base address, as of FABRIC-3.md §V.3 item 3's fix,
* 2026-09-05: the node-scoped ("reg" is not globally unique the way
* "timebase-frequency" is) devicetree lookup this file's own comment
* previously flagged as "a bigger addition than this item's stated scope"
* was subsequently built anyway -- `fdt_find_node_by_compatible()` +
* `fdt_find_prop_in_node()` (`fdt.c`/`fdt.h`), added for the aarch64 GIC
* base fix (FABRIC-3.md §IV.3 item 7) and reused here unchanged.
* `plic_base_from_dtb()` below tries `"sifive,plic-1.0.0"` first; on
* failure (no DTB, e.g. this system's QEMU/UEFI firmware -- same
* `fdt_valid()` failure mode apic.c's GIC discovery already documents) it
* falls back to the QEMU constant above, unchanged from this function's
* previous unconditional behaviour. Not yet confirmed against the Milk-V
* Mars's real DTB -- the compatible string is architectural (SiFive
* PLIC-1.0.0 is the standard binding), not board-specific, but the actual
* JH7110 `reg` value is unverified until real hardware arrives (2026-09-17).
*/
#include <stdint.h>
#include "starkernel/plic.h"
#include "starkernel/fdt.h"
#include "console.h"
#define PLIC_BASE_QEMU_DEFAULT 0x0C000000UL
#define PLIC_CONTEXT_S 1u /* S-mode, hart 0 -- verified, see file header */
/* Runtime PLIC base address -- set once by plic_init(), defaulting to the
* QEMU constant above until/unless plic_base_from_dtb() overrides it.
* uintptr_t, not a #define, precisely because this is no longer a
* compile-time constant on this path (same shape as aarch64 apic.c's
* s_gicd_base/s_gicc_base). */
static uintptr_t s_plic_base = PLIC_BASE_QEMU_DEFAULT;
#define PLIC_PRIORITY(irq) (s_plic_base + 4u * (irq))
#define PLIC_ENABLE(ctx) (s_plic_base + 0x2000u + 0x80u * (ctx))
#define PLIC_THRESHOLD(ctx) (s_plic_base + 0x200000u + 0x1000u * (ctx))
#define PLIC_CLAIM(ctx) (s_plic_base + 0x200004u + 0x1000u * (ctx))
static inline volatile uint32_t *reg32(uintptr_t addr) {
return (volatile uint32_t *)addr;
}
/**
* @brief Read one big-endian 32-bit cell from a devicetree property blob.
*
* Duplicated from aarch64 apic.c's identical helper rather than shared --
* same "a few lines is simpler than a new shared dependency" precedent
* that file's own comment already set for this exact situation.
*/
static uint32_t be32_cell(const unsigned char *b) {
return ((uint32_t) b[0] << 24) | ((uint32_t) b[1] << 16) |
((uint32_t) b[2] << 8) | (uint32_t) b[3];
}
/**
* @brief Read one two-cell (64-bit) big-endian address from a `reg` entry.
*
* QEMU-virt's `plic` node's own dumped `reg = <0x0 0xc000000 0x0 0x600000>`
* (see file header) is address(2 cells) + size(2 cells) = 4 cells total,
* same shape as aarch64's `gicv2` node -- reads only the two address
* cells, ignoring the size pair that follows.
*/
static uint64_t be64_addr_cell_pair(const unsigned char *b) {
return ((uint64_t) be32_cell(b) << 32) | (uint64_t) be32_cell(b + 4);
}
/**
* @brief Try to discover the real PLIC base address from the DTB.
*
* Looks up the `"sifive,plic-1.0.0"` node and reads its `reg` property's
* first (address) cell pair. Returns 0 (leaving @p base_out untouched) if
* @p dtb is not a valid FDT, no matching node exists, or `reg` is shorter
* than one address-cell pair -- the caller keeps the QEMU constant default
* in that case, exactly as before this function existed.
*
* @param dtb Candidate devicetree blob (@c BootInfo->dtb); NULL-safe.
* @param base_out Receives the PLIC base address on success.
* @return 1 on success, 0 if discovery failed for any reason.
*/
static int plic_base_from_dtb(const void *dtb, uintptr_t *base_out) {
const void *node;
const unsigned char *reg;
uint32_t len;
if (!fdt_valid(dtb)) return 0;
node = fdt_find_node_by_compatible(dtb, "sifive,plic-1.0.0");
if (!node) return 0;
reg = (const unsigned char *) fdt_find_prop_in_node(dtb, node, "reg", &len);
if (!reg || len < 8u) return 0;
*base_out = (uintptr_t) be64_addr_cell_pair(reg);
return 1;
}
int plic_init(const void *dtb) {
if (plic_base_from_dtb(dtb, &s_plic_base)) {
console_puts("PLIC: base discovered from DTB (0x");
for (int s = 60; s >= 0; s -= 4)
console_putc("0123456789abcdef"[(s_plic_base >> s) & 0xF]);
console_println(")");
} else {
console_println("PLIC: no DTB PLIC node -- using QEMU virt-machine default (base=0x0c000000)");
}
/* Threshold 0 = maximally permissive. Safe: nothing is enabled at any
* source yet, so nothing can actually reach claim() until a caller
* explicitly enables it. */
*reg32(PLIC_THRESHOLD(PLIC_CONTEXT_S)) = 0;
__asm__ volatile ("csrs sie, %0" :: "r"(1UL << 9) : "memory"); /* SEIE */
console_println("PLIC: init (S-mode context 1, threshold=0)");
return 0;
}
void plic_set_priority(uint32_t irq, uint32_t priority) {
*reg32(PLIC_PRIORITY(irq)) = priority;
}
void plic_enable(uint32_t irq) {
volatile uint32_t *reg = reg32(PLIC_ENABLE(PLIC_CONTEXT_S) + 4u * (irq / 32u));
*reg |= (1u << (irq % 32u));
}
void plic_disable(uint32_t irq) {
volatile uint32_t *reg = reg32(PLIC_ENABLE(PLIC_CONTEXT_S) + 4u * (irq / 32u));
*reg &= ~(1u << (irq % 32u));
}
uint32_t plic_claim(void) {
return *reg32(PLIC_CLAIM(PLIC_CONTEXT_S));
}
void plic_complete(uint32_t irq) {
*reg32(PLIC_CLAIM(PLIC_CONTEXT_S)) = irq;
}
kernel/src/arch/riscv64/switch.S
+162
View File
@@ -0,0 +1,162 @@
/*
* switch.S (riscv64) - Cooperative VM context switch primitive.
*
* FABRIC-3.md §XXVIII (Stage 2, preemptive-context-switch save/restore
* primitive, 2026-09-13). Deliberately NOT isr.S's interrupt trap frame
* (Stage 0) -- that saves everything because an async trap has no call
* site. sk_vm_switch_to() is an ordinary `call`, so the LP64D psABI
* already makes every caller-saved register (ra excepted -- see below --
* t0-t6, a0-a7, ft0-ft11, fa0-fa7) the caller's own problem across the
* call. Only the callee-saved set needs explicit handling: s0-s11,
* fs0-fs11, and ra itself (ra is not "callee-saved" in the usual sense,
* but this function does not really return to its own caller -- it swaps
* stacks and later resumes as if returning from whichever earlier call
* parked this context, so ra must be captured explicitly here rather than
* left to a normal prologue/epilogue).
*
* The FP half is saved only when sstatus.FS != Off -- same reasoning and
* bit test as isr.S's trap frame, checked once per call and reused for
* both the save and restore decision (nothing changes FS in between; it
* is genuine single global CPU state, not part of what's "switched," so
* this function reads it but never writes it).
*
* void sk_vm_switch_to(uint64_t *save_sp_out, uint64_t new_sp);
* a0 = save_sp_out, a1 = new_sp
*
* For a VM that has never run before, new_sp must point at a synthetic
* 208-byte frame built by the C glue (switch.c) with ra set to the
* trampoline entry address and everything else zeroed, so the final
* `ret` below (jalr via ra) jumps there.
*/
.section .text
.balign 4
.equ SW_S0, 0
.equ SW_RA, 96
.equ SW_FS0, 104
.equ SW_SIZE, 208
.equ SSTATUS_FS_SHIFT, 13
.equ SSTATUS_FS_MASK, 3
.hidden sk_vm_switch_to
.global sk_vm_switch_to
sk_vm_switch_to:
addi sp, sp, -SW_SIZE
sd s0, SW_S0+0(sp)
sd s1, SW_S0+8(sp)
sd s2, SW_S0+16(sp)
sd s3, SW_S0+24(sp)
sd s4, SW_S0+32(sp)
sd s5, SW_S0+40(sp)
sd s6, SW_S0+48(sp)
sd s7, SW_S0+56(sp)
sd s8, SW_S0+64(sp)
sd s9, SW_S0+72(sp)
sd s10, SW_S0+80(sp)
sd s11, SW_S0+88(sp)
sd ra, SW_RA(sp)
csrr t0, sstatus
srli t1, t0, SSTATUS_FS_SHIFT
andi t1, t1, SSTATUS_FS_MASK
beqz t1, 1f
fsd fs0, SW_FS0+0(sp)
fsd fs1, SW_FS0+8(sp)
fsd fs2, SW_FS0+16(sp)
fsd fs3, SW_FS0+24(sp)
fsd fs4, SW_FS0+32(sp)
fsd fs5, SW_FS0+40(sp)
fsd fs6, SW_FS0+48(sp)
fsd fs7, SW_FS0+56(sp)
fsd fs8, SW_FS0+64(sp)
fsd fs9, SW_FS0+72(sp)
fsd fs10, SW_FS0+80(sp)
fsd fs11, SW_FS0+88(sp)
1:
sd sp, 0(a0) /* *save_sp_out = sp (this context's parked SP) */
mv sp, a1 /* sp = new_sp (switch stacks) */
ld s0, SW_S0+0(sp)
ld s1, SW_S0+8(sp)
ld s2, SW_S0+16(sp)
ld s3, SW_S0+24(sp)
ld s4, SW_S0+32(sp)
ld s5, SW_S0+40(sp)
ld s6, SW_S0+48(sp)
ld s7, SW_S0+56(sp)
ld s8, SW_S0+64(sp)
ld s9, SW_S0+72(sp)
ld s10, SW_S0+80(sp)
ld s11, SW_S0+88(sp)
ld ra, SW_RA(sp)
beqz t1, 2f
fld fs0, SW_FS0+0(sp)
fld fs1, SW_FS0+8(sp)
fld fs2, SW_FS0+16(sp)
fld fs3, SW_FS0+24(sp)
fld fs4, SW_FS0+32(sp)
fld fs5, SW_FS0+40(sp)
fld fs6, SW_FS0+48(sp)
fld fs7, SW_FS0+56(sp)
fld fs8, SW_FS0+64(sp)
fld fs9, SW_FS0+72(sp)
fld fs10, SW_FS0+80(sp)
fld fs11, SW_FS0+88(sp)
2:
addi sp, sp, SW_SIZE
ret
/*
* uint64_t sk_vm_switch_prime(uint64_t stack_top, void *entry);
*
* Builds the synthetic frame above at the top of a fresh native stack, so
* the first ever sk_vm_switch_to() into it lands at `entry` (ra) via its
* final `ret`. Kept in this file (not C) so the layout can never drift
* from sk_vm_switch_to() above -- one set of offsets, not two. FP slots
* are zeroed via plain integer stores (sd, not fsd) -- this is just
* writing zero bytes to memory, not an FP computation, so it needs no
* FS-enabled check at all, unlike every other f-register touch in this
* file.
*/
.hidden sk_vm_switch_prime
.global sk_vm_switch_prime
sk_vm_switch_prime:
addi t0, a0, -SW_SIZE
sd zero, SW_S0+0(t0)
sd zero, SW_S0+8(t0)
sd zero, SW_S0+16(t0)
sd zero, SW_S0+24(t0)
sd zero, SW_S0+32(t0)
sd zero, SW_S0+40(t0)
sd zero, SW_S0+48(t0)
sd zero, SW_S0+56(t0)
sd zero, SW_S0+64(t0)
sd zero, SW_S0+72(t0)
sd zero, SW_S0+80(t0)
sd zero, SW_S0+88(t0)
sd a1, SW_RA(t0)
sd zero, SW_FS0+0(t0)
sd zero, SW_FS0+8(t0)
sd zero, SW_FS0+16(t0)
sd zero, SW_FS0+24(t0)
sd zero, SW_FS0+32(t0)
sd zero, SW_FS0+40(t0)
sd zero, SW_FS0+48(t0)
sd zero, SW_FS0+56(t0)
sd zero, SW_FS0+64(t0)
sd zero, SW_FS0+72(t0)
sd zero, SW_FS0+80(t0)
sd zero, SW_FS0+88(t0)
mv a0, t0
ret
kernel/src/arch/riscv64/timer.c
+283
View File
@@ -0,0 +1,283 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James. All rights reserved.
Licensed under the StarForth License, Version 1.0.
*/
/**
* timer.c (riscv64) - Timer using the `time` CSR.
*
* RISC-V provides `cycle` (per-hart clock cycles) and `time` (fixed-frequency
* wall clock). The primary timestamp source is `time`, read via rdtime, and
* its rate comes from the devicetree property timebase-frequency.
*
* This was `cycle` at an assumed 1 GHz until punch-list item 0.3. Two things
* forced the change: the SBI TIME extension arms deadlines against `time`, so
* mixing the two counters would compare unrelated clocks; and `cycle` has no
* discoverable frequency, so every heartbeat variance and TIME-TRUST figure
* riscv64 produced before this was measured against a wrong expected interval
* (FABRIC-0.md §16.2). Figures from before and after are not comparable.
*/
#include "timer.h"
#include "console.h"
#include "q48_16.h"
#include "uefi.h"
#include "starkernel/fdt.h"
#include <stdint.h>
#include <string.h>
/* FABRIC-2.md §I.5, 2026-09-04: real hypervisor-vs-hardware detection.
* s_cal.vm_mode was hardcoded to 1 unconditionally below -- see
* aarch64/timer.c's own running_under_hypervisor() doc comment for why
* that's wrong to reuse as a general "are we in QEMU" signal elsewhere
* (contrib-capsule trust-tier enforcement, §I.5). RISC-V has no ACPI
* here (this file's own devicetree-only timebase-frequency discovery
* above is the proof) but does have a devicetree, already parsed for
* exactly one other property -- the root node's "compatible" property
* carries QEMU's own machine-model string ("qemu" appears in it for the
* virt board) on every QEMU riscv64 target; real hardware vendors set
* their own compatible strings, never this one. bytes_contain() is a
* tiny local substring search -- no strstr dependency assumed available
* in this translation unit. */
static int bytes_contain(const uint8_t *hay, uint32_t haylen, const char *needle) {
size_t nlen = strlen(needle);
if (nlen == 0 || haylen < nlen) return 0;
for (uint32_t i = 0; i + nlen <= haylen; i++) {
if (memcmp(hay + i, needle, nlen) == 0) return 1;
}
return 0;
}
static int running_under_hypervisor(BootInfo *boot_info) {
if (!boot_info || !fdt_valid(boot_info->dtb)) return 0;
uint32_t len = 0;
const void *prop = fdt_find_prop(boot_info->dtb, "compatible", &len);
if (!prop) return 0;
return bytes_contain((const uint8_t *) prop, len, "qemu");
}
/**
* @brief Read the RISC-V wall-clock counter (@c rdtime, CSR @c time 0xC01).
*
* @c time is the memory-mapped real-time counter mandated by the privileged
* spec: fixed frequency, common to all harts, and — unlike @c cycle — its
* rate is discoverable, published by firmware as the devicetree property
* @c timebase-frequency.
*
* This replaces the earlier @c rdcycle() source. Two reasons, and the first
* is not optional:
*
* - **The SBI TIME extension is defined against @c time.** @c sbi_set_timer()
* takes an absolute value on this counter, so arming a timer from a
* @c cycle reading would compare two unrelated clocks.
* - **@c cycle's frequency is not discoverable**, which is why the previous
* implementation hardcoded an assumed 1 GHz. Every heartbeat variance and
* TIME-TRUST figure riscv64 has produced was therefore computed against a
* wrong expected interval.
*
* @return Current 64-bit @c time value.
*/
static inline uint64_t rdtime(void)
{
uint64_t val;
__asm__ volatile (
"rdtime %0" : "=r"(val));
return val;
}
/* Fallback when firmware publishes no devicetree, or none carrying
* timebase-frequency. 10 MHz is the QEMU virt machine's value. Named rather
* than inlined so that a boot running on the fallback is greppable and
* obviously distinct from a discovered rate. */
#define RISCV_TIMEBASE_HZ_FALLBACK 10000000ULL
/**
* @brief Print @p val in decimal via @c console_putc().
*
* Same shape as the file-local helper in @c arch/amd64/timer.c — there is no
* shared decimal printer in @c console.h, and the freestanding build has no
* @c printf(). Prints "0" for zero.
*
* @param val Value to print.
*/
static void print_dec(uint64_t val)
{
char buf[32];
int i = 0;
if (val == 0)
{
console_putc('0');
return;
}
while (val > 0 && i < (int)sizeof(buf))
{
buf[i++] = (char)('0' + (val % 10));
val /= 10;
}
while (i-- > 0)
{
console_putc(buf[i]);
}
}
static uint64_t s_counter_hz = RISCV_TIMEBASE_HZ_FALLBACK;
static uint64_t s_ns_per_tick = 0;
static uint64_t s_base_count = 0;
static uint64_t s_base_ns = 0;
static timer_calibration_record_t s_cal;
/*
* @brief Initialise the RISC-V timer subsystem (M5 milestone).
*
* RISC-V has no CSR reporting the counter frequency (unlike AArch64's
* @c CNTFRQ_EL0), so the rate is read from the devicetree property
* @c timebase-frequency, located via the blob the firmware publishes under
* @c EFI_DTB_TABLE_GUID and carried in @c BootInfo::dtb.
*
* Steps performed:
* 1. Reads @c timebase-frequency from @c boot_info->dtb when present;
* otherwise keeps @c RISCV_TIMEBASE_HZ_FALLBACK and says so on the console.
* 2. Snapshots @c rdtime() into @c s_base_count as the ns origin.
* 3. Computes @c s_ns_per_tick as @c (1e9 << 16) / @c s_counter_hz in
* Q16.16 fixed-point to avoid floating-point in the freestanding build.
* 4. Fills @c s_cal, setting @c TIMER_TRUST_ABSOLUTE only when the rate came
* from firmware. On the fallback the counter is still invariant, but its
* scaling to real time is a guess — which is @c TIMER_TRUST_RELATIVE.
*
* @param boot_info Kernel @c BootInfo; @c ::dtb supplies the counter rate.
* NULL, or a NULL/invalid blob, selects the fallback.
* @return 0 always.
*/
int timer_init(BootInfo *boot_info)
{
uint32_t hz = 0;
int discovered = 0;
/* Discover the counter rate rather than assuming it. The devicetree is
* the only source: RISC-V has no CNTFRQ_EL0 equivalent. boot_info->dtb is
* NULL when firmware published no devicetree, in which case the named
* fallback stands and the banner says so. */
if (boot_info && fdt_valid(boot_info->dtb))
{
if (fdt_prop_u32(boot_info->dtb, "timebase-frequency", &hz) && hz != 0)
{
s_counter_hz = (uint64_t)hz;
discovered = 1;
}
}
s_base_count = rdtime();
s_base_ns = 0;
s_ns_per_tick = (1000000000ULL << 16) / s_counter_hz;
s_cal.tsc_hz_mean = s_counter_hz;
s_cal.hpet_hz = 0;
s_cal.pit_hz_mean = 0;
s_cal.converged = 1;
s_cal.vm_mode = running_under_hypervisor(boot_info) ? 1 : 0;
/* ABSOLUTE only when the rate came from firmware. On the fallback the
* counter is still monotonic and invariant, but its scaling to real time
* is a guess, which is exactly the RELATIVE case. */
s_cal.trust = discovered ? TIMER_TRUST_ABSOLUTE : TIMER_TRUST_RELATIVE;
console_puts("Timer: RISC-V time CSR @ ");
print_dec(s_counter_hz);
console_println(discovered ? " Hz (devicetree)" : " Hz (FALLBACK, no devicetree)");
return 0;
}
/**
* @brief Return the `time` counter frequency in Hz.
*
* Returns @c s_counter_hz — discovered from the devicetree's
* @c timebase-frequency when available, otherwise
* @c RISCV_TIMEBASE_HZ_FALLBACK. Used by @c apic_timer_init() to compute the
* expected inter-tick period and by the shim time backend to convert `time`
* counts to nanoseconds.
*
* @return `time` counter frequency in Hz.
*/
uint64_t timer_tsc_hz(void)
{
return s_counter_hz;
}
/**
* @brief Return a monotonic nanosecond timestamp.
*
* Reads @c rdtime(), computes the elapsed delta from @c s_base_count
* (captured at @c timer_init()), and converts to nanoseconds using the
* Q16.16 fixed-point @c s_ns_per_tick:
*
* @code
* ns = s_base_ns + ((delta * s_ns_per_tick) >> 16)
* @endcode
*
* The 64-bit product of @c delta × @c s_ns_per_tick wraps once elapsed real
* time reaches 2^48 ns (~3.26 days) — this bound is **independent of
* @c s_counter_hz**, since @c s_ns_per_tick scales inversely with frequency
* and the two effects cancel. Verified by direct computation, not carried
* over from the earlier @c cycle-based comment here, which claimed "~18
* seconds" and did not match the arithmetic either. Beyond ~3.26 days
* uptime @c timer_now_ns() wraps and produces incorrect values — a
* theoretical concern only for this single-hart POST-only kernel.
*
* @return Monotonic nanosecond count since @c timer_init(), starting at 0.
*/
uint64_t timer_now_ns(void)
{
uint64_t delta = rdtime() - s_base_count;
return s_base_ns + ((delta * s_ns_per_tick) >> 16);
}
/**
* @brief Check for `time` counter drift (RISC-V stub — always returns 0).
*
* On x86-64 this cross-checks the TSC against the HPET. The RISC-V @c time
* counter has no independent reference to check against at this milestone —
* and per the privileged spec it is architecturally required to be a fixed,
* synchronised rate across harts, unlike @c cycle. Always returns 0 (no
* drift) to satisfy the common call site.
*
* @return 0 always.
*/
int timer_check_drift_now(void) { return 0; }
/**
* @brief Return a pointer to the timer calibration record.
*
* Returns @c &s_cal, populated by @c timer_init() with the discovered (or
* fallback) frequency and the corresponding trust level — @c
* TIMER_TRUST_ABSOLUTE when @c timebase-frequency came from the devicetree,
* @c TIMER_TRUST_RELATIVE on the fallback. Consumed by @c kernel_main() for
* boot-log reporting and by the heartbeat subsystem's trust init.
*
* @return Pointer to the module-static calibration record; valid for the
* lifetime of the kernel.
*/
const timer_calibration_record_t *timer_calibration_record(void)
{
return &s_cal;
}
/**
* @brief Read the raw counter the riscv64 heartbeat is paced against.
*
* Item 0.8 (FABRIC-0.md §25.1): the shared heartbeat.c now owns
* heartbeat_init()/heartbeat_tick()/heartbeat_service()/heartbeat_ticks()/
* heartbeat_trust()/heartbeat_state(). This is the one piece that stays
* per-architecture -- the same @c rdtime() the timer deadline is armed
* against, not @c rdcycle() or any other source. Must read the same
* counter the deadline was programmed against: @c expected_delta is
* derived from timebase-frequency and is therefore in @c time units;
* measuring the interval with @c cycle instead would difference two
* unrelated clocks.
*
* @return Current `time` CSR value.
*/
uint64_t heartbeat_read_counter(void)
{
return rdtime();
}