Multilevel default cache: HBM (L1) → DRAM (L2) → disk/CXL/remote (L3), by default

The fak kernel owns its KV cache as a kernel object rather than renting it from vLLM or SGLang. That ownership is the whole reason a hardware-aware, multilevel cache is fak’s job and not the serving engine’s: when you own residency, you own which tier a span lives in and when it moves. This epic finishes the job — it makes the kernel place every cached span on the tier that fits it, demote one tier colder under real memory pressure instead of dropping it, and do so by default, against the device that actually exists, on every local level of the hierarchy.

It is the completion-and-downward-extension of the hardware-capacity bridge, epic #706. It does not restate #706’s positioning or rebuild its planes. Read these first:

Status of this document. This is a planning and progress artifact, not a measured benchmark. Every “shipped” claim below names a tested primitive in this repo (file:line), and every rung states the exact dos verb that proves or refutes it. The epic is not done: the honest baseline is that L1/HBM placement is wired into the policy plane but the executor is library-only, and L2/DRAM + L3/disk plan against representative defaults rather than the live box. Those are the rungs that follow.

The crux

There are two planes, and after #706 they meet at exactly one tier.

After #706 the two planes touch only at HBM, and only as a library:

  1. The executor is never called on a live serve loop. CapacityAdapter.Execute / RunCapacityPressureSweep are invoked only from tests and cmd/hwcachedemo / cmd/cxlpooldemo. Nothing in internal/gateway / internal/agent / the served decode path runs the sweep when KV pressure rises. (This was the #706 baseline; the live serve-path call site landed in #1073 and is the kernel default per #987 — see the MLCACHE3 rung below.)
  2. Only HBM has real pressure. DeviceHBMPressure (internal/engine/capacity_pressure.go) derives live HBM fullness from compute.DeviceMemoryInfo. There is no DeviceDRAMPressure / disk-pressure equivalent — L2/DRAM, disk, and CXL plan against cachemeta.DefaultTierProfiles’ representative numbers and an assumed-empty pressure of
    1. So the demote ladder is hardware-aware at its top rung and blind below it.
  3. storedrv routes orthogonally to the policy. Router picks a tier by payload size and a Hint (plane/scope/taint/durability); it consumes no PlacementDecision and no TierPressure. The KV-residency policy and the payload-storage tiering are two systems that never consult each other.
  4. The whole path is off by default. storedrv only registers when FAK_STORE is set; the capacity sweep has no enablement at all because no serve loop calls it.

So the one-line crux: the hardware-aware cache is a decision the kernel computes but does not yet act on, knows only for the top tier, and does not run by default. This epic closes those four, lowest-risk-and-highest-proof first.

The tier map — what is shipped-live, shipped-but-library-only, unbuilt

Levels named like CPU caches: L1 = HBM (GPU memory, when present), L2 = DRAM (host RAM), L3 = disaggregated (local SSD/disk, then CXL far-memory, then a remote pool).

Tier Capacity report Live pressure → policy Executor wired to serve Status
L1 / HBM ✅ cuda.go DeviceMemory() → cudaMemGetInfo ✅ DeviceHBMPressure ❌ library-only (CapacityAdapter uncalled on serve) shipped policy, executor not live
L2 / DRAM ✅ HostSystemMemoryInfo() (per-OS probe) ❌ no DeviceDRAMPressure ❌ probe exists, pressure-wire unbuilt
L3 / disk ❌ no filesystem-space probe ❌ ⚠️ storedrv blobfs tiers by size, not by pressure unbuilt (policy plans against defaults)
L3 / CXL ❌ no CXL HAL backend ❌ ⚠️ profile + share-kind exist; no device metadata-only; physical move unbuilt
L3 / remote n/a (unbounded) n/a ⚠️ storedrv blobhttp; not policy-driven transport exists, not policy-linked

The shipped foundations every rung rides on, so no rung rebuilds them:

The progress spine — prove or refute, rung by rung

Each rung names a claim, the build that would make it true, and the witness: the exact dos verb (and the assertion behind it) that proves it shipped or refutes a premature “done.” A rung is not done until dos verify <plan> <phase> returns shipped:true and the ship commit passes dos commit-audit (claim matches diff). The plan id for the whole spine is MLCACHE.

Ordered by proof-per-hour: cheapest, hardware-free, highest-certainty proofs first; the GPU-gated and cross-node rungs last.

Rung 1 — L2/DRAM live pressure (MLCACHE1) — hardware-free, ships from any box

Claim. The placement policy plans the DRAM tier against the host’s real RAM fullness, not the DefaultTierProfiles placeholder — so under host-memory pressure a span that would have demoted into DRAM instead skips to a colder tier with room.

Build. A DeviceDRAMPressure(b, residentBytes) → (pressure, capacity, known) mirroring DeviceHBMPressure, sourced from compute.HostSystemMemoryInfo() (backend-free, so it works on the cpu-ref floor), folded into the request by a withDRAMPressure / withDRAMCapacity copy-on-write exactly like the HBM wire. Fail-open: known=false → request used verbatim.

Witness. A test proving the flip: with TierDRAM pressure forced to 1.0 the coldestColderWithRoom walk skips DRAM and the decision’s ToTier is the next colder tier (NUMA-far/CXL/disk) — the same shape as #707’s HBM keep→demote flip test. Refute condition: if the test passes with the wire absent (i.e. the default already produced the colder target), the rung proves nothing and must be reframed.

Status: SHIPPED 8245e5de (#985, #986–988 track the rest). internal/engine/capacity_dram.go

Honest witness note. dos commit-audit 8245e5de returns ABSTAIN — not because the diff is empty (it adds the wire + tests) but because the subject verb “derive” is outside the referee’s claim-kind verb set, so it makes no checkable claim to the auditor. dos review over the range is CLEAN (the abstain is unverifiable, not a residual). And dos verify MLCACHE MLCACHE1 is shipped:false because MLCACHE/MLCACHE1 are this doc’s own plan/phase convention with no registry row or commit marker in the stamp grammar. So the rung’s real evidence is the passing refute-guarded test + the diff-witnessed file set, and the lesson for later rungs is concrete: lead the ship subject with a referee-recognized verb (add/feat … add) and carry the MLCACHE<n> marker, so both dos commit-audit and dos verify bind without an amend on the shared trunk.

First-ship target. This is the leading rung precisely because it needs no GPU and no serve loop — it is provable on the Windows dev box and in CI, and it removes the “only HBM is hardware-aware” asymmetry that makes the whole ladder blind below its top rung.

Rung 2 — L3/disk live pressure (MLCACHE2) — hardware-free

Claim. The spill decision (demote to disk) plans against the real free space on the spill filesystem, so a full disk forces eviction instead of a spill that would fail.

Build. A DiskPressure(path) → (pressure, capacity, known) over a cross-platform statfs/GetDiskFreeSpaceEx probe (new internal/compute file with per-OS build tags, matching the hostmem_* shape), wired into the request’s TierDisk profile/pressure.

Witness. A test: disk pressure 1.0 → a span that beats recompute but has no colder room evicts (no_colder_tier_with_room) instead of spilling. dos verify MLCACHE MLCACHE2.

Rung 3 — executor on a live loop (MLCACHE3) — the keystone

Claim. The served decode path runs the capacity sweep under real pressure: when HBM (or DRAM) crosses a high-water mark, the live KV candidates are demoted/spilled via CapacityAdapter, and the move is visible on /metrics as a real fak_engine_cache_* transition — not a synthetic demo.

Build. Call RunCapacityPressureSweep from the in-kernel served decode boundary (internal/gateway / the engine decode loop) with the live resident KV candidates and the high-water target, behind a default-on policy with a documented disable. This is the “executor wired to serve” cell the tier map marks ❌ for every tier today.

Witness. A served-path test (or a cmd/ harness on a real backend) showing a demote transition recorded through the cache-event stream during decode under pressure, plus the honest fence in the explainer flipped from “Nothing here claims the serving loop demotes KV under live pressure” to a cited, tested claim. dos verify MLCACHE MLCACHE3 + a dos commit-audit confirming the diff touches the serve path, not just a test.

This rung is gated on a backend that can hit real pressure (the datacenter GPU/L4 fleet via the Slack bridge, or a synthetic-pressure backend in test). It is the highest-value rung and the one most likely to be refuted by “it only ran in a demo” — so its witness must show a serve-path call site, not a library call.

Status: SHIPPED ca5aad6a (#1073, the live serve-path call site). internal/gateway/ kvmmu_pressure_relief.go adds maybeRelieveKVPressure, called from complete()’s post-decode success tail (the boundary right after observePlannerRequestMemory), driving two host-injected, import-clean seams — KVPressureCandidateProvider (live resident spans) and KVPressureSweeper (the engine.CapacityAdapter closure). cmd/fak/kvmmu_pressure_bridge.go is the host glue (twin of kvmmu_slot_bridge.go) that builds the sweeper over a live compute.Backend + CapacityAdapter and lowers candidates into engine.CapacityPressureCandidate. Gated behind FAK_INKERNEL_KVMMU, fail-open (nil seams / empty candidates ⇒ a no-op, byte-identical to before), FAK_KV_HIGHWATER overrides the 0.80 mark. The demote folds into the fak_engine_cache_* cache-event stream automatically via the adapter’s recorder.

Witness. TestMaybeRelieveKVPressureDemotesUnderPressure proves demote-not-drop under simulated HBM pressure (StageSpan → Evict, AppliedMoves==1, a ddr_cache cache-event row), with a refute guard (…GateOffIsNoOp) that asserts the gate-off / no-seams path is inert — so the test proves the WIRE, not a default-on behavior change. dos commit-audit ca5aad6a returns OK / diff-witnessed (the add-led subject made the claim checkable, where MLCACHE1’s “derive” abstained), dos review over the ship range is CLEAN (zero residual), and the diff touches the serve path (internal/gateway/gateway.go), not just a test.

Update — default-on policy + fence flip (#987). The post-decode sweep is now the kernel default: the FAK_INKERNEL_KVMMU opt-in #1073 first shipped it behind is lifted, replaced by the documented disable FAK_KV_PRESSURE_RELIEF=off (internal/gateway kvPressureReliefEnabled) — the “default-on policy with a documented disable” this rung’s Build asks for. The explainer’s honest fence (docs/explainers/hardware-limits-and-capacity.md) is flipped from “Nothing here claims … the serving loop demotes KV under live pressure” to the cited, tested claim that the served decode boundary demotes a hot span by default under simulated pressure (TestMaybeRelieveKVPressureDemotesUnderPressure, now run with no enablement env set so it witnesses the default). The diff touches the serve path (internal/gateway/kvmmu_pressure_relief.go), not just a test.

Honest fence (still in force). The production provider is nil at the serve.go call site: InKernelPlanner keeps residency in a radix reuse tree and builds a kvmmu.Context ephemerally per eviction, so there is no durable resident-span list to enumerate yet. So this rung ships the LIVE, non-test call site + the synthetic-pressure demote test; it does NOT yet assert the served loop demotes a real span under real GPU pressure. That last step — a persistent span enumerator over kvmmu.Segment{From,Len,KV} feeding a non-nil provider — is the follow-on #1074 / #987. As with MLCACHE1, dos verify MLCACHE MLCACHE3 is shipped:false because MLCACHE/MLCACHE3 are this doc’s own plan/phase convention with no marker in the stamp grammar; the rung’s real evidence is the diff-witnessed commit-audit + the passing refute-guarded test.

Rung 4 — default profiles from the live box (MLCACHE4) — hardware-free

Claim. On startup the kernel replaces DefaultTierProfiles’ representative order-of-magnitude numbers with the tiers the box actually has (HBM capacity from the CUDA probe, DRAM capacity from the host probe, disk capacity from rung 2’s probe) — so a box with no GPU has no HBM tier in its ladder, and a box with 1.5 TB RAM plans against 1.5 TB.

Build. A ProbedTierProfiles(b) map[ResidencyTier]TierProfile that starts from the defaults and overrides each tier’s CapacityBytes from the live probes, dropping tiers the box cannot prove. Surfaced on /debug/vars so the operator can see the ladder the kernel chose.

Witness. A test asserting a no-GPU profile set omits TierHBM and a probed DRAM capacity differs from the placeholder; dos verify MLCACHE MLCACHE4.

Rung 5 — storedrv consults the placement policy (MLCACHE5)

Claim. The payload-storage router and the KV-residency policy stop being orthogonal: a Router.Put of a large, durable, fleet-scoped payload lands on the tier the placement policy would choose under current pressure, not purely on its byte size.

Build. A thin adapter that lets storedrv.Router take an optional cachemeta-derived placement hint (pressure-aware tier selection) without changing the frozen abi.Resolver default path — additive, off unless the caller opts in, the same posture PutHinted already takes.

Witness. A test: under high hot-tier pressure a payload that Accept(size) would keep hot instead routes durable; dos verify MLCACHE MLCACHE5. Refute condition: if this only duplicates PutHinted’s size/durability routing, it adds no tier-awareness and must be cut.

Rung 6 — L3 CXL / disaggregated physical move (MLCACHE6) — cross-node, gated

Claim. A span demoted to CXL/remote is physically relocated (not just marked) and stays attendable/recoverable, closing the gap the L3 study (#79) and the CXL pool doc name.

Build. Tracked by #79’s children B–E (the L3RegionBackend behind the frozen Resolver seam). This epic does not rebuild the external L3 store; it consumes it. Deferred behind rungs 1–5 and an external L3 instance.

Witness. Per #79’s definition of done; dos verify against the child that lands it.

Honest boundaries (carry into every rung)

Parent epic: #706 (hardware-capacity bridge). Sibling: #79 (L3 disaggregated positioning).