How fak interfaces with hardware limits — capacity is the eighth assumption

Status: this is an incremental bridge, not a finished OOM subsystem. compute now exposes a classed capacity fit contract (DeviceCapacity, optional HostCapacity, MemoryPlan), the policy plane has a real device-pressure seam, the engine has a demote/spill executor, and fak serve --gguf --backend refuses a known-too-large GGUF before the device load allocates. For quantized-upload backends that fit check is mixed precision: lean-Q8 resident weights plus f32 HAL KV, activation, and scratch estimates; f32 weights are now only the fallback for backends that cannot consume quantized uploads yet. --cpu-offload-experts uses a split plan: dense/router/attention weights plus KV and HAL transients consume device capacity while expert weights are host-scoped offload bytes. Host-scoped offload/DDR bytes can refuse when a native GPU backend explicitly advertises HostCapacityProbe from the OS host-memory probe; otherwise they stay visible and fail open. On a successful eager GGUF device load, the same classed admission profile is visible on /metrics (fak_model_load_memory_plan_bytes, fak_model_load_memory_plan_dtype_bytes, fak_model_load_memory_capacity_*, fak_model_load_memory_fit_bytes) and /debug/vars (detailed rows with dtype/storage labels plus the capacity snapshot). Runtime device allocation failures now recover as a typed, classed in-kernel OOM on the served path, and served backend requests now run a prompt+max_tokens KV/HAL-transient capacity precheck before device decode when capacity is known. Both failure paths publish classed counters on /metrics plus last-site drilldown on /debug/vars, and the latest served backend request’s admission plan is visible as fak_gateway_in_kernel_request_memory_* gauges, including per-scope want/budget/margin rows, plus /debug/vars.request_memory; cumulative observed request-plan totals and high-water rows are exposed as fak_gateway_in_kernel_request_memory_*_total / *_high_water* series and /debug/vars.metrics.request_memory*. The one served decode retry is also explicit: fak_gateway_in_kernel_oom_retry_total reports idle-pool trim retry attempts, successes, and failures by backend/class, while /debug/vars keeps the last trigger site. The request precheck can also make one pre-decode idle-pool trim when known current-free pressure is already over budget or inside the low-margin band; those actions are counted as fak_gateway_in_kernel_memory_pressure_trim_total with sizing gauges and /debug/vars.metrics.in_kernel_pressure_trims. The served decode boundary now runs the capacity-pressure sweep by default (#987, the MLCACHE3 keystone): after a served turn mutates the KV cache, internal/gateway’s maybeRelieveKVPressure demotes a hot KV span to a colder tier (StageSpan → Evict) once HBM pressure crosses the high-water mark, and the move lands on /metrics as a fak_engine_cache_* transition — witnessed end to end by the served-path test TestMaybeRelieveKVPressureDemotesUnderPressure under simulated HBM pressure, with FAK_KV_PRESSURE_RELIEF=off the documented disable. What is still not claimed: that allocator OOM can spill/demote live KV or weights, that this default-on edge demotes a real resident span under real GPU pressure in production today (its resident-span provider is nil until the durable kvmmu.Segment enumerator lands — the gated follow-on #1074), or that every model loader has a complete memory plan yet. Those remain tracked planks below.

Who this is for: contributors reasoning about what fak does when it runs out of memory — on a GPU, on the host, across a tier hierarchy. Prerequisites: the hardware-portability explainer (the internal/compute HAL seam) and the hardware-aware cache doc (the cachemeta placement plane). This explainer sits between them.

1. Shape versus limit

The hardware-portability explainer makes a precise claim: the internal/compute HAL neutralizes seven host-CPU assumptions — float32 monoculture, host-pointer aliasing, x86 build-tag dispatch, synchronous return, goroutine-only parallelism, row-major layout, and eager full-RAM residency. Read the list again and notice what they have in common: every one describes the shape of the hardware — its dtype, its address space, its ISA, its execution model, its memory layout, how it obtains weights. Lift all seven and a new backend is a registration, not a fork.

None of the seven describes the one thing every accelerator actually runs out of: finite, exhaustible memory. A GPU is not just a different shape of compute; it is a smaller pool of it. The faithful-model ceiling the hardware matrix reports — “fak faithful ≤ 7B on the 36 GB Mac” — is not a shape fact. It is a capacity fact. And capacity is the assumption the HAL has not yet lifted.

This matters more for fak than for a typical serving engine, because fak owns the KV cache as a kernel object rather than renting it from vLLM or SGLang. When you own the cache, you own its residency — and residency is governed by capacity. So “how much fits” is not a detail fak can delegate; it is load-bearing for the central design claim.

2. Two planes that do not touch

Search the tree for how fak reasons about running out of memory and you find two separate worlds, each real, with no control path between them.

Plane A — the policy plane (internal/cachemeta)

This plane is genuinely good, and it is built and tested. It names where a cached span can live (ResidencyTier: HBM → DRAM → NUMA-far → CXL → Disk → Remote) and it carries the physical character of each tier in a TierProfile — CapacityBytes, BandwidthMBPerSec, ReadLatencyNanos, ByteAddressable, Coherent. Its PlanPlacement makes the one decision a blind LRU cannot: under pressure, demote a hot prefix one tier colder rather than evict it and pay a full re-prefill (RetainCheaperThanRecompute quantifies the trade-off; Lifecycle.DemoteOnExpiry ages a span down per-tier). The hardware-aware cache doc explains it in full, and cmd/hwcachedemo runs it: a 4000-token prefix relocating demote → demote → demote → spill → evict under escalating pressure, with 0 tokens re-prefilled versus 28000 for blind LRU.

The honest boundary that doc states about itself is the whole point of this one:

the policy/metadata plane … decides where a cached span should live and when it should move … The physical byte movement … is performed by the engine adapter that consumes those directives — this plane touches no bytes.

So the plane plans. Something else is supposed to act. Read on for what that something is.

Plane B — the physical plane (the compute HAL allocators)

This is where bytes are actually allocated and actually run out: cudaBackend.dalloc, metalBackend.dalloc, vulkanBackend.dalloc. Here is what they do when the device is full:

Historically the capacity information the physical plane had was not part of the HAL contract: CUDA’s totalGlobalMem was read and discarded, and Vulkan’s device-local heap size stayed behind the shim. compute.DeviceCapacity is the typed way out: Caps.DeviceMemory keeps its old shape meaning (“resident tensors are not host-addressable”), while Caps.CapacityProbe plus DeviceMemory() says “this backend can report N bytes.”

The gap: the planes meet only at the meter, never at the control

   POLICY PLANE (cachemeta)                 PHYSICAL PLANE (compute HAL)
   ┌───────────────────────────┐           ┌───────────────────────────┐
   │ PlanPlacement             │           │ cudaBackend.dalloc  panic │
   │  demote / spill / evict   │           │ metalBackend.dalloc panic │
   │ TierProfile.CapacityBytes │           │ vulkanBackend.dalloc spill│
   │ TierPressure (fullness)   │           │ totalGlobalMem  discarded │
   │ Lifecycle.DemoteOnExpiry  │           │ Caps: no capacity field   │
   └───────────┬───────────────┘           └─────────────┬─────────────┘
               │                                          │
        plans against                              allocates against
   DefaultTierProfiles (placeholder              the real device, blindly,
   GB) + SYNTHETIC pressure                      and PANICS when it's full
   (cmd/hwcachedemo feeds it)
               │                                          │
               └──────────────► (no control path) ◄───────┘
                         meet only here:
               internal/engine CacheEventRecorder  (metrics)

The policy plane plans demote-not-evict beautifully — against DefaultTierProfiles, whose own comment calls them “representative order-of-magnitude defaults, not measurements of any particular box,” and against TierPressure values that used to be synthetic only: cmd/hwcachedemo injected escalating pressure by hand. Plank 3 now derives HBM pressure from the real backend and RunCapacityPressureSweep binds that signal to the Plank-4 adapter for caller-supplied candidates, and the live serving loop now invokes that sweep by default at the post-decode boundary (#987, internal/gateway’s maybeRelieveKVPressure) — only the production resident-span provider that would feed it real GPU candidates is still nil (the gated follow-on #1074). Meanwhile the physical allocators that do know the real pressure — they hold totalGlobalMem, they are the ones that hit nullptr — still do not spill/demote from an OOM site by themselves. The served layer can now make one idle-pool trim retry around a typed allocation OOM, but allocator-site spill/demote and automatic live admission policy remain the integration gap.

3. The eighth assumption

Stated in the form the portability explainer uses for the first seven:

# Assumption Where it lives today What it shuts out
8 uniform / infinite capacity — a backend can hold whatever it is handed; “out of memory” is a process abort, not a value legacy raw dalloc panics; Caps.DeviceMemory is a shape bool, not a capacity report; unprobed backends still fail open any device too small for the model — i.e. every real serving target above its faithful ceiling

Lifting it follows the same discipline as the other seven: neutralize it in the type system first, even though only some backends can satisfy it today. That is what compute.DeviceCapacity does (§4). The portability doc’s own framing applies verbatim — the contract “already assumes none of [the assumptions], even though only the CPU reference is implemented today.” Capacity is now one more capability a backend may report, discovered the same way CollectiveBackend is (a Caps bool plus a type-assert), absent by default, with the core path failing open when it is absent so the portable floor is never blocked.

4. The bridge, plank by plank

The fix is not one change; it is a bridge with several planks. Several planks now ship, and the remaining ones are named here so the deferral is tracked, not forgotten.

Plank 1 — backends REPORT their ceiling (compute.DeviceCapacity). Shipped. A backend that can probe its own memory implements DeviceMemory() (total, free int64, known bool). DeviceMemoryInfo(b) and FitsOnDevice(b, wantBytes, headroom) let any caller ask without knowing the concrete backend, and they fail open: a backend that cannot answer (the pure-Go cpu-ref floor, a wasm target) reports known=false, and the fit check returns FitUnknown (“proceed”), never FitTooBig. CUDA now reports cudaMemGetInfo total/free bytes, the Windows Vulkan backend reports the sum of its device-local heaps plus current free budget when VK_EXT_memory_budget is available, and Metal reports the device’s recommended working-set size as a known total when the driver exposes it. Backends whose current free bytes are unavailable return FreeUnknown, so FitsOnDevice falls back to the total ceiling and still catches a model that cannot fit the whole device. This is the report half of the bridge: the wire by which a real backend’s capacity can feed cachemeta.TierPressure instead of placeholder profiles. The same contract now has a classed form: MemoryPlan / MemoryDemand keeps weights, KV cache, DDR cache, offload staging, activations, and scratchpad bytes distinct, while FitsMemoryPlan / RefuseMemoryPlanIfTooBig still fail open on unknown capacity and carry the class breakdown into the typed FitError. Device-scoped demands are checked against DeviceCapacity; host-scoped demands are checked against optional HostCapacity only when Caps.HostCapacityProbe is set, otherwise those host bytes remain visible but do not refuse. The native GPU backends now advertise that host side when the stdlib OS probe succeeds: Windows uses GlobalMemoryStatusEx, Linux uses sysinfo, and Darwin reports the physical-memory ceiling via hw.memsize with free=FreeUnknown.

Plank 2 — OOM becomes a typed value, not a panic. Partially shipped. dalloc should return a typed allocation fault — the exact shape abi.KVResidencyFault already models for the off-box KV direction (“a failed restore is NEVER a silent recompute … typed, never a hang”). The shipped slice is the in-kernel served path: compute.DeviceAllocError carries the failed byte count, allocator site, and memory class (weights, kv_cache, offload, scratchpad, activation, or unknown); CUDA, Metal, and Vulkan allocation choke points raise it instead of a raw string panic where the backend allocator returns nil; agent.InKernelPlanner recovers only that typed allocation fault; and the gateway maps it to a 503 in_kernel_oom with a client-safe, class-visible remedy. The same served planner also refuses a known-too-large request before allocation when the prompt+max_tokens KV window plus HAL activation/scratch/logit transients exceed the reported device budget; it also includes resident weights when the backend reports total capacity but not current free memory. That refusal returns agent.InKernelCapacityError with site capacity-precheck. The visibility slice is also live: /metrics publishes fak_gateway_in_kernel_oom_total, fak_gateway_in_kernel_oom_failed_bytes_total, and fak_gateway_in_kernel_oom_last_failed_bytes by memory class for both allocation OOMs and capacity precheck refusals, while /debug/vars includes the last site for drilldown without turning dynamic sites into Prometheus labels. The narrow served decode retry is visible too: fak_gateway_in_kernel_oom_retry_total publishes attempted/succeeded/failed counts by backend and memory class, fak_gateway_in_kernel_oom_retry_last_failed_bytes records the triggering allocation size, and /debug/vars.metrics.in_kernel_oom_retries carries the last trigger site. The pre-allocation side now has a narrow action too: when current free bytes are known and the request plan is already over its headroom-adjusted budget or close to it, the served planner asks the backend to release idle pools before decode (Recycle, Trim, or TrimLarge(0)) and reruns a refused capacity check once. That pre-decode action is visible as fak_gateway_in_kernel_memory_pressure_trim_total by backend/scope/class/ reason/outcome plus fak_gateway_in_kernel_memory_pressure_trim_last_bytes and /debug/vars.metrics.in_kernel_pressure_trims. Honest fences: allocation OOM recovery is still panic/recover below the HAL, unknown capacity still fails open, generic allocation sites can still be unknown, resource-cap validation panics are not OOM, and no caller yet spills or demotes live KV based on the class. Both the pre-decode pressure trim and the post-OOM retry only release idle transient pools; they do not move live KV spans, demote weights, or create a host DDR cache. A truly-too-large request still returns the typed OOM/capacity refusal.

Plank 3 — feed real pressure into the policy plane. Shipped (#707). internal/engine.PlanPlacementForDevice is the report→policy wire: it derives a live cachemeta.TierPressure (and the HBM tier’s real CapacityBytes) from the active backend’s device memory via compute.DeviceMemoryInfo(b), folds them into the request, and calls PlanPlacement — so the policy plans against the device that actually exists instead of DefaultTierProfiles’ representative numbers and the demo’s hand-injected TierPressure. It fails open: a backend that cannot probe (the cpu-ref floor, a wasm target; known=false) contributes no pressure and no capacity override, so the profile default stands and no path that works today regresses. When a producer reports current free bytes (CUDA, and Vulkan when VK_EXT_memory_budget is exposed), HBM pressure derives from current headroom; when free is unknown it falls back to total vs fak’s tracked-resident bytes. engine.RunCapacityPressureSweep now binds that report→policy wire to the Plank-4 adapter for a bounded list of KV candidates: it scales observed pressure to an operator high-water mark, plans each span, executes demote/spill/evict through CapacityAdapter, and stops once estimated HBM pressure falls below target or the candidate set is exhausted. Witness: go test ./internal/engine -run "Test(CapacityPressureSweep|PlanPlacementForDevice)" (a high-water device flips a hot prefix from keep → demote, stages it to DRAM, and records the move as ddr_cache; cpu-ref is unchanged). The live serving loop now calls that sweep by default at the post-decode boundary (#987); what remains is the production provider supplying real pressured KV candidates from a durable resident-span ledger (the gated follow-on #1074).

Plank 4 — the engine adapter that EXECUTES a placement directive. Shipped (#708). internal/engine.CapacityAdapter.Execute is the control path the diagram in §2 was missing: it turns a PlanPlacement demote/spill decision into a real abi.KVBackend.StageSpan (stage to the colder tier, addressed by digest) PLUS an abi.KVBackend.Evict (the re-RoPE/renumber eviction from the live KV tier) against the kernel-owned cache. It is fail-safe — the live copy is staged before it is dropped, so a typed staging MISS/FAULT retains the span rather than losing it — and it records every transition through the same CacheEventRecorder as a typed offload event, so a staging fault is never a silent recompute. Witness: go test ./internal/engine -run TestCapacityAdapter. The mechanical primitives it leans on were already in place — kvmmu.Context.ApplyPlan does bit-exact eviction to match a plan, and model/paging.go: pagedKernel pages a weight in on demand — but neither was driven by live capacity pressure. This adapter is the demote/spill executor that consumes a placement directive, and the reusable pressure sweep now performs the Plank-3 → Plank-4 bind for caller-supplied candidates. The served decode path now invokes that sweep automatically and by default (#987, maybeRelieveKVPressure); what remains is the production resident-span provider that supplies real candidates (the gated follow-on #1074).

Plank 5 — load-time and request-time fit pre-checks. Partially shipped. FitsOnDevice before the make/append in the model loaders (model: Load, LoadSafetensors, ggufload.LoadModel) turns “OOM panic mid-load” into “this needs ~W GB, the device has ~A GB” before a byte is allocated. The shipped slice is the GGUF path: ggufload.WeightSource can estimate the raw payload plan (EstimateLoadMemoryPlan) and the f32-resident plan used by LoadModel (EstimateF32LoadMemoryPlan) from the header, and FitOnDevice / FitF32OnDevice route that plan through the classed compute refusal. The HAL also exposes EstimateKVStoreMemoryPlan, which counts the current device KV layout (Kraw, K, and V as f32 rows) from KVConfig plus a planned token window, and EstimateHALTransientMemoryPlan, which adds a conservative per-token HAL activation and scratchpad estimate from the same model geometry. fak serve --gguf --backend now resolves the backend before eager model load and, for the non-offload device path, refuses a known-too-large classed plan. If the backend advertises quantized upload, the plan uses the lean/raw Q8 proxy plus f32 KV, activation, and scratch estimates from the GGUF context_length (or the smaller --context-budget-tokens window); a backend without quantized upload falls back to f32-resident weights with the same f32 runtime demands. --cpu-offload-experts uses the same header-only path but partitions the GGUF tensor directory through the runtime offload predicate: dense/router/attention weights stay device-scoped, routed/shared experts become host-scoped offload demands, and KV plus HAL transients stay device-scoped. If a backend also advertises HostCapacityProbe, the same classed pre-check can reject an expert-offload plan that exceeds reported host capacity; if the OS probe is unavailable the host-scoped bytes still fail open. The remaining 15% device headroom is still reserved for allocator fragmentation, backend pools, and unmodeled runtime uploads. After a successful eager device load, the gateway publishes the exact admission profile it used: /metrics aggregates fak_model_load_memory_plan_bytes by class+scope, adds fak_model_load_memory_plan_dtype_bytes by class+scope+dtype so mixed-precision plans are explicit, exports device/host capacity known/free-known/byte gauges, records the headroom ratio, and adds fak_model_load_memory_fit_bytes want/budget/margin rows by scope so the headroom-adjusted fit is visible without operator arithmetic; /debug/vars keeps the detailed memory-plan rows, dtype labels, capacity snapshot, and memory_fit rows for drilldown. The served in-kernel backend path now reuses the same classed fit contract per request: after tokenizing the transcript and applying the request’s max_tokens, it checks the planned KV window plus HAL activation/scratchpad demand before entering Prefill/Step. When the backend reports current free bytes, resident weights are not counted again because that free value already reflects the loaded model; when only total capacity is known (FreeUnknown) or capacity is unknown, the check includes the resident-weight estimate or fails open respectively. The gateway also exposes the latest observed backend request plan as fak_gateway_in_kernel_request_memory_plan_bytes by backend+class+scope+dtype, token-window gauges, capacity-known/free-known gauges, fak_gateway_in_kernel_request_memory_fit_bytes want/budget/margin rows by backend+scope, and /debug/vars.request_memory, so ordinary successful requests are inspectable before a refusal occurs. In parallel it keeps process-lifetime request-plan aggregates: observed-plan counters, per-class byte totals, per-class high-water gauges, token totals/high-water gauges, and fit want high-water plus known-capacity margin low-water rows under the same Prometheus prefix and /debug/vars.metrics.request_memory*. Honest fences: unknown capacity still fails open, host memory is physical/OS memory rather than cgroup/container quota, safetensors and generic model.Load are not wired yet, runtime upload pre-checks are still allocator-side, and the request-memory telemetry is not a spill/demote policy or a retry mechanism.

Plank 6 — the capacity ESCAPE when one device is not enough. When the model does not fit any single device, capacity is solved by spreading it: multi-GPU tensor/expert parallelism and CPU/NVMe offload. That track is real and separately planned in the native-753B staged plan (Pillars 3 and 4), with its own honesty ledger: the collective seams exist (CollectiveBackend, LocalCollective, DistComm) but a real cross-device NCCL collective does not yet, and today’s “CPU-offload” is compute-placement, not tensor paging. This explainer is the single-device capacity story; that plan is the multi-device one.

5. What already exists (so the gap is stated honestly)

Capacity is not unhandled in fak — it is handled in scattered, isolated places that do not add up to a HAL contract. Crediting them is what keeps the gap claim honest:

Mechanism What it does Why it is not the bridge
vulkan.go: dallocWeight / cuda.go: dallocWeight (FAK_GPU_BUDGET_MB) spills the cold weight tail when a device-local budget is exceeded: host-visible on Vulkan, managed memory on CUDA weights-only; env-set, not a device query; not in the HAL contract
cachemeta.PlanPlacement + TierPressure full demote-not-evict placement policy wired into cmd/hwcachedemo/cmd/cxlpooldemo + tests, not the live model/serving allocation path
residency.Manager + polymodel.Pool (ErrTooLarge) a backend-agnostic resident-weight-byte budget with LRU page-out policy-only; caller-supplied budget, not VRAM-derived; not wired to CUDA/Metal Upload
model/paging.go: pagedKernel upload → compute → free, an on-demand page-in primitive standalone; bit-equal to resident, but not integrated into the live weight HAL
engine.CacheEventMetrics memory-class projection KV residency events expose memory_class alongside to_tier: HBM is kv_cache, DRAM/NUMA-far/CXL are ddr_cache, and Disk/Remote/Provider are offload; byte/token breakdowns are exported as fak_engine_cache_*_breakdown_total series visibility only; it does not yet drive spill/retry policy or allocate a DRAM cache
engine.RunCapacityPressureSweep + CapacityAdapter bounded report→policy→execute loop for caller-supplied KV candidates: derives HBM pressure from the backend, treats an operator high-water mark as “full”, plans demote/spill/evict with cachemeta.PlanPlacement, executes via StageSpan then Evict, and records DRAM/CXL moves as ddr_cache and disk/remote/provider moves as offload now called by the live serving loop by default (#987, maybeRelieveKVPressure), but its real-candidate provider is nil in production until the durable resident-span enumerator lands (#1074); candidate ordering, resident-byte accounting, restore, and retry policy still belong to the serving/cache owner
compute.DeviceAllocError / agent.InKernelCapacityError → gateway in_kernel_oom + /metrics runtime device allocation failure becomes a typed served 503 with bytes + memory class; served backend requests also refuse a known-too-large prompt+max_tokens KV/HAL plan before allocation with site capacity-precheck, but first make one idle-pool trim when current-free pressure is over budget or in the low-margin band and rerun the fit check; both remaining refusals and allocation OOMs are counted by class on /metrics and drillable by last site on /debug/vars; after an allocation OOM, the served decode path can make one idle-pool trim retry and exposes attempted/succeeded/failed counts as fak_gateway_in_kernel_oom_retry_total; pre-decode trims are exposed as fak_gateway_in_kernel_memory_pressure_trim_total; the latest served backend request plan is separately visible as fak_gateway_in_kernel_request_memory_*, including per-scope want/budget/margin fit rows, and process-lifetime request-plan totals/high-water rows are exposed under the same prefix plus /debug/vars.metrics.request_memory*; device weight, KV-cache, offload, scratchpad, HAL activation upload, GLM-DSA backend activation, and paged-weight page-in sites carry dedicated classes where the backend knows the purpose classification/admission, one pre-decode idle-pool trim, one post-OOM idle-pool retry, and visibility only; no live spill/demote policy yet, ambiguous generic alloc sites stay unknown, detailed request-plan gauges remain last-request snapshots, aggregate request rows reset with the process and are not histograms, and the request precheck fails open when capacity is unknown
ggufload.WeightSource.FitOnDevice / FitF32OnDevice / FitCPUOffloadExpertsOnDevice + compute.EstimateKVStoreMemoryPlan / EstimateHALTransientMemoryPlan + gateway model-load memory telemetry header-only GGUF load refusal before the Go process allocates resident weights, with classed HAL KV-cache, activation, and scratchpad estimates when GGUF config exposes context geometry; expert-offload plans keep host expert bytes visible without counting them against device capacity, and optional HostCapacity can refuse known-too-large host-scoped offload/ddr_cache plans; successful eager device loads expose the admission profile on /metrics and /debug/vars, including bounded dtype/storage labels and per-scope want/budget/margin fit rows shipped for GGUF serve’s device load paths; quantized-upload backends are explicitly mixed precision (Q8 resident weights, f32 runtime state); f32 weights are only the fallback for backends without quantized upload; host capacity is OS physical memory, not cgroup/container quota; telemetry is visibility only, not spill/retry; not yet safetensors, generic model.Load, upload pre-checks, or backend-specific transient peaks
agent.KVMemoryReporter + gateway fak_gateway_kv_memory_* local in-kernel KV-prefix residency reports as kv_cache: bytes per KV position from compute.EstimateKVStoreBytes, true resident PrefixTokens, estimated resident bytes, configured radix LRU budget tokens, current LRU edge-token count, tree shape, splits, and LRU vs policy evictions; it also emits capacity-known/free-known, capacity bytes, headroom, and resident KV want/budget/margin fit rows; fak_gateway_kv_memory_dtype_info and /debug/vars carry the KV row dtype (f32 today) and the same fit snapshot visibility over the CPU-backed radix KV prefix cache and its known budget-vs-true-footprint gap; the host budget includes free + resident when free memory is known so the current cache is not counted against itself; device-HAL serve currently reports per-token geometry/capacity with enabled=false because it uses per-request backend sessions, not device-side radix reuse; no pressure-driven demote/spill, quantized KV, or KV-pressure retry yet
fak memgate an operator pre-flight RAM gate still useful outside the model process, but no longer Python-only
glm52_serve_preflight.py: required_vram_gb VRAM-vs-quant fit + per-rank TP shard sizing models the fit of external engines (SGLang/vLLM), not fak’s own backends

Every one is real. None lets the forward loop, or the layer above it, ask the backend it is running on whether the next allocation will fit — and act on the answer. That is the bridge, and it is what DeviceCapacity begins.

6. Honest boundaries

See also