Poly-model serving — host many, share the prefill, decode one

Poly-model serving is fak’s design for hosting tens of models warm in one kernel, sharing prefill across them, and serializing decode to a single lane — because prefill is compute-bound and parallelizable while decode is HBM-bandwidth-bound, so you host many and decode one. This page is that design decision, paired with a shipped, GPU-free deterministic core (internal/polymodel: the residency pool, the serial decode-lane scheduler, and the speculative-accept arithmetic) plus the cross-model prefill-share splice and single-pass verify execution (#532–#535), all off by default behind FAK_POLYMODEL. The live multi-model residency on a real backend is explicitly sequenced, not yet shipped, and no benchmark number is claimed — the speedup model in §5 is closed-form arithmetic gated on a measured run.

Design decision doc for the multi-model axis of in-kernel serving: hosting tens of models in one kernel, sharing/amortizing the prefill, and serializing decode to a single lane — plus the cache-led, next-generation multi-token-prediction (MTP) that the co-resident models unlock. This is the orthogonal axis to the disaggregated-serving epic (#50): that plan scales one model across many nodes; this one packs many models onto one kernel.

Scope: design + a shipped deterministic core (internal/polymodel) and the shipped ProvisionalSink rollback seam (internal/spec, rung #532 — CPU-lossless, off-defconfig), the cross-model prefill-share verdict + splice (#534), and the single-pass verify EXECUTION — batched chain + tree-attention masks (#533, internal/model.VerifyForward). The remaining GPU/engine wiring (real multi-model residency on a backend) is explicitly sequenced, not claimed here.

Provenance: every [SHIPPED]/[PARTIAL]/[SEAM-ONLY]/[GAP] mark carries a file:line pointer verified against the working tree on 2026-06-22. Line numbers drift; re-anchor with the rg snippets in §9. No benchmark number is asserted — the speedup model in §5 is closed-form arithmetic, gated on a measured run before any tokens/sec claim.

1. TL;DR — the decision

  1. Host many, decode one. Keep tens of models prefill-warm in one kernel, but let only one model decode at a time. This is not a compromise — it falls straight out of the hardware: prefill and decode have opposite bottlenecks.
  2. The asymmetry is the whole design. Prefill is compute-bound (one weight stream amortized over every prompt token — a batched GEMM), so prefilling many models is parallelizable and interleavable. Decode is HBM-bandwidth-bound (each token streams the model’s entire weight set from memory), so two models decoding at once need 2× memory bandwidth against a fixed bus. Residency is cheap (capacity); decode is scarce (bandwidth). So you host many and decode one.
  3. The cache is the lever — twice. (a) fak’s prefix cache already eliminates redundant prefill for one model; the poly-model extension shares prefill across models (the radix tree is already model-agnostic in structure). (b) The bit-exact KV Clone/Evict make speculative MTP correct and cheap: clone is the fork, Evict is the exact rollback of rejected draft tokens.
  4. Next-gen MTP = the idle models become the speculation ensemble. When model X decodes, the other warm models (or X’s own cheaper quant) draft tokens that X verifies in one prefill-shaped pass — turning the spare residency into decode-lane throughput instead of dead weight.
  5. What lands now: internal/polymodel, the deterministic, GPU-free core — residency pool, serial decode-lane scheduler, and the speculative-accept arithmetic — proven by a witness suite. Everything that touches a real backend is sequenced in §7.

2. The asymmetry, in the code (not in the abstract)

The two regimes are already distinct execution paths in internal/model:

Regime Bottleneck How it runs today Anchor
Prefill compute (FLOPs) one batched GEMM sweeps all P prompt tokens through each weight row once — matMulBatch, prefillBatched internal/model/parallel.go:339; internal/model/prefill_batch.go:21
Decode HBM bandwidth one token at a time; each step streams the whole weight set — Session.Step → token. The Q4/Q4K flags exist only to cut decode bytes/token internal/model/kv.go:1076; internal/model/kv.go:258

The decode path is defined by bandwidth: the entire reason int4 decode exists (internal/model/kv.go:258, “int4 streams ~1.8× fewer bytes/token”) is that decode is memory-bound, not compute-bound. That single fact forces “decode one”: you cannot serialize fewer than one model’s weight stream, and you cannot afford to run several at once. Prefill, being compute-bound, is the half you can fan out and share.

There is even multi-user static batching already — StepBatch runs B users’ decode tokens in lockstep, reusing one weight stream B-fold (internal/model/batch.go:1122), and its own comment says it “is exactly what a continuous-batching scheduler would call” (batch.go:1147). But that is one model, many users. The poly-model lane is many models, time-shared — a caller-side scheduler over the same primitive.

3. Host many, decode one — the architecture

   prefill-warm residency (capacity-bound: HOST MANY)
   ┌───────────────────────────────────────────────────────────┐
   │  model A   model B   model C   …   model N   (weights warm) │
   │   KV/prefix cache per model, shared where prefixes match    │
   └───────────────┬───────────────────────────────────────────┘
                   │  exactly one at a time
                   ▼
        ┌──────────────────────┐
        │   THE DECODE LANE     │  bandwidth-bound: DECODE ONE
        │  (one weight stream)  │  round-robin / priority / FCFS
        └──────────────────────┘

4. The cache is the lever — prefill sharing across models

fak already does intra-model prefill elimination: a computed prefix’s KV is cloned into the next session bit-identically, skipping its prefill (internal/model/kv.go:123 Clone, :221 SessionFromPrefix). The poly-model question is can two different models share prefill? The honest taxonomy:

Sharing mode What’s shared Status Note
Same model, same prefix the whole KV prefix [SHIPPED] Clone/SessionFromPrefix; RadixAttention reuse (internal/radixkv/radixkv.go:72)
Cross-model, same prefix, structurally the radix tree can already index one prefix node regardless of model [SEAM-ONLY] the tree keys on token-ids only (radixkv.go:51,96,173); model identity lives in the verdict layer, not the structure
Cross-model reuse, served actually serving model B from model A’s KV [SHIPPED] (verdict + splice) cachemeta.MaterializeVerdict lifts the exact-ModelID barrier via WithPrefillShare (#534); internal/spec.SplicePrefillShare is the bit-exact KVCache.Clone splice. The prefix cache was always model-agnostic at the data-structure level; the barrier was verdict-layer only
Shared lower-layer bands (adapter / distilled siblings) the prefill of layers two models genuinely share [GAP] the per-layer KVCache.K[][]float32 (kv.go:20) makes layer-banded sharing expressible; correctness needs the models to share those layers’ weights + RoPE convention

The key, non-obvious finding: the prefix cache is model-agnostic at the data-structure level and single-model only at the policy level. Cross-model prefill share is a verdict-layer decision (lift the exact-ModelID barrier for a declared-compatible family), not a cache rewrite. That is the cheapest unlock on this axis and the first real-backend rung (§7).

5. Next-gen MTP — the co-resident models are the speculation ensemble

Classic MTP / speculative decoding: a cheap draft proposes K tokens; the expensive target verifies all K in ONE forward pass. That verify is prefill-shaped (parallel over K tokens, compute-bound) — so the target pays one compute-bound pass instead of K bandwidth-bound decode steps. On the single serial decode lane, that is the whole speedup.

The poly-model twist: the spare warm models are the drafters. When X decodes, a cheaper same-family resident (or X’s own Q4 twin) drafts; X verifies. The spare residency stops being idle capacity and becomes decode throughput.

Two things make this correct and cheap in fak specifically, and they are fak’s existing moat:

The honest speedup model (polymodel.EffectiveTokensPerVerify): with draft length K and per-token acceptance probability a, one verify advances E = Σ_{i=0}^{K} a^i = (1 − a^{K+1})/(1 − a) real tokens (Leviathan et al.). So the lane yields ~E tokens per bandwidth-bound pass instead of 1 — before subtracting draft cost. This is arithmetic, not a measurement; a real number needs the bench harness (#44).

The next-gen form is a token TREE, not a chain. polymodel.AcceptGreedy is the linear case (one draft sequence). polymodel.AcceptTree generalizes it to a speculation tree (Medusa / EAGLE-2 / SpecInfer): many candidate continuations share a KV prefix, are verified in one pass, and only the accepted path is kept. A wide tree explores several futures for the price of one verify, raising the acceptance length per pass — and it is the caching idea made literal: a branch forks the cache (KVCache.Clone), rejected branches are removed bit-exactly (KVCache.Evict). AcceptTree is the deterministic accept/keep/evict core (witnessed); the tree-attention verify execution on the model (positional masks for the tree) is shipped (#533, internal/model.VerifyForward with the ancestor mask) — spec.VerifyTree/SpeculativeTree drive it end-to-end and prove the accepted path is token-identical to plain greedy decode.

6. Capability honesty table

Legend: [SHIPPED] real & proven · [PARTIAL] real but incomplete · [SEAM-ONLY] interface/seam exists, no production impl · [GAP] absent.

Reconciled 2026-07-10. The internal/spec leaf (the end-to-end spec-decode driver: Sink, SpeculativeGreedy, VerifyTree/SpeculativeTree, SplicePrefillShare) was retired as unwired on main (f7083ed1d refactor(spec): retire the unwired internal/spec speculation leaf) — it is no longer on disk or in HEAD, so its go test ./internal/spec witnesses no longer run. What SURVIVED and stays proven: the abi.ProvisionalSink seam, the polymodel accept core (AcceptGreedy/AcceptTree/CanShare), the single-pass verify EXECUTION primitive (model.VerifyForward), and the cachemeta prefill-share verdict layer (WithPrefillShare). The rows below are corrected to that surviving substrate; any remaining internal/spec reference elsewhere in this doc (§5, §7 rungs 4/4b/5) is superseded by this note. Wiring the surviving substrate natively into the decode loop is tracked in #3197.

Capability Status Anchor
Bit-exact KV Clone (the speculative fork) [SHIPPED] internal/model/kv.go:123
Bit-exact middle-span Evict (the speculative rollback) [SHIPPED] internal/model/kv.go:60; proof internal/model/evict_test.go:86
Speculate→inspect→KV-rollback path (precision policy) [SHIPPED] internal/model/kv.go:272; internal/model/dynamic_precision.go
Prefill batched / decode-per-token asymmetry [SHIPPED] internal/model/parallel.go:339; internal/model/kv.go:1076
Multi-user static decode batch (StepBatch) [SHIPPED] internal/model/batch.go:1122
RadixAttention prefix reuse (model-agnostic tree) [SHIPPED] internal/radixkv/radixkv.go:72,51,96
Speculation/MTP ABI envelope (frozen) [SEAM-ONLY] internal/abi/types.go:112,126,128,144; internal/abi/registry.go:552
Multi-model residency pool + serial decode lane + accept core [SHIPPED] internal/polymodel/ (this work)
Continuous-batching admit/evict loop [SEAM-ONLY] internal/model/batch.go:1147 (comment); no scheduler
Multiple models hosted in one process [GAP] single modelengine.Default, one *model.Model (internal/modelengine/modelengine.go:54,226); gateway binds one planner (internal/gateway/gateway.go:253)
Multi-model weight residency — policy + *model.Model binding [SHIPPED] internal/residency — Manager hosts many *model.Model under one weight-byte budget with LRU page-out, reusing polymodel.Pool as the budget + eviction policy and binding each residency descriptor to the weights it governs (off-defconfig, a library type like polymodel)
Multi-model weight residency / whole-model eviction ON a backend [GAP] the policy + binding layer is shipped above; the real per-backend weight load/evict (per-weight budget internal/compute/vulkan.go:164; process-wide gpulease lease.go:36) is the deeper rung a future wiring drives through Manager.Admit/Evict
Runtime resident-budget adapt (re-budget a live pool) [SHIPPED] (off-defconfig) polymodel.Pool.Resize + residency.Manager.SetBudget (#3999) — shrink evicts the coldest-unpinned residents (LRU) until it fits, handing the weight handles back for page-out; grow evicts nothing; a shrink below the pinned footprint refuses ErrPinnedNoRoom with the pool unchanged (Admit’s all-or-nothing). The runtime knob for the gpulease memory-pressure cascade / hot-add, without rebuilding the pool
Cross-model prefill share (served) [PARTIAL] (verdict-layer shipped; splice retired) cachemeta.MaterializeVerdict + WithPrefillShare/PrefillSharePolicy (ModelID-axis-only barrier lift, #534) — SHIPPED, on main. The KVCache.Clone splice half lived in the now-retired internal/spec (SplicePrefillShare/CrossModelPrefillShare, removed as unwired in f7083ed1d); rebuild it when the substrate is wired (#3197). Live multi-model backend residency (#531) still [GAP]
ProvisionalSink implementation / internal/spec [SEAM-ONLY] (impl retired) The abi.ProvisionalSink seam is frozen and present (internal/abi/registry.go), but the internal/spec implementation (Sink + OpSpecCommit/OpSpecSquash; Rollback→bit-exact KVCache.Evict; SpeculativeGreedy) was retired as unwired in f7083ed1d — the leaf and its go test ./internal/spec witness no longer exist on main. The accept DECISION (polymodel.AcceptGreedy) and the rollback (model.KVCache.Evict) survive; a native re-wiring is tracked in #3197
Single-pass verify EXECUTION — batched chain + tree-attention masks [SHIPPED] (primitive; driver retired) internal/model.VerifyForward(ids, pos, allow) — the rung-#533 execution primitive — SURVIVES on main. CHAIN (nil pos/nil allow): bit-identical to P sequential Session.Step calls (same logits + appended K/Kraw/V/pos); TREE (depth-based pos, ancestor allow): tree-attention masks — each node attends only to its ancestor chain + prefix, never siblings. Witness: go test ./internal/model. The internal/spec end-to-end drivers that consumed it (SpeculativeGreedy, VerifyTree/SpeculativeTree) were retired as unwired (f7083ed1d); native re-wiring is #3197. CPU PreNorm regime only — no GPU/tokens-per-sec (bench harness #535)

7. The child map / sequencing

Ordered so each rung stands on a shipped one. None of these land in this doc. Tracked as epic #529 (children #530–#535); each rung is gated per §7a.

  1. internal/polymodel — the deterministic core. ✅ shipped. Residency pool, serial decode lane, speculative-accept arithmetic (AcceptGreedy + AcceptTree), ensemble drafter selection, CanShare, the FAK_POLYMODEL gate. GPU-free, witness-proven. Runnable witness: cmd/polymodelbench -selfcheck hosts 10 synthetic models under a budget, drives the serial decode lane over real model.Session decode, and proves greedy speculative decode is token-identical to plain greedy even when an adversarial draft forces a rollback every round — the end-to-end proof that the rejected-draft KVCache.Evict rollback is bit-exact.
  2. Caller-side decode-lane scheduler over StepBatch/Session — #530. Wire polymodel to actually drive a multi-model decode loop (single backend) — the continuous-batching seam the batch.go:1147 comment invites.
  3. Multi-model residency on a backend — #531. ✅ policy + binding shipped. internal/residency.Manager lifts the single-*model.Model assumption (modelengine.Default is one *model.Model): a pool of *model.Model under a weight-byte budget with LRU page-out, reusing polymodel.Pool as the budget + eviction policy and binding each residency descriptor to the weights it governs (off-defconfig; witness go test ./internal/residency). The real per-backend weight load/evict (the compute-HAL per-weight budget, the process-wide gpulease) is the deeper rung a future wiring drives through Manager.Admit/Evict.
  4. A ProvisionalSink + internal/spec implementation — #532. ✅ shipped. internal/spec is the first ProvisionalSink registrant + the reserved OpSpecCommit/OpSpecSquash ops; its Rollback drives the bit-exact KVCache.Evict, with polymodel.AcceptGreedy as the accept decision (the native verify/accept #23 and #284 defer). SpeculativeGreedy is proven token-identical to plain greedy through the seam, even under an adversarial draft that forces a rollback every round (vacuity-guarded). Off-defconfig + FAK_POLYMODEL-gated. The verify EXECUTION (single-pass batched + tree-attention masks) is the throughput half — #533, shipped (see rung 4b below). 4b. The single-pass verify EXECUTION — batched chain + tree-attention masks — #533. ✅ shipped. internal/model.VerifyForward(ids, pos, allow) runs the PreNorm-standard batched forward over the P candidate tokens in ONE pass and returns each position’s next-token logits. The CHAIN case (nil pos/nil allow) is bit-identical to kk sequential Session.Step calls (same logits AND appended K/Kraw/V/pos in every layer), so spec.SpeculativeGreedy’s verify loop is now a single VerifyForward instead of kk Steps — still token-identical to greedy. The TREE case (depth-based pos: siblings share base+depth-1; ancestor allow mask) is tree-attention: each node attends only to its ancestor chain + the committed prefix, never to sibling branches. spec.VerifyTree/SpeculativeTree drive it through AcceptTree and commit the accepted path (rewind the speculation with one bit-exact KVCache.Evict, recommit the accepted chain as one VerifyForward); the accepted path is token-identical to plain greedy decode. Witness: go test ./internal/model (TestVerifyForwardChainMatchesSerial, TestVerifyForwardTreeMaskIsolatesBranches); go test ./internal/spec (TestSpeculativeGreedyLossless now exercises the batched verify, TestSpeculativeTreeLossless*, TestVerifyTreeRewindsAndCommitsCleanly). Honest fence: CPU synthetic PreNorm regime only — no GPU, so no tokens/sec (the speedup is EffectiveTokensPerVerify arithmetic; a measured number needs the bench harness #535); the chain falls back to sequential Steps on a non-batched regime; the tree recomputes the accepted path’s KV (a tree-aware KV-compaction primitive is the sequenced cost, not a correctness gap).
  5. Cross-model prefill share (verdict-layer) — #534. ✅ shipped. Lift the exact-ModelID barrier for a declared-compatible family in cachemeta — the cheap structural unlock from [§4]. The DECISION half is polymodel.CanShare (same Family + byte-identical PrefixDigest ⇒ the reused KV is bit-identical, so reuse is lossless); the verdict- layer wiring is cachemeta.PrefillSharePolicy + WithPrefillShare threaded into MaterializeVerdict (ModelID-axis-only lift; every other binding axis still verified), and the KVCache.Clone splice is internal/spec.SplicePrefillShare / CrossModelPrefillShare — the off-defconfig bridge (cachemeta never imports the off-defconfig polymodel leaf) that forks the provider’s prefix bit-exactly into the consumer. Witness: go test ./internal/cachemeta ./internal/spec (the splice continuation is bit-exact to a direct prefill). The live multi-model backend residency (#531) remains sequenced; the verify execution (#533) is shipped (rung 4b).
  6. Bench harness numbers — #535. ✅ shipped. Gate every speedup claim on a measured run (#44): E vs draft cost, decode-lane utilization, residency hit-rate. The harness is cmd/polymodelbench -bench [-out FILE] (benchHarness in cmd/polymodelbench/bench.go): it drives the REAL polymodel primitives (Pool, Schedule, AcceptGreedy, model.KVCache.Evict) over deterministic synthetic workloads and emits every number as MEASURED, re-asserting the correctness invariants (losslessness, MaxConcurrentDecode==1, pinned-survives, budget-never-exceeded) on the measured runs — a number on a broken core is invalid, so *ok lowers on any gate failure. The three axes: (a) speculation — the acceptance rate is measured by running a real draft/target pair (and an adversarial proposer that forces the bit-exact Evict rollback path), so E is reported as a MEASURED spread E ∈ [1, K+1] bracketed by the two regimes’ acceptance, never a single curated point; (b) decode lane — tokens/step, decode steps, and MaxConcurrentDecode==1 off a real Schedule, plus the HBM traffic via DecodeBandwidthBytes; (c) residency — a hot-set/cold-tail request stream against a real Pool under a tight budget yields the measured hit-rate, eviction count, and pinned-survival. Honest fence: these are deterministic synthetic-workload measurements on the policy/accounting core — no GPU, so no tokens/sec-on-hardware claim, only the acceptance/throughput/hit-rate numbers every speedup claim reduces to. Witness: go test ./cmd/polymodelbench; reproduce the artifact: polymodelbench -bench -out report.json.

7a. Safety — off by default until ready (feature-flag gating)

This lane must not affect the shipped fak binary until each rung is production-ready. Two layers enforce that, matching fak’s existing flag conventions:

  1. Defconfig opt-in (the strongest gate). A leaf reaches a live request path only if it is blank-imported in internal/registrations (the “defconfig”; see architest’s request-path closure). internal/polymodel is deliberately NOT there, and nothing on the request path imports it — the fak binary does not even link it today (only the standalone cmd/polymodelbench demo does). So the shipped kernel’s behavior is byte-unchanged.
  2. Runtime env gate (the second layer, for when it is wired). When an integration rung does put the lane on a request path, that wiring must guard on polymodel.Enabled() — the FAK_POLYMODEL env flag, default off (opt in with FAK_POLYMODEL=on). This mirrors FAK_AUDIT_JOURNAL (a leaf that ships imported but inert until its flag is set) and FAK_GITGATE. The pure helpers (Pool, Schedule, AcceptGreedy, …) never consult the flag — they are deterministic library calls; only the live-path integration does.

The acceptance bar for moving any rung past these gates is a green make ci and a measured bench run (#44) — never a flag flip alone.

8. Non-goals

9. How to re-verify

rg -n 'func matMulBatch'                       internal/model/parallel.go      # prefill: compute-bound batched GEMM
rg -n 'func \(s \*Session\) Step|Q4 streams'   internal/model/kv.go            # decode: per-token bandwidth path
rg -n 'func \(c \*KVCache\) (Clone|Evict)'     internal/model/kv.go            # the fork + the bit-exact rollback
rg -n 'is exactly what a continuous-batching'  internal/model/batch.go         # the scheduler seam (SEAM-ONLY)
rg -n 'SpeculationContext|ProvisionalSink|TxnID' internal/abi/types.go         # frozen MTP envelope
rg -n 'func CheckResidentClaim'                internal/cachemeta/manifest.go  # the exact-ModelID barrier (cross-model GAP)
rg -n 'node.key|func \(t \*Tree\) walk'        internal/radixkv/radixkv.go     # model-agnostic prefix tree
go build ./internal/polymodel/ && go vet ./internal/polymodel/                 # the shipped core

Sibling axis: docs/serving/dual-track-serving-plan.md (one model, many nodes). · Shipped core: internal/polymodel.