llm-d repository study

Verdict

TL;DR: Use llm-d as FAK’s Kubernetes inference control plane. Borrow its typed routing and executable-guide discipline; keep worker placement, P/D topology, and autoscaling under one owner.

llm-d is a Kubernetes-native distributed inference assembly. FAK should interoperate with it, learn from its routing and operations contracts, and avoid duplicating its control plane. Its strongest idea is the decomposition. Each concern has a separate control-plane contract: routing; model-server deployment; prefill/decode placement; KV-state observation; flow control; autoscaling; and guide verification. FAK already has the lower-level kernel seams for an llm-d backend, cache-aware worker choice, native P/D, and KV-transfer governance. The useful remaining work is therefore narrow:

  1. Update FAK’s integration guide for llm-d v0.9 and assign one owner at each routing boundary.
  2. Add a live compatibility witness. The existing OpenAI-shape smoke test proves protocol shape only. Cluster interoperability still needs a live run.
  3. Watch llm-d’s workload-variant autoscaler and exact KV-event routing. Copy neither until FAK has stronger fleet evidence.

This is an Enabling study.

Scope and evidence discipline

What llm-d actually is

The repository is predominantly an integration and operations distribution. Scheduler implementations live in the assembled components. The pinned tree has about 740 guide files and 136 architecture-doc files. It assembles versioned external components through Helm and Kustomize. Gateway API resources, model-server images, and scripts complete the deployment. The architecture page makes the separation explicit:

That design serves platform teams running accelerator fleets on Kubernetes. They need portable APIs, replaceable components, observable SLOs, and reproducible workload guides. This deployment boundary differs from FAK’s. Each project should stay focused on its own boundary.

Load-bearing mechanisms

1. Routing is a filter/score plugin pipeline

EPP first filters infeasible endpoints, then scores the survivors with independently configurable plugins. Prefix locality is one signal among load, queue state, accelerator availability, and deployment topology; the scheduler can normalize and weight signals rather than hard-code one universal policy. The data layer deliberately separates request-derived state from endpoint/model-server state (data layer).

The transferable principle is typed signal composition with an observable reason for the winner. llm-d’s current weights are context, not a template. FAK’s current CacheAwarePolicy already implements power-of-two choice over prefix overlap and load, and its later Dynamo-derived issues added richer load/tier terms. The llm-d evidence strengthens the need to keep these terms modular and explainable.

2. Approximate and precise prefix routing are distinct products

The ordinary prefix-aware path derives locality from request tokens and cached state. The precise path enables model-server KV-event publication, aligns block hashing, tokenizes through a render service, and feeds exact create/evict state to the router (precise prefix routing).

Merged PR #2203 is particularly useful: it adds a wide-EP/LWS precise-routing variant rather than silently replacing the approximate default. Its manifests wire a precise-prefix-cache-producer, a render service, per-rank KV-event publication, and load-gated affinity. That is a good bounded-superset pattern. Exactness adds several dependencies: state, tokenization, ports, and new failure modes. llm-d therefore keeps it as an explicit variant.

3. Flow control is an admission contract, not just load balancing

llm-d’s flow-control design gives the router a request queue and concurrency bounds, supports priority through request metadata, and exports queue/admission metrics (flow control). Merged documentation PRs #2213 and #2249 tightened verification and observability, which matters more than merely exposing knobs: overload behavior has to be visible from the same guide used to deploy it.

FAK has continuous-batching queue gates, priority/preemption leaves, and receiver-granted KV-transfer credits, but those are several planes. The useful lesson is to describe which queue is controlled and which metric proves shedding or admission; “flow control exists” is too conflated to be useful.

4. P/D disaggregation is topology plus state transfer

llm-d describes prefill and decode as separately scalable workloads connected by KV transfer, with routing and deployment resources selecting roles (P/D architecture). The repository carries runnable P/D guides; role split alone does not count as proof.

FAK’s NativePDService and transfer-governance seams are native kernel mechanisms. llm-d is a strong external orchestrator option. FAK should retain both behind the EngineDriver and integration boundaries, not grow a second Kubernetes operator.

5. Workload-variant autoscaling is deliberately higher-order

The Workload Variant Autoscaler observes request characteristics and routes/scales among variant deployments, and avoids forcing one replica count to serve every prompt/output mix (autoscaling architecture). That idea is relevant to FAK’s model/engine routing, but Kubernetes replica control is outside the kernel’s best-default boundary. Treat WVA as an optional platform module and a watch item until FAK has a measured demand-to-placement contract worth exporting.

6. The guides are executable compatibility contracts

The repository’s center of gravity is workload guides plus release/nightly matrices. The v0.9 notes call out promoted guides, platform/accelerator variants, routing and observability changes, and known limitations; the workflow tree runs guide-specific e2e/nightly lanes. The frozen v0.9.0-rc.1 rows in release/README.md@48fa8c0 are mixed. Precise-prefix lanes pass on several platforms. P/D includes dry-run and guide-error cells; GKE flow control is prereqs-error; CKS WVA is failing. The live badges above those frozen rows can move after this observation, so neither the guide’s presence nor one green platform generalizes to every deployment. Recent fixes such as #2296 (P2P cache-sharing guide and GLM results) and #2307 (FMA launcher RBAC) show why manifests and live verification are load-bearing code. A client-shape unit test covers only the protocol edge. Cluster interoperability needs a live witness.

Candidate borrow matrix

The fak_feature_query MCP surface was dogfooded with five queries. The queries covered precise prefix routing, admission and priority, and heterogeneous autoscaling. Two more covered P/D transfer and adapter interop. It returned only lexical false-neighbor skill/tool cards for the first four queries and did not discover the existing engine adapter for the fifth. Per the field-borrow rule, that lexical score serves only as discovery evidence. The classifications below come from direct code, tests, docs, and issue-state readback.

Candidate FAK state Coverage decision Evidence and action
OpenAI-compatible llm-d backend adapter PRESENT DEFAULT interop seam internal/engine/llmd.go, its tests, cmd/fak/llmd_smoke.go, and docs/integrations/llm-d.md resolve llm-d URLs, preserve auth/header policy, and probe OpenAI endpoints. Keep it as the supported composition path.
Prefix-overlap plus load-aware worker routing PRESENT structurally; fleet benchmark evidence is narrower DEFAULT internal/gateway/residency_router.go ingests add/drop events and scores overlap against load; issues #41, #2238, #5272, #5274, and #5275 document the evolution.
Do not clone EPP. Keep modular score terms and require routed-vs-blind witnesses for broad performance claims.
Exact KV-event-backed global index PARTIAL OPTIONAL-MODULE / WATCH FAK ingests external KV events and has cache metadata, but llm-d’s precise variant couples tokenizer/render and block-hash identity across pods.
Preserve the external event seam; do not make that operational bundle a FAK default without a live heterogeneous-fleet witness. Existing #5260/#2238 cover the concept; no duplicate issue.
Request admission, priority, and overload observability PARTIAL across planes DEFAULT kernel policy, RECIPE for llm-d Native scheduler/preemption and gateway policies cover core control; llm-d owns Kubernetes gateway queues when it is the backend.
Document one owner per queue and propagate priority intentionally. Do not stack two opaque queues.
Native P/D role split and governed KV transfer PRESENT DEFAULT native; RECIPE external internal/modelengine/native_pd.go and tests cover native roles; transport/credit packages govern transfer. When llm-d is selected, let llm-d own pod topology and use FAK as front gate/agent kernel.
Workload-variant replica autoscaling ABSENT as a FAK-owned closed loop OPTIONAL-MODULE / WATCH FAK routes models and engines but does not need to own Kubernetes replica reconciliation. Interoperate with WVA; reconsider only when a FAK demand signal and measured placement objective justify an exported controller contract.
ModelService/InferencePool lifecycle controller ABSENT by design EXCLUDE This is llm-d/Kubernetes control-plane territory. FAK should consume a stable endpoint or emit deployment recipes. CRDs and controllers stay with llm-d.
Guide-as-tested-product release matrix PARTIAL DEFAULT for integrations FAK has smoke/unit docs but no llm-d cluster lane. Borrow the practice: a named compatibility witness should deploy or target a pinned llm-d release, run /v1/models plus streaming chat through FAK, capture routing/health evidence, and record component versions.
Pluginized request/endpoint signal pipeline PARTIAL DEFAULT principle FAK has typed routing seams but some policies remain bespoke. New routing signals should name source, freshness, normalization, weight, fail-open/closed behavior, and explanation output.
This is a design constraint; FAK does not need an EPP clone.

Best-default frontier and bounded superset

Surviving work after deduplication

Two gaps survive. Existing issues cover five nearby areas. They include fleet KV routing and P/D. Richer scoring, event streams, and transport control are also covered. This study does not reopen them.

  1. Refresh the llm-d integration contract for v0.9 (#8016). The current page is useful but lacks a pinned version, precise ownership table, queue/double-routing warning, and clear distinction between unit smoke and live cluster proof. This is documentation/productization work.
  2. Add a live llm-d compatibility witness (#8017). The minimum witness uses a pinned llm-d deployment or sanctioned cluster. It configures FAK through --engine llm-d, calls /v1/models, and sends a streaming chat request. The artifact captures component versions plus auth, error, and streaming behavior. Routing performance needs a separate locality-vs-blind experiment. The compatibility witness cannot establish it.

The WVA and exact-prefix variants remain WATCH, not silently deferred FAK commitments: they have explicit refresh triggers above and are intentionally assigned to llm-d unless measured FAK-specific demand changes the boundary.

Claims and limits

Source trail

Exhaustive inventory refresh (2026-08-25)

Issue #8988 refreshes the study denominator without replacing the mechanism analysis above. The exhaustive map is inventory/llm-d-llm-d.json, generated from llm-d/llm-d@3243fcf1191348b55c7811267a98117f8b7a6910. It indexes 1,129 files, 593 directories, 30,601,578 bytes, and 11 top-level subsystems. The revision is 17 commits after the prior checked revision; the intervening changes are predominantly v0.9 guide, artifact, CI, template, and platform corrections, not a reversal of the ownership decisions above.

Complete source-class read-back

The pinned map’s non_tree_study block records the commands, aggregate counts, roadmap treatment, license decision, self-query results, and candidate matrix. This keeps mutable forge observations separate from claims about the immutable tree.

Refreshed FAK decisions and follow-ons

Candidate Decision Durable owner
v0.9 ownership and double-routing contract Already shipped. FAK owns policy/model/cache decisions; llm-d owns cluster endpoint selection and Kubernetes assembly. #8016, docs/integrations/llm-d.md
Pinned live-cluster compatibility receipt Keep open. Static inventory evidence cannot replace a live cluster witness. #8017
KV-event locality, speculative cache-warm metadata, and matched prefix-reuse measurement Borrow as bounded inspiration, preserving fak-native cache ownership. #3888, #6082
Endpoint-scoring plugins, flow control, P/D topology, workload-variant autoscaling, Gateway API, and deployment guides Do not port. These are llm-d control-plane responsibilities; FAK should expose signals and compatibility seams instead of building a second Kubernetes inference platform. Interoperation boundary above

Every surviving action is shipped or already tracked, so this refresh files no duplicate follow-on. The row-specific completion witness is:

fak study-monitor --registry docs/research/monitored-repositories.json --inventory-check --json
repository=llm-d/llm-d mode=exhaustive ready=true indexed_revision=3243fcf1191348b55c7811267a98117f8b7a6910