fak + LiteLLM

LiteLLM gives one OpenAI-compatible endpoint in front of 100+ providers, with load-balancing, failover, budgets, and key management. fak is the agent kernel: a default-deny capability floor that adjudicates every tool call, plus first-class per-aspect model routing and ensembles. They solve different problems, so they compose cleanly — the only question is where fak sits relative to the proxy.

TL;DR. LiteLLM is connectivity; fak is governance + routing. They speak the same wire (OpenAI Chat Completions), so wiring them together is a one-line base-URL change in whichever direction you need. Put fak in front of LiteLLM to govern everything it routes; put fak behind LiteLLM as a governed model node; or let fak’s own per-aspect routing dispatch each aspect through LiteLLM so you never reimplement a provider. A payload that leaves the box is treated as remote by the residency floor in every case.

The one insight: it’s one integration, not a hundred

The OpenAI Chat Completions wire is the field’s lingua franca. LiteLLM, OpenRouter, Together, Groq, Fireworks, vLLM, SGLang, llm-d, llama.cpp, Ollama, and most clouds all expose it. So “support LiteLLM” is not a custom adapter — it is the OpenAI wire pointed at a different base_url. fak already speaks that wire on both sides (as a server to clients, as a client to upstreams), which is why every topology below is a base-URL change, not code.

What fak adds above that wire is exactly what an aggregator does not do:

Three topologies (and what each one means)

1. fak in front of LiteLLM — govern everything the proxy routes

your agent ──▶ fak serve ──▶ LiteLLM proxy ──▶ {OpenAI, Anthropic, Bedrock, …}
            (capability floor,  (connectivity,
             quarantine, audit)  failover, budgets)

LiteLLM speaks the OpenAI wire, so point fak serve at it like any upstream:

fak serve --addr 127.0.0.1:8080 \
  --provider openai \
  --base-url http://127.0.0.1:4000/v1 \   # your LiteLLM proxy
  --model gpt-4o \                          # a model id your LiteLLM config serves
  --api-key-env LITELLM_KEY \               # the proxy's master/virtual key
  --policy floor.json                       # omit for the fail-closed default floor

Then point your agent at fak (OPENAI_BASE_URL=http://127.0.0.1:8080/v1). What it means: you keep every LiteLLM feature — provider fan-out, retries, spend caps — and add a capability floor and audit trail in front of all of it. The tool calls your agent proposes cross the floor before LiteLLM ever routes them, and a poisoned tool result is quarantined out of context before it reaches the model. This is the most common ask and is fully shipped.

2. fak behind LiteLLM — a governed model node in your routing fabric

your agent ──▶ LiteLLM router ──┬──▶ fak serve ──▶ model     (the governed lane)
                                └──▶ provider direct          (everything else)

Register fak serve as one OpenAI-compatible deployment in LiteLLM’s model_list:

# litellm config.yaml
model_list:
  - model_name: governed-gpt-4o
    litellm_params:
      model: openai/gpt-4o            # LiteLLM's "openai/" custom-provider prefix
      api_base: http://127.0.0.1:8080/v1   # fak serve
      api_key: os.environ/FAK_TOKEN        # if you set --require-key-env

What it means: fak becomes “the governed model” inside the routing fabric you already run. You can send the high-risk agent, the sensitive tenant, or the untrusted workload through the governed-* deployment and leave the rest direct — selective governance with no re-architecting. Shipped, because fak is just an OpenAI-compatible endpoint here.

3. fak’s per-aspect routing dispatching through LiteLLM — the differentiator

your agent ──▶ fak  ── route per aspect / ensemble ──▶ per member ──▶ LiteLLM ──▶ provider
                 │                                                     (connectivity)
                 └─ owns: the decision, the floor, determinism, residency

This is the case the design is built for: you use fak’s kernel and ensemble, and want LiteLLM to connect each chosen model to the actual provider. fak decides which model — or which ensemble, folded by a reduction — serves each aspect of a request (the categorical capability no request-level router has). To execute that decision, each member dispatches to a backend — and since LiteLLM speaks the OpenAI wire, a member’s backend is simply the OpenAI wire pointed at your LiteLLM proxy. fak never reimplements a provider; LiteLLM is the connectivity fabric for the members fak chose.

The division of labor is the point:

Concern Owner
Which model / ensemble serves each aspect, and how outputs fold fak (internal/modelroute, fak route)
The capability floor, quarantine, audit on each member call fak (the kernel)
Determinism of the decision + the reduce; engine residency fak
Connecting each chosen model to a concrete provider LiteLLM

Status (honest). The routing decision and the ensemble reduce are shipped and pure (fak route, witnessed by go test). The live multi-backend dispatch that runs each member on its bound backend and folds the results is the tracked [STUB] in CLAIMS.md — see the wiring contract in model routing. Authoring the routing policy and previewing the decision works today; binding each routed model id to a LiteLLM-backed account and executing the ensemble is the model-routing roadmap.

Residency: a payload that leaves the box is remote — including through LiteLLM

The load-bearing safety property when you connect your routing to any aggregator: fak’s residency floor (internal/engine) is fail-closed. It treats an engine route it cannot prove is on-box — a direct provider wire, a LiteLLM/OpenRouter/aggregator proxy, or your own gateway — as remote, and denies a tenant-scoped or sensitivity-tagged payload bound for it before dispatch. So routing a member through a LiteLLM proxy does not quietly open an exfiltration path: a sensitive aspect routed off-box is refused, an on-box engine (inkernel, a local/on-device route) is exempt. The route must be written to the call before adjudication (route-before-adjudicate), which is exactly how the routing wiring is specified.

This is why “first-class LiteLLM support” is safe by construction rather than a hole: the floor classifies an unknown backend as remote, not as trusted.

Prove the wire with no LiteLLM install (60 seconds)

You can confirm topology #1’s gate is real before standing up a proxy — fak serve with no --base-url runs a deterministic offline mock, so the floor is exercisable with no model and no key:

python3 examples/wire-proof/verify.py   # -> PASS, exit 0

Then swap the mock for your LiteLLM proxy by adding --base-url http://127.0.0.1:4000/v1; nothing else about your agent changes.

Cross-references