Model serving engines fak supports

fak does not generate tokens for production. It fronts an engine that does. The gateway speaks the OpenAI-compatible and Anthropic Messages wires, adjudicates every proposed tool call, and proxies the request to whatever serves the model.

So “supported engine” has a precise meaning here. An engine is supported when

fak serve --provider openai --base-url <engine /v1>

puts the gate in front of it. The engine keeps serving tokens its own way; fak adds the capability floor, the result-side quarantine, and the audit trail in front. This page lists the local engines that wiring covers, then the one engine fak runs itself — the in-kernel reference engine — and finally the catch-all for anything else that speaks the wire.

Start here: front an engine and confirm the seat in one check

If you are an operator, your job is to pick the engine that serves your tokens and confirm fak is governing it. The wiring is one shape for every engine in section 1 (and anything else that speaks the OpenAI-compatible wire) — point fak at the engine’s /v1, then check fak’s own health:

fak serve --provider openai --base-url http://<host>:<port>/v1 --model <served-model>
curl -s http://127.0.0.1:8080/healthz   # -> {"ok":true,"model":"…","engine":"…"}

The one exception is fak’s own in-kernel reference engine (section 2): select it with --engine inkernel, and the engine field reads inkernel instead of an upstream. Support is witnessed in the compatibility matrix (the section-1 engines, each with its base URL) and the Claims ledger (the in-kernel [SHIPPED] tag). Everything below is the per-engine base URL and the exact wiring.

1. Local / self-hosted engines over the OpenAI-compatible wire

These run on your own box and expose an OpenAI-compatible /v1 surface. You point fak serve --base-url at the engine, then point your agent at fak. The base URLs below are the engine defaults from the compatibility matrix and the Claude Code guide; swap host and port to match your own deployment.

Engine Default base URL fak wiring
Ollama http://localhost:11434/v1 fak serve --provider openai --base-url http://<host>:11434/v1 (host/port via OLLAMA_HOST)
vLLM http://localhost:8000/v1 fak serve --provider openai --base-url http://<host>:8000/v1 (server launched with vllm serve, host/port via --host/--port). For the registered vllm lifecycle adapter (KV-events, Prometheus, cache-control), see the vLLM V1 driver doc.
llm-d cluster Gateway API route, usually http://<gateway-host>/v1 fak serve --provider openai --base-url http://<llm-d-gateway>/v1 for chat proxy mode. For kernel-dispatched calls and route manifests, use the registered engine id llm-d with FAK_LLMD_BASE_URL, FAK_LLMD_MODEL, and optional FAK_LLMD_API_KEY / FAK_LLMD_METRICS_URL.
SGLang http://localhost:30000/v1 fak serve --provider openai --base-url http://<host>:30000/v1 (launched via python3 -m sglang.launch_server)
llama.cpp (llama-server) http://localhost:8080/v1 fak serve --provider openai --base-url http://<host>:8080/v1 (llama-server -m model.gguf --host 0.0.0.0 --port 8080)
LM Studio http://localhost:1234/v1 fak serve --provider openai --base-url http://<host>:1234/v1 (start the server in the Developer tab, port configurable in the app)
MLX (mlx-lm / vllm-mlx) http://localhost:8080/v1 (mlx-lm mlx_lm.server), http://localhost:8000/v1 (vllm-mlx) fak serve --provider openai --base-url http://<host>:8080/v1 (Apple Silicon; the current-SOTA local Mac runtime). For the registered mlx lifecycle adapter (OpenAI dispatch, vllm-mlx vllm:* Prometheus normalized under engine="mlx", honest whole-prefix cache-control), select the engine id: fak serve --engine mlx with FAK_MLX_BASE_URL and FAK_MLX_MODEL (optional FAK_MLX_API_KEY / FAK_MLX_METRICS_URL).
A local transformers shim (Windows dogfood path) set by the dogfood launcher The committed dogfood-claude.ps1 launcher starts a transformers-backed local_shim.py (expected at experiments/agent-live/local_shim.py) instead of Ollama, defaulting to SmolLM2-135M for CPU-friendly serving. The launcher is committed; the shim helper itself is not.

Once the engine answers, the wiring is the same for all of them. Verify the upstream with curl http://<host>:<port>/v1/models, start fak serve against it, then check fak’s own health at /healthz. The Claude Code guide has the full manual two-terminal walkthrough, including the engine launch commands and the Claude Code environment variables; the dogfood-claude.sh / dogfood-claude.ps1 launchers automate the same stack with one command.

If the engine needs provider-specific request fields (for example vLLM, llm-d, or SGLang sampling knobs), pass them through with FAK_PROVIDER_EXTRA_BODY_JSON. The serve config reference covers that plus the auth, policy, and timeout knobs you set for a network-facing deploy — a slow local CPU model in particular needs the write and planner timeouts raised together.

llm-d details

llm-d is a Kubernetes serving stack, not a single local worker. It fronts vLLM workers through the Gateway API / Endpoint Picker Provider path and exposes an OpenAI-compatible route. Put fak in front of that route when your agent speaks Chat Completions:

fak serve --addr 0.0.0.0:8080 \
  --provider openai \
  --base-url http://<llm-d-gateway>/v1 \
  --model <served-model> \
  --policy floor.json \
  --require-key-env FAK_GATEWAY_KEY

For routes that use fak’s syscall dispatch or model-routing manifest, select the first-class engine id:

export FAK_LLMD_BASE_URL="http://<llm-d-gateway>/v1"
export FAK_LLMD_MODEL="<served-model>"
fak serve --engine llm-d --model "<served-model>"

FAK_LLM_D_* aliases are accepted for the same variables. The adapter deliberately uses llm-d’s public OpenAI-compatible frontend and Prometheus/vLLM-style worker signals; it does not import llm-d internals or claim exact remote KV-span eviction.

Run fak llmd-smoke --base-url http://<llm-d-gateway>/v1 --model <served-model> before putting traffic behind the route. Add --metrics-url when the deployment exposes the worker Prometheus endpoint and you want the smoke report to verify fak’s engine="llm-d" metrics normalization too.

The route-manifest preset at examples/routing-presets/llm-d.json uses llm-d as the default dispatch target while keeping common sensitivity labels on inkernel. Validate it with fak route --check examples/routing-presets/llm-d.json before using it with --route-manifest.

MLX details (Apple Silicon)

MLX is Apple Silicon’s current SOTA serving path. Both mlx-lm’s server and vllm-mlx expose an OpenAI-compatible /v1 frontend, so fak fronts them the same “ride, don’t fork” way it fronts vLLM/SGLang/llm-d:

export FAK_MLX_BASE_URL="http://<host>:8080/v1"
export FAK_MLX_MODEL="<served-model>"
fak serve --engine mlx --model "<served-model>"

The mlx adapter deliberately uses only the public OpenAI-compatible frontend plus, when vllm-mlx exposes it, the vLLM-format Prometheus surface (normalized under engine="mlx"). It is a ride adapter — it fronts an external MLX server, not a native MLX/Metal reimplementation inside fak’s kernel (that is separate work). It does not import MLX internals and does not claim exact remote KV-span eviction: like the other three ride adapters its control-plane boundary is whole-prefix reset only (enginecache.SupportsExactSpan stays false for it).

2. The in-kernel reference engine

fak also ships an engine of its own: a pure-Go model runner fused into the kernel. You select it with --engine inkernel. Instead of proxying to an upstream, an allowed tool call is completed by a real greedy decode over a kernel-owned KV cache (model.Session.Generate), wired in as a RegisterEngine backend (internal/modelengine). [SHIPPED] in the claims ledger.

What it is for:

The honest fence: the in-kernel engine is a correctness reference, not a production-throughput server. The int8/Q8_0 SIMD lane is an in-flight increment, not yet a [SHIPPED] row, and the watt source / token-per-watt telemetry is labelled SIMULATED. When you need fast production serving, front one of the engines in section 1 instead. For the architectures the in-kernel engine runs and which rungs are proven bit-exact, see the Models page.

3. Any other OpenAI-compatible server

The list in section 1 is not a closed set. The OpenAI-compatible /v1/chat/completions wire is the field’s common denominator, and fak’s engine client is base-URL-swappable local-or-remote with bounded timeout and backoff. [SHIPPED] So any server that exposes that surface is fronted the same way — point fak serve --provider openai --base-url at its /v1 and point your agent at fak.

Rather than list engines the repo cannot source, the honest claim is the rule itself: if the server speaks the OpenAI-compatible wire, fak fronts it. The compatibility matrix is the sourced reference for that — its “Model backends & gateways” section carries each engine with its exact base URL and a source link, and the integration index has the universal “repoint one base URL” recipe and a 60-second offline proof.

One thing to keep honest in the comparison: against a fast engine, fak’s difference is operational surface, not throughput. fak adds the capability floor, the result-side quarantine, and the decision journal in front of the tokens; it does not make the engine generate them faster.

Reference (the witnessed sources behind this page)