DeepSeek V4 self-hosted (vLLM/SGLang) baseline runbook

Status: wire-readiness plan. This document states the reproducible upstream shape, the keyless live-smoke command, and the minimum evidence for a “supported self-hosted route.” It carries no throughput/latency headline — any such number requires a real tuned baseline per Comparability boundary, and none is claimed here.

Resolves #3013. Parent program: #3006. Sibling DeepSeek-V4 leaves: #3009–#3012 / #3014 / #3015.

Why front a tuned engine first, not a native loader

DeepSeek-V4-Pro is a published 1.6T-total / 49B-active MoE with 1M context (model card). The repo’s prior-art / default-spine rule says the practical support path is to front a tuned serving engine first and only then decide whether native in-kernel loader/backend work is worth a separate scoped effort. That means: prove fak can sit in front of a vLLM/SGLang DeepSeek-V4 server exposing the OpenAI Chat Completions surface — the same surface the hosted DeepSeek API exposes (vendor docs) — before a single line of native 1.6T-weight loading is filed.

This runbook is the baseline floor. A native fak MoE follow-on (tokenizer / chat-template mapping, expert placement, FP4/FP8 weight loading, 1M-context cache pressure) may be filed only if this baseline surfaces a concrete fak-owned gap — not from model hype.

The self-hosted upstream shape

fak fronts the engine as a generic OpenAI-compatible upstream. The gateway’s proxy planner posts to BaseURL + "/chat/completions" and reads the roster at BaseURL + "/models", so BaseURL is the engine’s OpenAI root (conventionally ending in /v1).

# vLLM (BF16/FP8, day-0 V4 support per lmsys.org/blog/2026-04-25-deepseek-v4)
vllm serve deepseek-ai/DeepSeek-V4-Pro \
  --served-model-name deepseek-ai/DeepSeek-V4-Pro \
  --port 8000
# → OpenAI root: http://<host>:8000/v1

# SGLang (equivalent OpenAI-compatible surface)
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro \
  --port 8000
# → OpenAI root: http://<host>:8000/v1

Front it with fak serve (keyless from fak’s perspective — auth, if any, is the engine’s concern and passed through):

fak serve --provider openai-compatible \
  --base-url "http://<host>:8000/v1" \
  --model deepseek-ai/DeepSeek-V4-Pro

Turnkey GPU-node bring-up script

scripts/dgx-deepseek-serve.sh does the bring-up turnkey: it detects the node’s GPUs, gates on the V4 architecture floor (below), launches an OpenAI-compatible server for the chosen engine (FAK_DGX_ENGINE=vllm|sglang|nim) and model (FAK_DGX_MODEL, default V4-Flash) on :8000, health-checks GET /models, then prints the exact fak serve line and the keyless wire-witness command below.

# On the GPU node (vLLM default; V4-Flash for a single node):
scripts/dgx-deepseek-serve.sh
# NVIDIA NIM instead of vLLM (confirm the V4 image tag exists in NGC first):
FAK_DGX_ENGINE=nim FAK_DGX_NIM_IMAGE=nvcr.io/nim/deepseek-ai/<v4-tag> \
  NGC_API_KEY=... scripts/dgx-deepseek-serve.sh
# Dry-run the plan (detect GPUs + print the launch/serve commands, launch nothing):
FAK_DGX_DRY_RUN=1 scripts/dgx-deepseek-serve.sh

The script refuses to launch below sm_90 (Hopper) unless FAK_DGX_FORCE=1, because a Volta-class node (V100, sm_70) cannot serve a stock V4 FP4/FP8+DSA path (see the architecture floor below). This is the same gate stated in the DeepSeek V4 supported page.

Hardware assumptions (generic capacity language only)

No private bridge/control details are committed. In generic terms:

Which node can actually serve V4 — the honest floor

The floor above is confirmed for stock engines, not hypothetical, for this repo’s fleet — but a fork-based sm_80 path has since opened (see the update below):

Minimum evidence for a “supported self-hosted route”

The optional live smoke collects, in order, exactly these rungs:

rung check witness
readiness GET {base}/models → 200 with a model roster TestDeepSeekV4SelfHostReadiness
non-streaming POST {base}/chat/completions (stream:false) → content + a usage block TestDeepSeekV4SelfHostNonStreaming
streaming POST {base}/chat/completions (stream:true) → SSE deltas terminated by [DONE], non-empty content TestDeepSeekV4SelfHostStreaming
tool-call (optional, engine-dependent) one function-call fixture if the engine advertises tool support documented here; not gated
usage/counters token counts reported honestly; an engine that omits them is a recorded gap, never a synthesized number logged by the non-streaming rung

Optional live smoke — command and skip conditions

The smoke lives in internal/gateway/deepseek_selfhost_smoke_test.go and is keyless from fak’s perspective. It skips cleanly (never fails) when DEEPSEEK_SELFHOST_BASE_URL is unset, so the gateway test suite stays green on a box with no model server:

# Skips cleanly — no upstream configured:
go test ./internal/gateway -run TestDeepSeekV4SelfHost -v
#   --- SKIP: TestDeepSeekV4SelfHostReadiness (DEEPSEEK_SELFHOST_BASE_URL unset …)
#   --- SKIP: TestDeepSeekV4SelfHostNonStreaming
#   --- SKIP: TestDeepSeekV4SelfHostStreaming

# Runs the readiness/completion/streaming rungs against a live engine:
DEEPSEEK_SELFHOST_BASE_URL="http://<host>:8000/v1" \
DEEPSEEK_SELFHOST_MODEL="deepseek-ai/DeepSeek-V4-Pro" \
  go test ./internal/gateway -run TestDeepSeekV4SelfHost -v

Environment contract:

env var required default meaning
DEEPSEEK_SELFHOST_BASE_URL to run (unset ⇒ skip) OpenAI-compatible root, e.g. http://host:8000/v1
DEEPSEEK_SELFHOST_MODEL no deepseek-ai/DeepSeek-V4-Pro served model id to route
DEEPSEEK_SELFHOST_API_KEY no (none) passed as Authorization: Bearer … only if set

Comparability boundary (do not confound)

Completion bar

The self-host baseline is complete only when:

The wire is now witnessed live against a real DeepSeek-V4 endpoint (see Live witness); the remaining honest gap is a dedicated-node performance headline, still deferred to the tuned EP/EPLB baseline.

Live witness — wire proven on a real V4 endpoint

On 2026-07-09 (repo 4d13dc6c9) all three rungs ran live against a real DeepSeek-V4 server — deepseek-ai/deepseek-v4-flash behind NVIDIA’s OpenAI-compatible surface (https://integrate.api.nvidia.com/v1), the exact wire this runbook specifies — and all three passed:

rung result evidence
readiness PASS /models → 121-model roster including deepseek-ai/deepseek-v4-pro and deepseek-ai/deepseek-v4-flash
non-streaming PASS finish="stop"; usage witnessed prompt=11 completion=24 total=35 (counters relayed by the engine, not fak-authored)
streaming PASS SSE deltas assembled 90 chars of content, terminated by [DONE] 
DEEPSEEK_SELFHOST_BASE_URL="https://integrate.api.nvidia.com/v1" \
DEEPSEEK_SELFHOST_MODEL="deepseek-ai/deepseek-v4-flash" \
DEEPSEEK_SELFHOST_API_KEY="$NVIDIA_API_KEY" \
  go test ./internal/gateway -run TestDeepSeekV4SelfHost -v

What this does and does not establish. It establishes that fak’s openai-compatible route relays readiness, a non-streaming completion with honest usage counters, and a streamed completion to [DONE] against a real DeepSeek-V4 server — the wire-readiness bar above. It is not a dedicated-node performance headline: the NVIDIA-hosted surface is a shared, pooled endpoint (it returns HTTP 503 ResourceExhausted under load), so no TTFT/TPOT/tok-s number is drawn from it — that still requires the tuned EP/EPLB profile on a dedicated node per the comparability boundary.

Shared-endpoint robustness (smoke hardening). Because a pooled endpoint throttles, the smoke now treats an upstream capacity signal (HTTP 429/503 or a ResourceExhausted body) as a clean skip, not a failure — a busy node is not a fak wire defect — and accepts a DeepSeek thinking-mode turn whose tokens land in reasoning_content with empty content as a live wire. On a dedicated dedicated Hopper-class node there is no such contention and all three rungs pass without skips.