DeepSeek V4 Pro/Flash TTFT/TPOT/context-scaling scorecard

Status: harness + keyless fixture shipped; live latency numbers unwitnessed. The fak deepseekbench command, its locked row schema, and the speedup-refusal gate are checked in and CI-green with no key. No latency headline is claimed: every dry-run number is a labelled placeholder, and the live arm has not been run on a keyed node yet. This is the same honesty posture as the self-host smoke in DEEPSEEK-V4-SELFHOST-BASELINE-RUNBOOK.md, which is “wired and skips cleanly, ready to run.”

Resolves #3014. Parent program: #3006. Complements the self-host baseline #3013.

What it measures

One JSONL row per scorecard cell across the matrix the issue locks:

Each row carries: ttft_ms, tpot_ms, e2e_ms, output_toks_per_s, prompt_tokens, completion_tokens, reasoning_tokens, prompt_cache_hit_tokens, prompt_cache_miss_tokens, cache_attribution, speculative, accepted_token_ratio, plus the route identity (model_id, provider_route, engine_route, hosting) and the two comparability keys (prompt_shape, quality_parity). The full locked list is RequiredFields() in internal/deepseekbench/deepseekbench.go; the field-lock test TestRequiredFields fails if any row drifts from it. The fak deepseekbench command in cmd/fak/deepseekbench.go is only the thin flag/I-O wire over that pure, isolation-testable package.

Speculative axis (#3020)

Every row is labelled with a speculative mode from the locked vocabulary off | mtp | dspark (SpeculativeModes in the core package): off for plain autoregressive decode, mtp for DeepSeek’s native multi-token-prediction draft head, dspark for an external draft-model speculative stack. All current rows — the dry-run fixture and the plain streamed live measurement — are honestly labelled off.

Alongside it rides accepted_token_ratio, the acceptance evidence for a speculative run: a string, "unknown" whenever the serving engine does not expose an acceptance counter (the same provider-observed-or-unknown discipline as cache_attribution) — never a guessed number.

The speedup gate covers the axis: CompareSpeedup refuses ([NOT COMPARABLE: missing speculative label]) when either row’s speculative label is empty or off-vocabulary, and a speculative (mtp/dspark) speedup can only print when quality parity is verified for both rows — a spec-decode speed headline is never reported without its parity check, and the printed line always carries the subject’s speculative mode and accepted_token_ratio beside the ratio. Background + eval plan: ../notes/DEEPSEEK-V4-MTP-SPECULATIVE-EVAL-2026-07-08.md.

The keyless dry-run fixture (CI-safe, no key, no network)

fak deepseekbench                 # JSONL rows -> stdout; coverage + honesty -> stderr
fak deepseekbench --out rows.jsonl

Every dry-run row is labelled "measurement":"dry-run-fixture" and "speed_provenance":"fixture-placeholder-not-measured". The numbers are a pure deterministic function of the axes — they exist to lock the schema and prove the harness runs keyless, exactly like a fixture. They are not measurements and must never be quoted as performance.

The opt-in live run (behind a key + an explicit spend ack)

# Hosted DeepSeek API:
DEEPSEEK_API_KEY=sk-… fak deepseekbench --live --spend --model deepseek-v4-pro

# Self-hosted OpenAI-compatible endpoint (vLLM/SGLang, per the #3013 runbook):
DEEPSEEK_API_KEY=… fak deepseekbench --live --spend \
  --base-url http://host:8000/v1 --model deepseek-ai/DeepSeek-V4-Pro

The live arm refuses before any network call unless DEEPSEEK_API_KEY is set and --spend is passed (an explicit acknowledgement that the run costs money), so default CI never touches the network. A live row — and only a live row — carries "measurement":"live" and "speed_provenance":"provider-observed", with TTFT timed to the first content delta, TPOT as the mean inter-delta gap, and the token + prompt-cache counters read from the provider’s final usage block.

OBSERVED provider speed is NOT a fak-authored saving

This scorecard reports what the provider’s endpoint did — TTFT/TPOT/E2E/tok-s and the provider’s own prompt-cache hit/miss split. None of it is a fak-authored saving:

The speedup-refusal gate

The report generator refuses to print a speedup unless both compared rows share a prompt_shape and both carry quality_parity:"verified" and both are live measurements. A dry-run fixture, a shape mismatch, or an unverified parity yields a [NOT COMPARABLE: …] line instead of a number (see CompareSpeedup, tested by TestSpeedupRefusal). A DeepSeek row is compared against an existing fak route baseline only when that baseline was produced by this same harness under a matching shape — otherwise the row is [NOT COMPARABLE] and says so.