serving — the dimensions that matter, and where fak stands

← back to the scorecard index · part of the industry-first scorecard. Each dimension is a thing the field competes on; the fak column is honest — mostly no-claim gaps for a focused reuse kernel.

Throughput & batching (throughput)

○ Continuous / in-flight (iteration-level) batching: max aggregate token throughput at high concurrency — fak: no-claim

Why it matters: Iteration-level batching admits and retires requests mid-flight so a long sequence never blocks short ones, and a fixed GPU serves far more simultaneous sequences. It is the single largest lever on tokens/s per GPU and on serving cost, so it is the first thing any operator benchmarks.

○ Absolute aggregate tokens/s at scale on standardized hardware (MLPerf Inference) — fak: no-claim

Why it matters: Cross-vendor, audited absolute throughput on a fixed model and SLO is what a buyer uses to compare full systems and compute cost-per-token. MLPerf Inference is the only neutral, reproducible apples-to-apples leaderboard with both offline (max throughput) and server (latency-constrained) scenarios, so it anchors the absolute SOTA bar that engine micro-benchmarks cannot.

○ Latency-vs-throughput Pareto frontier — fak: no-claim

Why it matters: There is no single operating point - every system trades batch size against per-request latency. The buyer’s real question is the shape of the whole frontier: how much throughput can I buy before TTFT/TPOT breaches my SLO, and how gracefully does the curve bend. A system that advances the frontier serves more users per GPU at the SAME latency guarantee.

▼ Peak served throughput per accelerator (model-served-by-engine, MLPerf-grounded) — fak: trails

Why it matters: Throughput per GPU is the denominator of cost-per-token and the headline capacity number every operator sizes a fleet against. MLPerf is the only neutral, audited cross-vendor source, so it anchors the SOTA bar buyers actually trust over vendor blogs.

Scheduling & routing (scheduling)

○ Chunked / piggyback prefill and prefill-decode interference (stall-free batching) — fak: no-claim

Why it matters: When a long prefill shares an iteration with ongoing decodes it stalls every decode, spiking inter-token latency (TPOT/TBT). Chunked prefill splits a prefill into token-bounded chunks and piggybacks them onto decode batches so decodes never stall, decoupling the throughput/latency tradeoff. It is the core mechanism that lets a co-located engine hit a tight TPOT SLO at high load.

≈ Request scheduling and cache-aware routing (FCFS vs priority vs prefix/KV-cache-aware) — fak: parity

Why it matters: The scheduling discipline decides which requests batch together and where they route. A naive FCFS engine ignores that requests sharing a prefix (system prompts, RAG context, multi-turn chat, agents) can reuse KV cache; a cache-aware scheduler co-locates and orders them to maximize reuse. This directly multiplies effective throughput on shared-context workloads and trades off against load balance.

Latency & SLO (latency)

○ Interactive latency SLO attainment: TTFT and TPOT/ITL tail under load — fak: no-claim

Why it matters: For chatbots, coding assistants, and agents the contract is felt latency: time-to-first-token and steady per-token cadence at the tail (p99), not the mean. A serving system is differentiated by how high it can push concurrency before TTFT or TPOT breach the SLO. This is the latency face of the same tradeoff goodput measures on the throughput side.

○ Time-to-first-token (TTFT) under realistic prefill — fak: no-claim

Why it matters: TTFT is the perceived responsiveness of an interactive system - the wait before any output appears. It is dominated by queueing delay plus prefill compute, so it is the headline latency metric for chat, voice, and copilot UX and the gating constraint in every serving SLO. Buyers compare TTFT at a fixed input length and concurrency because a single-user TTFT (just network + one forward pass) tells you almost nothing about production behavior.

○ Inter-token latency / time-per-output-token (ITL/TPOT) — fak: no-claim

Why it matters: After the first token, ITL/TPOT governs how fast text streams - the felt ‘typing speed.’ It must clear a per-user threshold (e.g. faster than human reading) and stay smooth; stalls produce visible jank even at good averages. TPOT excludes the first token by construction (denominator subtracts 1) so it isolates the decode phase from prefill.

▼ End-to-end request latency (E2EL) — fak: trails

Why it matters: E2EL is the full submit-to-final-token time and is what a downstream agent or pipeline actually budgets against. It bundles queueing, prefill, all decode steps, batching effects, and network, so it is the only metric that captures the compound cost of long generations and is the right unit for multi-step agent SLA math.

▼ Tail latency distribution (P50/P95/P99/P99.9) — fak: trails

Why it matters: Averages hide the requests that ruin trust. A degrading P99 while P50 holds steady is the early-warning signature of queue buildup or KV-memory pressure. At scale a 1-in-100 bad request is a continuous problem (100/hr at 10k req/hr), so operators size capacity and write SLOs against tail percentiles, not means.

○ Prefill-decode interference & chunked-prefill TTFT/TPOT tension — fak: no-claim

Why it matters: When a long-context prefill shares a GPU batch with short interactive requests, it starves decode steps and spikes ITL/TPOT for everyone in the batch - and conversely, protecting decode can balloon TTFT. How a scheduler resolves this tension (chunked prefill, stall-free batching) is a primary differentiator of mixed-workload latency stability, which naive single-shape benchmarks never reveal.

Single-stream (one chat) (single-stream)

▼ Single-stream (one chat) decode throughput on CPU — fak: trails

Why it matters: The regime local/edge stacks (llama.cpp, Ollama, MLC) compete hardest on: one user, one chat, raw tokens/s on commodity hardware. fak explicitly does NOT target this — showing the loss keeps the scorecard honest.

▼ Single-stream prefill throughput, apples-to-apples CPU-vs-CPU — fak: trails

Why it matters: Prefill (prompt ingestion) speed on CPU is a core local-inference axis. fak’s arm64 build lacks the register-blocked int8 GEMM tile that x86 already has — a disclosed, narrowing loss.

≈ Single-stream decode throughput on a consumer GPU — fak: parity

Why it matters: Local GPU single-stream decode (the RTX-class laptop/desktop regime) is where llama.cpp’s CUDA backend sets the bar; fak reaches parity here at higher precision via a reusable CUDA graph.