The industry dimension catalog
Every dimension the field competes on, researched industry-first (not derived from what fak measured). ← back to the index.
Dimensions are grouped by the part of the stack they belong to. Each group is a section below; follow the group link for the full per-dimension analysis with fak’s position.
agent
See the full agent analysis for fak’s honest position on each dimension.
| Category | Dimension | SOTA bar | Leading systems | Source | Date |
|---|---|---|---|---|---|
| agent-fleet | Cross-agent fleet serving time (N agents × T turns): work eliminated by shared-prefix fusion | The competing floor remains tuned warm per-agent KV reuse (vLLM Automatic Prefix Caching / SGLang RadixAttention / OpenAI prompt caching); fak’s own measured 50x5 fleet-serving result (19 min vs 78 min warm-KV baseline, 4.1x from cross-agent prefix fusion) is the live number on this dim and no external SOTA has displaced the warm-KV-reuse baseline class. | vLLM Automatic Prefix Caching, SGLang RadixAttention, OpenAI prompt caching (warm-KV reuse floor) | link | 2026-01 |
| agent-fleet | Marginal value of cross-agent prefix fusion ON TOP of an already-warm per-agent cache | The conservative SOTA point is fak’s own already-warm per-agent KV arm (baseline 1.0x); cross-agent prefix fusion on top of it yields a measured ~2.4-2.7x marginal gain. No external benchmark redefines the already-warm-cache baseline for this dim. | Tuned warm per-agent KV cache, already hot (vLLM APC / SGLang RadixAttention class) | link | 2026-01 |
| agent-fleet | Cross-agent reuse marginal value vs a LIVE tuned shared-prefix engine (head-to-head) | Head-to-head marginal value of cross-agent reuse vs a LIVE tuned shared-prefix engine (vLLM APC / SGLang RadixAttention, which also fuses a shared prefix once) remains UNMEASURED; fak’s own WebVoyager stratification puts the marginal-vs-tuned win at only ~1.0-1.10x on that workload. | SGLang RadixAttention (live process), vLLM Automatic Prefix Caching (live process) | link | 2026-01 |
| client-compaction | Long-session history compaction that preserves the provider prompt-cache prefix (drop-and-splice vs summarize-and-resend) | The field splits two ways. Summarizers (Aider, LangChain ConversationSummaryMemory, Codex CLI, Copilot CLI) and Anthropic’s API context-editing REWRITE/clear content and break the prefix cache (OpenAI docs: ‘when you drop, summarize or compact earlier turns … you’ll break the cache’; Anthropic context-editing docs: ‘Invalidates cached prompt prefixes when content is cleared’). The cache-preserving sub-field is small: LangChain trim_messages (pure sliding-window drop, no rewrite) and Copilot’s DELIBERATE cache-boundary reset reason about it; no surveyed tool ships a PROVEN byte-identity splice on by default. Context-editing pairs clearing with a memory tool for recovery (a different axis: recoverability), reporting up to ~84% token reduction on an Anthropic agent eval. | Anthropic API context-editing (clear_tool_uses_20250919), OpenAI Codex CLI (summary compaction), GitHub Copilot CLI (cache-boundary compaction), Aider (ChatSummary), LangChain (trim_messages / ConversationSummaryMemory) | link | 2026-01 |
| model-routing | Model routing granularity: per-aspect + first-class ensemble routing vs whole-request single-model selection | Surveyed 2025-2026 routers/gateways (RouteLLM, Martian, NotDiamond, Unify, OpenRouter, Portkey, LiteLLM Router) all route the WHOLE request to ONE model; OpenRouter Fusion is the only shipped model ensemble and it is a fixed parallel-synthesize recipe, not a configurable per-aspect reduction. None routes a sub-request aspect to its own model. (Aurelio Semantic-Router routes to an intent, not a model; vLLM/SGLang routers balance replicas of ONE model for KV locality — a different layer.) | RouteLLM (LMSYS), Martian, NotDiamond, Unify.ai, OpenRouter (+ Fusion), Portkey, LiteLLM Router | link | 2026-06 |
| request-routing | Cache-aware request routing across replicas (route to the worker that already holds the prefix) | Cross-region KV-locality-aware routing (GORGO-proxy) makes network latency an explicit routing term and cuts median TTFT ~2.5x vs least-load / prefix-trie baselines (224ms vs 568ms) while preserving prefix-cache locality across regions. | GORGO / GORGO-proxy (cross-region, arXiv 2602.11688), SkyWalker (locality-aware cross-region LB, EuroSys 2026, arXiv 2505.24095) | link | 2026-02 |
| time-to-solution | Long-horizon agent time-to-solution: wall-clock time for an agent (or agent fleet) to actually FINISH a multi-hour engineering task | FrontierSWE (Proximal Labs, published 2026-04-16) is the field ultra-long-horizon coding benchmark: 17 real tasks (5 implementation, 9 performance-engineering, 3 ML-research) with a 20-hour-per-task budget, graded on a zero-to-one scale rather than binary pass/fail. It is UNSATURATED - the top model (Claude Fable 5, avg rank 2.35 / 90 percent dominance) still fails to solve almost all tasks, and agents burn many hours (e.g. Opus 4.6 ~6.6-13.8h/task) making little progress. There is no published time-to-solution winner: the SOTA bar is that long-horizon completion is an OPEN problem the field measures but nobody yet finishes fast. | FrontierSWE (Proximal Labs) - the ultra-long-horizon coding benchmark, Claude Fable 5 / Opus 4.6 + agent scaffold (top FrontierSWE ranks), GPT-5.4 + agent scaffold | link | 2026-04 |
cost
See the full cost analysis for fak’s honest position on each dimension.
| Category | Dimension | SOTA bar | Leading systems | Source | Date |
|---|---|---|---|---|---|
| cost-efficiency | Inference unit economics ($ per 1M tokens) at realistic utilization | GPT-4-equivalent quality now serves at ~$0.40 / 1M tokens vs ~$20 in late 2022 (≈10x/yr decline). Real production GPU utilization is only 30-60%, so realistic cost-per-token runs 2-3x above spreadsheet estimates; FP8/FP4 quantization roughly halves cost-per-token on H100/H200. Self-hosting a 7B breaks even only above ~50% utilization. | Hosted frontier APIs, vLLM/SGLang self-host on H100/H200, FP8/FP4 quantization | link | 2025-09-01 |
| cost-efficiency | End-to-end effective request capacity / cost gain from the KV system under SLO | Mooncake’s KVCache-centric disaggregated architecture increases effective request capacity by 59%-498% on real traces (up to 525% in simulation) while meeting SLOs; this remains the headline effective-capacity-under-SLO bar, now republished in ACM Transactions on Storage (2026). | Mooncake (Moonshot AI / Kimi), NVIDIA Dynamo (KVBM + NIXL KV transfer) | link | 2024-07 |
| cost-efficiency | Energy efficiency (Wh per token / tokens-per-watt) and power-capped throughput | Empirical inference energy is ~0.0001-0.002 Wh per output token depending on model size; a ~500-token GPT-4o-class query is ~0.3 Wh (~3e-4 Wh/token). FP4/FP8 and Blackwell-class hardware are the main levers. No neutral audited tokens-per-watt leaderboard yet exists, so the bar is a measured range, not a single SOTA number. | NVIDIA Blackwell (GB200/GB300, FP4), AMD MI355X (FP4) | link | 2026-03-01 |
| cost-efficiency | Goodput under an SLA (requests/s meeting TTFT and TPOT SLOs simultaneously) | DistServe’s headline 7.4x more goodput (or 12.6x tighter SLO) over prior systems remains the canonical bar; newer 2025-26 work (TaiChi, PD-Multiplexing, DOPD, NVIDIA Dynamo) reports only incremental gains relative to disaggregation itself rather than a new baseline-relative record. | DistServe, NVIDIA Dynamo (incremental: 38% over best aggregated config, 2026), PD-Multiplexing/GreenContext (1.62x over PD-disagg, 2025-09) | link | 2024-01 |
| cost-efficiency | SLO-constrained goodput (per-GPU) | DistServe’s 4.48x SLO-constrained goodput (the conservative per-workload figure under strict TTFT+TPOT SLOs) still stands as the canonical bar; no 2025-26 system reports a higher SLO-constrained goodput multiplier against the same baseline. | DistServe, DOPD (>99% SLO attainment, up to 1.5x over vLLM/DistServe, 2026-03), TaiChi (90% SLO attainment, 29-77% over PD-disagg, 2025-08) | link | 2024-01 |
decoding
See the full decoding analysis for fak’s honest position on each dimension.
| Category | Dimension | SOTA bar | Leading systems | Source | Date |
|---|---|---|---|---|---|
| speculative-decoding | Draft acceptance length / acceptance rate (mean accepted tokens per verify step) | EAGLE-3 reports per-task acceptance rate ~0.80-0.88 (coding/instruction) and mean accepted length ~4.5-5.0 tokens per draft-verify cycle, exceeding the older DeepSeek-V3 MTP ~0.85 second-token figure. | EAGLE-3 (SafeAILab), DeepSeek-V3 MTP, FastMTP / FastEagle (2025 successors) | link | 2025-03 |
| speculative-decoding | Breadth of speculative methods supported (draft-model, EAGLE/-2/-3, Medusa, MTP, lookahead, n-gram/prompt-lookup) | TensorRT-LLM supports draft-target, Medusa, ReDrafter, Lookahead, and EAGLE(-1/-2, EAGLE-3 via two-model/disaggregated); vLLM supports EAGLE/EAGLE-3, Medusa, MTP, n-gram/prompt-lookup; SGLang supports EAGLE/EAGLE3 and standard draft | TensorRT-LLM, vLLM, SGLang | link | 2025-12 |
| speculative-decoding | Retrieval / prompt-lookup (n-gram) drafting for input-grounded tasks | SuffixDecoding (NeurIPS 2025, model-free suffix-tree drafting) is the new prompt-lookup-family SOTA: outperforms classic PLD/n-gram by 1.3-3x and delivers up to 5.3x speculative speedup (and ~4x end-to-end on SWE-Bench agentic tasks) over vanilla decoding. | SuffixDecoding (Snowflake Arctic Inference + vLLM), vLLM PLD/n-gram (prior bar), TensorRT-LLM Lookahead | link | 2025-05 |
| speculative-decoding | Speculative-decoding behavior under high concurrency / large batch (break-even) | EAGLE-3’s measured high-concurrency behavior: positive throughput uplift sustained to batch ~64 in SGLang (1.38x at batch 64), with the speedup approaching break-even beyond that as baseline throughput rises; EAGLE-2 already regresses (0.93x at batch 24). | EAGLE-3 (SGLang), MagicDec (fixed-window draft, prior reference), vLLM/SGLang continuous batching | link | 2025-03 |
| speculative-decoding | Output-distribution losslessness of speculation (greedy and stochastic) | EAGLE 3.1 (vLLM/EAGLE/TorchSpec, May 2026) is the current SOTA lossless speculative-decoding method: up to 2x longer acceptance length and up to ~2.03x per-user output throughput vs EAGLE-3 (Kimi-K2.6-NVFP4, GB200), fixing ‘attention drift’ robustness; correctness is preserved by rejection-sampling verification that matches the target distribution (greedy spec == greedy non-spec). | EAGLE 3.1 (vLLM + TorchSpec), EAGLE-3, SpecForge training framework | link | 2026-05 |
| speculative-decoding | Speculative-decoding wall-clock speedup (single-stream / low-concurrency) | EAGLE-3 reports up to 4.79x speedup on LLaMA-3.3-70B with no quality loss and is the de-facto industrial standard supported in vLLM and SGLang; vLLM’s P-EAGLE reaches ~4-5x over standard decoding (20-30% over EAGLE-3 alone) on coding benchmarks | EAGLE-3 (SafeAILab), vLLM, SGLang, P-EAGLE (vLLM) | link | 2025-09 |
| speculative-decoding | Speculative decoding effect on ITL (acceptance-rate-bound) | EAGLE-3 reports ~0.80-0.88 acceptance on coding/instruction tasks giving ~3-4x decode token-rate (ITL) gains on H100/H200, and outperforms EAGLE-2/Medusa-2 by 15-25% tokens/s at batch size 1. Speedup primarily reduces ITL, not TTFT. | EAGLE-3, Medusa-2, TensorRT-LLM / vLLM speculative decoding | link | 2026-01 |
| speculative-decoding | Speculative decoding throughput effect across batch sizes (does it survive at high concurrency?) | EAGLE-3 remains the named SOTA: ~1.38x throughput at batch 64 in SGLang on a single H100, peaking at ~1.81x at batch 2 and up to ~6.5x single-stream, ~1.4x over EAGLE-2. | EAGLE-3 (SafeAILab), SGLang speculative backend, vLLM/TensorRT-LLM spec backends | link | 2025-03 |
| structured-output | Constrained/guided decoding per-token overhead (JSON-schema, regex, grammar/CFG) | XGrammar-2 (arXiv 2601.04426, Jan 2026) drops grammar-mask per-token overhead to ~12-20us (Llama tool-calling ~12-13us, OpenAI Harmony ~15-20us) with <6% end-to-end overhead vs unconstrained and ~10ms (>100x faster) dynamic compilation; it is the new SOTA, far below LLGuidance’s ~250us (Harmony) / >1000us (tool-calling) on the agentic workloads. | XGrammar-2 (vLLM/SGLang/TensorRT-LLM default backend), XGrammar, LLGuidance (Rust Earley) | link | 2026-01 |
| structured-output | Grammar/schema compilation latency and cache (cold-start for dynamic agentic schemas) | XGrammar-2 compiles in ~10 ms vs >1000 ms for XGrammar (~80x faster), with >6x tool-calling compilation speedup via Earley-based adaptive mask cache + cross-grammar cache + JIT; integrated into vLLM and SGLang | XGrammar-2, XGrammar, LLGuidance/Guidance | link | 2026-05 |
| structured-output | Sampler coverage and correctness (temperature, top-p, top-k, min-p) | Reference samplers cover the full mainstream set: temperature, top_p, top_k, min_p, typical, presence/frequency/repetition penalties, logit_bias, seed/n/best_of, plus pluggable custom logits processors (argmax-invariant classification). vLLM and llama.cpp converge on Temperature+Min-P as the 2026 default power-user combo. | vLLM SamplingParams + logits processors, llama.cpp sampler chain, SGLang sampler | link | 2026-01 |
| structured-output | Structured-output validity vs reasoning-quality tax (constraint-induced degradation) | Constrained decoding guarantees ~100% syntactic/schema validity (JSON-Pass/Type-Safety >.95) but does NOT guarantee semantic quality and can REDUCE accuracy: ExtractBench (2026) shows structured mode dropping overall validity 51%->37% and GPT-5 credit-agreement pass rate 86.9%->70.0% vs prompt-based extraction. The honest SOTA bar is ‘validity is solved; value accuracy is the open gap (13.7pt spread across models).’ | XGrammar (vLLM/SGLang/TRT-LLM default), LLGuidance (guidance-ai), OpenAI Structured Outputs | link | 2026-02 |
| structured-output | Tool/function-call parsing coverage and robustness (per-model formats) | vLLM ships per-family parsers (hermes, llama3_json, mistral, pythonic, internlm, granite, plus qwen3/qwen3_coder); when constraints aren’t enforced it falls back to extracting from raw text, so args can be malformed - making grammar-backed (constrained) tool calling the robustness frontier | vLLM tool-call parsers, SGLang, XGrammar-2 (grammar-backed tool calling) | link | 2025-12 |
distributed
See the full distributed analysis for fak’s honest position on each dimension.
| Category | Dimension | SOTA bar | Leading systems | Source | Date |
|---|---|---|---|---|---|
| parallelism | Data-parallel (DP) attention / hybrid attention-FFN parallelism | SGLang’s DP Attention is the production technique that eliminates KV-cache duplication across devices for DeepSeek-V3, enabling the EP32/EP72 deployment to scale memory-efficiently; combined with PD disaggregation and MTP it delivers up to ~60% higher output throughput. | SGLang, vLLM | link | 2025-07-17 |
| parallelism | Large-scale expert parallelism (EP) for giant MoE models | EP320: Huawei CloudMatrix-Infer runs DeepSeek-R1 with 320-way expert parallelism (one expert per NPU die: 32 shared + 256 router + 32 redundant experts) over the UB network, superseding the 96-GPU large-scale EP (effective EP~72) demonstrated by LMSYS/SGLang. | Huawei CloudMatrix-Infer (CloudMatrix384, DeepSeek-R1), SGLang + DeepEP (96-GPU, prior bar), vLLM expert-parallel deployment | link | 2025-06 |
| parallelism | Multi-node scale-out (combined TP x PP x EP x DP across hosts) | A single GB300 NVL72 is 72 Blackwell-Ultra GPUs in one NVLink domain; MLPerf Inference v6.0 (2026-03-30) composed FOUR of them – 288 GPUs over Quantum-X800 InfiniBand, NVIDIA’s largest-ever submission – into one coherent DeepSeek-R1 671B MoE serving run at ~2.494M tok/s aggregate offline. Disaggregated serving via NVIDIA Dynamo composes the TPxPPxEPxDP dimensions across the fabric (up to 1.5x on Llama-3.1-405B interactive). | NVIDIA GB300 NVL72 x4 (288 GPUs over Quantum-X800 InfiniBand, MLPerf v6.0), SGLang/TensorRT-LLM + NVIDIA Dynamo multi-node, GB200 NVL72 (prior generation) | link | 2026-04-01 |
| parallelism | Prefill-decode disaggregation & per-phase SLO isolation | Phase disaggregation for SLO isolation (separating compute-bound prefill from memory-bound decode so each phase’s SLO is met independently) is now production-standard via NVIDIA Dynamo, which reports 38% goodput over the best aggregated config and 7x with wide expert parallel on GB200 NVL72; DistServe’s 7.4x remains the founding goodput figure. | NVIDIA Dynamo 1.0, DistServe, TaiChi (unifies agg+disagg to fix both TTFT and TPOT violations, 2025-08) | link | 2026-03 |
| parallelism | Prefill-decode disaggregation: independent scaling of compute-bound prefill and memory-bound decode | SGLang PD disaggregation achieves up to 6.9x more goodput (Chatbot 3200in/800out, TTFT<=1000ms, TPOT<=25ms) on MI300X; this remains the headline open-framework PD-disaggregation figure, now corroborated by NVIDIA Dynamo’s 2026 production results (38% goodput over best aggregated config; 7x with wide expert parallel on GB200 NVL72). | SGLang PD (MI300X), NVIDIA Dynamo 1.0 (38% over best aggregated config; 7x w/ wide-EP on GB200), vLLM disaggregated prefilling (NIXL) | link | 2025-08 |
memory
See the full memory analysis for fak’s honest position on each dimension.
| Category | Dimension | SOTA bar | Leading systems | Source | Date |
|---|---|---|---|---|---|
| kv-cache | Attention architecture impact on KV footprint (MQA/GQA/MLA bytes-per-token) | Multi-head Latent Attention (MLA, DeepSeek-V3) holds ~70 KB/token KV footprint vs 192-328 KB/token for GQA models (a 2.7-4.7x reduction, ~9x vs MHA, ~2x vs 8-group GQA) at higher quality; the frontier is now MLA + learned sparsity (DeepSeek NSA, 2025) pushing toward sub-100 floats/token for long contexts. | MLA (DeepSeek-V3), MLA + Native Sparse Attention (NSA, DeepSeek 2025), Qwen-Latent / MiniMax-Text-01 MLA-style | link | 2025-02 |
| kv-cache | Automatic prefix caching / RadixAttention prefix reuse and cache hit rate | Automatic prefix caching now reaches ~85-95% cross-request hit rates on shared-prefix workloads (SGLang RadixAttention few-shot 85-95%, agentic ~88%; vLLM APC ~87% warm-cache hit, ~88% faster TTFT); the original RadixAttention 6.4x throughput claim (LMSYS 2024-01) remains the canonical headline number. | SGLang RadixAttention, vLLM Automatic Prefix Caching | link | 2024-01 |
| kv-cache | Automatic cross-request prefix / KV-cache reuse (shared system prompts, agent scaffolds, few-shot) | Provenance: SGLang-published KV plane. SGLang RadixAttention’s automatic cross-request KV reuse via an LRU radix tree gives up to 6.4x throughput on shared-prefix workloads with 50%-99% cache hit rates; this remains the canonical cross-request prefix-reuse bar in 2026. | SGLang RadixAttention, vLLM Automatic Prefix Caching, Mooncake KV pool | link | 2024-01 |
| kv-cache | KV-cache-aware / prefix-aware request routing across a fleet | KV-cache-aware routing across a replica fleet now delivers ~50% lower TTFT, ~34% lower TPOT, ~61% more RPS and ~62% more output tok/s (Baseten on NVIDIA Dynamo, Qwen3 480B), with llm-d reporting 3x throughput / 2x TTFT vs round-robin and GKE Inference Gateway 13.9x faster TTFT than a 3rd-party service. | Baseten + NVIDIA Dynamo KV router, llm-d v0.5 (GAIE/EPP prefix-cache-aware), GKE Inference Gateway prefix-cache-aware routing | link | 2025-10 |
| kv-cache | KV-cache memory efficiency and max concurrent sequences (paging / fragmentation) | PagedAttention keeps KV-cache memory waste under 4% (near-zero internal/external fragmentation), enabling 2-4x throughput at the same latency; this remains the canonical paged-KV efficiency reference as of mid-2026. | vLLM PagedAttention, SGLang RadixAttention (radix-tree paged KV) | link | 2023-09 |
| kv-cache | KV-cache quantization (FP8 / INT8 / KIVI sub-4-bit) accuracy at long context | Ultra-low-bit KV quant frontier (2026): Together AI’s OSCAR reaches 2.28 effective bits within 1.42 points of BF16 (Qwen3-8B; ~0.02-pt gap on Qwen3-32B) with ~8x KV memory reduction at 100K context and up to 7.83x job-level throughput; TurboQuant (ICLR 2026) is near-lossless at 3-4 bits / full-precision NIAH recall at 4x. | OSCAR (Together AI, 2026), TurboQuant (Google/NYU, ICLR 2026), Kitty 2-bit dynamic channel-wise (arXiv 2511.18643) | link | 2026-06 |
| kv-cache | KV compression / token eviction under a fixed cache budget | Reasoning-aware eviction now beats the old H2O/Quest budget cliff: R-KV achieves lossless compression at 34% KV budget on MATH-500 and 10% on AIME-2024, and at 16% budget exceeds FullKV at 105% accuracy; Apple’s EpiCache (2026) gets near-full-cache accuracy at 4-6x compression with up to 40% higher accuracy than eviction baselines and 3.5x lower peak memory. | R-KV (redundancy-aware, reasoning, arXiv 2505.24133), EpiCache (Apple, 2026), SnapKV / H2O / Quest (baselines now superseded) | link | 2025-05 |
| kv-cache | KV-cache offloading to CPU/host/NVMe/remote (multi-tier hierarchy depth) | LMCache (graduated to production Jan 2026, OSDI/arXiv paper Oct 2025) delivers up to 15x throughput improvement over vanilla vLLM by tiering KV cache across HBM/DRAM/NVMe with a global prefix index; llm-d production benchmarks report up to 57x faster TTFT and 2x throughput vs round-robin under high prefix reuse on 16 H100s. | LMCache (vLLM, production Jan 2026), llm-d native KV offload (57x TTFT), Mooncake Store | link | 2025-10 |
| kv-cache | KV-cache transfer for prefill/decode disaggregation (NIXL / NCCL / UCX) | NVIDIA Dynamo 1.0 (GA March 16, 2026 at GTC) delivers up to 7x higher throughput with disaggregated prefill/decode serving combined with wide expert parallelism, measured on DeepSeek-R1 on GB200 NVL72; KV blocks move prefill->decode over RDMA via NIXL at wire speed. | NVIDIA Dynamo 1.0 + NIXL (GA Mar 2026), Mooncake Transfer Engine, SGLang EPD disaggregation | link | 2026-03 |
| kv-cache | Non-prefix / cross-document KV reuse (RAG chunk caching) | LMCache CacheBlend reuses NON-prefix KV caches (any RAG chunk, not just the first) and selectively recomputes ~15% of tokens, achieving near-100% KV-cache hit rate with 2.2-3.3x lower TTFT and 2.8-5x higher throughput vs full recompute; won ACM EuroSys’25 Best Paper. | LMCache CacheBlend, Mooncake (non-prefix KV pool) | link | 2025 |
| kv-cache | Paged KV block management and memory utilization (anti-fragmentation) | PagedAttention achieves ~96.3% KV-cache memory utilization (the complement of <4% waste); no 2026 system reports a higher published paged-KV utilization figure. | vLLM PagedAttention, SGLang RadixAttention | link | 2023-09 |
| kv-cache | Prefix/KV-cache reuse impact on TTFT (cache-hit latency) | Qualitative: APC reuses exact KV pages via hash-based block matching, cutting prefill tokens and TTFT for repeated/long-prefix prompts with negligible steady-state overhead; overhead from hash compute grows with input length and concurrency. Distributed prefix-cache scheduling (llm-d) extends the win across replicas. | vLLM Automatic Prefix Caching (APC), SGLang RadixAttention, llm-d distributed prefix cache | link | 2025-01 |
| kv-cache | Prefix caching / automatic KV reuse across requests (shared-context workloads) | SGLang RadixAttention delivers ~29% higher end-to-end throughput than vLLM on H100 (16,200 vs 12,500 tok/s, Llama 3.1 8B) on shared-prefix workloads, with up to 6.4x on prefix-heavy RAG/multi-turn traffic; the 29% SGLang-vs-vLLM edge still holds in 2026 benchmarks. | SGLang RadixAttention, vLLM Automatic Prefix Caching | link | 2026 |
models
See the full models analysis for fak’s honest position on each dimension.
| Category | Dimension | SOTA bar | Leading systems | Source | Date |
|---|---|---|---|---|---|
| hardware-coverage | Hardware backend breadth (NVIDIA, AMD, Intel, TPU, AWS, Apple, CPU) | vLLM runs on NVIDIA, AMD ROCm (MI300X/MI325X/MI350X/MI355X), x86/ARM/PowerPC CPU, plus plugin backends for Google TPU, Intel Gaudi 2/3, AWS Inferentia/Trainium (Neuron), Apple Silicon, Huawei Ascend, IBM Spyre/Z, and more; llama.cpp targets the widest portability surface (CUDA, HIP/ROCm, Metal, Vulkan, SYCL, CANN, OpenCL, CPU SIMD on x86 AVX-512/AMX and ARM Neon/SVE/SME). MLC-LLM uniquely adds WebGPU/browser via TVM compilation. | vLLM, llama.cpp, SGLang, MLC-LLM | link | 2026-01-01 |
| hardware-coverage | Hardware / format coverage and portability of quantized kernels | FP8 W8A8 is hardware-accelerated on NVIDIA Hopper/Ada/Blackwell and AMD MI300X (hipBLASLt FP8 GEMM + aiter attention); NVFP4 is Blackwell-only; MXFP4 acceleration on MI300X/MI355X is on the roadmap, not yet broadly shipped. GGUF k-quants give the widest CPU/consumer-GPU portability. | vLLM (FP8 on Hopper/Ada/Blackwell + MI300X via hipBLASLt/aiter), SGLang (FP8 MI300X), TensorRT-LLM (NVFP4 Blackwell), llama.cpp (broad CPU/GPU GGUF + FP4) | link | 2026-01 |
| hardware-shape-neutrality | Hardware-shape neutrality: explicit fences for host-CPU assumptions | A hardware-neutral stack exposes backend contracts, capability discovery, layout/dtype/residency boundaries, and explicit fallback behavior so new accelerators do not require a fork of the model loop. Mature serving stacks cover more device families, but most public scorecards still report breadth without a seven-assumption shape-neutrality ledger. | vLLM plugin backends, llama.cpp backend matrix, MLC-LLM / TVM compilation | link | 2026-06-30 |
| model-coverage | Embedding & reranker (non-generative) model coverage | vLLM and SGLang both serve embedding and reranker/cross-encoder models alongside generative models with pooling support and OpenAI-compatible embeddings endpoints, covering the common open embedding/reranker families (BGE, E5, GTE, Qwen-embedding, etc.) on the same engine. | vLLM, SGLang, TensorRT-LLM | link | 2026-01-01 |
| model-coverage | Long-context serving (128K-1M tokens) & KV/prefix-cache reuse | Million-token windows are production-ready (Qwen2.5-1M open weights; 128K-1M is mainstream). SGLang’s RadixAttention prefix caching delivers up to ~6.4x gains on prefix-heavy RAG/multi-turn workloads and a ~29% throughput edge over vLLM on H100 (16.2K vs 12.5K tok/s) on shared-prefix traffic. | SGLang (RadixAttention), vLLM (PagedAttention/prefix caching) | link | 2026-01-01 |
| model-coverage | LoRA / multi-LoRA hot-swap serving | The peer-reviewed SOTA is S-LoRA (MLSys’24, arXiv:2311.03285): ~2000 LoRA adapters served over one base model on a single A100-80GB at up to 4x vLLM throughput, via unified paging of adapter weights + custom MBGMM/MBGMV kernels. vLLM Multi-LoRA integrated S-LoRA-style techniques as a follow-on (max_loras GPU-resident + max_cpu_loras LRU CPU cache + runtime hot-swap); SGLang/TensorRT-LLM offer comparable multi-adapter batching. Practical resident count is bounded by GPU memory and adapter rank. | S-LoRA (MLSys’24, ~2000 adapters/A100-80GB), vLLM Multi-LoRA, SGLang, TensorRT-LLM | link | 2025-08 |
| model-coverage | Maximum parameter count & frontier-MoE architecture coverage | Kimi K2 (Moonshot AI) – 1 trillion total parameters / 32B active, 384 experts (8+1 active) – is the largest open-weight MoE routinely served by vLLM and SGLang, exceeding DeepSeek-V3/R1’s 671B. | Kimi K2 / K2.x (Moonshot AI, 1T MoE; vLLM + SGLang + KTransformers), DeepSeek-V3/R1 671B (prior bar), Qwen3 MoE / Llama-4 (other frontier MoE) | link | 2025-07 |
| model-coverage | Multimodal / vision-language (VLM) model coverage | SGLang is explicitly a serving framework ‘for large language models and multimodal models’ and extends RadixAttention prefix caching to multimodal inputs with vision-embedding reuse; vLLM also provides broad multimodal model support. Coverage spans the major open VLM families (Qwen-VL, Llama Vision, InternVL, Pixtral, etc.). | SGLang, vLLM | link | 2026-01-01 |
| model-coverage | Capability/quality of the served model (SWE-bench Verified, GPQA, AIME, LiveCodeBench) | Frontier models served by these engines define the quality bar buyers cite: SWE-bench Verified ~80.6% (Gemini 3.1 Pro) / 79.2% (Claude Opus 4.5 + Live-SWE-agent, Nov 2025); GPQA Diamond effectively saturated ~93-94% (Gemini 3 Deep Think 93.8%, GPT-5.4 92.8%); AIME 2025 up to 100% (GPT-5.4). These are model-served-by-engine results, not engine claims. | Gemini 3.x (SWE-bench/GPQA), Claude Opus 4.5 + agent scaffold, GPT-5.x (AIME) | link | 2025-11-24 |
numerics
See the full numerics analysis for fak’s honest position on each dimension.
| Category | Dimension | SOTA bar | Leading systems | Source | Date |
|---|---|---|---|---|---|
| numerical-correctness | Deterministic / bitwise-reproducible inference (batch invariance) | Batch-invariant kernels (invariant normalization, matmul, attention reductions) yield 1000/1000 bitwise-identical completions under dynamic batching, vs dozens of distinct outputs without; cost ~60% slower than default kernels. Now exposed as a vLLM batch-invariance feature. | Thinking Machines batch-invariant-ops (on vLLM FlexAttention), vLLM batch-invariance mode, LLM-42 (deterministic speculation) | link | 2025-09 |
| numerical-correctness | Numerical-correctness error metric vs reference (perplexity, KL-divergence, recovery %) | Best practice pairs perplexity with KL-divergence against the FP16 baseline (llama.cpp emits both); KLD captures per-token distribution shift perplexity hides. Empirically: 5-bit GGUF is a strong near-lossless trade-off, 3-bit shows clear degradation; imatrix is essential below 3-bit, marginal at Q4_K_M+. Generation text-similarity is reported as a supplementary correctness check. | llama.cpp (perplexity + KL-divergence tooling), Neural Magic / vLLM recovery-% methodology, ModelOpt accuracy harness | link | 2026-01 |
| numerical-correctness | Dense GPU compute correctness with ZERO vendor GEMM (cuBLAS-free) | cuBLAS is the tuned vendor GEMM every other stack carries; matching it bit-for-bit (cosine=1.0) is parity by construction. | cuBLAS (NVIDIA tuned vendor GEMM) | link | 2026-06 |
| quantization | Full 4-bit (W4A4 + KV4) via rotation/outlier removal | Learned-rotation methods now lead 4-bit weight+activation: SpinQuant (ICLR 2025) closes the gap to full precision by up to 45.1% relative to QuaRot on hard-to-quantize Llama-3 8B, and 2025-2026 successors (Qronos, ResQ PCA-rotations, TurboQuant for KV cache, WUSH adaptive transforms) advance further; Hadamard/learned rotations are the standard outlier-removal primitive feeding NVFP4/W4A4. | SpinQuant (learned rotations), Qronos (2025), ResQ PCA-rotation (2025), TurboQuant (2026, KV cache), QuaRot (Hadamard baseline) | link | 2025-04 |
| quantization | 4-bit microscaling float (NVFP4 / MXFP4 / FP4) accuracy vs FP8 on Blackwell | NVFP4 (4-bit, 16-value blocks, FP8 E4M3 fractional block scales) now achieves <=1% accuracy drop vs FP8 on large reasoning models (DeepSeek-R1-0528: MMLU-PRO 85->84, GPQA 81->80) and ~99% recovery vs BF16 for 70B-235B models, at ~2.3x higher throughput than weight-only 4-bit; clearly the SOTA 4-bit format on Blackwell, well ahead of MXFP4 (32-value, power-of-two scales). | NVFP4 via NVIDIA TensorRT Model Optimizer + LLM Compressor (vLLM, Blackwell B200/GB200), NVFP4 (vLLM Marlin) | link | 2025-09 |
| quantization | FP8 (W8A8, E4M3) accuracy retention vs bf16 reference | FP8 (W8A8, E4M3) weight+activation quantization is effectively lossless: ~99.9% accuracy recovery on HumanEval and <=1% degradation vs BF16 across all model scales, including Llama 3.1 405B fully recovering accuracy. | FP8 W8A8 via LLM Compressor (vLLM), NVIDIA TensorRT Model Optimizer FP8, AMD Quark FP8 | link | 2024-10 |
| quantization | INT4 weight-only (W4A16: AWQ, GPTQ) accuracy and calibration sensitivity | INT4 weight-only (W4A16) via AWQ or GPTQ keeps accuracy degradation within negligible-to-moderate range (<4% typical, often <1-2% on HumanEval); AWQ is the de-facto best-practice INT4 format for vLLM in 2026 and generally edges out GPTQ on perplexity. | AWQ (AutoAWQ) via vLLM, GPTQ / GPTQModel + Marlin (vLLM), W4A16 LLM Compressor | link | 2026-01 |
| quantization | int8 / Q8_0 SIMD decode throughput vs the same-rung int8 peer | HuggingFace dynamic-int8 is the standard same-rung CPU int8 reference (1×); llama.cpp Q8_0 CPU is the harder peer (~parity). | HuggingFace dynamic-int8, llama.cpp Q8_0 | link | 2026-06 |
| quantization | INT8 W8A8 (SmoothQuant) accuracy under activation outliers | INT8 W8A8 (per-channel static weight + per-token dynamic activation, optionally SmoothQuant) recovers ~99% of BF16 accuracy, i.e. ~1% average loss; vLLM/LLM Compressor report negligible Open-LLM-leaderboard drop. | INT8 W8A8 SmoothQuant via LLM Compressor (vLLM), SmoothQuant (Xiao et al.) | link | 2025-03 |
| quantization | KV-cache quantization (FP8 / INT8 / INT4 KV) | FP8 (e4m3) KV-cache quantization is the production standard: ~46% KV memory reduction (per-token cost ~54% of BF16), +14.9% output throughput / -14.8% median ITL on Llama-3.1-8B, recovering 97-98% of baseline long-context AUC at 128k and fully recovering aggregated AUC@1M. INT4 KV gives ~75% reduction but is research-stage; vLLM ships FP8 only. | FP8 e4m3 KV (vLLM production, Apr 2026), INT4 KV (research: ~75% reduction), FP8 e5m2 (vLLM) | link | 2026-04 |
| quantization | Accuracy-constrained throughput on a neutral benchmark (MLPerf Inference) | Under MLPerf’s accuracy- and latency-constrained server scenario (TTFT and TPS/user SLOs enforced at p99), MLPerf Inference v6.0 (2026-03-30) DeepSeek-R1 reaches 8,064 tokens/second/GPU on GB300 NVL72 (vs 9,821 tok/s/GPU unconstrained offline); DeepSeek-R1’s server SLO is 2s TTFT with a 12.5 tok/s/user p99 target while holding the required accuracy bar, and the GB300 server result is ~25% higher per-GPU than GB200 and up to 2.7x the GB300 debut via software. | NVIDIA GB300 NVL72 (Blackwell Ultra, TensorRT-LLM/Dynamo), NVIDIA GB200 NVL72 (Blackwell), CoreWeave / Nebius GB300 NVL72 submissions | link | 2026-04-01 |
| quantization | Quantization format & low-precision datatype coverage | Reference serving stacks now cover the full ladder: vLLM (2026) supports FP8 (W8A8), INT8 (W8A8 + W4A8), INT4 (W4A16), NVFP4 and MXFP4 (via Marlin), AWQ, GPTQ, BitsAndBytes (4/8-bit), AMD Quark, and online runtime quantization, across Turing->Hopper->Blackwell plus x86/Arm CPU for INT8. | vLLM quantization (LLM Compressor, Marlin, AutoAWQ, GPTQModel, bitsandbytes, Quark), TensorRT Model Optimizer (FP8/INT4/NVFP4) | link | 2026-01 |
| quantization | Quantization granularity (per-tensor / per-channel / per-group / microscaling block size) | Per-block (block-32) microscaling beats per-tensor/per-channel; measured SNR gains of 3.0-3.4 dB over group-128 and 9.2-9.4 dB over per-tensor for two-level microscaling. Common group sizes 32/64/128; group-128 is the de-facto 4-bit weight-only default. NVFP4’s block-16 + FP32 scale is finer than MXFP4’s block-32 power-of-two. | MX formats (OCP MXFP4/MXFP8, block-32), NVFP4 (block-16, FP32 second-level), group-128 INT4 (AWQ/GPTQ), MOSS two-level microscaling | link | 2025-11 |
operability
See the full operability analysis for fak’s honest position on each dimension.
| Category | Dimension | SOTA bar | Leading systems | Source | Date |
|---|---|---|---|---|---|
| operability | Cold-start / autoscaling latency (scale-to-zero & scale-out) | GPU-snapshotting cold start is now sub-200 ms: RunPod FlashBoot achieves sub-200 ms for ~48% of requests (<250 ms generally) and Modal GPU memory snapshots cut Ministral-3 3B median cold start ~10x (118 s -> 12 s) - the corroborable bar. InferX additionally claims an industry-leading 177 ms on H100 via GPU snapshotting, but that single number traces to one vendor source and is not independently confirmable, so the defensible figure is sub-200 ms. | RunPod FlashBoot (sub-200 ms / <250 ms), Modal GPU memory snapshots (10x: 118 s -> 12 s), InferX (177 ms vendor claim, GPU snapshot, uncorroborated) | link | 2026 |
| operability | Fairness, priority, and tenant isolation in batch formation / scheduling | No single audited cross-system number; SOTA is qualitative: FairBatching reformulates batch formation to bound per-request unfairness, and serving stacks add priority classes plus preemption (KV-page eviction with recompute-on-resume in SGLang) to isolate tenants. Buyers evaluate head-of-line-blocking behavior and priority-preemption support directly. | FairBatching (fairness-aware batch formation), SGLang (FCFS + page-eviction preemption), hybrid real-time/best-effort schedulers (e.g. arXiv 2504.09590) | link | 2025-10 |
| operability | Multi-node fault tolerance & failure recovery (resilient serving) | Research systems define the bar: KevlarFlow reports ~20x lower mean-time-to-recovery via decoupled parallel init, dynamic traffic rerouting, and background KV-cache replication; FailSafe sustains high-performance tensor-parallel serving under irregular GPU availability; ReviveMoE targets fast recovery from hardware failures specifically in large-scale MoE inference. Production stacks (Dynamo) add dynamic GPU scheduling and rerouting but full resilient serving is still maturing. | FailSafe (research), KevlarFlow (research), ReviveMoE (research), NVIDIA Dynamo | link | 2026-01-01 |
| operability | Kubernetes-native deployment + LLM-aware autoscaling (metric-driven, scale-to-zero) | KServe + KEDA autoscale vLLM on LLM-specific signals (e.g. vllm:num_requests_running) rather than CPU, supporting scale-from/to-zero with a default 5-min cooldown; the 2025-2026 production stack is vLLM + KServe/llm-d + Ray + Kueue + KEDA on K8s. Ray Serve and KServe provide canary, traffic-split, and multi-model endpoints. The SOTA bar is custom-LLM-metric autoscaling with scale-to-zero, not generic HPA. | KServe + KEDA, Ray Serve, llm-d / vLLM on Kubernetes | link | 2025-09-23 |
| operability | Multi-tenant fairness, per-tenant quotas, and SLO attainment under contention | VTC (the 2024 first fair scheduler) is now superseded by a 2025 family: Equinox (whole-system fair scheduling), FairBatching (fairness-aware batch formation), PROSERVE (multi-priority SLO-aware), and DLPM (locality-aware fair scheduling) – all addressing VTC’s gaps on prefix-locality, SLO, and diverse workloads. | Equinox (2025), FairBatching (2025), PROSERVE (2025), DLPM locality-aware fair scheduling (2025), VTC (2024, prior bar), LiteLLM virtual-keys/budgets | link | 2025-08 |
| operability | Production observability: Prometheus metrics (TTFT/TPOT/queue) + OpenTelemetry tracing | vLLM v1 exposes an extensive Prometheus /metrics surface (vllm:time_to_first_token_seconds, TPOT/inter-token, vllm:num_requests_running, vllm:kv_cache_usage_perc, prefix-cache hit) plus native OpenTelemetry distributed tracing via –otlp-traces-endpoint – the de facto serving-observability contract. | vLLM v1 (Prometheus + OpenTelemetry), SGLang, NVIDIA Dynamo / GenAI-Perf | link | 2026-01 |
| operability | OpenAI-compatible API surface + reliable structured output / tool-calling | Constrained/guided decoding now guarantees 100% schema-valid structured output by construction (XGrammar-2 is the default engine across vLLM/SGLang/TensorRT-LLM, <40 us/token), with the OpenAI-compatible /v1/chat/completions + response_format json_schema surface as the de facto contract. | vLLM + XGrammar-2 (default), SGLang, TensorRT-LLM, OpenAI Structured Outputs (response_format) | link | 2026-01 |
| operability | Cross-instance / persistent KV cache sharing and coherence | LMCache enables KV reuse across different vLLM engine instances plus multi-GPU peer-to-peer KV sharing and disaggregated prefill/decode; Mooncake Store exposes a global, deduplicated KVCache pool addressable across the cluster | LMCache, Mooncake Store | link | 2025-10 |
| operability | Latency degradation under concurrency / load (saturation behavior) | Under concurrency, single-GPU TTFT P99 explodes (e.g. ~8.6 s at C=180, ~34 s at C=420); pipeline parallelism (PP=2) holds it to ~3 s at C=180 and ~13 s at C=420 (~2.5-3x improvement). At 100 concurrent requests TensorRT-LLM P95 TTFT ~1,280 ms vs vLLM ~1,450 ms; vLLM sustains QoS to C=32 and stays usable to C=64. | TensorRT-LLM (lowest P95 TTFT under load), SGLang, vLLM (most robust queueing/scheduling) | link | 2026-01 |
security
See the full security analysis for fak’s honest position on each dimension.
| Category | Dimension | SOTA bar | Leading systems | Source | Date |
|---|---|---|---|---|---|
| security | Prompt-injection defense and agent security (attack-success-rate vs utility-under-attack) | On AgentDojo, design-by-construction defenses now reach provable security: CaMeL drives successful indirect-prompt-injection attacks to 0 (vs 8 for the next-best tool-filter) while still solving 77% of tasks (vs 84% undefended); MELON (ICML’25) likewise beats prior SOTA defenses. Undefended best agents still see ASR <25%. | CaMeL (Defeating Prompt Injections by Design, 2025), MELON (ICML’25), Meta SecAlign (secure foundation LLM), tool-filter / SecAlign baselines | link | 2025-03 |
| security | Tool/agent sandboxing, structural containment, and PII/exfil prevention | Capability/IFC-based structural containment is the 2025 SOTA: CaMeL (Defeating Prompt Injections by Design) extracts control/data flow so untrusted data can never alter program flow and enforces capability policies at tool-call time (0 successful AgentDojo attacks); MELON adds masked re-execution + tool-comparison provable defense. | CaMeL (capability + control/data-flow IFC, 2025), MELON (masked re-execution, ICML’25), container/microVM sandboxes, IFC / capability allow-lists | link | 2025-03 |
serving
See the full serving analysis for fak’s honest position on each dimension.
| Category | Dimension | SOTA bar | Leading systems | Source | Date |
|---|---|---|---|---|---|
| latency | End-to-end request latency (E2EL) | End-to-end request latency (TTFT + total decode) is reported per request by the standard tools (NVIDIA GenAI-Perf, vLLM bench serve, LLMPerf) with P99 as the default percentile; there is no single SOTA number - it is a tunable composite governed by the per-model MLPerf v6.0 TTFT+TPOT SLOs (e.g. a 1000-token DeepSeek-R1 interactive response budgets ~1.5 s TTFT + ~15 s decode). | NVIDIA GenAI-Perf, vLLM bench serve (–metric-percentiles, default p99), LLMPerf | link | 2026-01 |
| latency | Inter-token latency / time-per-output-token (ITL/TPOT) | MLPerf Inference v6.0 reasoning-model interactive scenarios now require TPOT <= 15 ms P99 (GPT-OSS 120B and DeepSeek-R1), down from the prior 80 ms TPOT bar; the legacy Llama-2-70B interactive bound is 40 ms TPOT (~25 tok/s/user) and server is 80-200 ms. | MLPerf Inference v6.0 (GPT-OSS 120B, DeepSeek-R1 interactive), MLPerf Inference v5.1 | link | 2026-03-24 |
| latency | Interactive latency SLO attainment: TTFT and TPOT/ITL tail under load | MLPerf Inference v6.0 (Apr 2026) tightened the interactive SLO floor: the most stringent latency-aware bounds are now TPOT <= 15 ms with TTFT <= 1.5-2.0 s P99 (GPT-OSS 120B: TTFT <= 2.0 s / TPOT <= 15 ms; DeepSeek-R1: TTFT <= 1.5 s / TPOT <= 15 ms), superseding the v5.1 30 ms-TPOT / 0.5 s-TTFT interactive bar. | MLPerf Inference v6.0 (NVIDIA Blackwell Ultra GB300, AMD MI355X), MLPerf Inference v5.1 | link | 2026-03-24 |
| latency | Prefill-decode interference & chunked-prefill TTFT/TPOT tension | Chunked prefill cuts P95 inter-token latency by ~68% under mixed long/short workloads (e.g. 2,800 ms -> 890 ms P95 ITL at 32K inputs) by prioritizing decode and chunking prefill into the token budget. | vLLM chunked prefill (decode-prioritized scheduling), SGLang chunked prefill | link | 2026-06 |
| latency | Tail latency distribution (P50/P95/P99/P99.9) | P99 (99th percentile) is the canonical tail SLO across MLPerf Inference v6.0 (all TTFT/TPOT constraints are ‘99th percentile <= X’), NVIDIA GenAI-Perf/NIM, and vLLM bench (default –metric-percentiles p99). P99 remains the industry tail metric; some leaderboards also surface P95/P50. | MLPerf Inference v6.0 (99th-percentile TTFT/TPOT constraints), vLLM bench (p99 default), NVIDIA GenAI-Perf / NIM | link | 2026-03-24 |
| latency | Time-to-first-token (TTFT) under realistic prefill | MLPerf Inference v6.0 defines per-model P99 TTFT SLOs: GPT-OSS 120B interactive TTFT <= 2.0 s (server <= 3.0 s), DeepSeek-R1 interactive TTFT <= 1.5 s (server <= 2.0 s), Llama-3.1-405B interactive TTFT <= 4.5 s (server <= 6.0 s). The 4.5 s P99 405B bar from v5.1 still stands for the largest model. | MLPerf Inference v6.0 (DeepSeek-R1 interactive), MLPerf Inference v5.1 (Llama-3.1-405B interactive) | link | 2026-03-24 |
| scheduling | Request scheduling and cache-aware routing (FCFS vs priority vs prefix/KV-cache-aware) | SGLang’s cache-aware load balancer still holds the cited bar: up to 1.9x throughput and 3.8x higher prefix-cache hit rate in multi-node deployments, via an approximate radix tree mirroring each worker’s cache. | SGLang router / cache-aware load balancer (v0.4+, current v0.5.x) | link | 2024-12 |
| scheduling | Chunked / piggyback prefill and prefill-decode interference (stall-free batching) | Proactive intra-GPU prefill/decode disaggregation (Nexus) now beats chunked-prefill, delivering up to 2.2x higher throughput, 20x lower TTFT, and 2.5x lower TBT vs vLLM (and up to 2x vs SGLang), superseding Sarathi-Serve’s 5.6x chunked-prefill capacity claim. | Nexus (proactive intra-GPU PD disaggregation, arXiv 2507.06608), DuetServe (adaptive GPU multiplexing, arXiv 2511.04791), Sarathi-Serve (chunked prefill, original bar) | link | 2025-08 |
| single-stream | Single-stream (one chat) decode throughput on CPU | llama.cpp Metal/CPU leads single-stream decode on the same box (e.g. Qwen2.5-7B Q8 ~17.3 tok/s Metal, M3 Pro). | llama.cpp, Ollama, MLC-LLM | link | 2026-06 |
| single-stream | Single-stream decode throughput on a consumer GPU | llama.cpp Q8_0 ~120 ± 15 tok/s on an RTX 4070 (-ngl 99). | llama.cpp (CUDA) | link | 2026-06 |
| single-stream | Single-stream prefill throughput, apples-to-apples CPU-vs-CPU | llama.cpp CPU sets the prefill bar on the same box + Q8 weights (1× reference); fak arm64 is ~0.12× until the int8 tile lands. | llama.cpp | link | 2026-06 |
| throughput | Absolute aggregate tokens/s at scale on standardized hardware (MLPerf Inference) | MLPerf Inference v6.0 (results 2026-03-30 / published 2026-04-01) supersedes v5.0 and v5.1, and the headline model is now DeepSeek-R1 (671B MoE reasoning), not Llama 3.1 405B: NVIDIA’s largest-ever submission of 288 Blackwell Ultra GPUs (4x GB300 NVL72 over Quantum-X800 InfiniBand) hit ~2.494 million tokens/second aggregate in the offline scenario, at 9,821 tok/s/GPU offline (8,064 tok/s/GPU server) — up to 2.7x higher than the GB300 debut six months prior via TensorRT-LLM/Dynamo software. | NVIDIA GB300 NVL72 x4 (288 Blackwell Ultra GPUs, TensorRT-LLM/Dynamo), NVIDIA GB300 NVL72 single rack, CoreWeave / Nebius GB300 NVL72 | link | 2026-04-01 |
| throughput | Continuous / in-flight (iteration-level) batching: max aggregate token throughput at high concurrency | Continuous (in-flight) batching is now table stakes across vLLM/SGLang/TensorRT-LLM; the live competitive number is no longer ‘vLLM 24x vs Orca/TGI’ but cross-engine: SGLang’s RadixAttention holds roughly a 29% aggregate-throughput edge over vLLM on H100 (~16,200 vs ~12,500 tok/s, Llama 3.1 8B), widening to up to ~6.4x on prefix-heavy RAG/multi-turn and shrinking to near-zero on unique-prompt batches; continuous batching itself still delivers ~3-4x over naive/static batching. | SGLang (RadixAttention), vLLM v1 (continuous batching + automatic prefix caching), TensorRT-LLM (in-flight batching) | link | 2026 |
| throughput | Latency-vs-throughput Pareto frontier | The latency-throughput Pareto frontier is now framed around goodput (max request rate meeting both TTFT and TBT/TPOT SLOs), and the dominant lever has moved from Sarathi-Serve chunked prefill (2024) to prefill-decode (PD) disaggregation: DistServe reports up to 7.4x more goodput or 12.6x stricter SLOs than prior systems while keeping >90% of requests in budget, and 2025-2026 work (DynaServe, MuxWise, DuetServe, RAPID-Serve) pushes intra-GPU PD multiplexing to recover disaggregation’s isolation without dedicating GPU pools. | DistServe (PD disaggregation), Mooncake / Splitwise (PD disaggregation), Sarathi-Serve (chunked prefill, now a baseline point), DuetServe / MuxWise / DynaServe (intra-GPU PD multiplexing, 2025-2026) | link | 2024-2026 |
| throughput | Peak served throughput per accelerator (model-served-by-engine, MLPerf-grounded) | MLPerf Inference v6.0 (2026-03-30) is the current round and the audited per-accelerator peak: DeepSeek-R1 671B on GB300 NVL72 reaches 9,821 tok/s/GPU offline (8,064 tok/s/GPU accuracy+latency-constrained server); the full 288-GPU (4x GB300 NVL72) submission aggregates ~2.494M tok/s (see absolute-tokens-per-second-mlperf). The audited single-H200 Llama-2-70B peak of ~33,000 tok/s (40% over H100) still stands for that older/smaller workload. The prior v5.1/v5.0 ~1.1M tok/s/rack Azure GB300 figure was an unverified submission and is superseded. | NVIDIA GB300 NVL72 (DeepSeek-R1 671B, MLPerf v6.0, TensorRT-LLM/Dynamo), NVIDIA H200 (Llama-2-70B audited), MangoBoost/AMD MI300X | link | 2026-04-01 |