Skip to the content.

memory — the dimensions that matter, and where fak stands

← back to the scorecard index · part of the industry-first scorecard. Each dimension is a thing the field competes on; the fak column is honest — mostly no-claim gaps for a focused reuse kernel.

KV cache & prefix reuse (kv-cache)

○ KV-cache memory efficiency and max concurrent sequences (paging / fragmentation) — fak: no-claim

Why it matters: GPU memory left for KV cache, after weights, sets the hard ceiling on how many sequences can run concurrently, which caps batch size and therefore throughput. Pre-reserving max-context KV cache wastes 60-80% of memory to fragmentation; paged allocation cuts waste to under 4%, letting the same GPU hold far more concurrent sequences.

≈ Prefix caching / automatic KV reuse across requests (shared-context workloads) — fak: parity

Why it matters: Real workloads (long system prompts, RAG, few-shot, multi-turn chat, agent fleets) repeat large prefixes. Automatic prefix caching reuses the already-computed KV for shared prefixes, skipping that prefill entirely. On these workloads it is a large, distinct throughput and TTFT multiplier that is invisible to single-request benchmarks but dominates production economics.

≈ Prefix/KV-cache reuse impact on TTFT (cache-hit latency) — fak: parity

Why it matters: Real workloads (shared system prompts, RAG documents, multi-turn chat) share long prefixes. Automatic prefix caching skips recomputing the shared KV, slashing prefill tokens and therefore TTFT on cache hits. The cache-hit vs cache-miss TTFT gap, and the hit rate under a realistic prompt mix, are decisive for chatbot/RAG/agent latency yet are absent from cold synthetic benchmarks.

○ Paged KV block management and memory utilization (anti-fragmentation) — fak: no-claim

Why it matters: KV cache is the dominant runtime memory consumer in LLM serving. Before paging, engines wasted 60-80% of KV memory to internal fragmentation and over-reservation, capping batch size and throughput. Block-based (paged) allocation determines how close an engine gets to theoretical KV capacity, which directly sets max concurrency and tokens/sec on fixed hardware. This is the foundational dimension every serious operator evaluates first.

≈ Automatic prefix caching / RadixAttention prefix reuse and cache hit rate — fak: parity

Why it matters: Shared prompt prefixes (system prompts, few-shot exemplars, multi-turn history, RAG templates) are extremely common. Reusing their KV instead of recomputing prefill is the single largest throughput lever for prefix-heavy workloads. The hit rate an engine actually attains under a given workload, and whether reuse is automatic (no app changes), is a primary buyer differentiator between serving stacks.

○ Non-prefix / cross-document KV reuse (RAG chunk caching) — fak: no-claim

Why it matters: Prefix caching only helps when the shared text is a literal prefix. RAG concatenates retrieved chunks in varying order, so prefix reuse misses most of the opportunity. Engines that can reuse KV for arbitrary cached chunks (recomputing only the small fraction needed to fix cross-attention) unlock a second, larger tier of reuse for the most common enterprise workload. This is a sharp differentiator that a naive prefix-only scorecard omits.

○ KV-cache offloading to CPU/host/NVMe/remote (multi-tier hierarchy depth) — fak: no-claim

Why it matters: GPU HBM holds only a small KV working set. Offloading cold KV to CPU DRAM, local SSD/NVMe, and remote object/RDMA stores expands effective cache capacity by orders of magnitude, raising hit rates and enabling ‘trade storage for compute’. The depth of the tier hierarchy, supported backends, and load bandwidth from each tier are what separate a production KV layer from a single-GPU cache.

○ KV compression / token eviction under a fixed cache budget — fak: no-claim

Why it matters: For very long contexts, even quantized KV is too large. Eviction/sparsity methods (keep attention-heavy or recent/sink tokens) shrink the resident KV to a fixed token budget, trading a controlled accuracy loss for memory and decode speed. Operators serving long-context or reasoning workloads must evaluate how aggressively a system can compress before quality breaks, and which algorithm it uses.

○ KV-cache transfer for prefill/decode disaggregation (NIXL / NCCL / UCX) — fak: no-claim

Why it matters: Disaggregated serving splits prefill and decode onto separate GPUs, requiring the prefill KV to be shipped to the decode engine. The transport library, supported fabrics (NVLink/InfiniBand/RoCE/NVMe-oF/S3), and whether transfer is non-blocking and overlapped layer-by-layer with compute decide whether disaggregation adds TTFT or hides the cost. This is the connective tissue that makes large-scale disaggregation viable.

≈ KV-cache-aware / prefix-aware request routing across a fleet — fak: parity

Why it matters: A cache is only as good as the routing that lands a request on the GPU that already holds its KV. Across a multi-node fleet, a cache-aware router computes prefix/block overlap and balances it against load to maximize cluster-wide hit rate. Without it, replicas duplicate KV and recompute prefill. This is the dimension that turns per-engine caching into fleet-level economics.

○ Attention architecture impact on KV footprint (MQA/GQA/MLA bytes-per-token) — fak: no-claim

Why it matters: The model’s attention design sets the floor on KV size before any serving trick applies. MQA/GQA share K/V heads; MLA compresses K/V into a low-rank latent. A 3-5x smaller KV footprint per token compounds with every other lever (paging, offload, quant) to multiply servable concurrency and context. Operators selecting models for serving cost must weigh this, and engines must implement the architecture’s KV layout efficiently.

○ KV-cache quantization (FP8 / INT8 / KIVI sub-4-bit) accuracy at long context — fak: no-claim

Why it matters: KV-cache dominates memory at long context and high concurrency; halving it (FP8) or going to 2-4 bits (KIVI/KVQuant) directly buys longer contexts or more concurrent requests at fixed hardware. The buyer question is exactly where it stays lossless vs where it degrades (prefill-heavy, head_dim=256, hybrid sliding-window layers).

≈ Automatic cross-request prefix / KV-cache reuse (shared system prompts, agent scaffolds, few-shot) — fak: parity

Why it matters: Agent and fleet workloads repeat large identical prefixes (system prompts, tool schemas, few-shot exemplars). Reusing their KV across requests is the single largest lever on prefill cost and TTFT, and is now table-stakes; buyers compare hit-rate and the throughput gain it delivers.