distributed — the dimensions that matter, and where fak stands

← back to the scorecard index · part of the industry-first scorecard. Each dimension is a thing the field competes on; the fak column is honest — mostly no-claim gaps for a focused reuse kernel.

Parallelism & disaggregation (parallelism)

○ Prefill-decode disaggregation: independent scaling of compute-bound prefill and memory-bound decode — fak: no-claim

Why it matters: Prefill is compute-bound and decode is memory-bandwidth-bound; co-locating them forces one hardware/parallelism config to serve both and couples their scaling. Disaggregating onto separate instances (over a KV-cache transport) lets each phase scale and tune independently and removes interference entirely. It is now the default architecture for large-scale, SLO-tight deployments.

○ Prefill-decode disaggregation & per-phase SLO isolation — fak: no-claim

Why it matters: Prefill is compute-bound (TTFT-sensitive) and decode is memory-bandwidth-bound (TPOT-sensitive); colocating them couples their tails. Disaggregating onto separate GPU pools lets each phase scale and meet its own SLO independently, which has become the production standard for hitting tight TTFT AND tight TPOT simultaneously at scale.

○ Large-scale expert parallelism (EP) for giant MoE models — fak: no-claim

Why it matters: Sparse MoE models (DeepSeek-V3/R1 671B, Qwen MoE, Mixtral) activate only a few experts per token, so naive tensor parallelism replicates expert weights and wastes memory bandwidth. Scaling EP across dozens of GPUs (one or few experts per device) is the only way to serve 600B+ MoE economically, and demands all-to-all expert dispatch, expert-load balancing, and overlap of comms with compute. EP scale and balance is the differentiating capability for frontier-MoE serving.

○ Data-parallel (DP) attention / hybrid attention-FFN parallelism — fak: no-claim

Why it matters: When EP fans the FFN out across many GPUs, replicating the KV cache under tensor parallelism for attention duplicates memory and caps batch size. DP attention runs attention data-parallel (each rank owns distinct sequences, no KV duplication) while the MoE FFN stays expert-parallel. This is what lets a deployment hold large batches and long contexts at high EP without exploding KV memory, and it is a distinct, non-obvious axis from raw TP/PP/EP degrees.

○ Multi-node scale-out (combined TP x PP x EP x DP across hosts) — fak: no-claim

Why it matters: Serving a frontier model means composing tensor, pipeline, expert, and data parallelism across many hosts and an InfiniBand/RoCE fabric. The relevant operator question is not ‘does it run on 8 GPUs’ but how many GPUs/nodes a single coherent deployment spans, what interconnect it requires, and how cleanly the parallelism dimensions compose. Datacenter-scale orchestration (rack-scale NVLink domains, cross-node fabric) is the table-stakes axis for frontier serving.