The fak Fleet Benchmark Suite — explore it yourself

The fak Fleet Benchmark Suite is five model-agnostic kernel demos — fanbench, fleetbench, fak turntax, radixbench, and ctxdemo — that you reproduce in minutes with go run, with no model weights, no GPU, and no API key. Each drives the real fak kernel and reads its own counters, so the headline numbers are deterministic and seeded down to a fixed (profile, grid, trials, seed). They measure an axis orthogonal to raw throughput: how much redundant work a fleet of agents can safely delete through cross-agent cache reuse, turn-tax elimination, and shared-prefix fan-out. Every figure is graded against the best already-shipped baseline and keeps measured kernel events strictly apart from modeled cost economics — any naive/cold multiple is labeled as such, never as a SOTA win.

What this page is. A single place to run the five benchmarks that show what fak buys a fleet of agents — cross-agent cache reuse, turn-tax elimination, and fan-out — and to read each one honestly. All five are model-agnostic kernel demos: they drive the real fak kernel (k.Syscall, the process-global vDSO cache, the ctx-MMU, NewBatchFromPrefix) but need no model weights, no GPU, and no API key, so the headline numbers reproduce on any laptop in minutes. Every figure here traces to BENCHMARK-AUTHORITY.md (the single source of truth) and the on-box witnessed run in GLM52-PURE-KERNEL-AND-AGENT-TURN-DEMOS-RESULTS-2026-06-21.md §3.

If you want the why behind the win first, read KV cache for agentic context and SOTA optimizations fak sits on top of. If you want to watch them in a browser instead of running them, jump to Watch them live.

TL;DR — the five headline numbers

Each row is one command you can paste from the repo root. The honest baseline column is the one that matters: the eye-catching multiples are mostly against a naive / cold reference, while the real fak-only win is the cross-agent reuse on top of an already-warm per-agent cache (the baseline a tuned vLLM / SGLang / provider stack gives you). Both are shown — never just the flattering one.

Demo Headline result The honest baseline (read this)
fanbench fan-out, N=1024 1,005 sibling-only tool-result saves · 61.7% of the multi-agent token tax clawed back · 72.8× parallel critical-path speedup The cross-agent dedup is measured on top of a warm per-agent prompt cache. The 61.7% is modeled cache economics (saturates there); the 72.8× is latency that saturates past N≈256 as the fold dominates.
fanrun LIVE fan-out, N=1024 1,024 real agent sessions complete one goal in 364 ms (no GPU, no model) · 3,069 real cross-agent vDSO dedup hits · vdso_fills flat at 3 for every N The MEASURED capstone: real agent.RunArm loops, not a synthetic stream. Serial (the world-version is process-global), so the number is not a parallel rate and it does not claim fanbench’s 72.8×. The win is prefill elision + real dedup — a per-agent cache would fill 3·N; cross-agent fills 3.
fleetbench 50×50 corner deletes 2,344 / 2,500 tool calls · +370 cross-agent turns over isolated worlds The +370 cross-uplift is the real fleet-only win (a measured tier-2 vDSO path-swap); it is read-fleet only — even a ~1% write rate flips it negative under the coarse eraser.
fak turntax airline / happy 9 turns saved on airline (forced 5 + elision 4) · 0 on the clean control · safety floor: injections 1→0, destructive 1→0 The 9 is a cache-favorable slice (~64% addressable), not the ~0.7% real-world rate; turn-savings are self-host-only. The safety floor is the moat — engine-agnostic, on a separate axis.
radixbench 77–88% cache hit across workloads · agents: FCFS 62.1% → 86.7% cache-aware · policy-evict freed 8 tokens, kept the sibling Hit rate is hardware-/model-independent, so fak-on-CPU vs SGLang-on-GPU is a fair axis; 86.7% is inside SGLang’s published 50–99% band. The token-speedup is vs a cold baseline.
ctxdemo fleet-5×50 1.26M cold tokens → 35,495 with fak (35.5× vs cold) The 35.5× is vs the cold no-cache reference. Against the honest serving baseline (warm per-agent KV) the win is 1.1× — both printed side by side.

One framing law for the whole suite. Compare against the best already-shipped baseline, state the absolute number, and mark every naive/cold multiple so it can never read as a SOTA win. fak does not beat vLLM / SGLang / llama.cpp on raw tokens-per-second and never claims to — see one binary is the whole surface. What these benchmarks measure is the orthogonal axis: how much redundant work a fleet’s kernel can delete, exactly and safely.

Before you run anything

Prerequisites — that’s the whole list:

These are the kernel demos. They are distinct from the model-ladder benchmarks (sessionbench, modelbench, the live radixbench -hf … arm), which do need a real checkpoint on disk to produce wall-clock tok/s — those are indexed in BENCHMARK-AUTHORITY.md. This page stays on the model-agnostic floor so anyone can reproduce it.

Two honesty axes, kept strictly apart (the same discipline runs through every demo):

  1. Measured on the real kernel — cross-agent dedup, turn-tax levers, hit rate, and the safety floor are kernel events the model did not author (VDSOHits, Transforms, Quarantines, Denies), proven by a real ON/OFF path-swap or a shared-vs-isolated ablation, with an exactly-zero anti-inflation control.
  2. Modeled by a transparent cost model — only the price of a turn (tokens, dollars, latency) and the prefix-cache economics are modeled, with every knob exposed as a flag. The two halves are never blended.

1. fanbench — one master goal → N sub-agents (the fan-out topology)

What it measures. The orchestrator-worker pattern (one lead decomposes a goal, spawns N sub-agents, folds their results), swept from N=1 to N=1024 — the regime no public benchmark maps. It prices the cross-agent tool-result dedup the fan-out structure deletes, plus the exact (N−1)·prefix prefill the kernel never redoes because NewBatchFromPrefix prefills the shared master-goal prefix once and clones it bit-identically into all N sub-agents.

Why it matters in a fleet. When N agents decompose one goal, they read the same shared sources. A naive framework re-ships the full system+goal prompt per sub-agent; fak does the shared prefill once for the whole wave.

Run it:

go run ./cmd/fanbench -agent-max 1024 -grid log

The N-ladder corner (research profile, the headline surface):

N calls shared isolated (warm) cross tax clawed back parallel speedup
256 1,028 785 536 255 61.7% 57.7×
512 2,052 1,569 1,069 483 61.7% 66.9×
1024 4,100 3,155 2,152 1,005 61.7% 72.8×

At N=1024 the interleaved fan-out deletes 3,155 of 4,100 calls (77%), of which +1,005 is the cross-agent bonus the same sub-agents run solo could not get.

Honest fences.

Full results: docs/benchmarks/FANOUT-BENCH-RESULTS.md.

2. fleetbench — the 2-D turn-tax surface (turns × agents)

What it measures. fak turntax prices one agent; this sweeps the full 1..50 × 1..50 grid — 2,500 cells — of A independent agents that happen to overlap. The kernel’s tier-2 vDSO cache is keyed (tool, args-sha256, world-version) and is process-global, so when A agents read the same reference data the first pays a cold round-trip and every other agent’s identical read is a tier-2 hit the kernel counts itself (Counters.VDSOHits). Each cell is ablated shared-world fleet vs per-agent-isolated worlds.

Why it matters in a fleet. A research / monitoring / support-lookup fleet mostly reads shared reference data. The cross-agent uplift is the turns sharing buys that A independent agents cannot get — and it is linear in agent count but saturating in turns.

Run it (the 50×50 read-heavy corner, as witnessed):

go run ./cmd/fleetbench -agents 50 -turns 50 -trials 24 -profile read-heavy -granularity resource

T=50 A=50  calls=2500  shared=2344  isolated=1974(warm)  cross=370
tokens_saved_shared=3,094,080   $12.66 saved (shared)

The read-fleet corner deletes 2,344 / 2,500 calls (94%) with +370 cross-agent turns over isolated (warm per-agent KV) worlds. Run it without -agents/-turns to sweep the full 2,500-cell heatmap.

Honest fences.

Full results: docs/benchmarks/FLEET-SWEEP-RESULTS.md.

3. fak turntax — the turn-tax A/B and the safety floor

What it measures. Two distinct things, kept structurally apart:

  1. The safety floor (the moat). On a 14-call airline-support slice, the kernel quarantines a poisoned tool result out of context (Quarantines) and refuses a destructive delete_account (Denies) — a deterministic completion/integrity delta the model did not author, reproducible on any backend including a frontier API you do not own.
  2. The efficiency upside (self-host only). When a SOTA tool-calling loop hits an error code, malformed args, or a duplicate read, the documented recovery is to re-prompt — an extra turn. fak’s 1-shot path resolves the same condition inside the syscall.

Why it matters in a fleet. The safety floor is the non-optional reason to run the kernel at all, and it scales to every agent regardless of which engine answers the call.

Run it (the demonstration slice, then the anti-inflation control):

go run ./cmd/fak turntax --suite turntax-airline
go run ./cmd/fak turntax --suite turntax-happy
Suite turns saved breakdown vDSO ON / OFF safety floor (separate axis)
turntax-airline 9 forced 5 (grammar + dedup) + elision 4 (pure + static) 9 / 2 → vDSO = 7 turns injections admitted 1 → fak 0; destructive executed 1 → fak 0
turntax-happy 0 — the clean-path control: it inflates nothing 0 / 0 base 0 / fak 0

The vDSO contribution (7) is proven by a real ON/OFF path swap (SetVDSO(false) drops the win to grammar-only 2), and it equals the live Counters.VDSOHits — not arithmetic.

Honest fences.

Full results: docs/benchmarks/TURN-TAX-RESULTS.md.

4. radixbench — RadixAttention prefix reuse + cache-aware scheduling

What it measures. fak’s KV-cache prefix reuse against SGLang’s RadixAttention (arXiv:2312.07104 / NeurIPS 2024) on the metric SGLang’s own paper headlines: cache hit rate — the fraction of prompt tokens served from cache instead of recomputed. That metric is hardware- and model-independent (a function of workload × matching algorithm only), so fak-on-CPU vs SGLang-on-GPU is a fair head-to-head on this axis. fak runs the same algorithm (radix tree + longest-prefix match + LRU-leaf eviction, internal/radixkv).

Why it matters in a fleet. Cache-aware scheduling recovers hit rate a naive FCFS order thrashes away — exactly the fleet-scheduling lever that turns shared prefixes into saved work.

Run it (synthetic workloads, no model needed):

go run ./cmd/radixbench -scale 1
Workload reqs cache hit cross-subtree reuse bounded sched (FCFS → cache-aware)
few-shot 16 88.2% 1.00× 88.2% → 88.2% (100% of optimal)
multi-turn-chat 8 79.5% 2.50× 79.5% → 79.5%
tree-of-thought 27 77.2% 1.40× 77.2% → 77.2%
agents (5×6) 30 86.7% 1.48× 62.1% → 86.7% (cache-aware lift)

The agents hit rate of 86.7% is inside SGLang’s published 50–99% band; the cache-aware scheduler lifts FCFS’s 62.1% → 86.7% (100% of the DFS-optimal bound the paper proves).

Honest fences.

Full results: docs/benchmarks/RADIXATTENTION-RESULTS.md · authority: RadixAttention model ladder.

5. ctxdemo — context-changing fleet token accounting

What it measures. The exact, timing-free prefill-token work each strategy performs in the multi-agent, multi-turn, long-context regime — the one where the context changes every turn as tool calls land heterogeneous, variable-sized results. Decode is excluded (it’s generated, not re-read), so this is a load-independent, hardware-independent floor.

Why it matters in a fleet. It puts the three strategies side by side in one number per scenario: cold re-prefill (naive), warm per-agent KV (the honest serving baseline), and fak (cross-agent prefix sharing on top of the warm cache).

Run it (instant, no model, CI-usable):

go run ./cmd/ctxdemo -print

scenario       C   T    P   no-cache    warmKV     fak    fak-win  (ref×)  maxCtx
fleet-5x50     5  50 1024  1,259,857    39,591   35,495    1.1×    35.5×    9569
deep-research  4   5 1536     40,188     9,358    4,750    2.0×     8.5×    2642

The 5-agent × 50-turn fleet re-reads 1.26M tokens cold; fak does 35,495 — 35.5× vs cold, and 1.1× on top of an already-warm per-agent KV cache.

Honest fences.

Details: the command’s own header (cmd/ctxdemo/main.go) and docs/benchmarking/README.md.

Watch them live

If you’d rather see these drive the kernel in a browser than run them locally, the live demos page hosts three of them on a single GCP VM (NVIDIA L4): the turn-tax race (turntaxdemo), the multi-agent context-reuse proof (ctxdemo), and a live model reuse race (demorace) — each driving the real kernel, not a recording. Run any of them locally instead:

go run ./cmd/turntaxdemo   # http://127.0.0.1:8150 — turn-tax race, no model
go run ./cmd/ctxdemo       # http://127.0.0.1:8153 — context reuse (live model if one is on disk)
go run ./cmd/demorace      # the reuse race + the reuse curve

The honesty discipline (one place)

Every number on this page obeys the same rules, enforced in CI and in the per-demo tests:

Where to go deeper

Last updated: 2026-06-21