Awesome Token Efficiency

A single index of the methods that make an LLM workload cost fewer tokens — billed input/output tokens and the resource “tokens” of KV-cache memory and recompute. This is a reference catalog (an “awesome list”), not a roadmap: it lists methods even if fak does not and will not implement them, so an operator or contributor can see the whole field on one page and know where fak sits on each.

How to read this

Every entry carries up to four tags:

• Loss — lossless (the model sees identical tokens / output is provably unchanged) vs lossy (the model sees less or different content; outputs may change). For trained-in architecture choices we write native (lossy vs vanilla MHA, but no fidelity loss relative to that model’s own weights).
• Kind — mechanical (no task understanding needed) vs semantic (requires understanding the content or task). Mechanical + lossless methods are the ones safe to turn on by default; semantic/lossy methods need a guardrail and usually a flag.
• Layer — where it lives: prompt construction, provider feature, gateway/proxy, inference engine, model architecture, agent harness.
• fak — fak’s status on this method (see legend). Omitted when the method lives at a layer fak does not own (e.g. the prompt author’s wording, a provider’s batch tier).

fak status legend: ✅ shipped · 🟡 partial / seam present · 🔭 plan or design doc only · ❌ not in fak (and maybe never) · ➖ out of fak’s layer (client/provider/author side) · ⭐ a place fak does something the standard stacks do not.

The load-bearing axis is lossy vs lossless. The biggest mechanical, lossless wins — prefix caching, paged/radix KV reuse, batch tiers, lazy tool loading, tokenizer choice — change cost without changing what the model sees, so they are the ones that can be default-on. Lossy methods (compression, eviction, summarization, quantization) trade fidelity for size and belong behind a flag with a witness.

---

fak at a glance — what is on by default

fak is an agent kernel: a gateway you put in front of the model that keeps the provider’s prompt-cache prefix byte-identical while shedding old turns, and (in the fused path) runs the model with an addressable, bit-exact KV cache. Its token-saving defaults are audited by fak token-defaults-scorecard (scorecard doc). The budget-size policy for those bounded defaults lives in the long-context defaults doctrine: the advertised context window is a hard cap, not a resident-context target. On by default today:

Saver	Loss	What it does	Lever
Provider prompt-cache passthrough	lossless	ships a byte-identical cache prefix so the provider discount survives	structural
Tool-floor pruning	lossless	drops provably-unreachable tool definitions from the request	structural
vDSO dedup	lossless	collapses identical repeated calls	--vdso
Compact-history	bounded	sheds the un-cacheable middle past a 48k-token budget by splicing original bytes	--compact-history-budget
Oversized-result elision	bounded	shrinks a scrolled-past tool_result to head+tail at 16 KB	--elide-result-bytes
Structured-output passthrough	lossless	forwards response_format / json_schema / logit_bias to the ride engine	structural (#907)
ctxplan O(1) view	bounded	re-materializes history under a budget	--ctx-view-budget (default-on at 8000)

The rest of this page is the wider field. Where a method could be a safe fak default and isn’t yet, it links to a tracking issue (see default-on candidates).

---

Part A — Prompt & API-level (request-side)

Methods that shape the request before/around the model. fak owns the gateway here, so many of these are fak levers; others are the prompt author’s or the provider’s.

A1. Prompt caching (lossless, mechanical)

• Anthropic prompt caching — explicit cache_control: ephemeral breakpoints (≤4) over tools→system→messages; min ~1024 tokens; read ≈ 0.1× input, write 1.25×/2.0× (5-min/1-h TTL). docs. fak: ✅ byte-faithful passthrough is fak’s core thesis.
• OpenAI automatic prefix caching — zero-config, caches longest seen prefix ≥1024 tokens; up to 90% input discount; prompt_cache_key pins routing. docs. fak: ✅ passthrough.
• Gemini context caching — implicit (auto on 2.5+, no storage fee) and explicit (CachedContent + TTL, storage fee). docs. fak: ✅ passthrough.
• Cache-prefix stability (“static-first” layout) — invariant content (instructions, tool schemas, few-shot) first, volatile content (user turn, timestamps) last, to maximize the cacheable prefix. lossless · mechanical · prompt/gateway. fak: ✅⭐ compact-history splices on the original bytes (a memcpy, never a re-marshal) so the prefix stays byte-identical while the middle is shed.
• Cache-aware routing / sticky sessions — route same-prefix requests to the same warm server. lossless · mechanical · gateway. fak: 🟡 sticky trace_id routing (advanced-topics).
• Cache economics / break-even — only cache prefixes read back many times; a write-once prefix costs more than no cache. semantic · gateway policy. fak: ✅⭐ see O(1) context-window economics and compounding benefits of a saved call.

A2. Prompt compression (lossy, semantic)

• LLMLingua — small LM scores token perplexity; coarse-to-fine budget controller drops low-information tokens (~20×). arXiv:2310.05736. fak: ❌ (lossy; tracked as a research item, issue #110).
• LongLLMLingua — question-aware long-context compression + reordering to fight “lost in the middle.” arXiv:2310.06839. fak: ❌.
• LLMLingua-2 — task-agnostic compression as token classification (distilled encoder), 3–6× faster. arXiv:2403.12968. fak: ❌.
• Selective Context — prune low self-information lexical units (the seminal hard-pruning method). arXiv:2304.12102. fak: ❌.
• Gisting / soft-prompt compression — train the LM to compress a prompt into a few cacheable “gist” tokens (≤26×). arXiv:2304.08467. fak: ❌ (needs a gist-trained model).
• Prompt distillation / instruction rewriting — offline-rewrite a verbose system prompt into a tighter equivalent, reused thereafter. lossy · semantic · one-time authoring. fak: ➖ author-side.

A3. Context pruning / selection (mostly lossy, semantic)

• Retrieval-Augmented Generation (RAG) — fetch only top-k relevant chunks instead of the full corpus. lossy vs full corpus · semantic · app. fak: ➖ above fak’s boundary, but promptmmu (#751) curates what enters the context window cache-prefix-safely.
• Cross-encoder / LLM reranking — retrieve broadly, jointly score query×passage, keep top-K so you pass fewer chunks. Contextual Retrieval. fak: ➖.
• Post-rerank evidence pruning — strip redundant sentences from retrieved passages (Provence/XProvence, information-gain pruning). fak: ➖.
• Context deduplication — collapse repeated/near-identical context blocks (same doc pulled twice, repeated tool output). lossless if exact-dup · mechanical→semantic · gateway. fak: ✅ vDSO exact-call dedup + simhash near-dup detection; extending exact-dedup to inbound message content is a default-on candidate.
• History truncation / windowing — drop or summarize old turns past a budget. lossy · mechanical/semantic · gateway. fak: ✅ compact-history (shed, not summarize, to stay cache-safe).
• Rolling / hierarchical summarization — replace long history with a running summary + recent verbatim tail. lossy · semantic · app. fak: 🟡 many open issues (#65, #459, #1075); fak prefers cache-safe shedding over summarization.
• Tool-result pruning / elision — replace a stale/oversized tool output with a short reference once consumed. lossy · semantic · gateway. fak: ✅ oversized-result elision (head+tail at 16 KB), on by default.

A4. Output-side control (caps lossless; concision lossy)

• max_tokens cap — hard output ceiling. lossy if it truncates real content · mechanical · provider param. fak: ➖ passthrough.
• Stop sequences — halt at a delimiter. lossless · mechanical · provider param. fak: ➖.
• Concise / “be terse” instructions — tell the model to skip preamble/filler. lossy · semantic · prompt. fak: ➖ author-side.
• Structured Outputs / JSON Schema (strict) — constrained decoding masks invalid next-tokens against a compiled grammar → schema-valid output, no prose wrapper. OpenAI. fak: ✅ structured-output passthrough (#907) forwards the constraint to vLLM guided_json / SGLang json_schema.
• Grammar-constrained / guided decoding — GBNF, Outlines, XGrammar, vLLM guided decoding force output into a regex/CFG/enum. XGrammar. fak: ✅ (passthrough, as above).
• Reasoning-effort / thinking-budget control — cap hidden reasoning tokens on reasoning models. lossy · semantic · provider param. fak: ➖ passthrough.

A5. Batching / request shaping (lossless, mechanical)

• Batch API (50% off) — async jobs at half price on input and output (OpenAI Batch, Anthropic Message Batches). OpenAI Batch. fak: ➖ provider feature, client-side (fak is a sync gateway). Stacks with prompt caching.
• Flex / lower service tier — same ~50% discount via the live API, latency-tolerant. fak: ➖ passthrough.
• Multi-item prompts — pack several independent items into one call sharing a cached instruction prefix. lossless · mechanical · app/gateway. fak: ➖ app-side.
• Embedding-based request dedup / semantic response cache — serve a cached response for exact/near-identical prompts (GPTCache pattern). lossless (exact) / lossy (near) · gateway. fak: 🟡 simhash is for trajectory similarity, not yet a response cache.
• Model cascade / FrugalGPT — try a cheap model first, escalate on low confidence. lossless if escalation is sound · semantic · gateway. fak: 🟡 per-aspect routing + ensembles (vote/best_of) is the routing spine; auto-confidence escalation is not wired.

A6. Tool / function-definition efficiency

The tool schemas are a per-turn input tax paid whether or not a tool is called.

• Tool-schema pruning (load only relevant tools) — send only the plausibly-needed subset. lossless if pruned tools unused · semantic · gateway. fak: ✅⭐ tool-floor pruning drops provably-unreachable tool defs by default (structural, not a classifier).
• Tool Search / lazy schema loading (MCP) — load a search interface; fetch a tool’s full schema only when the model picks it (Anthropic reports ~85% tool-token cut). code-execution with MCP. lossless · mechanical · host/protocol. fak: ✅ — shipped as the fak_tools_search MCP tool, supporting name/description/full detail levels with query filtering (see internal/gateway/mcp.go).
• Compact tool descriptions / schema minification — trim verbose docstrings; let the tool name carry meaning. lossy · semantic · authoring. fak: ❌.
• One-router-tool pattern — fold many tools behind a single dispatcher with an op param. lossy (loses per-tool typing/permissions) · mechanical · gateway. fak: ➖ (fak prefers per-tool adjudication for the capability floor).
• Code-execution tool exposure (skills) — present MCP servers as importable code files, not upfront JSON (Anthropic/Cloudflare report ≤98.7% cut). fak: ➖ harness-side.

A7. Tokenizer-level (lossless, mechanical)

• Choose a denser tokenizer — token-per-word varies 2–10× across vocabularies. lossless · mechanical · model selection. fak: ➖ model choice.
• Avoid token-wasteful formats — prefer terse JSON/CSV/TSV over verbose XML/pretty-printed JSON; strip needless whitespace. lossless · mechanical · prompt. fak: ➖ author-side.
• Language/script choice — express instructions in a tokenizer-efficient language where the task allows. lossless · mechanical · prompt. fak: ➖.

---

Part B — KV-cache & inference-serving (resource-side)

These reduce the resource tokens — KV-cache memory, attention compute, recompute. fak owns the governance band on top of an engine and, in the fused path, runs its own reference engine with an addressable, bit-exact KV cache — so some of these are shipped in-kernel, many are the engine’s job that fak fronts, and a few are uniquely fak’s.

B1. KV-cache quantization (lossy; int8 ≈ neutral)

• KIVI — tuning-free 2-bit: Keys per-channel, Values per-token. arXiv:2402.02750. fak: 🟡 KV precision tiers q8/f32 planner half shipped (#1047); engine GPU-gated.
• KVQuant — per-channel Key (pre-RoPE) + per-token Value, non-uniform levels; ~10M ctx. arXiv:2401.18079. fak: 🟡 (as above; q8 keeps pre-RoPE K in f32 for exact eviction).
• TurboQuant (Google, ICLR 2026) — random orthogonal rotation per vector + per-coordinate Lloyd-Max scalar quant; training/calibration-free, per-token, with asymmetric key>value bits (K4/V2 ≈ 3-bit avg). github:tonbistudio/turboquant-pytorch. fak: 🟡 candidate quantizer for the lossy #1047 q8 tier — fak already keeps Keys more precise (f32-Kraw + q8_0-K/V), the same asymmetry, but for exactness (the bit-exact evictor re-rotates survivors from the lossless pre-RoPE Kraw), so running it on the key conflicts with the exact path. Its README’s own caveat — 99.5%+ attention cosine “does not guarantee working generation,” 5x@3-bit shows generation failures — is why fak gates the lossy tier behind a served quality witness. See triage #1266.
• INT8 / INT4 KV (engine-native) — LMDeploy/TensorRT-LLM/vLLM online KV quant. fak: ➖ engine-side.
• FP8 / NVFP4 KV — hardware-native low-precision KV (Hopper/Blackwell). fak: ➖ engine-side.

B2. KV-cache eviction / sparsity (lossy)

• StreamingLLM (attention sinks) — keep sink tokens + a sliding window. arXiv:2309.17453. fak: ❌ engine-level; fak’s eviction is addressable & exact, a different axis.
• H2O (Heavy-Hitter Oracle) — retain recent + high-attention tokens. arXiv:2306.14048. fak: ❌.
• Scissorhands — exploit importance persistence to evict. arXiv:2305.17118. fak: ❌.
• SnapKV — vote on important prefix positions from an end-of-prompt window (targets prefill). arXiv:2404.14469. fak: ❌.
• FastGen — per-head hybrid eviction policy chosen at end-of-prefill. arXiv:2310.01801. fak: ❌.
• EpiKV (epiphany score) — score tokens by the change in the model’s internal representation read from the forward pass (no attention matrix), not by attention weight; training-free, fused-kernel-compatible. arXiv:2606.26472. fak: 🟡 importance policy (fak’s exact evictor is policy-agnostic; the shipped attention-weight policy is #856, fed by internal/model/attn_observer.go). The epiphany score is the fused-kernel-friendly alternative on fak’s fused GPU decode path — filed, not built (needs a hidden-state-delta observer + a served long-CoT eviction-quality witness). See triage #1123.
• Sliding-window / local attention — Longformer, Mistral SWA. arXiv:2310.06825. fak: ➖ model-native.
• PyramidKV / PyramidInfer — larger KV budget to lower layers. arXiv:2406.02069. fak: ❌.
• Quest — query-aware sparsity: keep full KV, load only top-K relevant pages per step. arXiv:2406.10774. fak: ❌.
• DuoAttention — split retrieval heads (full KV) from streaming heads (sinks+recent). arXiv:2410.10819. fak: ❌.
• Addressable, bit-exact mid-run causal eviction — evict one span (a poisoned tool result, an expired secret) and leave the KV cache bit-for-bit identical to a run that never saw it (max|Δ| = 0). fak: ✅⭐ UNIQUE — no shipped serving engine offers policy-driven exact mid-run eviction. See addressable KV cache.

B3. Attention architecture for KV reduction (native — lossy vs MHA, no fidelity loss vs own weights)

• Multi-Query Attention (MQA) — all query heads share one K/V head. arXiv:1911.02150. fak: ✅ in-kernel engine reads multi_query.
• Grouped-Query Attention (GQA) — g K/V groups; MHA↔MQA interpolation. arXiv:2305.13245. fak: ✅ in-kernel engine reads num_key_value_heads (Llama/Qwen/Gemma).
• Multi-head Latent Attention (MLA) — low-rank joint KV compression to one latent per token (DeepSeek-V2/V3). arXiv:2405.04434. fak: 🟡 GLM-MoE proven; MLA-specific kernel not separately witnessed.
• Cross-Layer Attention (CLA) — share K/V across adjacent layers. arXiv:2405.12981. fak: ❌.
• YOCO — one global KV cache reused by a cross-decoder. arXiv:2405.05254. fak: ❌.
• MLKV — share KV heads across layers (below MQA). arXiv:2406.09297. fak: ❌.
• TPA (Tensor Product Attention) — factorize Q/K/V into low-rank tensors; cache factors. arXiv:2501.06425. fak: ❌.
• GoldFinch — RWKV/Transformer hybrid, 756–2550× smaller KV. arXiv:2407.12077. fak: ❌.
• Sigma / DiffQKV — asymmetric Q/K/V rescaling. arXiv:2501.13629. fak: ❌.
• Interleaved local/global + GQA — Gemma 2/3. arXiv:2408.00118. fak: ➖ model-native.

B3b. Low-rank KV compression (post-hoc SVD, lossy)

• Palu — offline SVD of K/V projection weights. arXiv:2407.21118. fak: ❌.
• Eigen Attention — attention in a calibrated low-rank space. arXiv:2408.05646. fak: ❌.
• ShadowKV — online SVD of pre-RoPE keys + value offload. arXiv:2410.21465. fak: ❌.
• LoRC / xKV — progressive / cross-layer SVD KV compression. arXiv:2410.03111 · arXiv:2503.18893. fak: ❌.

B4. Prefix / cache sharing (lossless, exact reuse)

• PagedAttention (vLLM) — OS-style paging of KV blocks shared via block tables. arXiv:2309.06180. fak: ✅⭐ in-kernel kvmmu context-MMU with span-level management + policy-aware invalidation (not just memory pressure).
• RadixAttention (SGLang) — KV in an LRU radix tree, longest-prefix auto-reuse. arXiv:2312.07104. fak: ✅⭐ in-kernel radixkv (86.7% hit on agents) + policy-driven eviction by quarantine verdict.
• Automatic Prefix Caching — vLLM / TensorRT-LLM hash-keyed block reuse. fak: ➖ engine feature fak fronts; fak’s own equivalent is radixkv.
• Prompt Cache (modular attention reuse) — precompute attention states of reusable modules, reuse even non-contiguous. arXiv:2311.04934. fak: 🔭 related to the regenerable-KV plan.
• Hydragen — attention decomposition (shared prefix vs unique suffix) + inter-seq batching. arXiv:2402.05099. fak: 🟡 BatchFromPrefix does shared-prefix batched inference.
• FlashInfer cascade inference — hierarchical shared-prefix + per-request suffix attention. arXiv:2501.01005. fak: ➖ engine kernel.
• ChunkAttention — chunked K/V in a prefix tree for runtime prefix sharing. arXiv:2402.15220. fak: ➖ engine kernel.
• Cross-agent / cross-session shared prefill — do shared prefill once; later agents read it for free. fak: ✅⭐ the cache-efficient fleets result — ~4× fewer tokens vs a tuned warm-cache stack on a 50-turn × 5-agent run.

B5. KV offloading / transport / disaggregation (lossless movement)

• FlexGen — LP-scheduled GPU+CPU+disk placement of weights/KV/activations. arXiv:2303.06865. fak: ➖ engine.
• vLLM CPU offload — async GPU→CPU DRAM KV copy to avoid recompute. fak: ➖ engine.
• InstInfer — in-storage attention offload to computational SSDs. arXiv:2409.04992. fak: ➖.
• LMCache — external KV layer (GPU→DRAM→disk→Redis), cross-query non-prefix reuse. arXiv:2510.09665. fak: 🔭 governance only — see KV transport governance.
• Mooncake — KVCache-centric disaggregated architecture (pooled CPU/DRAM/SSD/RDMA). arXiv:2407.00079. fak: 🔭 governance band; see P/D disaggregation + KV-routing SOTA.
• NIXL + NVIDIA Dynamo — async P2P KV transport over RDMA/TCP/NVMe-oF. github. fak: 🔭 governance.
• DistServe — disaggregate prefill (TTFT) and decode (TPOT) onto distinct GPUs. arXiv:2401.09670. fak: 🔭 see PD doc above.
• Splitwise — split prompt vs token-gen phases onto heterogeneous pools. arXiv:2311.18677. fak: 🔭.
• KV-aware routing (Preble, vLLM Router) — route to the replica with the best KV hit rate. arXiv:2407.00023. fak: 🟡 sticky trace_id routing; per-aspect model routing.
• CacheGen — encode KV into compact bitstreams + adaptive network streaming. arXiv:2310.07240. fak: ➖.
• CacheBlend — fuse multiple precomputed KV caches with selective recompute (RAG). arXiv:2405.16444. fak: ➖.
• InfiniGen — full KV in CPU; speculatively prefetch next-layer critical tokens. arXiv:2406.19707. fak: ➖.

B6. Recompute vs storage tradeoffs (lossless)

• Chunked prefill (Sarathi) — split big prefills into chunks batched with decodes. arXiv:2308.16369. fak: ➖ engine scheduler.
• Recompute-on-preempt — recompute rather than store when KV space runs out. fak: ➖ engine.
• Compute-or-Load (hybrid prefix caching) — use GPU recompute and SSD-load bandwidth together. openreview. fak: 🔭 the regenerable-KV plan is fak’s take on the store-vs-recompute knob.

B7. Decoding-side compute savers (speculative family — mostly lossless)

• Speculative decoding — draft proposes K tokens, target verifies in parallel, exact distribution recovered. arXiv:2211.17192. fak: 🟡 ABI seam (internal/abi/speculate.go); no in-kernel engine impl yet.
• Medusa — extra heads + tree-attention. arXiv:2401.10774. fak: ❌.
• EAGLE / EAGLE-2 / EAGLE-3 — feature-level autoregressive drafting. arXiv:2401.15077. fak: ❌ (engine feature fak can front).
• Lookahead decoding — Jacobi-iteration parallel n-gram generation, no draft model. arXiv:2402.02057. fak: ❌.
• Self-speculative / LayerSkip — draft with a subset of the model’s own layers. arXiv:2309.08168. fak: ❌.
• Prompt Lookup Decoding (PLD) — draft tokens copied from the input via n-gram match (great for RAG/code-edit). github. fak: ❌.
• SpecInfer / Sequoia / REST / Ouroboros — tree- and retrieval-based speculative verification. arXiv:2305.09781. fak: ❌.

B8. Context-length extension (KV-relevant)

• YaRN / NTK-aware / Position Interpolation — rescale RoPE frequencies to extend context cheaply; zero runtime KV overhead but amplifies the KV-footprint problem that B1–B5 solve. arXiv:2309.00071. fak: ➖ model-native.

---

Part C — Agent / context-engineering (loop-level)

How agent harnesses keep multi-turn token usage down. “Context engineering” (the term Cognition and Anthropic popularized) is the discipline of curating the optimal set of tokens in the model’s limited attention budget at each step. fak shares prefill across agents and sheds old turns cache-safely; most of the rest is the harness’s job, though fak’s gateway position lets it do several mechanically. A useful field taxonomy is Drew Breunig’s “How to Fix Your Context” (RAG · Tool Loadout · Context Quarantine · Pruning · Summarization · Offloading).

C1. Context compaction / summarization (lossy, semantic)

• Auto-compaction at the window limit — summarize the transcript into a high-fidelity digest and restart the loop from it (one of Anthropic’s three long-horizon strategies). Effective context engineering. lossy · semantic · harness. fak: 🟡 fak prefers cache-safe shedding (compact-history) over summarization; summarization tracked in #65/#459/#1075. Watch the recursive re-nesting failure mode (#880-class).
• Dedicated compaction model — a separate fine-tuned LLM whose only job is to compress the full action trace, preserving key decisions for a single-threaded agent. Cognition — Don’t Build Multi-Agents. fak: ❌.
• Rolling / summary-buffer memory — last N turns verbatim + a continuously-updated summary of everything older. LangChain ConversationSummaryBufferMemory, LangGraph summarization node + RemoveMessage. fak: ➖ harness-side.

C2. Tool-result management

• Head+tail elision of large tool outputs — clip a big output to head+tail with a byte marker. lossy · mechanical · gateway/harness. fak: ✅ oversized-result elision, on by default. (Claude Code/Cursor cap bash/read/terminal output similarly.)
• Restorable compression (reference by handle) — drop the bulky body but keep a stable reference (path, URL, artifact id) so it can be re-fetched; recoverable, so effectively lossless. Manus (filesystem-as-context). semantic · harness. fak: 🟡 elision keeps head+tail; a full handle-store is a candidate.
• Mask, don’t remove (logit masking of tools) — keep all tool definitions in context (so the KV-cache and references stay valid) and instead mask the decoder logits to constrain which tools are selectable per state. Manus. lossless · semantic · engine/harness. fak: ➖ — note the contrast with tool-floor pruning (A6): masking trades tokens-saved for cache-stability; fak’s tool-floor removes provably-unreachable defs, which is lossless only because they were unreachable.
• Observation masking / scrollback dropping — drop superseded observations while keeping the actions/decisions. lossy · mechanical→semantic · harness. fak: 🟡 compact-history sheds the middle.

C3. Sub-agent / context isolation

• Orchestrator–subagent fan-out — a lead agent spawns parallel subagents, each with an isolated context window, that return only condensed findings; the orchestrator never carries the sub-transcripts (map-reduce over agents). Anthropic multi-agent research. lossy (only conclusions return) · semantic · harness. fak: ➖ harness-side (Claude Code, this very session) — but fak’s cross-agent shared prefill makes the fan-out cheap.
• Context quarantine — isolate contexts into separate threads so none grows long or accrues conflicting content. fak: ➖ (the name collides with fak’s security result-quarantine, which is unrelated — that holds untrusted tool results out of context for safety, not for token economy).
• Counterpoint: single-threaded linear agent — Cognition argues against naive fan-out (subagents can’t see each other’s work, so “actions carry implicit decisions” that clash); their fix is one continuous context + compaction. Bounds when isolation is safe (read-heavy, parallelizable) vs unsafe (coordinated writes). Cognition.

C4. Retrieval over full-context / external memory

• Just-in-time / agentic retrieval — don’t frontload data; let the agent fetch it at runtime (grep/glob/targeted queries) via “progressive disclosure”. recoverable · semantic · harness. fak: 🟡 promptmmu (#751) curates inbound context; Context is not memory is the durability model.
• RAG / vector-store retrieval — retrieve only relevant chunks per query. lossy · semantic · app. fak: ➖ above fak’s boundary.
• External / paged memory hierarchy — context = RAM, external store = disk; the agent pages memory blocks in/out (MemGPT/Letta, mem0, LangMem). recoverable · semantic · harness. fak: ➖; fak’s four layers of agent memory is the related model.
• Tool Loadout (RAG over tool definitions) — when tools number in the dozens, retrieve only the relevant subset of tool descriptions per turn (“RAG-MCP”, arXiv:2505.03275). lossy (a needed tool can be unselected) · semantic · gateway. fak: ❌ — related to the lazy tool-loading default-on candidate.

C5. History pruning / dedup

• Sliding-window buffer — keep only the last k turns, no summary. lossy · mechanical · harness. fak: ✅ compact-history is the cache-safe analogue (sheds the un-cacheable middle).
• Permanent deletion (RemoveMessage) vs non-destructive trim — LangGraph distinguishes dropping turns from the checkpoint vs making a shorter copy for one call. fak: ➖ harness-side.
• Tool-call / duplicate-turn dedup — reuse a prior result for an identical repeated call. lossless (identical) · mechanical · gateway. fak: ✅ vDSO; read-once file dedup tracked as #318.
• Compact the “uncacheable middle” — target compaction between the cached prefix and the live tail to preserve cache hits. lossy · semantic · gateway. fak: ✅⭐ exactly compact-history’s design.

C6. Planning / scratchpad externalization (lossless)

• Structured note-taking (NOTES.md / memory tool) — write durable state to a file, re-inject only the relevant slice. Anthropic. recoverable · semantic · harness. fak: ➖.
• Recitation (todo.md rewriting) — rewrite the todo list each step so the global plan stays in the most-recent attention span (counters “lost in the middle” over long loops). Manus. lossless (adds, doesn’t drop) · semantic · harness. fak: ➖.
• Context offloading (“think” tool / scratchpad) — store working notes outside the conversation via a tool. recoverable · semantic · harness. fak: ➖.

C7. Prompt / system-prompt hygiene

• Right-altitude system prompts — concise, high-signal; specific enough to guide, general enough not to over-constrain. Anthropic. lossless · semantic · author. fak: ➖.
• Keep the wrong stuff in (don’t scrub errors) — leave failed actions + error observations in context; seeing its own mistakes reduces repeats. Manus. lossless (intentionally retains) · semantic · harness. fak: ➖ (an anti-pruning caution).
• Stable serialization / minimal boilerplate — deterministic JSON/tool serialization so the cacheable prefix doesn’t churn. lossless · mechanical · gateway/author. fak: ✅ byte-faithful passthrough preserves whatever stable serialization the client sends.

C8. Caching at the agent layer

• KV / prompt-cache reuse via a stable, append-only prefix — Manus treats KV-cache hit rate as the #1 production metric (~10× cost difference on Sonnet); one changed prefix token invalidates it. lossless · mechanical (with prompt discipline) · gateway. fak: ✅⭐ the whole thesis — see Part A1.
• Semantic caching of repeated sub-queries (GPTCache) — embed the query, vector-search prior queries, return a stored response above a cosine threshold. GPTCache. lossy (near-match) · semantic · gateway. fak: 🟡 simhash similarity exists; a response cache is not wired.
• Response memoization / deterministic tool-call cache — exact-match response/tool-result reuse. lossless · mechanical · gateway. fak: ✅ vDSO (exact-call) is the in-kernel form.

---

Default-on candidates (tracked as issues)

The honest rule (from the token-defaults scorecard): default-on a saver only when it is demonstrably safe — lossless or in-code-guarded bounded loss, with a committed witness. The methods below are mechanical and lossless-or-bounded, sit at fak’s gateway layer, and aren’t on yet:

Candidate	Why it qualifies	Issue
Inbound content exact-dedup (same file/tool output sent twice in one request)	lossless, mechanical, extends vDSO from calls to message content	#1101

Note: Tool Search / lazy MCP tool-schema loading shipped as fak_tools_search in 2026-06 and is no longer a candidate — see A6. Tool / function-definition efficiency.

Lossy methods (LLMLingua compression, summarization, rerank, KV eviction/quant) are deliberately not default-on candidates — they change what the model sees and stay behind a flag with a witness.

Maintenance

This list is meant to be refreshed. To update it:

1. Re-run the field survey (the three research clusters: prompt/API, KV/serving, agent/context-engineering) and add new methods with a verified primary reference.
2. Re-derive fak’s status from source, not from this page — the fak token-defaults-scorecard is the on/off authority for the defaults; internal/ packages are the authority for the in-kernel methods.
3. For any newly-safe default-on candidate, file a tracking issue and link it here.

See also: SOTA optimizations fak sits on top of, fak vs vLLM / SGLang / provider KV caching, the frozen-trajectory cache cliff, and the broader Agent optimization methods survey (370 methods, 16 families) — this page is the token-efficiency-focused, fak-status-annotated slice of that field map.