Agent Optimization Methods — A Field Inventory, Survey & Index

Date: 2026-06-23

What this is. A field-wide inventory of publicly known techniques for making LLM-based agents better, faster, cheaper, or more reliable — from prompt-level reasoning tricks down to the serving kernels they ride on. It is built as three things at once: an index (a 16-family taxonomy you can jump from), a survey (each method gets a definition, a mechanism, what it optimizes, a representative reference, and a maturity tag), and an inventory (per-family tables plus one alphabetical master index you can grep). It deliberately spans layers most single surveys keep apart: a context-pruning heuristic and a PagedAttention kernel are both “agent optimizations” — they just optimize different terms of the same cost.

What “agent optimization” means here. An agent turn is not prompt in → tokens out; it is memory view → model proposal → tool/effect check → world read/write → result admission → memory update → cache/invalidation → audit. Every stage has its own optimization literature. Methods are grouped by what they act on and tagged by what they improve: accuracy/quality, latency, cost/tokens, memory, throughput, reliability, autonomy, or context-capacity.

Honesty line — read before citing. References are representative canonical pointers (“FirstAuthor et al., YEAR”), not a verified bibliography — treat them as search anchors, not as a claim that a specific identifier is correct. Method names are the durable, checkable part; confirm exact identifiers at the source before relying on them. Inclusion is not endorsement, and maturity tags are judgment calls as of mid-2026.

How this was built. The inventory was assembled by a structured fan-out — one researcher per family enumerating named methods, then a completeness-critic pass over the union to surface what was missing — then deduplicated (453 raw entries → 370 distinct methods) and re-homed into the taxonomy below. It is a knowledge survey: the methods and what they do are drawn from the public literature and practice through early 2026, not from new measurement. No number in this document is a fak benchmark; the families fak has actually measured link out to their own witnessed result docs.

How this maps to fak

This survey is the umbrella; the repo already has depth on several families. The fak thesis is not “we invented these” — a 29-claim prior-art audit scored 0/29 novel. The contribution is the assembly into one in-process gate where the tool call is the checkpoint, and where cache reuse becomes a kernel verdict (identity · state · freshness · authority · integrity · causality · economics) rather than a boolean. The field-to-repo map:

Family (this survey) fak’s deeper treatment
Inference / serving optimizations SOTA serving optimizations — the 10 already-shipped serving optimizations that define the honest baseline (“tuned SOTA”), each with fak’s status.
KV cache & prefix reuse Agentic caching SOTA — nine named cache layers mapped to SOTA + fak; Addressable KV cache — bit-exact mid-run span eviction (max\|Δ\|=0); KV cache as agentic context grows.
Context-window management O(1) context window economics; ctxwin baseline; the O(1) turn context planner (internal/ctxplan).
Multi-agent / orchestration Scaling laws of agents — work ≈ agents × turns × working-set × reread-rate × legality; ultra-long-context levels & levers.
Tool use & function calling Grammar-constrained tool-call decoding; the internal/vdso tool-result fast path.
Reliability / eval-as-optimization Policy in the kernel (default-deny floor); trajectory-replay / offline policy evaluation (internal/turnbench).
Training (quantization) AWQ quantization.

The honest read: fak owns the governance band (which effects are allowed, which results may enter context, when reuse is still legal) plus a few reuse/coherence levers on top of an engine it fronts. It does not try to beat vLLM/SGLang/llama.cpp at raw throughput. Most rows in this survey are levers fak sits on top of, not levers it claims.

The index — 16 families

370 distinct methods, grouped into 16 families across 5 parts. Counts are post-deduplication (a method that spans families is filed under its primary home and cross-noted in its row).

Part I — Reasoning & inference-time problem solving

Part II — Context, memory & knowledge

Part III — Action, planning & orchestration

Part IV — Cost, serving & training

Part V — Reliability & evaluation

Cross-cutting views

By what they optimize (a method may count under several):

Objective # methods
accuracy/quality 266
reliability 180
cost/tokens 150
latency 103
context-capacity 72
throughput 69
autonomy 62
memory 59

By maturity:

Maturity # methods
production-standard 137
emerging 117
research 116

By where they apply:

Applies to # methods
single-agent 128
any 92
serving 76
training 41
multi-agent 33

Part I — Reasoning & inference-time problem solving

1. Prompting & reasoning strategies

The oldest and cheapest lever: change what you ask, not the model. These methods restructure the prompt to elicit better reasoning — surfacing intermediate steps, decomposing the problem, abstracting before answering, or offloading computation to code. They cost only tokens and work on any model, which is why they stay the first thing to try.

20 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
Algorithm of Thoughts
AoT		 Guides the model through algorithmic, in-context exploration (mimicking search like DFS/BFS) within a single generation rather than many external queries. Embedding algorithmic exploration examples in the prompt lets the model internalize search-and-backtrack behavior with far fewer model calls than tree-search frameworks. accuracy/quality, cost/tokens research Sel et al., 2023, ‘Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models’
Analogical prompting
analogical reasoning prompting, self-generated exemplars		 Prompting the model to self-generate relevant exemplars or related solved problems before tackling the target problem. Generating tailored, problem-specific analogues replaces fixed few-shot examples with on-the-fly relevant demonstrations, improving accuracy without manual labeling. accuracy/quality, cost/tokens emerging Yasunaga et al., 2023, ‘Large Language Models as Analogical Reasoners’
Automatic Chain-of-Thought
Auto-CoT		 Automatically constructing CoT demonstrations by clustering questions and generating reasoning chains for representative samples, removing manual exemplar authoring. Diversity-based sampling plus zero-shot-CoT-generated rationales builds a varied demonstration set, avoiding the error-propagation of similar exemplars. accuracy/quality, cost/tokens research Zhang et al., 2022, ‘Automatic Chain of Thought Prompting in Large Language Models’
Chain-of-Thought prompting
CoT		 Prompting the model to produce intermediate reasoning steps before the final answer, typically via few-shot exemplars that show step-by-step worked solutions. Eliciting an explicit token-level reasoning trace gives the model more serial compute and a scaffold to decompose multi-step problems, raising answer accuracy on reasoning tasks.
↔ also: Test-time compute scaling & search over reasoning		 accuracy/quality, reliability production-standard Wei et al., 2022, ‘Chain-of-Thought Prompting Elicits Reasoning in Large Language Models’
Chain-of-Thought without prompting (CoT-decoding) Eliciting reasoning paths by altering the decoding (inspecting top-k first-token branches) rather than by any prompt, since CoT paths already exist in the distribution. Branch over alternative first tokens and select the path whose answer tokens carry the highest confidence margin; surfaces latent reasoning without a ‘think step by step’ instruction. accuracy/quality research Wang & Zhou, 2024 (Chain-of-Thought Reasoning without Prompting)
Complexity-based prompting
complex CoT		 Selects few-shot CoT exemplars with more reasoning steps (higher complexity) and votes among high-complexity sampled chains at decoding time. Conditioning on and voting over longer reasoning chains biases the model toward the more thorough reasoning that correlates with correct answers on hard tasks. accuracy/quality research Fu et al., 2022, ‘Complexity-Based Prompting for Multi-Step Reasoning’
Emotion Prompting (EmotionPrompt) Appending emotional stimulus phrases (e.g. ‘This is very important to my career’) to a prompt to elicit higher-quality model responses. Emotional / psychological-stimulus sentences appended to the instruction shift the model toward more careful, complete generations, measurably improving zero-shot performance. accuracy/quality research Li et al., 2023 (Large Language Models Understand and Can Be Enhanced by Emotional Stimuli)
Faithful Chain-of-Thought
Faithful CoT		 Translates a query into an interleaved natural-language and symbolic (code/logic) reasoning chain whose final answer is deterministically computed from the symbolic part. Binding the answer to a deterministic solver over the symbolic chain guarantees the stated reasoning actually produced the answer, improving faithfulness and accuracy. accuracy/quality, reliability research Lyu et al., 2023, ‘Faithful Chain-of-Thought Reasoning’
Few-shot prompting
in-context learning, ICL		 Providing several input-output exemplars in the prompt so the model infers the task format and pattern from demonstrations. Demonstrations condition the model’s next-token distribution toward the demonstrated mapping, improving format adherence and task accuracy without weight updates. accuracy/quality, reliability production-standard Brown et al., 2020, ‘Language Models are Few-Shot Learners’ (GPT-3)
Maieutic prompting Generates a tree of abductive explanations (recursively justified propositions and their negations) and resolves contradictions to infer a logically consistent answer. Building a maieutic tree of self-justifying statements and using a satisfiability solver over their logical relations yields answers robust to inconsistent model beliefs. accuracy/quality, reliability research Jung et al., 2022, ‘Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations’
Program-Aided Language models
PAL		 An approach where the model reads a problem and emits intermediate reasoning as code that an interpreter runs to produce the answer. The LLM handles natural-language understanding and produces program steps while the interpreter guarantees correct execution of the computation. accuracy/quality, reliability emerging Gao et al., 2022, ‘PAL: Program-aided Language Models’
Program-of-Thoughts
PoT		 Expressing the reasoning steps as an executable program (e.g., Python) and delegating the computation to an interpreter rather than to the language model. Offloading arithmetic and logic to a code interpreter separates reasoning from computation, eliminating the model’s calculation errors. accuracy/quality, reliability emerging Chen et al., 2022, ‘Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks’
Rephrase and Respond
RaR		 Prompts the model to first rephrase and expand the user’s question, then answer the clarified version, optionally in one pass. Self-rephrasing resolves ambiguity and aligns the question with the model’s understanding before answering, reducing misinterpretation errors. accuracy/quality, reliability research Deng et al., 2023, ‘Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves’
Self-Ask A prompting pattern where the model explicitly poses and answers follow-up sub-questions before composing the final answer, often invoking a search tool per sub-question. Decomposing into explicit follow-up questions makes the compositional structure visible and lets each sub-question be answered (or retrieved) independently, narrowing the compositionality gap. accuracy/quality, reliability emerging Press et al., 2022, ‘Measuring and Narrowing the Compositionality Gap in Language Models’
Step-Back prompting
abstraction prompting		 Prompting the model to first derive a high-level concept, principle, or abstraction from the question before reasoning toward the specific answer. Abstracting to governing principles grounds subsequent reasoning, reducing errors from getting lost in low-level details. accuracy/quality, reliability emerging Zheng et al., 2023, ‘Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models’
System 2 Attention (S2A) A two-pass prompting method where the model first regenerates the context to remove irrelevant/biasing content, then answers over the cleaned context. An explicit context-regeneration step filters distractors and opinion before the answering pass, reducing sycophancy and irrelevant-context errors. accuracy/quality, reliability research Weston & Sukhbaatar, 2023 (System 2 Attention)
Tab-CoT / Structured (tabular) reasoning
Tabular Chain-of-Thought, Tab-CoT		 Elicits reasoning in a structured table format with explicit columns (step, subquestion, operation, result) rather than free-form prose. Imposing a tabular schema on the reasoning trace makes intermediate steps explicit and self-aligned across rows and columns, improving structured task accuracy. accuracy/quality, reliability research Jin & Lu, 2023, ‘Tab-CoT: Zero-shot Tabular Chain of Thought’
Take a Deep Breath / optimized meta-prompts (OPRO-discovered instructions) Using LLM-discovered instruction phrases (e.g. ‘Take a deep breath and work on this problem step by step’) that empirically outperform hand-written CoT triggers. An optimizer LLM searches the instruction space; the best-found natural-language instruction is reused as a zero-shot trigger, lifting reasoning accuracy. accuracy/quality research Yang et al., 2023 (Large Language Models as Optimizers / OPRO)
Thread of Thought (ThoT) A prompting strategy for chaotic / long contexts that walks the context piece-by-piece, summarizing and carrying a running analysis before answering. An instruction to ‘analyze this part by part, summarizing as we go’ segments long, disordered context into manageable steps, improving retrieval-style QA. accuracy/quality, context-capacity research Zhou et al., 2023 (Thread of Thought Unraveling Chaotic Contexts)
Zero-shot Chain-of-Thought
Zero-shot CoT, Let’s think step by step		 Eliciting step-by-step reasoning without exemplars by appending a trigger phrase such as ‘Let’s think step by step’ to the prompt. A single instruction phrase induces the model to generate a reasoning trace before answering, capturing most CoT gains with no hand-crafted examples. accuracy/quality, cost/tokens production-standard Kojima et al., 2022, ‘Large Language Models are Zero-Shot Reasoners’

2. Test-time compute scaling & search over reasoning

Spend more inference compute for a better answer: sample many times, search over reasoning states, or let a verifier pick the winner. A fixed model gets smarter when it thinks longer or wider — the regime behind o1/R1-style reasoning models. The trade is latency and tokens for accuracy.

23 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
Adaptive / early-stopping sampling
adaptive consistency, early-stopping self-consistency, dynamic sample allocation		 Dynamically decides how many samples to draw per question, stopping early once the answer distribution is confidently converged to save compute. Monitors agreement among accumulated samples with a stopping criterion (e.g. Bayesian/beta confidence) and halts sampling when further samples are unlikely to change the vote. cost/tokens, latency, accuracy/quality emerging Aggarwal et al., 2023 (Let’s Sample Step by Step: Adaptive-Consistency)
Best-of-N sampling
BoN, rejection sampling at inference, repeated sampling		 Generate N independent candidate outputs and select the single best one according to a scorer (reward model, verifier, or self-evaluation). Trades inference compute for quality by sampling N completions and returning argmax under an external selector rather than relying on one decode.
↔ also: Reliability, structured generation & evaluation-as-optimization		 accuracy/quality, reliability production-standard Stiennon et al., 2020; Nakano et al., 2021 (WebGPT); Cobbe et al., 2021 (verifier-reranked BoN on GSM8K)
Budget forcing / long-thinking (s1-style)
s1, test-time budget forcing, wait-token forcing		 Explicitly controlling the length of the model’s reasoning by forcing it to continue thinking (e.g. appending ‘Wait’) or to stop, to spend a target compute budget. Manipulates the decode trajectory by suppressing the end-of-thinking token and injecting continuation cues so the model extends or truncates its chain to a budget. accuracy/quality, cost/tokens emerging Muennighoff et al., 2025 (s1: Simple Test-Time Scaling)
Compute-optimal test-time scaling
test-time compute-optimal allocation, FLOPs-matched test-time scaling		 Choosing per-prompt how to spend a fixed inference budget (which strategy, how many samples, how deep a search) to maximize accuracy, framed as compute-optimal allocation. Adapts strategy and sample/search depth to problem difficulty (often via a verifier/difficulty estimate), shifting between sequential refinement and parallel search. accuracy/quality, cost/tokens emerging Snell et al., 2024 (Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Parameters)
Forest-of-Thoughts / ensemble-of-trees
Forest of Thought, FoT, boosting-style tree ensembling		 Runs multiple reasoning trees in parallel and aggregates across them (with consensus/voting and dynamic self-correction) for sparse-activation collective reasoning. Ensembles several independent ToT-style trees and combines their leaf answers with consensus-guided decision and selective re-exploration of weak trees. accuracy/quality, reliability research Bi et al., 2024 (Forest-of-Thought: Scaling Test-Time Compute for Enhanced LLM Reasoning)
Graph of Thoughts
GoT		 Generalizes tree-structured reasoning to an arbitrary graph where thoughts can be aggregated, refined, and looped, with self-scoring at vertices. Models reasoning as a graph whose transformations (aggregate/refine/generate) and volume/latency-aware scheduling let the model merge and re-improve thoughts beyond a strict tree.
↔ also: Prompting & reasoning strategies; Self-improvement & feedback loops at inference; Planning algorithms & world models for agents		 accuracy/quality research Besta et al., 2024, ‘Graph of Thoughts: Solving Elaborate Problems with Large Language Models’
Least-to-Most / Decomposed Prompting
DecomP, decomposed prompting, Least-to-Most prompting		 Decomposes a complex task into a sequence of sub-tasks, each handled by a dedicated prompt or sub-handler that can itself be further decomposed or call tools. A modular, recursive decomposer routes sub-tasks to specialized prompt ‘modules’, enabling reuse, hierarchical decomposition, and tool delegation.
↔ also: Prompting & reasoning strategies		 accuracy/quality, reliability, autonomy research Khot et al., 2022, ‘Decomposed Prompting: A Modular Approach for Solving Complex Tasks’
Long chain-of-thought RL reasoning (o1/R1-style)
inference-time scaling via RL, long-CoT, test-time thinking models		 Models trained (typically via RL) to produce very long internal chains of thought at inference, scaling accuracy with the number of thinking tokens spent. RL on reasoning traces teaches the model to allocate many sequential ‘thinking’ tokens to self-correct and explore before answering, monotonically improving with test-time budget. accuracy/quality, reliability, autonomy production-standard OpenAI o1 system card, 2024; DeepSeek-AI, 2025 (DeepSeek-R1: Incentivizing Reasoning Capability via RL)
Lookahead / rollout-guided decoding
lookahead search, rollout scoring		 At each step, simulate one or more short continuations (lookahead rollouts) and use their estimated value to choose the current step. Performs limited forward simulation per candidate step and backs up a value estimate to bias the immediate choice toward promising branches. accuracy/quality research Snell et al., 2024 (lookahead variant); Zhang et al., 2023 (Planning with lookahead)
Outcome Reward Model reranking
ORM, outcome-supervised verifier		 A model trained to predict whether a full solution is ultimately correct, used to rerank or filter candidate completions. Scores each complete candidate with a learned correctness predictor and selects/weights by that final-answer reward. accuracy/quality, reliability production-standard Cobbe et al., 2021 (Training Verifiers to Solve Math Word Problems); Uesato et al., 2022
Quiet-STaR
token-level rationale generation		 Teaches a model to generate implicit rationales at every token to predict future text, generalizing STaR beyond QA to free-form reasoning. Inserts learned ‘thought’ tokens between text tokens and rewards thoughts that improve next-token prediction, trained with a REINFORCE-style objective. accuracy/quality research Zelikman et al., 2024 (Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking)
Repeated sampling / coverage scaling
large language monkeys, sampling-based scaling, pass@k scaling		 Drawing a very large number of samples per problem so that at least one correct solution appears, exploiting that coverage (pass@k) rises log-linearly with sample count. Amplifies the chance of hitting a correct answer by sampling hundreds to thousands of attempts, paired with a verifier or oracle to pick winners. accuracy/quality, cost/tokens emerging Brown et al., 2024 (Large Language Monkeys: Scaling Inference Compute with Repeated Sampling)
Self-consistency confidence (logit/entropy-weighted) aggregation
confidence-informed self-consistency, CISC, entropy-weighted voting		 Weights sampled answers by the model’s internal confidence (token probabilities / answer entropy) so that fewer, higher-confidence samples suffice. Derives a confidence score per sample from logits/answer entropy and performs a confidence-weighted vote, improving sample efficiency over uniform voting. accuracy/quality, cost/tokens research Taubenfeld et al., 2025 (Confidence Improves Self-Consistency in LLMs / CISC)
Self-Consistency early termination by entropy / agreement (ESC, Adaptive-Consistency adaptive sampling) Stopping self-consistency sampling early once a confidence/agreement criterion is met instead of drawing a fixed N samples. After each batch, a Dirichlet/agreement stopping rule estimates whether more samples could change the majority vote; sampling halts when the answer is locked in. cost/tokens, latency emerging Aggarwal et al., 2023 (Adaptive Consistency); Li et al., 2024 (Early-Stopping Self-Consistency)
Self-consistency over tool-augmented / program reasoning
PAL voting, PoT self-consistency, code-execution majority voting		 Applies repeated sampling and majority voting to program-aided / code-generating reasoning, where each sample is an executable program whose run yields the answer. Samples multiple programs, executes them to obtain answers, and votes/verifies over execution results to offload arithmetic and reduce reasoning errors. accuracy/quality, reliability emerging Gao et al., 2022 (PAL: Program-Aided Language Models); Chen et al., 2022 (Program of Thoughts)
Self-Verification / self-evaluation selection
LLM self-grading, self-evaluation guided decoding		 Using the model itself (rather than a separate trained verifier) to evaluate or rank its candidate solutions for selection or pruning. Prompts the model to check/score its own candidate answers (forward-backward verification or stepwise self-eval) and selects the best-scored one. accuracy/quality, reliability emerging Weng et al., 2022 (Large Language Models are Better Reasoners with Self-Verification); Xie et al., 2023 (Self-Evaluation Guided Beam Search)
Skeleton-of-Thought
SoT		 The model first generates a concise skeleton (outline of points) for the answer, then expands each point in parallel. Producing the outline first exposes independent sub-answers that can be decoded concurrently, cutting end-to-end latency while preserving structure.
↔ also: Prompting & reasoning strategies; Planning algorithms & world models for agents		 latency, accuracy/quality, throughput research Ning et al., 2023, ‘Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding’
Speculative reasoning / draft-then-verify reasoning skeletons (Skeleton-of-Thought parallel decode) Generating a reasoning/answer skeleton first and then expanding the points in parallel to cut end-to-end latency of long generations. A short skeleton outline is produced, then each bullet is expanded concurrently (batched), reducing sequential decode depth for the final long answer. latency, throughput research Ning et al., 2023 (Skeleton-of-Thought) — parallel-expansion variant
Step-level Beam Search over reasoning
reasoning beam search, PRM beam search		 Beam search applied at the granularity of reasoning steps, keeping the top-k partial solutions ranked by a process verifier rather than token log-prob. Expands each beam by sampling candidate next steps, scores them with a PRM/value model, and retains the top-k partial trajectories. accuracy/quality, cost/tokens emerging Snell et al., 2024 (Scaling LLM Test-Time Compute Optimally); Yu et al., 2023 (OVM)
Tree of Thoughts
ToT		 A deliberate problem-solving framework that explores a tree of intermediate ‘thoughts’, evaluating and backtracking among partial solutions via BFS or DFS search. The LLM generates multiple thought branches at each step and self-evaluates their promise, enabling lookahead and backtracking instead of a single left-to-right chain.
↔ also: Prompting & reasoning strategies; Self-improvement & feedback loops at inference; Planning algorithms & world models for agents		 accuracy/quality, reliability research Yao et al., 2023, ‘Tree of Thoughts: Deliberate Problem Solving with Large Language Models’
Universal Self-Consistency
USC, LLM-as-aggregator voting, LLM-as-aggregator self-consistency		 An extension of self-consistency to free-form/open-ended generations where exact-match voting is impossible, by having the LLM itself select the most consistent answer among samples. Instead of string-level majority vote, the model is prompted to read all sampled responses and pick the one most consistent with the rest, generalizing voting to non-extractable outputs.
↔ also: Self-improvement & feedback loops at inference; Reliability, structured generation & evaluation-as-optimization		 accuracy/quality, reliability emerging Chen et al., 2023, ‘Universal Self-Consistency for Large Language Model Generation’
Verifier-guided / reward-guided decoding
value-guided decoding, reward-guided search, VAS / value-augmented sampling		 Steering token- or step-level generation using a value/reward signal so that high-reward continuations are preferred during decoding. Reweights or prunes candidate continuations at each decoding/search step by a learned value or process-reward estimate. accuracy/quality, reliability emerging Yang & Klein, 2021 (FUDGE); Mudgal et al., 2023 (Controlled Decoding); Liu et al., 2024 (value-guided / VAS)
Weighted Self-Consistency
weighted majority voting, verifier-weighted voting, confidence-weighted voting		 A variant of self-consistency where votes from sampled paths are weighted by a score (verifier/PRM score, model confidence, or path probability) instead of counted equally. Aggregates answers as an argmax over the sum of per-path weights, so high-quality or high-confidence paths dominate the vote. accuracy/quality, reliability emerging Li et al., 2022 (On the Advance of Making Language Models Better Reasoners / ‘DiVeRSe’); Uesato et al., 2022

3. Self-improvement & feedback loops at inference

Let the model critique and revise its own output within a session, with no weight updates. A generator proposes, an evaluator (the same model, a tool, or a peer) finds faults, and the loop refines. Strong when an external signal — tests, execution, a verifier — grounds the critique; weak when the model grades itself unaided.

19 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
Best-of-N with reward/verifier model
BoN sampling, rejection sampling, verifier reranking		 Generate N candidate outputs and select the highest-scoring one according to a reward model or verifier (which may be the model itself). A scoring model ranks the N samples and the top-scored candidate is returned, converting sampling diversity plus an evaluation signal into a quality gain at the cost of extra inference. accuracy/quality, reliability production-standard Cobbe et al., 2021 (verifiers for GSM8K); Nakano et al., 2021 (WebGPT BoN); widely used since
CRITIC (tool-augmented self-correction)
CRITIC		 A framework where the LLM verifies and corrects its outputs by interacting with external tools (search engines, code interpreters) rather than relying solely on intrinsic judgment. The model generates an output, queries external tools to get grounded critique/feedback, and revises iteratively, compensating for the unreliability of pure self-critique. accuracy/quality, reliability emerging Gou et al., 2024, ‘CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing’
Iterative refinement / generate-evaluate-refine loop
draft-and-revise, refinement loop		 The general inference-time pattern of repeatedly generating, evaluating against a criterion, and revising an output until a quality threshold or iteration budget is reached. A scoring or critique function gates each revision, trading extra inference compute for higher output quality via a closed loop. accuracy/quality, reliability production-standard Madaan et al., 2023 (Self-Refine) and Welleck et al., 2023, ‘Generating Sequences by Learning to Self-Correct’ (Self-Correction / corrector model)
LLM-as-judge debate / multi-agent evaluation
debate-based evaluation, peer-review among agents, ChatEval		 Multiple LLM judges discuss/debate to reach an evaluation verdict, improving the reliability of automated assessment over a single judge. Judge agents exchange and contest their assessments across rounds, reducing single-judge bias and yielding a more human-aligned evaluation signal that can feed back into selection. accuracy/quality, reliability emerging Chan et al., 2023, ‘ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate’
Process Reward Model guided refinement
PRM-guided search, step-level verification		 A verifier that scores intermediate reasoning steps (not just final answers) is used to guide search, prune, or refine the reasoning trajectory at inference. Step-level reward signals steer tree/beam search or trigger re-reasoning at the first low-scoring step, providing denser feedback than outcome-only verification. accuracy/quality, reliability emerging Lightman et al., 2023, ‘Let’s Verify Step by Step’ (OpenAI PRM)
Recursive Criticism and Improvement
RCI		 A prompting scheme where the model first generates an output, then is prompted to criticize it, then to improve it based on the criticism, recursively. Explicit critique-then-improve prompt steps are chained recursively so each round conditions on the prior round’s self-identified faults, improving grounded computer-control and reasoning tasks. accuracy/quality, autonomy, reliability emerging Kim et al., 2023, ‘Language Models can Solve Computer Tasks’ (RCI prompting)
Reflection / verifier-in-the-loop agent reflection
actor-critic LLM loop, generator-discriminator loop		 A two-role pattern where a generator agent produces and a separate critic/reflector agent evaluates and returns actionable feedback for revision, looping until acceptance. Decoupling generation from critique into distinct prompts/agents yields more independent error detection than single-pass self-critique, with the critic’s feedback driving each revision. accuracy/quality, reliability, autonomy production-standard AutoGen reflection pattern (Wu et al., 2023) and LangGraph reflection agents; descends from Shinn et al., 2023
Reflective / metacognitive prompting
metacognitive prompting, step-back-and-reflect		 Prompting the model to reason about its own reasoning process (assess understanding, plan, monitor, evaluate) before and after answering. Explicit metacognitive stages make the model surface assumptions and self-monitor, catching errors that a direct answer would miss. accuracy/quality, reliability research Wang & Zhao, 2024, ‘Metacognitive Prompting Improves Understanding in Large Language Models’
Reflexion (verbal self-reflection memory)
verbal reinforcement learning, self-reflection, verbal reinforcement / self-reflection		 An agent that maintains an episodic memory of self-generated verbal reflections on prior failed attempts, using them as feedback to improve on subsequent trials of the same task. After a failed trajectory, the agent verbalizes what went wrong and stores that reflection; the reflection is prepended on the next attempt, acting as gradient-free reinforcement through language.
↔ also: Prompting & reasoning strategies; Test-time compute scaling & search over reasoning; Agent memory architectures; Multi-agent orchestration patterns; Planning algorithms & world models for agents; Reliability, structured generation & evaluation-as-optimization		 accuracy/quality, reliability, autonomy emerging Shinn et al., ‘Reflexion: Language Agents with Verbal Reinforcement Learning’, 2023
Reflexion with self-generated unit tests
test-time self-test generation, agentic test-then-fix		 An agent writes its own tests/checks for a task, runs them, and reflects on failures to drive correction, common in coding agents. Self-authored executable checks provide concrete, grounded failure signals that are reflected on and used to revise the solution, sharpening the otherwise-vague self-critique. accuracy/quality, reliability, autonomy emerging Common in SWE-agent / coding-agent literature; descends from Shinn et al., 2023 and Chen et al., 2024 (Self-Debugging)
Self-Consistency as feedback / verifier-guided refinement
agreement-as-reward, consistency feedback		 Using the (dis)agreement among multiple self-generated samples as an explicit feedback signal to trigger or guide a refinement step rather than just voting once. Inter-sample inconsistency is detected and surfaced to the model as a critique (‘your answers disagree on X’), prompting targeted re-reasoning on the contested portion. accuracy/quality, reliability research Derived from Wang et al., 2023 (self-consistency) and Madaan et al., 2023 (Self-Refine); a recognized composite pattern
Self-Contrast (multi-perspective divergent self-checking) Self-correction that first generates diverse solving perspectives, contrasts their discrepancies, and uses the differences as a checklist for revision. Explore-then-contrast: multiple solver views are diffed to surface inconsistencies that a single self-critique pass misses, then resolved into a revised answer. accuracy/quality, reliability research Zhang et al., 2024 (Self-Contrast)
Self-Correction with learned corrector (Self-Correct)
Self-Correct, learned corrector model		 Pairs a fixed generator with a separately trained corrector module that iteratively edits the generator’s output toward higher reward at inference. A corrector is trained on (hypothesis, improved-hypothesis) value-improving pairs and applied repeatedly at decode time to refine outputs. accuracy/quality, reliability research Welleck et al., 2023, ‘Generating Sequences by Learning to Self-Correct’
Self-critique / self-correction
self-critiquing, intrinsic self-correction		 The general technique of prompting a model to critique its own answer and then produce a corrected version, without external tools or ground truth. A critique prompt elicits the model’s identification of errors, and a revision prompt conditions a new answer on that critique within the same context. accuracy/quality, reliability production-standard Saunders et al., 2022, ‘Self-critiquing models for assisting human evaluators’; see also Huang et al., 2024 on the limits of intrinsic self-correction
Self-Debugging
self-debug, rubber-duck debugging for code		 A code-generation technique where the model explains and debugs its own generated program using execution results or its own line-by-line explanation, then fixes it. Execution feedback (errors, unit-test results) and self-generated code explanations are fed back to the model so it can localize and repair bugs without human guidance. accuracy/quality, reliability, autonomy emerging Chen et al., 2024, ‘Teaching Large Language Models to Self-Debug’
Self-Discover
self-composed reasoning structures		 A framework where the model self-composes a task-specific reasoning structure from a set of atomic reasoning modules before solving, then follows and can revise it. The model selects, adapts, and implements reasoning modules into an explicit structure that scaffolds the solution, improving over fixed prompting templates.
↔ also: Prompting & reasoning strategies		 accuracy/quality, cost/tokens research Zhou et al., 2024, ‘Self-Discover: Large Language Models Self-Compose Reasoning Structures’ (Google DeepMind)
Self-evaluation / self-assessment
self-eval, confidence self-estimation		 The model evaluates the quality or correctness of its own outputs (e.g., assigning a confidence or pass/fail), and that self-estimate gates or weights the answer. An evaluation prompt elicits a calibrated judgment of the model’s own answer, which is used to decide whether to accept, retry, or branch. accuracy/quality, reliability emerging Kadavath et al., 2022, ‘Language Models (Mostly) Know What They Know’; Ren et al., 2023 on self-evaluation for guided decoding
Self-Refine
iterative self-refinement, self-feedback refinement, tool error recovery		 A single model iteratively produces an output, generates feedback on its own output, and refines it, repeating until a stopping condition. The same LLM alternates between a generate step and a self-feedback-then-revise step in a closed loop, using only its own critiques (no external supervision) to improve the draft.
↔ also: Prompting & reasoning strategies; Test-time compute scaling & search over reasoning; Tool use & function calling optimization; Planning algorithms & world models for agents; Reliability, structured generation & evaluation-as-optimization		 accuracy/quality, reliability, autonomy production-standard Madaan et al., 2023, ‘Self-Refine: Iterative Refinement with Self-Feedback’ (NeurIPS 2023)
Self-verification
backward verification, self-verifier, generate-and-verify		 The model checks the correctness of its own candidate answer (often by reasoning backward or re-deriving), using the verification signal to accept, reject, or re-rank candidates. A separate verification pass scores or validates each candidate output (e.g., backward verification of conditions), and only verified candidates are kept or weighted higher.
↔ also: Prompting & reasoning strategies		 accuracy/quality, reliability emerging Weng et al., 2023, ‘Large Language Models are Better Reasoners with Self-Verification’

4. Decoding-time control

Steer generation at the token level by reshaping the output distribution itself — contrasting a strong model against a weak one, contrasting layers, amplifying the context, or reshaping the sampling tail. Model-internal knobs that lift factuality or quality without touching the prompt or the weights.

5 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
Context-aware / Context-Aware Decoding (CAD, PMI context amplification) Amplifying the influence of provided context by contrasting the model’s output distribution with and without the context (a pointwise-mutual-information adjustment). Logits-with-context minus logits-without-context up-weight tokens the context makes more likely, reducing reliance on stale parametric memory in RAG/summarization. reliability, accuracy/quality research Shi et al., 2023 (Trusting Your Evidence / Context-Aware Decoding)
Contrastive Decoding (expert vs amateur) A decoding method that subtracts a small ‘amateur’ model’s log-probabilities from a large ‘expert’ model’s to favor tokens the expert prefers more strongly. The contrast amplifies the capability gap between models, suppressing generic/degenerate continuations and improving reasoning/open-ended quality at decode time. accuracy/quality research Li et al., 2022 (Contrastive Decoding); O’Brien & Lewis, 2023 (CD improves reasoning)
DoLa (Decoding by Contrasting Layers) A factuality decoding method that contrasts the next-token distributions of later vs earlier transformer layers to surface factual knowledge. Mature (later-layer) logits are contrasted against premature (earlier-layer) logits, amplifying factual tokens and reducing hallucination with no extra model. reliability, accuracy/quality research Chuang et al., 2023 (DoLa)
Guidance / constrained generation interleaving (template-guided + acceleration) A programming model that interleaves fixed template text, control flow, and constrained model generations, reusing KV across the fixed parts. By emitting deterministic template tokens directly and only sampling the variable slots, it cuts generated tokens and guarantees format while keeping cache warm. cost/tokens, latency, reliability production-standard Microsoft Guidance (Lundberg et al., 2023)
Min-p sampling A truncation sampling scheme that keeps tokens whose probability is at least a fraction p of the top token’s probability, adapting the candidate set to confidence. The dynamic threshold widens when the model is uncertain and narrows when confident, giving better quality/diversity trade-offs than fixed top-p/top-k at high temperature. accuracy/quality emerging Nguyen et al., 2024 (Min-p sampling)

Part II — Context, memory & knowledge

5. Context-window management & compression

The context window is a scarce, expensive resource that grows every turn. These methods keep it bounded: summarize or compact old turns, compress the prompt, prune low-salience tokens, window long observations, or offload to external state. fak’s measured work here: O(1) context economics, the ctxwin baseline, and the context planner.

26 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
A-MEM (agentic memory / Zettelkasten-style)
episodic memory, reflection, memory stream		 A self-organizing agent memory that, on each new note, generates structured attributes (keywords, tags, context) and dynamically links it to related existing memories, forming an evolving interconnected knowledge network. Inspired by the Zettelkasten method, each new memory triggers LLM-driven link generation and memory evolution that updates the attributes of connected older notes, so the graph reorganizes itself over time.
↔ also: Agent memory architectures		 context-capacity, accuracy/quality, autonomy, cost/tokens, reliability research Xu et al., ‘A-MEM: Agentic Memory for LLM Agents’, 2025
Activation Beacon (sliding context condensation) Condensing raw activations into compact ‘beacon’ tokens at intervals so a short-context model effectively reads a much longer window. Beacon tokens summarize past activations and are kept while raw tokens are dropped, extending effective context with bounded memory. context-capacity, memory research Zhang et al., 2024 (Soaring from 4K to 400K: Extending LLM Context with Activation Beacon)
Attention sinks / StreamingLLM
attention sink, streaming LLM, sink tokens		 An inference technique that keeps the first few tokens (attention sinks) plus a sliding window of recent tokens in the KV cache to enable stable streaming over effectively unbounded inputs. Retaining initial sink tokens preserves the attention distribution while a rolling recent window bounds memory, avoiding the collapse seen when early KV entries are evicted. context-capacity, memory, latency, throughput production-standard Xiao et al., ‘Efficient Streaming Language Models with Attention Sinks (StreamingLLM)’, ICLR 2024
Contextual Compression / Retrieved-Context Pruning
cross-encoder reranking, context distillation filter, RECOMP		 Compresses or filters retrieved passages down to the query-relevant content before feeding them to the LLM, cutting tokens while keeping evidence. An extractor or abstractor (rule-based, LLM-based, or a trained compressor like RECOMP/LLMLingua) prunes sentences/tokens with low query relevance, shrinking the context window load.
↔ also: Retrieval & knowledge augmentation; Model routing, cascades & cost optimization		 accuracy/quality, cost/tokens, reliability, context-capacity, latency emerging Xu et al., 2023 (RECOMP); Jiang et al., 2023 (LLMLingua)
Conversation summarization / compaction
context compaction, history summarization, conversation summary memory		 Periodically replacing earlier turns of a conversation or agent trajectory with a model-generated summary so the working context stays within the window. An LLM condenses prior messages/tool-results into a shorter abstractive summary that is reinserted in place of the raw history. cost/tokens, context-capacity, latency, memory production-standard LangChain ConversationSummaryMemory; Anthropic/Claude Code auto-compact; widely deployed pattern
Deduplication / redundancy elimination in context
context dedup, near-duplicate removal, repetition collapse		 Detecting and collapsing repeated or near-duplicate content (boilerplate, re-pasted files, repeated tool outputs) within the context. Exact/near-duplicate spans are hashed or embedding-matched and replaced by a single instance plus references, removing redundant tokens. cost/tokens, context-capacity emerging Retrieval/context-engineering practice; near-duplicate detection (MinHash/embedding) applied to prompts
DMC / Dynamic Memory Compression (learned KV merging at decode) A retrofit that lets the model learn, per step, whether to append a new KV pair or accumulate it into the previous one, compressing the cache online. A learned gate decides append-vs-merge for each token’s KV, achieving multi-x KV compression with retained quality after light continued training. memory, throughput, context-capacity research Nawrot et al., 2024 (Dynamic Memory Compression)
Generative / parametric memory (memory tokens & soft prompts)
Gist tokens, gisting, AutoCompressor		 Encoding past context into a small set of learned memory vectors or summary tokens stored and re-injected, rather than keeping natural-language text. A model compresses long history into compact soft tokens / gist vectors that are appended to later contexts, recovering needed information at a fraction of the token cost.
↔ also: Agent memory architectures		 cost/tokens, memory, context-capacity, latency research Mu et al., ‘Learning to Compress Prompts with Gist Tokens’, 2023; Chevalier et al., ‘AutoCompressors’, 2023
Hierarchical / recursive summarization
recursive summarization, tree summarization, multi-level summary		 Building summaries at multiple levels of granularity (e.g., summaries of summaries) so very long histories or documents can be compressed in a structured tree. Chunks are summarized, then groups of summaries are recursively summarized up a tree until the result fits the budget. context-capacity, cost/tokens, accuracy/quality production-standard Wu et al. ‘Recursively Summarizing Books with Human Feedback’, 2021; RAPTOR (Sarthi et al., 2024)
In-context autoencoding / 500x context compression (ICAE — explicit), and prompt-distillation Training a learnable compressor to encode a long context into a handful of memory slots a frozen LLM can decode/condition on. A LoRA-adapted encoder compresses context into soft memory tokens; the target model attends to those slots instead of the raw text, slashing prompt length. cost/tokens, context-capacity, latency research Ge et al., 2023 (In-context Autoencoder for Context Compression)
LLMLingua
prompt compression, coarse-to-fine prompt compression		 A prompt-compression method that uses a small language model to drop low-information tokens from a long prompt while preserving the answer. A budget controller plus token-level iterative perplexity estimation from a small LM prunes tokens whose removal least raises model uncertainty. cost/tokens, latency, context-capacity emerging Jiang et al., ‘LLMLingua’, EMNLP 2023
LLMLingua-2
task-agnostic prompt compression		 A task-agnostic prompt-compression model trained via data distillation to classify each token as keep-or-drop. A bidirectional Transformer token-classifier, trained on GPT-4-distilled compression labels, predicts token retention faster and more robustly than perplexity pruning. cost/tokens, latency, context-capacity emerging Pan et al., ‘LLMLingua-2’, ACL 2024 (findings)
LongLLMLingua
long-context prompt compression		 An extension of LLMLingua targeting long-context RAG that compresses and reorders retrieved documents to mitigate lost-in-the-middle and reduce tokens. Question-aware coarse-to-fine compression plus document reordering by relevance, with a contrastive perplexity to keep question-relevant tokens. cost/tokens, accuracy/quality, context-capacity, latency emerging Jiang et al., ‘LongLLMLingua’, ACL 2024
Lost-in-the-middle mitigation via reordering
context reordering, relevance reranking placement, primacy/recency placement		 Placing the most important retrieved/context items at the beginning and end of the prompt to counter the model’s degraded use of mid-context information. Documents are reranked and positioned so high-relevance content occupies the high-attention head and tail positions rather than the neglected middle. accuracy/quality, reliability production-standard Liu et al., ‘Lost in the Middle: How Language Models Use Long Contexts’, TACL 2024
MemGPT (paged/virtual context memory)
MemGPT, Letta, context paging		 An agent memory architecture that treats the LLM context window like RAM and external stores like disk, paging information in and out via self-issued function calls so the agent can reason over data far larger than its context. The agent uses tool calls to read/write a tiered hierarchy (main context, recall storage, archival storage) and an interrupt-driven controller evicts and recalls memory pages as the working context fills.
↔ also: Agent memory architectures		 context-capacity, memory, autonomy, reliability, accuracy/quality, cost/tokens, latency production-standard Packer et al., ‘MemGPT: Towards LLMs as Operating Systems’, 2023
RAPTOR
tree-organized memory, hierarchical index, recursive abstractive processing		 Builds a tree of recursively clustered and summarized chunks, enabling retrieval at multiple levels of abstraction so both detailed and high-level questions are served. Recursively embeds, clusters, and summarizes text into a multi-level tree, then retrieves across tree nodes (collapsed-tree or traversal) to supply context at the right granularity.
↔ also: Retrieval & knowledge augmentation		 context-capacity, accuracy/quality, cost/tokens emerging Sarthi et al., 2024, ‘RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval’
Recurrent / segment-level memory transformers
RMT, Recurrent Memory Transformer, Transformer-XL		 Architectures that carry information across segments of a long input via recurrent memory tokens or cached/compressed past activations. Special memory slots or a (optionally compressed) recurrent state pass summarized context from one segment to the next, extending effective range beyond a single window. context-capacity, memory, accuracy/quality research Dai et al., ‘Transformer-XL’, 2019; Rae et al., ‘Compressive Transformers’, 2020; Bulatov et al., ‘Recurrent Memory Transformer’, 2022
Retrieval-based context selection
RAG context selection, retrieve-then-read, just-in-time retrieval		 Selecting only the most relevant external passages to inject into the window per query instead of stuffing the whole corpus. A retriever (dense/sparse/hybrid) scores and returns top-k passages by relevance to the query, bounding injected tokens. cost/tokens, accuracy/quality, context-capacity, latency production-standard Lewis et al., ‘Retrieval-Augmented Generation’, NeurIPS 2020
Scratchpad reasoning
scratchpad, filesystem as memory, external memory		 Training or prompting the model to emit intermediate computation steps to a ‘scratchpad’ before the final output, e.g., showing carries in multi-digit arithmetic. Externalizing intermediate state into generated tokens gives the model working memory it can attend to, enabling multi-step algorithmic computation.
↔ also: Prompting & reasoning strategies		 accuracy/quality, reliability, context-capacity, cost/tokens, autonomy research Nye et al., 2021, ‘Show Your Work: Scratchpads for Intermediate Computation with Language Models’
Selective Context
self-information pruning		 A prompt-pruning method that removes redundant lexical units (tokens, phrases, sentences) judged low-information by a model. A base LM computes self-information (negative log-prob) per unit and prunes the least informative units to fit a budget. cost/tokens, context-capacity, latency emerging Li et al., ‘Selective Context’ / ‘Compressing Context to Enhance Inference Efficiency’, 2023
Semantic chunking
semantic splitting, embedding-based chunking, proposition chunking		 Splitting source text into coherent units by meaning rather than fixed token length, so each chunk carries a self-contained idea. Sentence/segment embeddings detect topic boundaries (similarity drops) to cut the text where semantic shifts occur, yielding cleaner retrieval/compression units. accuracy/quality, cost/tokens, context-capacity production-standard Greg Kamradt ‘Levels of Text Splitting’; LlamaIndex/LangChain SemanticChunker; Chen et al. ‘Dense X Retrieval (propositions)’, 2023
Sliding-window / segment context processing
chunked long-document processing, map-reduce over context, windowed inference		 Processing an over-long input in overlapping windows or map-reduce passes and combining the partial results, rather than fitting it all at once. The document is split into windows the model can handle; per-window outputs are aggregated (map-reduce) or carried forward (refine) into a final answer. context-capacity, cost/tokens production-standard LangChain map-reduce / refine summarization chains; long-document QA practice
Structured / running state notes
agent scratchpad state, running summary, todo/plan file		 Maintaining a compact, structured note (plan, decisions, open items) that is updated each step and replaces re-reading the full trajectory. The agent keeps a small canonical state object it rewrites incrementally, so each turn conditions on a dense summary instead of raw history. context-capacity, cost/tokens, reliability, autonomy production-standard Anthropic context-engineering / ‘effective agents’; Cognition ‘Don’t Build Multi-Agents’ (single thread + state)
Sub-agent / context isolation (sub-context offload)
context isolation, sub-agent delegation, spawned context		 Delegating a bounded subtask to a separate agent/context that does heavy reading and returns only a distilled result to the parent’s window. A child context absorbs the token-heavy exploration and reports a small summary, keeping the orchestrator’s window lean. context-capacity, cost/tokens, reliability production-standard Anthropic ‘How we built our multi-agent research system’, 2025; context-engineering practice
Token / context pruning by saliency
context distillation, saliency token dropping, input token pruning		 Dropping input tokens deemed unimportant to the task before or during processing to shrink the context the model must handle. Importance scores (attention, gradients, or a learned scorer) rank tokens and prune low-saliency ones from the prompt or intermediate sequence. cost/tokens, latency, context-capacity, memory research Goyal et al., ‘PoWER-BERT’, 2020; LazyLLM (Fu et al., 2024); general token-pruning literature
Tool-result / observation truncation and windowing
observation pruning, tool-output truncation, result head+tail windowing		 Trimming verbose tool outputs and stale observations in an agent trajectory to a compact head/tail plus a pointer, keeping only what later steps still need. Large or superseded tool results are replaced with truncated excerpts or references so the running context does not accumulate dead bytes. cost/tokens, context-capacity, latency production-standard Agent context-engineering practice; ReAct observation management (Yao et al., 2022); Anthropic context-engineering guidance

6. Agent memory architectures

When a fact must outlive the context window, it goes to memory. These methods give agents typed, persistent stores — episodic, semantic, procedural — with policies for what to write, when to recall, and when to forget. Cross-cut: fak’s four layers of agent memory and context-is-not-memory.

23 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
Associative graph-traversal recall (HippoRAG-style)
HippoRAG, neurobiologically-inspired memory, personalized PageRank recall		 A long-term memory that builds a knowledge graph index and performs single-step multi-hop associative retrieval inspired by the hippocampal indexing theory. Passages are indexed into a schemaless KG; a Personalized PageRank pass over the graph from query-anchored nodes retrieves multi-hop-relevant memories in one shot rather than iterative search.
↔ also: Retrieval & knowledge augmentation		 accuracy/quality, latency, cost/tokens, context-capacity emerging Jimenez Gutierrez et al., ‘HippoRAG: Neurobiologically Inspired Long-Term Memory for LLMs’, 2024
Cognitive memory typing (working / episodic / semantic / procedural)
multi-store memory, CoALA memory taxonomy, typed agent memory		 An organizing framework that partitions agent memory into working memory (current context), episodic memory (past experiences/events), semantic memory (facts/knowledge), and procedural memory (skills/how-to), mirroring human memory systems. Each memory type gets its own store, write policy, and retrieval path so the agent reads procedural skills, episodic precedents, and semantic facts through specialized channels rather than one undifferentiated buffer. accuracy/quality, autonomy, context-capacity, reliability emerging Sumers et al., ‘Cognitive Architectures for Language Agents (CoALA)’, 2023
Conversational summary-buffer memory
summary buffer, recursive conversation summarization, rolling summary		 A memory that keeps recent turns verbatim while progressively summarizing older turns into a running condensed summary to fit a bounded context. When the buffer exceeds a token budget, the oldest spans are folded into an LLM-generated running summary, preserving gist while discarding token-heavy detail. cost/tokens, context-capacity, latency production-standard LangChain ConversationSummaryBufferMemory; recursive summarization (Wu et al., 2021)
Editable core / persona memory blocks
core memory, memory blocks, in-context persistent memory		 A small, always-in-context, agent-editable region holding durable facts (persona, user profile, key constraints) that the agent rewrites via dedicated functions. Reserved context slots are exposed to the model as editable blocks; the agent calls append/replace functions to keep this high-priority state current without re-retrieving it each turn. accuracy/quality, reliability, context-capacity, cost/tokens production-standard MemGPT/Letta core memory (Packer et al., 2023)
Entity / profile memory
entity memory, user profile memory, knowledge-base memory		 A structured memory keyed by entities (people, projects, preferences) that accumulates and updates per-entity facts across a long-running interaction. Mentioned entities are extracted and their attribute records are read into context and updated after each turn, giving the agent stable, queryable per-entity state. accuracy/quality, reliability, context-capacity production-standard LangChain entity memory; conversational-personalization literature
Experiential memory / case-based experience reuse
experience replay for agents, Expel, trajectory memory		 An episodic store of past task trajectories and the lessons/insights extracted from them, retrieved to guide the agent on analogous new tasks without weight updates. After a batch of tasks, the agent compares successes and failures to distill natural-language insights and stores representative trajectories; at inference it retrieves the most similar cases and insights as in-context exemplars. accuracy/quality, autonomy, reliability emerging Zhao et al., ‘ExpeL: LLM Agents Are Experiential Learners’, 2023
Generative / RAG-free memory via Cache-Augmented Generation distinction — RecurrentGPT (language-based recurrence) An interpretable long-form agent loop that keeps a natural-language short-term and long-term memory state updated each step like an RNN written in prose. The model reads/writes plain-text memory + a plan each turn, simulating recurrence to generate arbitrarily long, coherent outputs without hidden state. context-capacity, reliability research Zhou et al., 2023 (RecurrentGPT)
Graph-based agent memory
knowledge-graph memory, temporal knowledge graph memory, Zep/Graphiti		 An agent memory represented as a (often temporally-aware) knowledge graph of entities and relations extracted from the interaction history, queried via graph traversal plus semantic search. Conversations/observations are parsed into entity-relation triples with valid-time edges; retrieval combines graph traversal with embedding search to assemble a relevant, temporally-consistent subgraph for context. accuracy/quality, context-capacity, reliability production-standard Rasmussen et al., ‘Zep: A Temporal Knowledge Graph Architecture for Agent Memory’, 2025; Graphiti
Importance scoring / memory prioritization
salience scoring, memory importance weighting		 Assigning each memory an importance/poignancy score at write time that biases later retrieval, consolidation, and forgetting decisions. An LLM (or heuristic) rates how significant an observation is on write; the score is folded into retrieval ranking and reflection triggers, so consequential memories surface more readily and survive pruning. accuracy/quality, memory, cost/tokens emerging Generative Agents importance score (Park et al., 2023)
Mem0 (extraction-and-update memory layer)
Mem0, scalable memory layer		 A memory layer that extracts salient facts from conversations and maintains them through an explicit add/update/delete/no-op decision over existing memories, with an optional graph variant. An LLM extraction stage pulls candidate facts; a second update stage compares each against retrieved existing memories and chooses to insert, merge/update, or discard, keeping the store consolidated and non-redundant. accuracy/quality, cost/tokens, latency, context-capacity production-standard Chhikara et al., ‘Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory’, 2025
Memory consolidation
memory summarization, memory compaction, consolidation pass		 A background or threshold-triggered process that merges, deduplicates, and abstracts raw memories into compact higher-level representations to control store growth and improve retrieval signal. Periodically the system clusters or summarizes related raw memories into consolidated entries and may retire the originals, trading fine detail for density and reduced retrieval noise. cost/tokens, context-capacity, accuracy/quality, memory emerging MemGPT (Packer et al., 2023); generative-agents reflection lineage
Memory forgetting / decay
temporal decay, memory eviction, TTL forgetting		 Policies that downweight or remove memories over time or by disuse so stale, low-value entries do not dominate retrieval or bloat the store. Each memory carries a decay factor (e.g., exponential recency or access-count-based) and/or an eviction threshold, so unaccessed low-importance memories fade or are pruned while reinforced ones persist. cost/tokens, memory, accuracy/quality, latency emerging Generative Agents recency decay (Park et al., 2023); broader memory-management literature
Memory retrieval re-ranking / relevance gating
memory reranker, salience gating, two-stage memory retrieval		 A second-stage filter/reranker applied to candidate memories after first-pass retrieval to keep only the most salient, non-redundant entries before injection. Cheap recall fetches a broad candidate set, then a cross-encoder or LLM judge reorders and prunes them for relevance, recency, and diversity, raising precision of what enters context. accuracy/quality, cost/tokens, context-capacity emerging RAG reranking practice applied to agent memory (cross-encoder rerankers; HippoRAG lineage)
Memory writeback policy
write gate, memory update policy, ADD/UPDATE/DELETE/NOOP decision		 The decision policy that governs when and how new information is committed to long-term memory, including whether to insert, update an existing entry, delete, or ignore. Incoming candidate facts are compared against retrieved existing memories and an LLM (or rules) selects an operation, preventing duplication and drift while keeping the store consistent. accuracy/quality, cost/tokens, reliability, memory emerging Mem0 (Chhikara et al., 2025); MemGPT write functions (Packer et al., 2023)
MemoryBank with Ebbinghaus forgetting curve A long-term memory module that strengthens or decays stored memories per an Ebbinghaus-style forgetting schedule based on recall and recency. Each memory carries a strength that decays over time and is refreshed on access, prioritizing durable, frequently-recalled facts at retrieval. memory, accuracy/quality research Zhong et al., 2023 (MemoryBank: Enhancing LLMs with Long-Term Memory)
Procedural memory / learned skills writeback
agent workflow memory, procedural skill induction, AWM		 A mechanism that induces reusable procedures or workflows from an agent’s past successful trajectories and writes them back into memory for retrieval on similar future tasks. The agent mines recurring action patterns from solved tasks, abstracts them into named workflows/macros, and stores them so future planning retrieves the procedure instead of re-deriving the steps. accuracy/quality, cost/tokens, autonomy, latency emerging Wang et al., ‘Agent Workflow Memory’, 2024
Recurrent / Memorizing-Transformer memory
kNN memory, Memorizing Transformers, Transformer-XL recurrence		 Architectural long-term memory that stores past key-value pairs (or recurrent states) and retrieves them via attention/kNN to extend effective context beyond the attention window. Past hidden states or KV pairs are cached in a non-differentiable external store; a kNN-augmented attention layer attends over retrieved entries, giving the model long-range recall at sublinear cost. context-capacity, memory, accuracy/quality research Wu et al., ‘Memorizing Transformers’, 2022; Dai et al., ‘Transformer-XL’, 2019
Retrieval-augmented memory (retrieval-over-memory / RAG)
RAG over memory, vector-store memory, long-term memory retrieval		 Treating the long-term memory store as a retrieval corpus, embedding entries and fetching the top-k semantically relevant memories into the working context at each step. Memories are embedded and indexed (e.g., in a vector DB); a query embedding (often the current task or last turn) retrieves nearest neighbors that are injected into the prompt, bounding context to relevant content. context-capacity, cost/tokens, accuracy/quality, latency production-standard Lewis et al., ‘Retrieval-Augmented Generation’, 2020 (applied to agent memory)
Retrieval-scored memory (recency-importance-relevance)
RIR scoring, weighted memory retrieval		 A retrieval policy over a memory store that ranks candidate memories by a weighted sum of recency (exponential time decay), importance, and semantic relevance. Each memory carries a recency decay, an LLM-assigned importance score, and an embedding similarity to the query; the normalized weighted sum selects the top-k memories injected into context. accuracy/quality, cost/tokens, context-capacity emerging Park et al., ‘Generative Agents’, 2023
Self-editing / learning-to-memorize agents
self-edit memory, memory-as-learning, MemoryBank / SiliconFriend		 Agents that learn or are explicitly trained to decide what to remember and how to update their own memory store over long horizons, including human-memory-inspired update curves. The agent applies (sometimes Ebbinghaus-curve-based) update and reinforcement rules to its memory, strengthening repeatedly-recalled items and letting others decay, optimizing the store as a learned policy. accuracy/quality, autonomy, memory, reliability research Zhong et al., ‘MemoryBank: Enhancing LLMs with Long-Term Memory’, 2023
Shared / collective multi-agent memory
shared memory blackboard, team memory, collective experience pool		 A memory store shared across multiple agents in a system, letting one agent’s learned facts, skills, or experiences be retrieved and reused by peers. Agents write experiences/skills to a common indexed pool with provenance and access policies; teammates retrieve relevant shared entries, propagating knowledge without retraining. accuracy/quality, throughput, autonomy, cost/tokens research Multi-agent collaboration literature (e.g., generative-agents communities; MetaGPT/AutoGen shared context)
Typed multi-view memory
multi-view memory, multi-perspective memory store		 Storing the same underlying experience under several typed views (e.g., raw episodic, summarized, entity-indexed, intent/goal-indexed) so different retrieval needs hit the most suitable representation. On write, an event is projected into multiple parallel representations/indices; retrieval routes the query to the view that best matches its granularity or facet, improving recall precision. accuracy/quality, context-capacity, reliability research Generative-agents/CoALA-inspired multi-view designs (Sumers et al., 2023)
Voyager skill library
skill library, lifelong skill acquisition, code-as-skill memory		 A growing, retrievable library of executable skills (programs) that an agent authors, verifies, and reuses, enabling lifelong learning and compositional capability in an open-ended environment. Successful self-generated code routines are embedded by their description and stored; later tasks retrieve relevant skills as building blocks, so capability compounds and complex skills are composed from simpler ones.
↔ also: Planning algorithms & world models for agents		 accuracy/quality, autonomy, reliability, cost/tokens research Wang et al., ‘Voyager: An Open-Ended Embodied Agent with Large Language Models’, 2023

7. Retrieval & knowledge augmentation

Bring the right external knowledge into context at the right moment. RAG and its descendants vary on what to retrieve (chunks, graphs, hypothetical docs), when (statically, iteratively, on-demand), and how to trust it (re-ranking, correction, conflict resolution).

31 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
Adaptive-RAG (query-complexity-routed retrieval depth) A router that classifies query complexity and chooses between no-retrieval, single-step, and multi-step iterative retrieval accordingly. A trained complexity classifier directs simple queries to direct answering and hard ones to iterative retrieval, balancing accuracy against retrieval cost/latency. cost/tokens, latency, accuracy/quality emerging Jeong et al., 2024 (Adaptive-RAG)
Agentic RAG
iterative RAG, agent-driven retrieval, ReAct-style RAG		 Treats retrieval as a tool an agent invokes adaptively across multiple reasoning steps, deciding when, what, and how often to retrieve rather than retrieving once up front. An LLM agent interleaves reasoning and retrieval actions (often via ReAct), issuing follow-up queries based on intermediate findings until it has enough evidence to answer. accuracy/quality, autonomy, reliability emerging Builds on Yao et al., 2022 (ReAct) and IRCoT (Trivedi et al., 2023); ‘agentic RAG’ as a 2024-2025 design pattern
Astute RAG / knowledge-conflict resolution Explicitly reconciling the model’s internal/parametric knowledge against retrieved passages when they conflict, instead of blindly trusting retrieval. Internal knowledge is elicited, consolidated with external passages by source, and conflicts adjudicated before final answer generation. accuracy/quality, reliability emerging Wang et al., 2024 (Astute RAG)
Chain-of-Note (CoN) A robust-RAG method where the model writes per-document reading notes before answering, so noisy/irrelevant retrieved docs are explicitly down-weighted. Generating evidence notes per passage lets the model judge relevance and abstain (‘unknown’) when retrieval is unhelpful, cutting hallucination from bad context. accuracy/quality, reliability research Yu et al., 2023 (Chain-of-Note)
ColBERT Late-Interaction Retrieval
ColBERT, ColBERTv2, late interaction		 A retrieval model that stores per-token embeddings and scores documents via fine-grained token-level MaxSim interaction, giving cross-encoder-like precision at near bi-encoder speed. Encodes query and document into multiple token vectors and computes relevance as the sum of maximum cosine similarities (MaxSim) between query tokens and document tokens, enabling precise yet scalable search. accuracy/quality, latency emerging Khattab & Zaharia, 2020, ‘ColBERT’; Santhanam et al., 2022, ‘ColBERTv2’
Contextual Retrieval
contextual embeddings, contextual chunking		 Prepends a chunk-specific explanatory context (generated from the whole document) to each chunk before embedding/indexing, so isolated chunks retain document-level meaning. An LLM writes a short situating context for each chunk (and a BM25 variant), which is indexed with the chunk, sharply reducing retrieval failures from context-stripped chunks. accuracy/quality, reliability emerging Anthropic, 2024, ‘Introducing Contextual Retrieval’
Corrective RAG
CRAG		 Adds a lightweight retrieval evaluator that grades retrieved documents and triggers corrective actions (e.g. web search or knowledge refinement) when retrieval quality is poor. A retrieval evaluator labels results as correct/incorrect/ambiguous; on low confidence it decomposes-then-recomposes knowledge or falls back to web search before generation. accuracy/quality, reliability research Yan et al., 2024, ‘Corrective Retrieval Augmented Generation’
Cross-Encoder Re-ranking
re-ranking, neural reranker, two-stage retrieval		 A second-stage scorer that jointly encodes the query and each retrieved candidate to produce a precise relevance score, reordering an initial candidate set. Feeds query-passage pairs through a cross-attention transformer (e.g. a fine-tuned BERT or a hosted reranker) so each candidate is scored with full query-document interaction, unlike the bi-encoder used for first-stage recall. accuracy/quality, reliability production-standard Nogueira & Cho, 2019, ‘Passage Re-ranking with BERT’; commercial rerankers (Cohere Rerank)
Dense Passage Retrieval
DPR, bi-encoder dense retrieval		 A dual-encoder dense retriever that embeds queries and passages separately into a shared space and retrieves by nearest-neighbor similarity, the workhorse first-stage retriever for RAG. Trains two BERT encoders with contrastive learning so relevant query-passage pairs are close in embedding space, enabling fast ANN search over precomputed passage vectors. accuracy/quality, latency production-standard Karpukhin et al., 2020, ‘Dense Passage Retrieval for Open-Domain Question Answering’
FLARE (Forward-Looking Active Retrieval)
active retrieval, FLARE		 An active retrieval strategy that anticipates upcoming content and triggers retrieval mid-generation whenever the model is about to produce low-confidence tokens. Generates a tentative next sentence, and if its token probabilities fall below a confidence threshold, uses that sentence as a query to retrieve and regenerate, repeating throughout generation. accuracy/quality, reliability research Jiang et al., 2023, ‘Active Retrieval Augmented Generation’
Generate-then-Read / GenRead (LLM-as-retriever) Replacing or supplementing a retriever by having the LLM generate background documents that are then used as context to answer. The model produces several synthetic context documents (optionally clustered for diversity), which are fed back as evidence, leveraging parametric knowledge. accuracy/quality research Yu et al., 2022 (Generate rather than Retrieve / GenRead)
GraphRAG
graph-based RAG, knowledge-graph RAG		 Builds a knowledge graph (entities, relations, and community summaries) from the corpus and retrieves over graph structure to answer global/holistic questions a flat vector index cannot. Extracts entities and relationships into a graph, clusters into communities with hierarchical summaries, and answers by traversing/aggregating relevant graph elements rather than just top-k passages. accuracy/quality, context-capacity, reliability emerging Edge et al., 2024, ‘From Local to Global: A Graph RAG Approach to Query-Focused Summarization’ (Microsoft GraphRAG)
Hybrid Retrieval (sparse + dense)
BM25 + dense, hybrid search, lexical-semantic fusion		 Combines a lexical/sparse retriever (e.g. BM25) with a dense embedding retriever and fuses their results to capture both exact-term matches and semantic similarity. Runs both retrievers in parallel and merges ranked lists, commonly via Reciprocal Rank Fusion or weighted score combination, before passing candidates downstream. accuracy/quality, reliability production-standard Reciprocal Rank Fusion, Cormack et al., 2009; widely operationalized in modern RAG stacks
Hypothetical Document Embeddings
HyDE		 A zero-shot dense retrieval technique that has the LLM generate a hypothetical answer document for the query, then retrieves real documents similar to that synthetic document. Embeds an LLM-generated hypothetical answer (rather than the raw query) and searches the corpus with it, bridging the lexical/semantic gap between question phrasing and answer phrasing. accuracy/quality emerging Gao et al., 2022, ‘Precise Zero-Shot Dense Retrieval without Relevance Labels’
In-Context / Retrieval-Augmented Long-Context (kNN-LM family)
kNN-LM, retrieval-augmented LM, RETRO		 Augments next-token prediction at the model/decoding level by retrieving nearest-neighbor contexts from a datastore, blending retrieved continuations into the LM’s predictions. Interpolates the LM’s output distribution with a distribution over retrieved nearest-neighbor tokens (kNN-LM), or cross-attends to retrieved chunks via dedicated layers (RETRO), injecting knowledge below the prompt level. accuracy/quality, memory, context-capacity research Khandelwal et al., 2020 (kNN-LM); Borgeaud et al., 2022 (RETRO)
IRCoT (Interleaving Retrieval with Chain-of-Thought)
IRCoT, retrieval-interleaved CoT		 Interleaves retrieval with each step of chain-of-thought reasoning for multi-step questions, using the evolving reasoning to guide subsequent retrieval and vice versa. Alternates between generating a reasoning step and retrieving documents using the latest reasoning sentence as the query, so retrieval tracks the reasoning trajectory across hops. accuracy/quality, reliability research Trivedi et al., 2023, ‘Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions’
Late Chunking
late interaction chunking		 Embeds the entire document with a long-context embedding model first, then pools token embeddings into chunk vectors, so each chunk embedding carries full-document context. Runs the transformer over the long document once and only afterward splits the contextualized token embeddings into chunks via mean-pooling, preserving cross-chunk context without per-chunk LLM calls. accuracy/quality, cost/tokens emerging Günther et al., 2024, ‘Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models’ (Jina AI)
Maximal Marginal Relevance retrieval
MMR, diversity-aware retrieval		 Selects retrieved passages that balance relevance to the query against novelty relative to already-selected passages, reducing redundancy in the context. Greedily picks each next document to maximize a weighted trade-off between query similarity and dissimilarity to the already-chosen set, diversifying the retrieved evidence. accuracy/quality, cost/tokens production-standard Carbonell & Goldstein, 1998, ‘The Use of MMR for Reordering Documents’; standard option in modern RAG retrievers
Multi-Query Retrieval
query fan-out, multi-query expansion, RAG-Fusion		 Generates several diverse paraphrases of a single query, retrieves for each, and fuses the results to improve recall and robustness to query phrasing. An LLM produces multiple query variants, runs retrieval per variant, and combines the candidate sets (often with RRF) before re-ranking or generation. accuracy/quality, reliability production-standard RAG-Fusion (Raudaschl, 2023); LangChain MultiQueryRetriever
Query Decomposition
sub-question decomposition, multi-query, least-to-most retrieval		 Breaks a complex or multi-hop question into simpler sub-queries that are retrieved and answered independently, then composed into a final answer. An LLM splits the query into sub-questions, retrieves evidence per sub-question, and aggregates the partial answers, improving coverage on compositional/multi-hop questions. accuracy/quality, reliability production-standard Common in multi-hop RAG; cf. least-to-most prompting (Zhou et al., 2022) and decomposed retrieval pipelines
Query Expansion
query augmentation, pseudo-relevance feedback, Query2Doc		 Enriches the query with additional terms, synonyms, or LLM-generated context to broaden recall over the corpus. Appends generated or feedback-derived terms (e.g. an LLM expansion or pseudo-relevance-feedback terms) to the query so the retriever matches more relevant passages. accuracy/quality production-standard Wang et al., 2023, ‘Query2doc: Query Expansion with Large Language Models’
Query Rewriting
query transformation, Rewrite-Retrieve-Read		 Reformulates the user’s raw query into one or more retrieval-optimized queries before search, improving alignment with how relevant documents are phrased. An LLM (often a small trainable rewriter) rewrites the query, optionally guided by reinforcement signals from downstream answer quality, then the rewritten query drives retrieval. accuracy/quality, reliability production-standard Ma et al., 2023, ‘Query Rewriting for Retrieval-Augmented Large Language Models’
RankGPT / LLM listwise reranking Using the LLM itself to rerank retrieved passages via a listwise permutation prompt rather than a separate cross-encoder. Sliding-window listwise prompts ask the LLM to output a permutation of candidate doc ids by relevance, optionally distilled into a smaller reranker. accuracy/quality emerging Sun et al., 2023 (Is ChatGPT Good at Search? / RankGPT)
Reciprocal Rank Fusion
RRF, rank fusion		 A parameter-light method to merge multiple ranked retrieval lists (e.g. from different retrievers or query variants) into a single robust ranking. Scores each document by summing 1/(k + rank) across all input lists, rewarding items ranked highly by several retrievers without needing score calibration. accuracy/quality, reliability production-standard Cormack et al., 2009, ‘Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods’
REPLUG
retrieve-then-ensemble, black-box LM retrieval		 Augments a frozen black-box LLM by retrieving documents and ensembling the model’s output distributions across them, optionally tuning the retriever to the LM’s likelihood signal. Prepends each retrieved document separately, computes the LM’s next-token distribution per document, and marginalizes them weighted by retrieval scores; the retriever is trained to match LM perplexity gains. accuracy/quality research Shi et al., 2023, ‘REPLUG: Retrieval-Augmented Black-Box Language Models’
Retrieval Routing
query routing, adaptive RAG routing, retriever selection		 Routes each query to the most appropriate retrieval source, strategy, or no-retrieval path based on query characteristics or estimated complexity. A classifier or LLM router predicts query type/complexity and dispatches to the matching index, tool, or pipeline (e.g. simple direct answer vs single-step vs multi-step retrieval). accuracy/quality, cost/tokens, latency emerging Adaptive-RAG, Jeong et al., 2024, ‘Adaptive-RAG: Learning to Adapt Retrieval-Augmented LLMs through Question Complexity’
Retrieval-Augmented Fine-Tuning
RAFT, RFT, best-of-n distillation		 Fine-tunes the LLM to read retrieved documents that include distractors, teaching it to cite the relevant ones and ignore irrelevant passages in the RAG setting. Trains on question + golden documents + distractor documents with chain-of-thought answers, so the model learns domain-specific reading and robustness to imperfect retrieval.
↔ also: Training & adaptation for agentic capability		 accuracy/quality, reliability research Zhang et al., 2024, ‘RAFT: Adapting Language Model to Domain Specific RAG’
Retrieval-Augmented Generation
RAG, naive RAG, vanilla RAG		 A pattern that retrieves relevant documents from an external corpus and conditions the LLM’s generation on them, so the model answers from fetched evidence rather than parametric memory alone. Embeds the query, runs nearest-neighbor search over a vector index of chunked documents, and concatenates the top-k passages into the prompt context before generation. accuracy/quality, reliability, cost/tokens, context-capacity production-standard Lewis et al., 2020, ‘Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks’
Self-Querying Retrieval (metadata-filtered retrieval)
self-query retriever, metadata filtering, structured query construction		 Has the LLM translate a natural-language query into both a semantic search string and a structured metadata filter, combining vector search with attribute constraints. An LLM parses the query into a (query, filter) pair against a known metadata schema, so retrieval applies exact filters (date, author, type) alongside embedding similarity. accuracy/quality, reliability, latency production-standard LangChain Self-Query Retriever pattern, 2023
Self-RAG
self-reflective RAG		 An LLM trained to decide on demand whether to retrieve, then critique its own retrieved passages and generations using special reflection tokens. The model emits ‘retrieve’ and ‘critique’ (relevance/support/usefulness) reflection tokens during decoding, enabling adaptive retrieval and self-assessment that can gate or revise output. accuracy/quality, reliability, cost/tokens research Asai et al., 2023, ‘Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection’
Sentence-Window / Parent-Document Retrieval
small-to-big retrieval, parent-child chunking, auto-merging retrieval		 Retrieves on small precise units (sentences or child chunks) but returns the larger surrounding window or parent chunk to the LLM for fuller context. Indexes fine-grained chunks for accurate matching while linking each to a larger parent/window that is substituted in at generation time, decoupling retrieval granularity from context granularity. accuracy/quality, context-capacity production-standard LlamaIndex Sentence-Window / LangChain Parent-Document Retriever patterns, 2023-2024

8. KV cache & prefix reuse

The attention KV cache is the agent’s hottest reusable asset. These methods avoid recomputing it: reuse shared prefixes, page and offload it, share it across heads/layers/agents, quantize it, or evict the least-useful spans. fak’s deep treatment: agentic caching SOTA and the addressable KV cache with bit-exact mid-run eviction (max|Δ|=0).

25 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
Cache-aware load balancing / routing
KV-cache-aware routing, prefix-aware routing, sticky routing		 A request router that directs requests to the serving replica most likely to already hold the relevant KV prefix, maximizing cache hit rate across a fleet. The router tracks (or approximates) which prefixes each instance has cached and routes by longest-prefix affinity rather than pure round-robin, while balancing load. latency, throughput, cost/tokens emerging Preble (Srivatsa et al., 2024); SGLang / vLLM production-stack cache-aware routing
CacheBlend (segment-level / non-prefix KV reuse)
non-prefix caching, modular KV fusion		 A technique that reuses precomputed KV for arbitrary text chunks even when they are not a contiguous prefix (e.g., multiple retrieved RAG passages concatenated in different orders). It loads cached KV for each chunk and selectively recomputes only the small fraction of tokens whose attention is most affected by cross-chunk context, restoring accuracy at a fraction of full recompute cost. latency, cost/tokens, throughput emerging Yao et al., CacheBlend, 2024
Cascade / multi-tier KV inference (CascadeAttention)
cascade inference, shared-prefix cascade, FlashInfer cascade		 An attention execution that splits a shared prefix’s KV (computed once, kept in fast memory) from per-request suffix KV, running two attention passes and merging them. It computes attention over the large shared prefix with a memory-efficient kernel and over the small unique suffix separately, then combines via log-sum-exp merging, avoiding redundant prefix work. throughput, latency, memory emerging Ye et al., Cascade Inference / FlashInfer, 2024
ChunkAttention / prefix-aware fused attention
ChunkAttention, prefix-aware KV kernel		 An attention kernel that detects and batches shared KV prefixes across requests using a prefix-tree of KV chunks, then fuses the shared-prefix attention computation. KV is broken into chunks organized in a prefix tree; the kernel runs a single batched two-phase attention over shared chunks and per-request chunks, removing duplicated memory traffic. throughput, memory, latency research Ye et al., ChunkAttention, ACL 2024
Context caching for agent / tool-loop reuse
agent prefix reuse, tool-schema caching, trajectory KV reuse		 Applying prefix/KV caching specifically to the agent loop, where a long stable prefix (system instructions, tool definitions, prior turns) is reused across every step of a multi-turn trajectory. Keeping the agent’s stable context as a cached prefix (provider cache or local KV) so each tool-call iteration only prefills the new appended observation rather than the entire growing transcript. cost/tokens, latency, throughput emerging Agent framework caching guidance (e.g., Anthropic/OpenAI agent docs, 2024); common practice in agent infra
Cross-layer KV sharing (YOCO / CLA)
YOCO, Cross-Layer Attention, layer-shared KV		 Architectural schemes that share or reuse KV cache across transformer layers so the cache is stored once (or for few layers) rather than per layer. Later layers attend to KV computed at an earlier (or single global) layer, eliminating per-layer KV storage and roughly dividing cache size by the sharing factor. memory, context-capacity, throughput emerging Sun et al., You Only Cache Once (YOCO), 2024; Brandon et al., Reducing Transformer KV Cache Size with Cross-Layer Attention (CLA), 2024
Disaggregated KV transfer / KV-aware disaggregation
prefill-decode disaggregation KV transfer, NVIDIA Dynamo, Mooncake		 An architecture that separates prefill and decode onto different GPU pools and ships the computed KV cache between them, with a global KV store enabling cross-instance prefix reuse. Prefill workers compute KV and stream it over fast interconnect to decode workers, while a distributed KV cache pool lets any instance reuse another’s cached prefixes. throughput, latency, cost/tokens production-standard Mooncake (Qin et al., 2024); NVIDIA Dynamo (2025)
Distributed / global KV cache pool
shared KV pool, global prefix store, KV cache cluster		 A cluster-wide shared store of KV blocks (across GPUs and tiers) that lets any worker reuse prefixes computed by any other worker. KV blocks are content-addressed and registered in a shared index (often RDMA/NVLink-backed), so a cache lookup spans the whole fleet rather than a single instance’s local cache. throughput, cost/tokens, memory emerging Mooncake distributed KVCache (Qin et al., 2024); NVIDIA Dynamo KV manager
DuoAttention (retrieval vs streaming head split) Classifying attention heads into ‘retrieval heads’ (need full KV) and ‘streaming heads’ (need only recent + sink), keeping the full cache only for the former. Only retrieval heads retain the full KV cache; streaming heads use a small recent+sink window, sharply cutting long-context KV memory and latency. memory, latency, context-capacity emerging Xiao et al., 2024 (DuoAttention)
FastGen (adaptive KV compression)
H2O, Heavy-Hitter Oracle, Scissorhands		 An adaptive, head-aware KV compression method that assigns different retention policies to different attention heads based on their observed structure. A lightweight profiling pass classifies each head (e.g., local, special-token, punctuation, broad) and applies the cheapest policy that preserves that head’s behavior, compressing the cache per-head.
↔ also: Context-window management & compression		 memory, throughput, context-capacity, latency research Ge et al., Model Tells You What to Discard: Adaptive KV Cache Compression (FastGen), ICLR 2024
Grouped-Query Attention
GQA, MQA, shared KV heads		 An attention variant that interpolates between multi-head and multi-query attention by sharing each K/V head across a group of query heads. Uses G key/value heads (1 < G < num_heads) so the KV cache is smaller than MHA while retaining more quality than MQA, with a quick uptraining to convert an MHA checkpoint.
↔ also: Inference / serving optimizations beneath agents		 memory, throughput, context-capacity, latency, accuracy/quality production-standard Ainslie et al., 2023 (GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints)
Heavy KV merging (CaM / KVMerger)
cache merging, KV merging, CaM		 Compression methods that, instead of evicting tokens outright, merge soon-to-be-evicted KV entries into retained ones to preserve their information. Tokens marked for eviction are weighted-merged (by attention or similarity) into surviving KV slots, reducing cache size while limiting the accuracy loss of pure dropping. memory, accuracy/quality, context-capacity research Zhang et al., CaM: Cache Merging for Memory-efficient LLMs Inference, ICML 2024
Hydragen (shared-prefix attention decomposition)
Hydragen, high-throughput shared-prefix attention		 A high-throughput attention method for large batches that share a common prefix, decomposing attention into prefix and suffix parts to enable tensor-core-efficient batching. It batches the shared-prefix attention across all sequences into dense matmuls and handles unique suffixes separately, then combines them, turning memory-bound work into compute-efficient work. throughput, latency research Juravsky et al., Hydragen: High-Throughput LLM Inference with Shared Prefixes, 2024
KV cache deduplication / content-addressed block sharing
KV dedup, content-hash KV sharing, copy-on-write KV		 Eliminating duplicate KV storage by sharing identical KV blocks (same tokens, same position) across requests via content addressing and copy-on-write. Each KV block is hashed by its token content and position; identical blocks point to one physical copy until a write diverges (copy-on-write), collapsing redundant storage. memory, throughput production-standard vLLM block hashing / PagedAttention CoW (Kwon et al., 2023); vLLM prefix caching
KV cache offloading (CPU / disk tiering)
LMCache, KV offload, CPU/disk KV tier		 Extending KV cache capacity beyond GPU HBM by storing warm/cold KV blocks in CPU DRAM, local disk, or remote storage and fetching them back on demand. A tiered cache manager evicts cold KV blocks from GPU to slower tiers and reloads (or recomputes) them when a matching prefix recurs, trading bandwidth for far larger effective cache. memory, cost/tokens, context-capacity production-standard LMCache (project, 2024); NVIDIA Dynamo KV offload
KV cache quantization (KIVI)
KV cache quantization, KVQuant, KIVI		 Compressing the stored KV cache to very low bit-width (e.g., 2-bit) to shrink memory footprint while preserving generation quality. It quantizes keys per-channel and values per-token (asymmetric) to exploit their differing outlier distributions, keeping a small residual of recent tokens in full precision.
↔ also: Context-window management & compression; Inference / serving optimizations beneath agents		 memory, context-capacity, throughput emerging Liu et al., KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, 2024
MInference (dynamic sparse prefill attention) Accelerating long-context prefill by detecting per-head sparse attention patterns (A-shape, vertical-slash, block-sparse) and computing only those. Offline-classified head patterns drive a sparse attention kernel at prefill time, cutting the quadratic prefill cost on 100K+ contexts with little accuracy loss. latency, throughput, context-capacity emerging Jiang et al., 2024 (MInference 1.0)
Multi-head Latent Attention (MLA)
MLA, latent KV compression, DeepSeek MLA		 An attention variant that compresses keys and values into a low-rank latent vector cached per token, drastically reducing KV memory versus standard multi-head attention. K/V are projected down to a shared latent representation that is cached and up-projected at attention time, so only the small latent (not full per-head K/V) is stored.
↔ also: Inference / serving optimizations beneath agents		 memory, throughput, context-capacity production-standard DeepSeek-V2 (DeepSeek-AI, 2024)
PagedAttention
vLLM paged KV, vLLM KV paging		 A KV-cache memory manager that stores the cache in fixed-size non-contiguous blocks (pages), eliminating fragmentation and enabling near-zero-waste sharing of KV memory. Borrowing OS virtual-memory paging, it maps logical KV positions to physical blocks via a block table, allowing copy-on-write sharing of identical blocks across sequences (e.g., shared prefixes, parallel samples).
↔ also: Inference / serving optimizations beneath agents		 memory, throughput production-standard Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM), SOSP 2023
Prompt Cache (modular position-independent KV)
modular prompt caching, prompt schema reuse, exact-match cache		 A scheme that precomputes and reuses KV for reusable prompt modules (snippets) defined in a schema, allowing them to appear at varying positions across prompts. Position-independent KV is computed for each named module with discontinuous position-id assignment so cached modules can be spliced into different prompts without recomputation.
↔ also: Model routing, cascades & cost optimization		 latency, cost/tokens, throughput emerging Gim et al., Prompt Cache: Modular Attention Reuse for Low-Latency Inference, MLSys 2024
Provider prompt caching (OpenAI / Anthropic / Gemini)
KV prefix cache, prompt caching, RadixAttention		 Provider-side hosted caching of a request’s prefix KV state so repeated leading content (system prompt, tool schemas, few-shot examples, large documents) is billed at a steep discount and served faster on subsequent calls. The provider stores the attention KV for an already-processed token prefix keyed by its exact bytes and, on a cache hit, skips re-prefilling that prefix, charging cached-input rates instead of full input rates.
↔ also: Context-window management & compression; Tool use & function calling optimization; Model routing, cascades & cost optimization; Inference / serving optimizations beneath agents		 latency, cost/tokens, throughput, memory production-standard Anthropic prompt caching docs (2024); OpenAI prompt caching (2024); Google Gemini context caching (2024)
PyramidKV / PyramidInfer (layer-pyramidal KV budget)
PyramidKV, PyramidInfer, layer-wise KV budget allocation		 KV compression schemes that allocate a larger cache budget to lower layers and a smaller budget to higher layers, matching the pyramidal information-aggregation pattern across depth. Per-layer retention budgets decrease with depth (information funnels into fewer tokens at higher layers), so each layer keeps only its needed KV positions. memory, context-capacity, throughput research Cai et al., PyramidKV, 2024; Yang et al., PyramidInfer, 2024
Quest (query-aware KV page selection)
Quest, query-aware sparse KV		 A query-dependent sparse-attention method that, at each decoding step, selects only the most relevant KV pages to attend over, accelerating long-context decode. It keeps per-page min/max key summaries and uses the current query to estimate page criticality, loading and attending to only the top-k pages while leaving the rest in memory unused. latency, throughput, context-capacity research Tang et al., Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference, ICML 2024
Retrieval-head-aware KV eviction (RazorAttention / retrieval heads) Using the discovery that a few ‘retrieval heads’ carry long-range copying to protect their KV while compressing the rest. Identify retrieval heads (those that drive copy/induction over long range) and exempt their KV from eviction/compression, preserving long-context recall under a tight budget. memory, context-capacity research Tang et al., 2024 (RazorAttention); Wu et al., 2024 (Retrieval Heads)
StreamingLLM (attention sinks)
attention sinks, rolling KV with sink tokens		 An inference scheme enabling effectively unbounded streaming generation by keeping a small fixed set of initial ‘sink’ tokens plus a sliding window of recent KV. It retains the first few tokens (which absorb disproportionate attention as ‘sinks’) alongside a rolling recent-token window, preventing the perplexity blow-up that naive window truncation causes. memory, context-capacity, latency production-standard Xiao et al., Efficient Streaming Language Models with Attention Sinks, ICLR 2024

Part III — Action, planning & orchestration

9. Tool use & function calling optimization

An agent is only as good as its tool calls. These methods make tool use cheaper and more reliable: select the right tools from a large catalog, design schemas the model can follow, call in parallel, constrain output to valid calls, cache results, and recover from errors. Cross-cut: fak’s grammar-constrained tool-call decoding and the internal/vdso tool-result fast path.

26 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
Code generation for API composition (programmatic tool use)
program-of-thought tool use, ViperGPT-style, API graph as code		 Generate a program that calls multiple APIs/tools and intermediate values, instead of one call at a time, for complex multi-tool tasks. The model writes a script binding tool outputs to variables and passing them between calls, so multi-tool dependencies are resolved by execution rather than by re-prompting. accuracy/quality, cost/tokens, autonomy emerging Surís et al., ViperGPT, 2023; Gao et al., PAL, 2022
Code-as-Action (CodeAct)
CodeAct, code actions, executable action agents		 Have the agent express its actions as executable code (e.g., Python) rather than discrete JSON tool calls, letting one code block orchestrate many tool invocations, control flow, and data handling. Unifies the action space into a code interpreter so the LLM composes tools with loops/conditionals/variables in a single executable action, reducing round-trips. accuracy/quality, latency, cost/tokens, autonomy emerging Wang et al., ‘Executable Code Actions Elicit Better LLM Agents’ (CodeAct), 2024
Demonstration / few-shot tool-call exemplars
in-context tool examples, few-shot function calling		 Provide worked examples of correct tool calls (and tricky edge cases) in the prompt to steer the model’s tool selection and argument formatting. In-context demonstrations bias the model toward the demonstrated call format and selection logic without weight updates. accuracy/quality, reliability production-standard Brown et al., in-context learning, 2020; function-calling best-practice guides, 2023-2024
Forced / directed tool choice
tool_choice=required, function_call forcing, any/auto/none tool choice		 Constrain whether the model must call a tool, may choose, must call a specific tool, or must not call any tool on a given turn. An API parameter (auto/any/required/none/specific) biases or hard-restricts the decode toward the desired tool-calling behavior. reliability, accuracy/quality, latency production-standard OpenAI tool_choice / Anthropic tool_choice docs, 2023-2024
Gorilla / API-aware retriever finetuning
Gorilla, RAT (retriever-aware training)		 A finetuned LLM specialized for generating correct API/tool calls against large, evolving API catalogs, optionally coupled with a document retriever at inference. Trains the model on (instruction, API doc, call) triples and uses retriever-aware training so the model adapts to retrieved docs and reduces hallucinated API calls. accuracy/quality, reliability research Patil et al., ‘Gorilla: Large Language Model Connected with Massive APIs’, 2023
Irrelevance / no-call detection
abstention from tool use, relevance gating, should-not-call detection		 Train or prompt the model to recognize when no tool is needed (or no available tool fits) and answer directly instead of forcing an inappropriate call. Adds negative/irrelevance examples and a ‘none’ option so the policy learns to abstain, reducing spurious and unsafe tool invocations. accuracy/quality, reliability, cost/tokens production-standard BFCL irrelevance category (Yan et al., 2024); function-calling best practices, 2024
Lifting text/inline tool calls at the boundary
XML/inline tool-call parsing, tool-call extraction, non-native function calling		 Parse tool calls emitted as inline text (XML, JSON-in-text, or DSL) for models without native function-calling, normalizing them into structured calls at the runtime boundary. A parser at the gateway extracts the structured call from free-form generation (e.g., qwen3_coder XML), so any model can drive the tool loop. reliability, accuracy/quality production-standard SGLang/vLLM tool-call parsers (e.g., qwen3_coder, hermes), 2024
LLM-as-Compiler / LLMCompiler parallel tool orchestration Planning a DAG of tool calls up front and dispatching independent calls in parallel with a task fetcher and executor. A planner emits an inter-dependent task graph; a fetching unit resolves arguments and an executor runs independent nodes concurrently, cutting latency and token cost vs sequential ReAct. latency, cost/tokens, accuracy/quality emerging Kim et al., 2023 (An LLM Compiler for Parallel Function Calling)
Memoized / deduplicated tool execution caching
tool call memoization, idempotent-call cache, result dedup		 Cache the outputs of deterministic/idempotent tool calls keyed by (tool, args) so repeated identical calls skip re-execution. A keyed cache short-circuits the round-trip to the external tool when the same call signature recurs, returning the stored result. latency, cost/tokens, throughput production-standard Common agent-framework practice (LangChain/LlamaIndex tool caching), 2023-2024
Model Context Protocol (MCP) / tool standardization
MCP, tool server protocol, open tool standard		 An open protocol standardizing how agents discover and call external tools, resources, and prompts via interchangeable servers, decoupling tool implementations from the agent. Defines a client-server JSON-RPC interface for listing tools/resources and invoking them, so any compliant host can use any compliant tool server. autonomy, reliability production-standard Anthropic, ‘Model Context Protocol’, 2024
Parallel tool calling
parallel function calling, multi-tool calls, batched tool calls		 Allowing the model to emit multiple independent tool calls in a single turn so they can be executed concurrently rather than sequentially. The model returns a list of tool_use blocks in one response; the runtime dispatches the independent calls in parallel and feeds all results back together. latency, throughput, cost/tokens production-standard OpenAI parallel function calling, 2023; Anthropic tool use docs, 2024
ReAct
Reason+Act, reasoning and acting, thought-action-observation loop		 An agent framework that interleaves free-form reasoning (‘thoughts’) with tool/environment actions and observations in a single loop. Alternating reasoning traces and actions lets the model plan, ground reasoning in external observations, and recover from errors, making it the backbone of tool-using agents.
↔ also: Prompting & reasoning strategies		 accuracy/quality, reliability, autonomy production-standard Yao et al., 2022, ‘ReAct: Synergizing Reasoning and Acting in Language Models’
Reinforcement learning for tool use
ToolRL, RL tool-use training, execution-feedback RL		 Train tool-using policies with reinforcement learning where rewards come from tool-execution outcomes or task success rather than imitation alone. Uses outcome/format rewards from real tool execution to optimize when, which, and how the model calls tools via policy-gradient methods. accuracy/quality, reliability, autonomy research Qian et al., ToolRL, 2025; Feng et al., ReTool, 2025
Retry with backoff / circuit breaking for flaky tools
tool retry policy, exponential backoff, circuit breaker		 Operational policies that retry transient tool failures with backoff and trip a circuit breaker for persistently failing tools to keep the agent loop stable. The runtime wraps tool calls with retry/backoff and failure-rate thresholds that disable or route around a degraded tool. reliability, latency production-standard Standard distributed-systems practice applied to agent tool calls, 2023-2024
Sandboxed code-action execution
secure code interpreter, tool sandboxing, containerized actions		 Execute code-as-action and tool side effects inside an isolated, resource-limited sandbox to contain errors and untrusted behavior. Runs generated code/tool calls in a constrained container or VM with restricted filesystem/network and timeouts, returning captured stdout/errors as observations. reliability, autonomy production-standard Code-interpreter / E2B / sandbox patterns, 2023-2024
Speculative / predictive tool prefetching
tool prefetch, speculative tool execution, anticipatory calls		 Begin executing likely-needed tool calls before the model formally requests them, or speculatively run candidate calls, to hide latency. A predictor anticipates the next tool from context and kicks off execution in the background, discarding wasted work if the prediction is wrong. latency, throughput research Emerging agent-serving optimization; analogous to speculative execution, 2024-2025
Tool masking / dynamic gating
tool gating, allowed-tools constraint, state-dependent tool masking		 Restrict, at each step, which tools the model is permitted to call based on agent state, permissions, or workflow phase. The runtime supplies or logit-masks only the currently-valid tools (or rejects out-of-policy calls), shrinking the choice space and enforcing safety/sequencing. accuracy/quality, reliability, cost/tokens production-standard Agent-framework practice; OpenAI/Anthropic tool_choice controls, 2024
Tool pruning / fewer-tools curation
tool deduplication, minimal toolset, tool consolidation		 Reduce the number of exposed tools by merging overlapping ones, removing rarely-used ones, or designing a small high-coverage toolset. Curates the catalog so fewer, more orthogonal tools are presented, shrinking the selection space and the schema token cost while improving disambiguation. accuracy/quality, cost/tokens, context-capacity production-standard Practitioner guidance (Anthropic/OpenAI tool-design best practices), 2024
Tool Retrieval / RAG-based tool selection
tool RAG, dynamic tool retrieval, retrieval-augmented tool selection		 At scale (hundreds to thousands of tools), retrieve only the top-k most relevant tool schemas for a query and inject just those into the prompt instead of the full tool catalog. Embeds the user query and tool descriptions into a vector space (or uses a learned retriever) and selects the nearest tools so the model sees a small, relevant subset. accuracy/quality, cost/tokens, latency, context-capacity production-standard Patil et al., Gorilla, 2023; Qin et al., ToolLLM/ToolBench, 2023
Tool schema / documentation design
tool description engineering, function-spec design, JSON-schema tooling		 Authoring clear tool names, parameter descriptions, types, enums, examples, and constraints so the model selects and fills tools correctly. Encodes intent and constraints directly into the function schema (descriptive names, required/optional flags, enums, in-line examples) that the model conditions on at call time. accuracy/quality, reliability, cost/tokens production-standard Anthropic & OpenAI function-calling / tool-use documentation, 2023-2024
Tool-call validation / argument schema enforcement
argument validation, post-hoc schema check, guardrail validation		 Validate generated tool arguments against the schema (types, ranges, required fields, enums) before execution, rejecting or repairing invalid calls. A validator parses the emitted arguments against the declared JSON schema and either auto-repairs, re-prompts, or blocks malformed calls. reliability, accuracy/quality production-standard Pydantic/JSON-schema validation in agent frameworks (e.g., Instructor, Guardrails), 2023-2024
Tool-result compression / truncation
observation compression, result summarization, tool output trimming		 Compress, summarize, paginate, or truncate large tool outputs before they re-enter the context so they don’t blow the context budget. Applies extractive/abstractive summarization, head+tail windowing, pointers/handles, or schema-projection to the tool result prior to feeding it back to the model. cost/tokens, context-capacity, latency production-standard Agent-framework practice; LLMLingua-style compression (Jiang et al., 2023)
Tool-use benchmarking / capability gating
BFCL (Berkeley Function-Calling Leaderboard), API-Bank, ToolBench eval		 Standardized benchmarks that measure tool-selection accuracy, argument correctness, parallel/sequential calling, and irrelevance detection, used to gate model/prompt choices. Scores models on held-out tool-calling tasks (including ‘should not call’) so practitioners select models and prompts empirically rather than by intuition. accuracy/quality, reliability production-standard Berkeley Function-Calling Leaderboard (Yan et al., 2024); Li et al., API-Bank, 2023; tau-bench (Yao et al., 2024)
Toolformer A self-supervised method that teaches a model to decide which APIs to call, when, and with what arguments by inserting API calls into its own training text. Samples candidate API calls in-context, keeps only those whose results reduce LM loss on subsequent tokens, then fine-tunes on the filtered, API-annotated corpus.
↔ also: Training & adaptation for agentic capability		 accuracy/quality, autonomy research Schick et al., 2023, ‘Toolformer: Language Models Can Teach Themselves to Use Tools’
ToolkenGPT (tools as learned tokens)
toolkits, tool namespaces, meta-tools		 Representing each tool as a special ‘toolken’ embedding so the model can call tools by predicting a token, without per-tool fine-tuning of the whole model. Tool embeddings are added to the vocabulary; predicting a toolken triggers the call and switches to argument-filling mode, enabling massive tool sets via plug-in embeddings. accuracy/quality, context-capacity, cost/tokens research Hao et al., 2023 (ToolkenGPT)
ToolLLM / ToolBench (large-scale tool-use instruction tuning)
ToolLLM, ToolBench, DFSDT		 A framework and dataset for teaching LLMs to use 16k+ real-world REST APIs, including a depth-first search-based decision tree (DFSDT) for multi-step tool reasoning. Generates multi-tool instruction data with ChatGPT and trains models to plan and call tools, using DFSDT to expand and backtrack over candidate tool-call paths. accuracy/quality, autonomy, reliability research Qin et al., ‘ToolLLM’, 2023

10. Planning algorithms & world models for agents

Decide what to do before doing it. These methods span hybrid classical planners, tree-search over actions, and plan caching — separating deliberation from execution so the agent commits fewer, better-chosen steps.

26 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
AdaPlanner
Adaptive Planner		 A closed-loop LLM agent that adaptively refines its self-generated plan in response to environment feedback, supporting both in-plan and out-of-plan refinement. Generates a program-style plan, then uses two refiners that correct mistakes mid-trajectory and inject skills/code-based feedback, mitigating hallucination via a skill-discovery mechanism. accuracy/quality, reliability, autonomy research Sun et al., 2023, ‘AdaPlanner: Adaptive Planning from Feedback with Language Models’
ADaPT (As-Needed Decomposition and Planning)
ADaPT		 Recursively decomposes a task only when the executor fails, adapting decomposition depth to task complexity and the LLM’s capability. Interleaves a planner that splits a subtask into sub-subtasks on demand with an executor, avoiding over-decomposition of easy tasks while handling hard ones. accuracy/quality, cost/tokens, reliability research Prasad et al., 2023, ‘ADaPT: As-Needed Decomposition and Planning with Language Models’
Code as Policies
CaP		 Uses LLM-written code as the policy: the model emits executable robot-control programs that invoke perception and control APIs to realize a plan. Leverages code generation to express plans with precise logic, parameterization, and feedback handling, executing directly against an API layer. reliability, accuracy/quality, autonomy research Liang et al., 2022, ‘Code as Policies: Language Model Programs for Embodied Control’
Describe, Explain, Plan and Select (DEPS)
DEPS		 An interactive planning approach for open-world multi-task agents that describes the current plan-execution state, explains feedback on failures, replans, and selects the nearest feasible subgoal. A selector ranks parallel candidate subgoals by estimated proximity/feasibility and the describe-explain loop corrects long-horizon plans against environment feedback. accuracy/quality, reliability, autonomy research Wang et al., 2023, ‘Describe, Explain, Plan and Select: Interactive Planning with LLMs Enables Open-World Multi-Task Agents’
Dynamic replanning
Online replanning, Closed-loop replanning		 Revising or regenerating the agent’s plan during execution whenever observations diverge from expectations or a step fails, rather than committing to a fixed open-loop plan. Monitors execution feedback and triggers partial or full plan regeneration on deviation, turning open-loop planning into closed-loop control. reliability, autonomy, accuracy/quality production-standard Inner Monologue, Huang et al., 2022; general closed-loop agent control
Generative-agent reflection-and-planning memory
Generative Agents, Reflect-and-plan memory stream		 Believable agents that synthesize a memory stream into higher-level reflections and daily/hierarchical plans which guide and are recursively refined during behavior. Periodically reflects over salient memories to form abstractions, then generates broad-to-detailed plans that are reactively adjusted as new observations arrive. autonomy, accuracy/quality, context-capacity research Park et al., 2023, ‘Generative Agents: Interactive Simulacra of Human Behavior’
Goal decomposition / task decomposition
Subgoal decomposition, Task splitting		 Breaking a complex high-level goal into a set or sequence of smaller, individually solvable subgoals that the agent tackles one at a time. Reduces a hard reasoning/action problem into easier subproblems whose solutions compose, lowering per-step difficulty and error rate. accuracy/quality, reliability, context-capacity production-standard Least-to-Most Prompting, Zhou et al., 2022; HuggingGPT, Shen et al., 2023
Hierarchical Task Network planning
HTN, HTN planning		 Planning by recursively decomposing high-level compound tasks into ordered subtasks via predefined methods until only primitive executable actions remain. Uses a library of task-decomposition methods to expand abstract goals into concrete action sequences, constraining search to author-validated structures. reliability, accuracy/quality production-standard Erol, Hendler & Nau, 1994 (SHOP/SHOP2 lineage); applied to LLM agents
LATS (Language Agent Tree Search)
LATS, language agent tree search, MCTS over agents		 A framework unifying reasoning, acting, and planning by running Monte Carlo Tree Search over language-agent trajectories with environment feedback and self-reflection. Uses MCTS selection/expansion/evaluation/backpropagation where the LLM proposes actions, a value function scores nodes, and reflections from failed branches improve future rollouts.
↔ also: Multi-agent orchestration patterns		 accuracy/quality, reliability, autonomy research Zhou et al., 2023, ‘Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models’
LLM+P (LLM + Classical Planner)
LLMP, LLM plus Planner		 A hybrid that translates a natural-language task into a formal PDDL problem, hands it to a sound classical planner, and translates the returned plan back to natural language. Offloads the combinatorial search to a provably correct symbolic planner (e.g., Fast Downward) so the LLM only does NL-to-PDDL translation, not the planning itself. accuracy/quality, reliability research Liu et al., 2023, ‘LLM+P: Empowering Large Language Models with Optimal Planning Proficiency’
LLM-DP (LLM Dynamic Planner)
LLM-DP		 A neuro-symbolic embodied agent that combines an LLM with a symbolic planner, using the LLM to hypothesize world state and the planner to produce valid action sequences. The LLM proposes plausible PDDL world states/goals from observations and a classical planner solves them, leveraging the LLM’s commonsense plus planner soundness for fast valid plans. accuracy/quality, latency, reliability research Dagan et al., 2023, ‘Dynamic Planning with a LLM’
Monte Carlo Tree Search for LLM planning
MCTS planning, MCTS-for-LLM		 Applying the classic selection-expansion-simulation-backpropagation search loop to LLM-generated action/reasoning trees to find high-value trajectories. Balances exploration and exploitation via UCT over LLM-proposed branches, using a learned or LLM value estimate as the rollout reward to prune low-value paths. accuracy/quality, reliability research Kocsis & Szepesvari, 2006 (UCT); applied to LLMs in e.g. Hao et al., 2023 (RAP)
Monte Carlo Tree Self-refine / AlphaLLM-style self-improvement
AlphaLLM, MCTSr		 Couples MCTS with LLMs in a self-improving loop, using tree search to generate high-quality trajectories that train a better policy/value, then re-searching. Tree search with critic-based value estimation produces refined solution trajectories used as training/self-correction signal, iteratively bootstrapping planning quality. accuracy/quality, reliability research Tian et al., 2024, ‘Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (AlphaLLM)’; Zhang et al., 2024 (MCTSr)
Plan caching / plan templates
Plan reuse, Plan library, Cached plans		 Storing previously generated successful plans (or parameterized plan templates) and retrieving/reusing them for similar future tasks instead of replanning from scratch. Matches a new task to a cached plan by similarity and instantiates the template, amortizing planning cost and latency over repeated task types. latency, cost/tokens, reliability emerging Case-based planning lineage (Hammond, 1989); LLM agent plan-reuse patterns
Plan-and-Solve Prompting
PS, PS+		 A zero-shot prompting strategy that asks the model to first devise a plan dividing the task into subtasks, then carry out the subtasks step by step. Explicitly separates the ‘understand and plan’ phase from the ‘execute’ phase in a single prompt, reducing missing-step and calculation errors versus plain chain-of-thought.
↔ also: Prompting & reasoning strategies		 accuracy/quality, reliability emerging Wang et al., 2023, ‘Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning’
Planner-Executor (Plan-and-Execute)
Plan-and-Solve, Plan-and-Execute, decompose-then-act		 A two-role pattern where one agent produces a multi-step plan up front and a separate executor agent (or loop) carries out each step, optionally re-planning when steps fail. Separating planning from execution lets a strong model reason globally once, then delegate cheap per-step execution, reducing redundant re-reasoning and bounding wandering.
↔ also: Multi-agent orchestration patterns		 accuracy/quality, cost/tokens, reliability, autonomy, latency production-standard Wang et al., ‘Plan-and-Solve Prompting’, 2023; LangChain Plan-and-Execute agents
ProgPrompt (programmatic planning)
ProgPrompt		 Represents the plan as executable program code (functions and control flow) generated by the LLM, where each statement is a primitive action or assertion. Prompts the LLM with Pythonic program structure so plans inherit loops/conditionals/assertions, producing structured, situated, and verifiable action sequences. accuracy/quality, reliability research Singh et al., 2022, ‘ProgPrompt: Generating Situated Robot Task Plans using Large Language Models’
RAP (Reasoning via Planning)
Monte Carlo Tree Search reasoning, MCTS-LLM, RAP		 Repurposes the LLM as both a world model and a reasoning agent, doing principled MCTS planning over a tree of reasoning states with the LLM predicting next states and rewards. The LLM predicts the next world state given an action and self-evaluates rewards, and MCTS uses these to search a state-action reasoning tree toward high-reward solutions.
↔ also: Test-time compute scaling & search over reasoning		 accuracy/quality, reliability research Hao et al., 2023, ‘Reasoning with Language Model is Planning with World Model’
Reflexion-for-planning successor — AdaPlanner closed-loop refinement (explicit in-plan + out-of-plan) An LLM planner that refines its plan from environment feedback using both in-plan (continue) and out-of-plan (revise) corrections with code-style plans. Skill discovery plus dual feedback channels let the agent patch the current plan locally or replan globally, raising long-horizon success with few samples. accuracy/quality, autonomy research Sun et al., 2023 (AdaPlanner)
ReWOO (Reasoning WithOut Observation)
ReWOO, Reasoning WithOut Observation		 A plan-then-work architecture that produces the entire reasoning blueprint (a list of interdependent tool calls with variable substitution) up front, separating reasoning from observation/tool execution. A Planner emits a full plan with placeholder variables, a Worker fills them by executing tools once, and a Solver composes the final answer, removing repeated context re-feeding per ReAct step.
↔ also: Prompting & reasoning strategies		 cost/tokens, latency, accuracy/quality emerging Xu et al., 2023, ‘ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models’
SayCan (grounded affordance planning)
SayCan, Do As I Can, Not As I Say		 Grounds LLM-proposed actions in a robot’s real capabilities by combining the LLM’s task-relevance score with a learned affordance value for each skill. Selects the next skill by multiplying the LLM ‘say’ probability with a value-function ‘can’ affordance, ensuring the plan stays within feasible, executable actions. reliability, accuracy/quality research Ahn et al., 2022, ‘Do As I Can, Not As I Say: Grounding Language in Robotic Affordances’
ToolChain*
ToolChain-star		 An efficient tree-search-based planning algorithm for tool-use agents that navigates the action space with an A-style heuristic over decision trees of API calls. *Combines tree search with task-specific cost functions (a heuristic g+h) to expand the most promising tool-call branches first, pruning expensive exhaustive exploration. accuracy/quality, latency, cost/tokens research Zhuang et al., 2023, ‘ToolChain: Efficient Action Space Navigation in Large Language Models with A Search’
Tree-of-Mixed-Thought / Thought-of-Search and learned planning programs
Thought of Search, ToS		 Has the LLM author the planning components themselves (successor function and goal test) as code, then runs a sound classical search algorithm with those components. Shifts the LLM from generating plans to generating the search machinery, so a complete/sound search procedure guarantees valid plans across all instances. reliability, accuracy/quality, cost/tokens research Katz et al., 2024, ‘Thought of Search: Planning with Language Models Through The Lens of Efficiency’
Tree-of-Thoughts agent search successor — ToolChain* / A-guided tool-action search (explicit A) Treating multi-step tool action selection as A* search over an action tree with a learned/heuristic cost-to-go to prune the branching factor. An admissible-ish heuristic scores partial plans so the agent expands only promising tool-action paths, cutting cost vs exhaustive tree search. accuracy/quality, cost/tokens research Zhuang et al., 2023 (ToolChain*)
Tree-Planner
Tree Planner		 A plan-sample-then-search method that first samples a set of candidate whole plans, aggregates them into a single action tree, then grounds the tree by decision-making during execution. Decouples expensive plan generation (sample many plans once) from cheap tree-grounded decision-making, cutting redundant LLM calls and correcting mistakes via the merged action tree. cost/tokens, accuracy/quality, reliability research Hu et al., 2023, ‘Tree-Planner: Efficient Close-loop Task Planning with Large Language Models’
World-model / look-ahead simulation
Model-based look-ahead, Mental simulation		 Using a learned or LLM-internalized model of environment dynamics to simulate the consequences of candidate actions before committing to one. Predicts future states/rewards for hypothetical action rollouts and selects actions by their simulated outcomes, enabling planning without real-environment trial-and-error. accuracy/quality, reliability, cost/tokens research Ha & Schmidhuber, 2018 (‘World Models’); LLM instantiations e.g. RAP, Hao et al., 2023

11. Multi-agent orchestration patterns

Decompose work across several agents — a planner and workers, a debate, a pipeline, a routed swarm. The win is parallelism, specialization, and adversarial checking; the cost is coordination overhead and duplicated context. fak’s scaling laws of agents frames when fan-out pays.

27 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
Agent-as-Tool
agent-as-function, sub-agent tool, callable agent		 An entire agent is exposed to a calling agent as if it were a single tool/function, hiding its internal multi-step reasoning behind one invocation. Encapsulating a sub-agent behind a tool interface isolates its token-heavy internal context from the caller, preserving the parent’s context budget while reusing the sub-agent’s capability. context-capacity, cost/tokens, reliability, autonomy production-standard OpenAI Agents SDK ‘agents as tools’; Anthropic subagents; LangGraph subgraph-as-tool
AutoGen conversable-agent framework (programmable multi-agent conversation) A framework pattern of conversable agents that interleave LLM, tool, and human turns under programmable termination/auto-reply rules. Agents exchange messages with customizable reply functions and group-chat managers, composing flexible multi-agent workflows from a small primitive set. autonomy, accuracy/quality production-standard Wu et al., 2023 (AutoGen)
AutoGPT-style Autonomous Loop
autonomous agent loop, self-directed goal agent, BabyAGI		 A single autonomous agent that, given a high-level goal, iteratively generates tasks, executes them with tools/memory, and creates follow-up tasks until the goal is met. A self-generating task queue plus a perceive-plan-act-memory loop lets the agent pursue open-ended goals without per-step human direction. autonomy emerging AutoGPT (Significant Gravitas, 2023); BabyAGI (Nakajima, 2023)
Blackboard Architecture
shared-memory coordination, blackboard systems		 Agents (knowledge sources) communicate indirectly by reading from and writing to a shared structured workspace (the blackboard) rather than messaging each other directly. A shared mutable state plus a control component lets heterogeneous specialists opportunistically contribute when the current state matches their expertise, decoupling agents. reliability, context-capacity, autonomy emerging Hayes-Roth, ‘A Blackboard Architecture for Control’, 1985; HEARSAY-II (Erman et al., 1980)
ChatDev (Communicative Software Agents)
chat-powered software dev, phased multi-agent dev		 A virtual software company of communicative agents that collaborate through structured chats across design, coding, testing, and documentation phases. Chaining phase-specific dual-agent chats with a chat-chain and self-reflection keeps each phase focused and propagates validated decisions forward. accuracy/quality, reliability, autonomy research Qian et al., ‘ChatDev: Communicative Agents for Software Development’, 2023
ChatEval-style role-diverse multi-agent evaluation (referee-team) A debate-driven evaluator where multiple role-conditioned agents discuss before scoring, improving alignment with human judgments over single LLM-judge. Heterogeneous evaluator personas exchange critiques across rounds (one-by-one / simultaneous-talk) and converge to a verdict, reducing single-judge bias. accuracy/quality, reliability research Chan et al., 2023 (ChatEval)
Contract-Net / Auction Allocation
contract net protocol, CNP, bidding/auction task allocation		 A manager announces a task, candidate worker agents bid based on their fitness/cost, and the manager awards the contract to the best bidder. Market-style bidding dynamically matches tasks to the most capable or cheapest available agent, optimizing allocation without static assignment. cost/tokens, throughput, reliability, autonomy research Smith, ‘The Contract Net Protocol’, 1980 (applied to LLM agent task markets)
Cross-Model Mixture / Model Diversity Ensemble
heterogeneous-model panel, multi-LLM ensemble, model committee		 An orchestration that deliberately combines agents backed by different underlying models so their complementary strengths and uncorrelated errors are pooled. Using diverse base models reduces correlated failure modes, so aggregation (vote, debate, or synthesis) yields more reliable answers than any single model. accuracy/quality, reliability research Jiang et al., ‘LLM-Blender: Ensembling LLMs with Pairwise Ranking and Generative Fusion’, 2023
Dynamic / Self-Organizing Agent Topology
DyLAN, GPTSwarm, learnable agent graphs		 Frameworks that represent the multi-agent system as a graph and automatically learn or prune which agents participate and how they connect for a given task. Treating the agent communication topology as an optimizable graph (importance scoring or graph optimization) removes low-value agents and edges, cutting cost while preserving quality. cost/tokens, accuracy/quality, throughput research Liu et al., ‘DyLAN: Dynamic LLM-Agent Network’, 2023; Zhuge et al., ‘GPTSwarm’, 2024
Generator-Critic (Actor-Critic / Reviewer)
generator-evaluator, critic agent, evaluator-optimizer		 One agent produces a candidate output and a second critic agent evaluates it against criteria, returning feedback that the generator uses to revise, looping until acceptable. Externalizing evaluation into a dedicated critic gives an explicit acceptance signal and targeted feedback, driving iterative refinement toward higher-quality output. accuracy/quality, reliability production-standard Anthropic ‘Building effective agents’ (evaluator-optimizer), 2024; Madaan et al., ‘Self-Refine’, 2023
Group Chat (Conversational Multi-Agent)
group chat manager, round-robin agents, multi-agent conversation		 Multiple agents converse in a shared chat thread, with a manager (or selection policy) choosing which agent speaks next until the task is resolved. A speaker-selection policy over a shared conversation lets agents collaborate, build on each other, and self-organize without a rigid predefined pipeline. accuracy/quality, autonomy, reliability production-standard Wu et al., ‘AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation’, 2023
Handoff
agent handoff, transfer-to-agent, delegation handoff		 An agent dynamically transfers control (and conversation state) of an in-progress task to another agent better suited to continue it. Representing handoff as a callable transfer turns control flow into a runtime decision, letting the active agent always be the most relevant specialist without a central planner. accuracy/quality, autonomy, latency, reliability production-standard OpenAI Swarm / OpenAI Agents SDK handoffs, 2024-2025
Hierarchical / Recursive Agents
agent hierarchies, recursive decomposition, nested supervisors		 Agents arranged in a tree where higher-level agents spawn and supervise lower-level agents, which may themselves recursively decompose subtasks. Recursive task decomposition matches problem structure to agent structure, letting each level operate within a bounded, locally-coherent context. accuracy/quality, context-capacity, autonomy, reliability production-standard LangGraph hierarchical agent teams; AutoGen GroupChat nesting; classic HTN planning lineage
LLM-as-Judge Orchestration
judge agent, verifier agent, pairwise judge		 A dedicated judge agent scores, ranks, or accepts/rejects the outputs of other agents against a rubric to gate or select the final answer. A separate evaluation model applies explicit criteria to candidate outputs, providing a selection/gating signal that filters out low-quality or unsafe results. accuracy/quality, reliability production-standard Zheng et al., ‘Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena’, 2023
Map-Reduce over Agents
fan-out/fan-in, agent map-reduce, parallel decomposition		 A task is split into independent shards processed in parallel by worker agents (map), whose partial results are then aggregated by a reducer agent (reduce). Embarrassingly-parallel decomposition fans work across agents simultaneously and merges results, cutting wall-clock latency and letting each shard fit a smaller context. latency, throughput, context-capacity, accuracy/quality production-standard LangGraph map-reduce (Send API); classic MapReduce (Dean & Ghemawat, 2004) applied to agents
MetaGPT / SOP-Encoded Pipeline
standardized operating procedures, assembly-line agents, meta-programming framework		 A multi-agent software company where agents take human-style roles and follow encoded Standard Operating Procedures that produce structured intermediate artifacts. Encoding SOPs and structured outputs into the agent workflow constrains communication to validated artifacts, reducing error propagation across the assembly line. accuracy/quality, reliability research Hong et al., ‘MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework’, 2023
Mixture-of-Agents (for cost/quality)
mixture-of-agents, MoA, agent blending		 A layered architecture where multiple (often open/cheaper) LLMs each produce a response and an aggregator synthesizes them, sometimes matching a frontier model at lower cost. Proposer agents in each layer generate candidate answers that downstream aggregator agents iteratively refine, exploiting collaborative diversity instead of one large model.
↔ also: Self-improvement & feedback loops at inference; Model routing, cascades & cost optimization		 accuracy/quality, reliability, cost/tokens research Wang et al., ‘Mixture-of-Agents Enhances Large Language Model Capabilities’, 2024
Multi-agent debate
LLM debate, society of minds, self-consistency via debate		 Multiple LLM instances independently answer, then read each other’s answers and reasoning across several rounds, converging on a refined consensus answer. Each agent revises its response conditioned on the other agents’ arguments over multiple debate rounds, surfacing and correcting errors through cross-examination before a final answer is taken.
↔ also: Self-improvement & feedback loops at inference; Test-time compute scaling & search over reasoning		 accuracy/quality, reliability, autonomy emerging Du et al., 2023, ‘Improving Factuality and Reasoning in Language Models through Multiagent Debate’
Orchestrator-Worker (Manager-Workers)
coordinator-worker, manager-agent, supervisor pattern		 A central orchestrator agent decomposes a task and dispatches subtasks to specialized worker agents, then integrates their outputs into a final result. A single coordinating locus assigns work dynamically and synthesizes results, enabling parallelism and clean separation of concerns while keeping one accountable controller. accuracy/quality, throughput, autonomy, reliability production-standard Anthropic ‘Building a multi-agent research system’, 2025; Anthropic ‘Building effective agents’, 2024
Parallelization (Sectioning & Voting)
voting ensemble, sectioning, self-consistency across agents		 The same task is run across multiple agents in parallel either by splitting into independent sections or by sampling several attempts and aggregating via vote/merge. Aggregating diverse parallel attempts (majority vote or merge) cancels independent errors and improves both robustness and latency versus serial retries. accuracy/quality, reliability, latency production-standard Anthropic ‘Building effective agents’ (parallelization), 2024; Wang et al., ‘Self-Consistency’, 2022
Prompt-Chaining (Sequential Pipeline)
chain workflow, sequential agents, pipeline of agents		 A fixed sequence of agents/LLM calls where each step’s output feeds the next, optionally with programmatic gates between steps. Decomposing a task into ordered subtasks lets each step be simpler and individually validated, trading a little latency for higher per-step accuracy. accuracy/quality, reliability production-standard Anthropic ‘Building effective agents’ (prompt chaining), 2024
Role Specialization (Persona / Expert Agents)
role-play agents, expert personas, specialized roles		 Distinct agents are assigned focused roles or personas (e.g., coder, tester, reviewer, researcher) with tailored prompts, tools, and context windows. Narrowing each agent’s scope concentrates relevant instructions and tools per agent, raising per-task quality and keeping each context window focused. accuracy/quality, context-capacity, reliability production-standard Qian et al., ‘ChatDev: Communicative Agents for Software Development’, 2023; CAMEL (Li et al., 2023)
Role-Playing Cooperative Agents (Instructor-Assistant)
CAMEL, inception prompting, user-assistant role play		 Two agents are assigned complementary roles (e.g., a task-giving user and a task-solving assistant) and autonomously converse to complete a task without further human input. Inception prompting fixes the roles so the agents drive each other forward, generating a self-sustaining cooperative dialogue toward task completion. autonomy, accuracy/quality research Li et al., ‘CAMEL: Communicative Agents for Mind Exploration of Large Scale Language Model Society’, 2023
Routing / Dispatch
router agent, intent routing, classifier-router		 A lightweight router classifies an incoming request and forwards it to the most appropriate specialized agent, model, or workflow. Front-loading a cheap classification step sends each input to the cheapest sufficient handler, cutting cost and latency versus routing everything to the strongest agent. cost/tokens, latency, accuracy/quality, throughput production-standard Anthropic ‘Building effective agents’ (routing workflow), 2024; RouteLLM (Ong et al., 2024)
Self-Collaboration / Single-Model Multi-Persona
solo performance prompting, SPP, single-LLM multi-role		 A single model simulates multiple cooperating personas within one context to gain multi-agent benefits without running separate model instances. Eliciting distinct personas inside one model’s context provides internal division of labor and self-critique while avoiding the cost of multiple agent processes. accuracy/quality, cost/tokens research Wang et al., ‘Unleashing the Emergent Cognitive Synergy in LLMs: Solo Performance Prompting’, 2023; Dong et al., ‘Self-Collaboration Code Generation’, 2023
Shared Long-Term Memory / Knowledge Base Coordination
shared memory store, collective memory, vector-store coordination		 Agents read and write a persistent shared memory (vector store, KB, or scratchpad) so knowledge produced by one agent is reusable by others across time. Externalizing experience into a queryable shared store lets agents reuse prior findings instead of recomputing, saving tokens and improving cross-agent consistency. cost/tokens, memory, context-capacity, accuracy/quality emerging Park et al., ‘Generative Agents’, 2023 (memory stream); Generative Agents shared memory patterns
Stigmergic Coordination (Environment-Mediated)
stigmergy, trace-based coordination, shared-artifact coordination		 Agents coordinate indirectly by leaving and reacting to traces in a shared environment or artifact store rather than communicating directly. Modifications to shared state act as implicit signals that trigger other agents’ actions, enabling decentralized self-organization without explicit messaging. throughput, autonomy, reliability research Theraulaz & Bonabeau, ‘A Brief History of Stigmergy’, 1999 (applied to LLM agent file/scratchpad coordination)

12. Context engineering & agent-design patterns

Production patterns (mostly 2024–2025, from teams shipping agents at scale) for shaping context so the rest of the stack works better — keeping a cacheable stable prefix, masking tools instead of removing them, isolating sub-agent contexts, reciting goals to fight drift. The practitioner-facing complement to the academic methods above.

5 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
Context offloading via structured note-taking to a scratchpad tool (Anthropic ‘think’ tool) Giving the agent an explicit no-op ‘think’ / note tool to externalize intermediate reasoning and policy reminders mid-trajectory. A dedicated tool call records reasoning/plan to the transcript without side effects, improving multi-step tool-use adherence and reducing dropped constraints. accuracy/quality, reliability production-standard Anthropic, 2025 (The ‘think’ tool)
Prompt-cache-friendly prompt ordering (stable prefix / append-only design) Structuring prompts so static content (system, tools, few-shot) is a fixed prefix and only volatile content changes at the tail, maximizing prefix-cache hits. Keeping the cacheable prefix byte-stable and appending new turns at the end lets the serving stack reuse KV for the prefix, cutting TTFB and input cost. cost/tokens, latency production-standard Provider prompt-caching best practices (OpenAI/Anthropic, 2024); ‘KV-cache hit rate’ agent design (Manus, 2025)
Recitation / re-stating goals to combat goal drift (todo.md recitation) Periodically rewriting the task objective and remaining plan into the recent context so attention stays on the goal over long trajectories. Re-injecting an updated todo/plan at the tail exploits recency to keep the global objective salient, reducing lost-in-the-middle goal drift on long agent runs. reliability, accuracy/quality emerging Manus context-engineering writeup, 2025 (recitation / todo.md)
Sub-agent context isolation / orchestrator-with-clean-subcontexts (Claude Code / Cognition pattern) Spawning subagents that each receive a narrowly scoped fresh context and return only a compact result, keeping the orchestrator’s window clean. Heavy exploration/tool output is confined to a child context; only a distilled summary returns, preventing context pollution and preserving the main thread’s budget. context-capacity, cost/tokens, accuracy/quality production-standard Anthropic multi-agent research system, 2025; Cognition ‘Don’t Build Multi-Agents’ (context-engineering), 2025
Tool masking over tool removal for stable cache (logit-masked action space) Keeping the full tool definitions in the (cached) context and constraining the available actions via decode-time logit masking instead of editing the tool list. Masking disallowed tools at the token level preserves the cacheable tool-definition prefix while still gating actions per state, avoiding cache invalidation. cost/tokens, latency, reliability emerging Manus context-engineering writeup, 2025 (mask, don’t remove)

Part IV — Cost, serving & training

13. Model routing, cascades & cost optimization

Not every query needs the biggest model. These methods send each request to the cheapest model that can handle it — cascades that escalate on low confidence, learned routers, semantic caches that skip the model entirely, and token-budget controls.

22 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
AutoMix (self-verification mixing)
AutoMix, self-verify-and-route		 An approach where a smaller model both answers and few-shot self-verifies its own answer, and a meta-verifier (POMDP-based) decides whether to route the query to a larger model. Noisy self-verification signals feed a Markov-decision-process meta-controller that learns the cost-optimal escalation policy across model sizes. cost/tokens, accuracy/quality research Madaan et al., ‘AutoMix: Automatically Mixing Language Models’, 2023
Batch / off-peak request scheduling
batch API, asynchronous batch, off-peak discount routing		 Deferring latency-tolerant requests to a discounted batch/asynchronous tier instead of synchronous real-time inference. Requests are queued and processed in bulk during off-peak windows at a reduced per-token price (commonly ~50% off provider batch APIs). cost/tokens, throughput production-standard OpenAI Batch API / Anthropic Message Batches, 2024
Cache-augmented generation (CAG)
CAG, preloaded-context caching		 Preloading a fixed knowledge corpus into the model’s context and precomputing its KV cache so repeated queries against that knowledge skip both retrieval and prefill recomputation. The corpus KV state is computed once and reused across queries, replacing per-query retrieval+prefill with a cache load. cost/tokens, latency, throughput emerging Chan et al., ‘Don’t Do RAG: Cache-Augmented Generation’, 2024
Cascade threshold / abstention tuning
deferral threshold optimization, confidence-threshold cascade, Wisdom of Committees		 Optimizing the per-stage confidence/deferral thresholds (and which confidence signal to use) that govern when a cascade stops vs escalates. Thresholds on the model’s confidence (or a post-hoc calibrated score) are tuned on validation data to hit a target cost-accuracy operating point along the deferral curve. cost/tokens, accuracy/quality, reliability emerging Gupta et al., ‘Language Model Cascades: Token-Level Uncertainty And Beyond’, 2024; Wang et al. ‘Wisdom of Committees’, 2020 (cascade theory)
Difficulty / complexity-based routing
query difficulty routing, complexity routing, adaptive routing		 Routing that classifies incoming queries by estimated difficulty or complexity and assigns simpler queries to smaller/cheaper models and harder queries to larger models. A difficulty estimator (classifier, heuristic, or LLM self-assessment of required reasoning depth) produces a complexity score that maps to a model tier via thresholds. cost/tokens, accuracy/quality, latency production-standard General routing framework; see Šakota et al. ‘Fly-Swat or Cannon? Cost-Effective Language Model Choice’, 2024 and related routing surveys
Embedding/predictive router (zero-shot model selection)
semantic router, embedding router, ZOOTER		 A lightweight router that maps the query embedding directly to the best-suited model among many, without running any candidate model first. Query embeddings are scored against learned per-model competence regions (reward-distilled or nearest-neighbor) to pick the cheapest model expected to succeed. cost/tokens, latency, accuracy/quality emerging Lu et al., ‘Routing to the Expert (ZOOTER)’, 2023; semantic-router open-source library (Aurelio)
FrugalGPT LLM cascade
FrugalGPT, LLM cascade, cascade of LLMs		 A query is sent to a sequence of progressively more expensive LLMs, and a learned scorer decides at each stage whether the cheap model’s answer is good enough to return or whether to escalate to the next model. A per-query ‘generation scorer’ (DistilBERT-style reliability predictor) accepts the cheapest answer above a learned threshold, stopping the cascade early so only hard queries reach the expensive model. cost/tokens, accuracy/quality, latency research Chen, Zaharia & Zou, ‘FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance’, 2023
Hybrid LLM routing (cost-aware quality threshold)
Hybrid LLM, router with desired quality level		 A router trained to send only the queries that the small model would handle as well as the large model to the small model, exposing a knob for the user’s tolerable quality drop. A BERT-based router predicts per-query quality gap between models and routes to the cheap model when the predicted drop is within the allowed budget. cost/tokens, accuracy/quality research Ding et al., ‘Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing’, 2024
LLM ensembling for cost/quality
model ensembling, ensemble routing, LLM blending		 Combining outputs from several (often smaller/cheaper) models to reach quality competitive with a single large model at lower aggregate cost. Multiple models answer and a fusion/voting/selection step picks or merges responses, trading parallel cheap calls for one expensive call. accuracy/quality, cost/tokens, reliability research Jiang et al., ‘LLM-Blender: Ensembling LLMs with Pairwise Ranking and Generative Fusion’, 2023; ‘Blending Is All You Need’, 2024
Mixture-of-Thought / model-internal effort routing (reasoning-effort parameter) A provider-level control that routes a request to more or less internal reasoning compute (e.g. reasoning_effort low/medium/high) to trade cost for accuracy. A single model exposes a thinking-budget knob; the request specifies effort, and the serving stack allocates more hidden reasoning tokens for hard queries only. cost/tokens, latency, accuracy/quality production-standard OpenAI o-series / Anthropic extended-thinking reasoning_effort (2024-2025)
Model cascade / quantization-tier serving
quantized-model cascade, precision cascade, mixed-precision tiering		 Serving the same base model at multiple quantization/precision tiers and routing easy queries to the cheaper low-precision variant, escalating to higher precision when needed. A confidence or difficulty gate selects the lowest-precision tier likely to be correct, reserving full-precision compute for hard inputs. cost/tokens, latency, memory, throughput emerging Cascade/serving design pattern; generalization of FrugalGPT applied to precision tiers
Multi-LLM optimal assignment / portfolio routing
LLM portfolio, Fly-Swat or Cannon, meta-model selection		 Given a pool of many candidate LLMs/APIs, choose per query the model (or set) that maximizes expected quality under a cost budget, framed as an assignment/optimization problem. A meta-model predicts each model’s expected performance and price for the query and solves a constrained optimization (or scoring) to pick the best cost-quality candidate. cost/tokens, accuracy/quality research Šakota et al., ‘Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling’, 2024
Output-length control / token budgeting
length control, token budgeting, max-token capping		 Techniques that constrain or shorten model output (and sometimes reasoning) length to reduce generated tokens, which dominate cost and latency. Caps (max_tokens), length-conditioned prompts/instructions, or fine-tuned brevity steer the decoder to stop sooner while preserving answer content. cost/tokens, latency production-standard Common practice; see length-controlled generation and concise-CoT studies (e.g. Nayab et al. ‘Concise Thoughts’, 2024)
Provider/API cost routing (cheapest-provider arbitrage)
provider routing, API arbitrage, load-balanced model routing		 Routing a request among multiple providers or deployments hosting equivalent models to the cheapest, fastest, or most-available endpoint. A gateway compares per-token price, latency, and availability across providers and dispatches to the best endpoint, with fallback on failure. cost/tokens, latency, reliability, throughput production-standard Gateway practice; OpenRouter / LiteLLM / portkey-style routing, 2023-2025
Reasoning-budget control / token-budget-aware reasoning
thinking budget, reasoning effort control, TALE		 Methods that allocate or cap the number of chain-of-thought / ‘thinking’ tokens a reasoning model spends per query, matched to the query’s difficulty. A predicted or instructed token budget bounds the reasoning trace (e.g. ‘reasoning_effort’ levels or an estimated optimal budget) so easy queries do not over-think. cost/tokens, latency, accuracy/quality emerging Han et al., ‘Token-Budget-Aware LLM Reasoning (TALE)’, 2024; provider ‘reasoning effort’ controls, 2024-2025
RouteLLM (learned LLM router)
RouteLLM, learned router, binary strong/weak router		 A trained router predicts, per query, whether a cheap (weak) model or an expensive (strong) model should answer, sending each query to exactly one model rather than running them in sequence. Routers (matrix factorization, BERT classifier, similarity-weighted ranking) trained on preference data (e.g. Chatbot Arena) score query difficulty and dispatch to strong vs weak model at a tunable cost-quality threshold. cost/tokens, accuracy/quality, latency production-standard Ong et al., ‘RouteLLM: Learning to Route LLMs with Preference Data’, 2024
Self-consistency with adaptive sampling
CoT-SC, self-consistency decoding, sample-and-vote		 Sampling multiple reasoning paths and majority-voting the final answer, with adaptive variants that stop sampling early once the vote is confident to save tokens. A confidence/consensus stopping rule monitors the running vote distribution and halts generation of further samples when the answer is statistically settled.
↔ also: Prompting & reasoning strategies; Self-improvement & feedback loops at inference; Test-time compute scaling & search over reasoning; Reliability, structured generation & evaluation-as-optimization		 accuracy/quality, reliability, cost/tokens research Wang et al., ‘Self-Consistency Improves Chain of Thought Reasoning’, 2022; Aggarwal et al., ‘Adaptive-Consistency’, 2023
Semantic caching (GPTCache)
GPTCache, semantic cache, embedding cache		 A cache keyed on the semantic meaning (embedding) of a request so that semantically similar prompts return a stored answer instead of re-invoking the LLM. Incoming queries are embedded and a vector-similarity lookup above a threshold returns the cached response, skipping the model call entirely. cost/tokens, latency, throughput production-standard Bang, ‘GPTCache: An Open-Source Semantic Cache for LLM Applications’, 2023 (Zilliz)
Small-model-first escalation
model escalation, tiered fallback, weak-to-strong escalation		 Always try the smallest/cheapest model first and escalate to a larger model only when the small model’s answer fails a quality or confidence check. A verifier or confidence gate after the small model’s response triggers a retry with the next-larger model on failure, so escalation cost is paid only for hard cases. cost/tokens, latency, reliability production-standard Cascade/fallback design pattern; generalization of FrugalGPT and provider ‘fallback’ routing
Speculative cascades
speculative cascade, cascade + speculative decoding		 A hybrid that fuses model cascades with speculative decoding: a small model drafts tokens and a deferral rule decides token-by-token whether the large model needs to verify/take over, rather than deferring whole queries. Combines a cascade’s quality-aware deferral with speculative decoding’s parallel draft-and-verify, applying an optimal token-level deferral rule so the large model is invoked only where the small model is likely wrong. cost/tokens, latency, accuracy/quality research Narasimhan et al. (Google Research), ‘Faster Cascades via Speculative Decoding’ / speculative cascades, 2024-2025
Tool/retrieval gating for cost
adaptive retrieval, should-I-retrieve gating, selective tool use		 Deciding per query whether expensive auxiliary steps (retrieval, tool/API calls, web search) are needed at all, skipping them for queries the model can answer directly. A learned or self-reflective gate (e.g. a retrieve/no-retrieve token or classifier) suppresses unnecessary context-fetching calls and their token cost. cost/tokens, latency, accuracy/quality emerging Asai et al., ‘Self-RAG’, 2023; adaptive-retrieval literature (e.g. Jeong et al. ‘Adaptive-RAG’, 2024)
Verifier-gated cascade / answer verification deferral
verification cascade, judge-gated deferral, quality-gate routing		 A cascade where the decision to accept a cheap model’s output or escalate is made by an explicit verifier (LLM-as-judge, reward model, or rule check) rather than the generating model’s own score. An independent verifier scores the candidate answer and routes below-threshold answers to a stronger model or a re-generation step. accuracy/quality, cost/tokens, reliability emerging Cascade literature; LLM-as-judge deferral (e.g. AutoMix, Madaan et al. 2023)

14. Inference / serving optimizations beneath agents

The engine layer the agent rides on. Quantization, speculative decoding, batching, attention kernels, and parallelism set the throughput/latency floor — almost all implemented in the serving engine (vLLM/SGLang/llama.cpp/TensorRT-LLM), not the agent. fak’s honest baseline: SOTA serving optimizations; quantization: AWQ.

31 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
AWQ
Activation-aware Weight Quantization		 A post-training weight-only quantization method that protects a small fraction of salient weights to preserve accuracy at low bit-widths. Identifies salient weight channels from activation magnitude statistics and applies per-channel scaling so quantization error concentrates on less-important weights. memory, cost/tokens, throughput, accuracy/quality production-standard Lin et al., 2023 (AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration)
Chunked prefill
dynamic chunking, split-fuse, piggybacking		 A scheduling technique that splits a long prompt’s prefill into smaller chunks and interleaves them with ongoing decode steps. Breaks prefill into token-budgeted chunks and batches them together with decode iterations so long prompts no longer stall token generation for other requests, balancing compute-bound prefill and memory-bound decode. latency, throughput production-standard Agrawal et al., 2023/2024 (SARATHI / Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve)
Continuous / in-flight batching
iteration-level scheduling, dynamic batching, in-flight batching		 A serving scheduler that admits and retires requests at the granularity of individual decode iterations rather than whole-batch boundaries. After each token-generation step it removes finished sequences and slots waiting requests into the running batch, keeping the GPU saturated instead of waiting for the slowest sequence in a static batch. throughput, latency, cost/tokens production-standard Yu et al., 2022 (Orca: A Distributed Serving System for Transformer-Based Generative Models)
CUDA graphs
graph capture, CUDA graph replay		 A mechanism that records a sequence of GPU kernel launches as a graph and replays it with a single launch. Captures the static decode-step kernel sequence once and replays the whole graph per step, eliminating per-kernel CPU launch overhead that otherwise dominates small-batch autoregressive decode. latency, throughput production-standard NVIDIA CUDA Graphs (CUDA Toolkit documentation)
Disaggregated prefill/decode
P/D disaggregation, prefill-decode disaggregation		 A serving architecture that runs the prefill (prompt) phase and the decode (generation) phase on separate GPU pools. Routes prompts to compute-optimized prefill workers and streams the resulting KV cache to memory-bandwidth-optimized decode workers, eliminating prefill/decode interference and letting each phase use its own parallelism and hardware. latency, throughput, cost/tokens production-standard Zhong et al., 2024 (DistServe); Patel et al., 2024 (Splitwise)
EAGLE
Extrapolation Algorithm for Greater Language-model Efficiency, EAGLE-2, EAGLE-3		 A speculative decoding method that drafts at the feature (hidden-state) level rather than the token level for higher acceptance rates. A small autoregressive head predicts the next hidden feature given prior features plus the sampled token, then decodes draft tokens from it and verifies with the target, with later versions adding dynamic draft trees. latency, throughput emerging Li et al., 2024 (EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty)
Expert parallelism
EP, MoE expert sharding		 A parallelism strategy for Mixture-of-Experts models that distributes distinct experts across devices. Places different experts on different GPUs and uses all-to-all communication to route tokens to their selected experts and gather results, scaling parameter count without scaling per-token compute. memory, throughput, cost/tokens production-standard Lepikhin et al., 2020 (GShard); Fedus et al., 2021 (Switch Transformer)
FlashAttention
FlashAttention-2, FlashAttention-3		 An exact, IO-aware attention kernel that computes attention without materializing the full N×N score matrix in HBM. Tiles the attention computation into SRAM and uses online softmax to fuse the softmax and matmuls in a single pass, minimizing slow HBM reads/writes; later versions add better work partitioning and FP8/async support. latency, memory, throughput production-standard Dao et al., 2022 (FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness)
FlashDecoding / FlashDecoding++
Flash-Decoding		 An attention kernel specialization for the decode phase that parallelizes over the KV sequence length. Splits the long key/value sequence into chunks processed in parallel and combines partial softmax results, exposing parallelism when batch and query length are tiny (single-token decode). latency, throughput production-standard Dao et al., 2023 (Flash-Decoding for long-context inference); Hong et al., 2023 (FlashDecoding++)
FP8 quantization
E4M3, E5M2, 8-bit floating point		 Inference (and training) in 8-bit floating-point formats supported natively on Hopper/Ada and later GPUs. Represents weights and/or activations and KV cache in FP8 (E4M3/E5M2) so tensor cores execute matmuls at 8-bit with per-tensor or finer scaling factors to preserve dynamic range. memory, throughput, latency, cost/tokens production-standard Micikevicius et al., 2022 (FP8 Formats for Deep Learning)
GGUF / llama.cpp k-quants
GGUF, GGML, Q4_K_M		 A file format and family of mixed-precision block quantization schemes used by llama.cpp for CPU/edge and consumer-GPU inference. Stores weights in small blocks with per-block scales (and super-block scales for k-quants), mixing bit-widths across tensors to balance size and quality for memory-constrained hardware. memory, cost/tokens production-standard Gerganov et al., llama.cpp project (GGUF format)
GPTQ
Generative Pre-trained Transformer Quantization		 A post-training weight-only quantization method that compresses LLM weights to 3-4 bits with minimal accuracy loss. Uses approximate second-order (Hessian-based) information to quantize weights layer-by-layer, greedily updating remaining weights to compensate for rounding error. memory, cost/tokens, throughput production-standard Frantar et al., 2022 (GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers)
Hadamard / incoherence-processed quantization (QuIP and QuIP#) Post-training quantization that pre-processes weights with incoherence (random orthogonal) transforms to make them quantization-friendly at 2-3 bits. Incoherence processing + lattice codebooks let weights compress to ~2 bits with adaptive rounding, retaining usable quality at extreme compression. memory, cost/tokens research Chee et al., 2023 (QuIP); Tseng et al., 2024 (QuIP#)
Hydra / multi-token Medusa successors and SpecInfer tree speculation Tree-structured speculative decoding that verifies many candidate continuations in one forward pass via a token tree and tree attention. A draft (small model or heads) proposes a token tree; the target verifies all branches at once with tree-attention, accepting the longest valid prefix. latency, throughput emerging Miao et al., 2023 (SpecInfer); Ankner et al., 2024 (Hydra)
KV cache reuse across requests via offload + load (CacheGen / LMCache streaming load) Encoding and streaming precomputed KV caches between storage/network and the serving node so reused contexts skip recomputation even when not a live prefix. KV tensors are compactly encoded and loaded on demand, trading bandwidth for prefill compute when the same context recurs across sessions/nodes. latency, throughput, cost/tokens emerging Liu et al., 2024 (CacheGen); LMCache project
Lookahead decoding
Jacobi decoding, lookahead		 A draft-model-free parallel decoding method that breaks the sequential dependency of autoregressive generation. Runs a parallel Jacobi-style iteration that generates and caches n-gram candidates in a lookahead branch and verifies them in a verification branch within the same step, trading FLOPs for fewer decoding steps. latency emerging Fu et al., 2024 (Break the Sequential Dependency of LLM Inference Using Lookahead Decoding)
Medusa A self-speculative decoding method that adds multiple lightweight decoding heads to the model to predict several future tokens at once. Extra trained heads predict tokens at positions t+1, t+2, … and a tree-attention scheme verifies many candidate continuations in a single forward pass without a separate draft model. latency, throughput emerging Cai et al., 2024 (Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads)
MoE routing (sparse gating / top-k)
Mixture-of-Experts, sparse MoE, top-k gating		 A sparse architecture/inference pattern where a gating network activates only a few experts per token. A learned router selects top-k of N expert FFNs per token so only a small fraction of parameters compute per token, decoupling model capacity from per-token FLOPs; serving variants add capacity factors, load balancing, and expert caching. cost/tokens, throughput, accuracy/quality production-standard Shazeer et al., 2017 (Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer)
Pipeline parallelism
PP, inter-layer model parallelism		 A model-parallel strategy that partitions the model’s layers into stages placed on different GPUs. Passes activations stage-to-stage and processes micro-batches in a pipeline so multiple stages stay busy simultaneously, fitting models too large for one device with lower inter-device communication than tensor parallelism. memory, throughput production-standard Huang et al., 2019 (GPipe); Narayanan et al., 2019 (PipeDream)
Prompt Lookup Decoding / n-gram speculative decoding A draft-model-free speculative decoding that proposes continuation tokens by matching recent n-grams already present in the prompt/context. When generation is input-grounded (RAG, code edits, summarization), candidate spans are copied from the context and verified in one pass, accelerating decode for free. latency, throughput production-standard Saxena, 2023 (Prompt Lookup Decoding); LLMA / Yang et al., 2023 (Inference with Reference)
Quantization-aware / low-bit serving (W4A16, QLoRA-style)
W4A16, marlin kernels, bitsandbytes NF4		 End-to-end low-bit serving combining sub-4-bit weight formats with optimized mixed-precision GEMM kernels. Pairs aggressive weight quantization (e.g., NF4, 2-3 bit via incoherence processing) with custom dequant-fused GEMM kernels (e.g., Marlin) so low-bit weights are unpacked on-chip during the matmul for near-full-precision quality at much lower memory/bandwidth. memory, throughput, latency, cost/tokens emerging Dettmers et al., 2023 (QLoRA / NF4); Tseng et al., 2024 (QuIP#); Frantar et al., 2024 (Marlin)
QuaRot / SpinQuant (rotation-based outlier-free quantization) Applying learned/Hadamard rotations to weights and activations so outliers are spread out, enabling clean 4-bit weight+activation+KV quantization. Rotation matrices are fused into the network to remove activation outliers, after which uniform low-bit quantization (W4A4/KV4) keeps accuracy. memory, throughput, cost/tokens emerging Ashkboos et al., 2024 (QuaRot); Liu et al., 2024 (SpinQuant)
Ring Attention / Striped Attention / Blockwise context parallel (explicit) Distributing a single long sequence’s attention across devices in a ring so context length scales with the number of accelerators near-losslessly. Blockwise KV blocks are passed around a device ring overlapping compute with communication, enabling million-token contexts beyond single-device memory. context-capacity, memory, throughput emerging Liu et al., 2023 (Ring Attention with Blockwise Transformers); Brandon et al., 2023 (Striped Attention)
Self-speculative decoding
layer-skip drafting, Draft & Verify		 Speculative decoding that uses the target model itself (a subset of its layers) as the draft, requiring no separate model. Drafts tokens by skipping some intermediate/later layers (early exit) of the same model, then verifies them with the full model in one pass, accepting the consistent prefix. latency, memory emerging Zhang et al., 2023 (Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding); Elhoushi et al., 2024 (LayerSkip)
Sequence / context parallelism
SP, Ring Attention, context parallelism		 A parallelism strategy that shards the sequence (token) dimension across devices to support very long contexts. Each device holds a slice of the sequence and exchanges keys/values in a ring (or via all-gather) so attention can be computed over the full sequence without any single device holding all tokens. memory, context-capacity, throughput production-standard Liu et al., 2023 (Ring Attention with Blockwise Transformers for Near-Infinite Context); Korthikanti et al., 2022 (sequence parallelism in Megatron)
Sequence packing / sample packing for training & prefill efficiency Concatenating multiple short sequences into one packed sequence (with attention masks blocking cross-doc attention) to eliminate padding waste. A bin-packing of variable-length samples plus document-boundary attention masking raises hardware utilization without leaking attention across samples. throughput, cost/tokens production-standard Krell et al., 2021 (Efficient Sequence Packing); FlashAttention varlen packing
SmoothQuant A post-training quantization technique enabling accurate W8A8 (8-bit weight and activation) quantization for LLMs. Migrates quantization difficulty from hard-to-quantize activations to weights via a per-channel mathematically-equivalent scaling transformation, smoothing activation outliers. memory, throughput, latency production-standard Xiao et al., 2022 (SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models)
Speculative decoding (draft model)
speculative decoding, speculative sampling, draft model decoding		 An exact-distribution acceleration where a small fast draft model proposes multiple tokens that the large target model verifies in parallel. The draft model autoregressively generates a token chunk; the target model scores them in one forward pass and accepts the longest prefix consistent with its distribution via a rejection-sampling correction, preserving the target’s output distribution.
↔ also: Model routing, cascades & cost optimization		 latency, throughput, cost/tokens production-standard Leviathan et al., 2023 / Chen et al., 2023 (Speculative Decoding / Accelerating LLM Decoding with Speculative Sampling)
Structured pruning / sparsity
2:4 sparsity, Wanda, SparseGPT		 Removing weights (unstructured or hardware-friendly structured patterns like N:M) to reduce model size and compute. Selects prunable weights by magnitude and/or activation-weighted importance, often in one shot post-training, and uses sparse tensor-core support (e.g., 2:4) to accelerate matmuls at reduced precision/density. memory, throughput, cost/tokens emerging Frantar & Alistarh, 2023 (SparseGPT); Sun et al., 2023 (Wanda: A Simple and Effective Pruning Approach for LLMs)
Tensor parallelism
TP, intra-layer model parallelism, Megatron-style		 A model-parallel strategy that shards individual weight matrices (and their matmuls) across multiple GPUs within a layer. Splits attention and MLP matmuls column/row-wise across devices with all-reduce/all-gather collectives to combine partial results, fitting large layers and reducing per-GPU compute/latency. memory, latency, throughput production-standard Shoeybi et al., 2019 (Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism)
Weight offloading / heterogeneous inference
CPU offload, FlexGen, ZeRO-Inference		 Serving models larger than GPU memory by holding weights/KV in CPU RAM or disk and streaming them to the GPU as needed. Schedules layer-by-layer transfer of weights/activations/KV between GPU, CPU, and NVMe with overlap and tensor compression, trading bandwidth for the ability to run otherwise-too-large models. memory, cost/tokens production-standard Sheng et al., 2023 (FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU); Aminabadi et al., 2022 (DeepSpeed-Inference)

15. Training & adaptation for agentic capability

Bake agentic skill into the weights. Supervised tuning on trajectories, preference optimization (DPO and its family), RL with verifiable or execution rewards, self-training loops, and parameter-efficient adapters turn a base model into a competent tool-user and reasoner.

34 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
Adapter / Prefix / Prompt Tuning (PEFT family)
adapters, prefix tuning, P-tuning		 A family of parameter-efficient tuning methods that add small trainable modules or learned continuous prompts instead of updating full weights. Inserts bottleneck adapter layers, learnable prefix/prompt vectors, or learned activation scalings while keeping the backbone frozen, so few parameters carry the task adaptation. memory, cost/tokens, throughput production-standard Houlsby et al., 2019 (Adapters); Li & Liang, 2021 (Prefix Tuning); Lester et al., 2021 (Prompt Tuning); Liu et al., 2022 ((IA)^3)
Agentic RL with Environment Interaction
agentic RL, multi-turn tool-use RL, RL on agent trajectories		 Reinforcement learning where the LLM agent acts over multiple turns in a real or simulated environment (tools, code, web, OS) and is rewarded on task outcomes. The policy rolls out multi-step tool-calling episodes against an environment, and trajectory-level rewards (with credit assignment over turns) update the policy via PPO/GRPO-style RL. accuracy/quality, autonomy, reliability emerging Carta et al., 2023 (GLAM); Bai et al., 2024 (DigiRL); SWE-RL / agent-RL line, 2024-2025
AgentTuning
AgentInstruct (dataset)		 An approach that fine-tunes LLMs into generalist agents using a mixed corpus of agent-interaction trajectories (AgentInstruct) plus general instructions to preserve broad ability. Blends high-quality multi-task agent trajectories with general SFT data so agentic skill is gained without catastrophic loss of general capability. accuracy/quality, autonomy, reliability research Zeng et al., 2023, ‘AgentTuning: Enabling Generalized Agent Abilities for LLMs’
Constitutional AI (critique-and-revise)
CAI, RLAIF critique step, self-critique against principles		 A method where the model critiques and revises its own responses against an explicit set of written principles (a ‘constitution’), used both at inference and to generate training data. At the critique-revise stage, the model is prompted to find where a response violates a principle and rewrite it, replacing human harmlessness labels with principle-conditioned self-feedback.
↔ also: Self-improvement & feedback loops at inference		 accuracy/quality, reliability, cost/tokens production-standard Bai et al., 2022, ‘Constitutional AI: Harmlessness from AI Feedback’ (Anthropic)
DAPO / Dr.GRPO (decoupled-clip and bias-corrected GRPO variants) GRPO refinements that fix length/difficulty bias and clipping issues to stabilize large-scale RL for long-CoT reasoning. Decoupled clipping, dynamic sampling, token-level loss and removal of the length/std normalization bias improve sample efficiency and stop reward hacking in reasoning RL.
↔ also: RL for agents & verifiable reasoning		 accuracy/quality, autonomy emerging Yu et al., 2025 (DAPO); Liu et al., 2025 (Dr. GRPO)
Direct Preference Optimization
DPO		 An offline preference-alignment method that fine-tunes directly on preferred/dispreferred response pairs without training a separate reward model or running online RL. Reframes the RLHF objective as a closed-form classification loss over preference pairs, implicitly fitting a reward through the policy/reference log-ratio. accuracy/quality, reliability, cost/tokens production-standard Rafailov et al., 2023, ‘Direct Preference Optimization’
DoRA (weight-decomposed low-rank adaptation) A PEFT method that decomposes pretrained weights into magnitude and direction, adapting direction with LoRA and learning magnitude separately. Separating magnitude/direction lets the low-rank update behave more like full fine-tuning, closing the LoRA-vs-FT gap at similar parameter cost. accuracy/quality, memory emerging Liu et al., 2024 (DoRA)
FireAct Fine-tuning language model agents on ReAct-style trajectories generated by a stronger teacher across multiple tasks and prompting methods. Distills diverse GPT-4-produced ReAct/CoT-with-tools trajectories into a smaller open model via SFT, improving robustness over few-shot prompting. accuracy/quality, reliability, cost/tokens research Chen et al., 2023, ‘FireAct: Toward Language Agent Fine-tuning’
Group Relative Policy Optimization
GRPO		 A critic-free policy-gradient algorithm that estimates advantages by normalizing rewards within a group of sampled responses to the same prompt. Samples a group of completions per prompt, uses the group’s mean/std reward as the baseline (no value network), and applies a PPO-style clipped update. accuracy/quality, memory, cost/tokens emerging Shao et al., 2024 (DeepSeekMath GRPO)
Identity Preference Optimization
IPO		 A DPO variant that adds an explicit regularizer to prevent overfitting to deterministic/near-certain preferences. Optimizes a squared-loss objective on the preference log-ratio gap to a target margin, bounding the implicit reward and reducing preference overfitting. accuracy/quality, reliability emerging Azar et al., 2023, ‘A General Theoretical Paradigm to Understand Learning from Human Preferences’
Instruction Tuning for Tool Use
tool-use instruction tuning, function-calling fine-tuning		 Instruction-format fine-tuning on (instruction, tool-call, tool-result, answer) examples to teach a model when and how to emit structured function/tool calls. Trains on instruction-response pairs whose responses include well-formed API/function-call syntax so the model generalizes the calling format and tool-selection policy. accuracy/quality, reliability, autonomy production-standard Gorilla (Patil et al., 2023); ToolLLM/ToolBench (Qin et al., 2023)
Iterative / Online DPO
iterative DPO, online preference optimization, online DPO		 An on-policy variant of preference optimization that repeatedly samples fresh model responses, labels new preferences, and re-runs DPO each round. Each iteration regenerates on-policy pairs from the latest policy and an updated reward/judge, narrowing distribution shift versus a single offline DPO pass. accuracy/quality, reliability emerging Xu et al., 2023 / Guo et al., 2024 (online/iterative DPO line)
Kahneman-Tversky Optimization
KTO		 A preference-alignment method that learns from single examples labeled merely desirable or undesirable, without needing paired comparisons. Applies a prospect-theory-inspired utility loss to per-example binary signals relative to a reference, removing the need for matched pairs. accuracy/quality, reliability, cost/tokens emerging Ethayarajh et al., 2024, ‘KTO: Model Alignment as Prospect Theoretic Optimization’
Knowledge Distillation
teacher-student distillation, KD		 Transfers capability from a larger teacher model to a smaller student by training the student to mimic the teacher’s outputs or distributions. The student minimizes divergence to teacher soft logits (or imitates teacher-generated outputs), compressing teacher behavior into a cheaper model. cost/tokens, latency, memory, throughput production-standard Hinton et al., 2015, ‘Distilling the Knowledge in a Neural Network’
Low-Rank Adaptation
LoRA, PEFT		 A parameter-efficient fine-tuning method that freezes base weights and learns small low-rank update matrices injected into selected layers. Represents weight deltas as a product of two low-rank matrices trained while the base model stays frozen, drastically cutting trainable parameters and optimizer memory. memory, cost/tokens, throughput production-standard Hu et al., 2021, ‘LoRA: Low-Rank Adaptation of Large Language Models’
Odds Ratio Preference Optimization
ORPO		 A monolithic method that combines instruction SFT and preference alignment in a single training stage with no reference model. Adds an odds-ratio preference penalty to the SFT loss, jointly raising chosen-response likelihood while penalizing the rejected response. accuracy/quality, reliability, cost/tokens emerging Hong et al., 2024, ‘ORPO: Monolithic Preference Optimization without Reference Model’
Outcome-Reward Reinforcement Learning for Reasoning
RLVR, reinforcement learning with verifiable rewards, RL on reasoning		 RL that fine-tunes a model using automatically verifiable outcome rewards (e.g., correct answer, passing tests) to elicit long-form reasoning. A programmatic verifier returns sparse 0/1 outcome reward over sampled solutions, and policy-gradient updates increase the probability of solution traces that verify. accuracy/quality, reliability, autonomy emerging DeepSeek-AI, 2025 (DeepSeek-R1); Lambert et al., 2024 (Tulu 3 RLVR)
Process Supervision / Process Reward Models
PRM, process-supervised verifier, step-level reward model		 Trains reward models to score the correctness of each intermediate reasoning step rather than only the final outcome. Human or automatically labeled step-level correctness signals supervise a PRM, which then guides search, reranking, or RL toward valid reasoning chains.
↔ also: Test-time compute scaling & search over reasoning		 accuracy/quality, reliability emerging Lightman et al., 2023, ‘Let’s Verify Step by Step’; Uesato et al., 2022
Quantized Low-Rank Adaptation
QLoRA		 Memory-efficient fine-tuning that backpropagates LoRA adapters through a frozen 4-bit quantized base model. Stores the base in 4-bit NF4 with double quantization and paged optimizers, training only LoRA adapters so large models fine-tune on a single GPU. memory, cost/tokens production-standard Dettmers et al., 2023, ‘QLoRA: Efficient Finetuning of Quantized LLMs’
Rank Responses to Align Human Feedback
RRHF		 A lightweight alignment method that aligns model scores with a ranking of multiple sampled responses using a ranking loss. Scores candidate responses by length-normalized log-probability and applies a pairwise ranking loss so preferred responses outrank dispreferred ones, plus an SFT term. accuracy/quality, cost/tokens research Yuan et al., 2023, ‘RRHF: Rank Responses to Align Language Models with Human Feedback’
Reasoning Distillation
CoT distillation, distilling reasoning chains, chain-of-thought distillation		 Distills a strong reasoning/agent model into a smaller one by training on the teacher’s long chain-of-thought and tool-use traces. SFT the student on teacher-generated step-by-step reasoning (and tool trajectories), transferring the reasoning policy without running RL on the student. accuracy/quality, cost/tokens, latency emerging DeepSeek-AI, 2025 (R1 distilled models); Hsieh et al., 2023 (Distilling Step-by-Step)
Reflection / Verbal-Feedback Self-Improvement Tuning
Reflexion-style tuning, self-correction training, SCoRe		 Methods that train agents to improve via self-generated natural-language critiques and corrections of their own prior attempts. Collects (attempt, reflection, improved attempt) episodes and fine-tunes (often with RL) so the model learns to self-critique and revise toward correct outcomes. accuracy/quality, reliability, autonomy emerging Shinn et al., 2023 (Reflexion); Kumar et al., 2024 (SCoRe)
Reinforced Self-Training
ReST, ReST-EM, ReST^EM		 An offline batched RL/self-training scheme that alternates generating a dataset of model samples (Grow) and fine-tuning on the reward-filtered subset (Improve). Iteratively samples outputs, scores them with a reward/verifier, filters to high-reward samples, and re-fines on them (expectation-maximization view), repeating. accuracy/quality, reliability, cost/tokens research Gulcehre et al., 2023 (ReST); Singh et al., 2023 (ReST-EM)
Reinforcement Learning from Human Feedback
RLHF, PPO-RLHF		 Aligns a model to human preferences by training a reward model from human comparisons and optimizing the policy against it with RL. A learned reward model scores outputs and a policy-gradient algorithm (typically PPO) updates the LLM to maximize reward while a KL penalty keeps it near the reference policy. accuracy/quality, reliability production-standard Ouyang et al., 2022 (InstructGPT); Christiano et al., 2017
ReST-MCTS* / self-training with tree-search-generated rationales Bootstrapping reasoning data by running MCTS with a process reward to mine high-quality solution traces, then fine-tuning the policy (and PRM) on them. Tree search produces per-step value estimates and verified traces without human step labels; these supervise both the policy and the process reward model iteratively. accuracy/quality, autonomy research Zhang et al., 2024 (ReST-MCTS*)
Reward-Ranked / Conditioned Fine-Tuning
quark, reward-conditioned SFT, control-token tuning		 Fine-tuning that conditions generation on reward/quality control tokens so the model can be steered toward high-reward behavior at inference. Labels training samples by reward quantile with control tokens (or unlearns low-reward behavior), letting the model learn reward-conditioned policies without online RL. accuracy/quality, reliability research Lu et al., 2022 (Quark); Korbak et al., 2023 (conditional training)
RLEF / execution-feedback RL for code & tool agents Reinforcement learning where the reward comes from executing the agent’s code/tool calls (tests pass, environment success) over multi-turn trajectories. Grounding rewards in real execution outcomes across turns teaches the model to iterate on failures, improving code/agent success beyond single-shot supervised tuning.
↔ also: RL for agents & verifiable reasoning		 accuracy/quality, autonomy emerging Gehring et al., 2024 (RLEF: Grounding Code LLMs in Execution Feedback)
Self-Instruct
instruction self-generation		 A bootstrapping method that uses a model to generate its own diverse instruction-following examples for instruction tuning with minimal seed data. Prompts the model to expand a small seed set into new instructions and responses, filters for quality/diversity, and fine-tunes on the synthesized dataset. accuracy/quality, cost/tokens production-standard Wang et al., 2022, ‘Self-Instruct: Aligning Language Models with Self-Generated Instructions’
Self-Play preference / SPIN and SPPO self-play fine-tuning Iterative self-play where the current model generates responses treated as ‘losers’ against prior human/own data, optimizing toward a Nash equilibrium policy. Each round the model plays against its previous self under a preference objective, distilling more capability from a fixed dataset without new human labels. accuracy/quality, autonomy research Chen et al., 2024 (SPIN); Wu et al., 2024 (Self-Play Preference Optimization / SPPO)
Self-Rewarding / LLM-as-Judge Self-Training
self-rewarding LLM, self-rewarding language models, iterative self-judging		 An iterative alignment loop where the same model both generates candidate responses and judges them to create its own preference data. The model acts as its own reward judge to label preference pairs each round, then trains via DPO/preference optimization on the self-generated labels, iterating.
↔ also: Self-improvement & feedback loops at inference		 accuracy/quality, autonomy, reliability research Yuan et al., 2024, ‘Self-Rewarding Language Models’
Self-Taught Reasoner (inference-time bootstrapping)
STaR, rationale bootstrapping, rejection-sampling fine-tuning loop		 A loop where the model generates rationales, keeps those that lead to correct answers (rationalizing the rest from the answer), and learns from them; the generate-filter-by-correctness loop is the feedback core. Correctness of the final answer filters self-generated reasoning traces, and rationalization repairs wrong ones, creating a self-improving feedback loop over reasoning.
↔ also: Self-improvement & feedback loops at inference; Test-time compute scaling & search over reasoning		 accuracy/quality, autonomy research Zelikman et al., 2022, ‘STaR: Bootstrapping Reasoning with Reasoning’
SimPO (reference-free simple preference optimization) A preference-optimization objective using length-normalized average log-probability as the implicit reward with a target margin, removing the reference model. Drops DPO’s reference model and adds a reward margin on length-normalized sequence likelihood, simplifying training and improving alignment efficiency. accuracy/quality, cost/tokens emerging Meng et al., 2024 (SimPO)
Spectrum / GaLore (memory-efficient full-rank training) Training/fine-tuning at full rank while keeping optimizer memory low by projecting gradients into a low-rank subspace (GaLore) or training only high-SNR layers (Spectrum). GaLore periodically updates a low-rank gradient projection so Adam states stay small; Spectrum freezes low-information layers, both cutting memory vs full FT. memory, accuracy/quality emerging Zhao et al., 2024 (GaLore); Hartford et al., 2024 (Spectrum)
Supervised Fine-Tuning on Agent Trajectories
SFT, behavioral cloning, trajectory SFT		 Fine-tuning a base or instruct model on curated multi-step agent trajectories (observation-thought-action-tool-result sequences) so it imitates competent tool-using, multi-turn behavior. Minimizes next-token cross-entropy over recorded successful trajectories, copying expert/teacher action sequences into the policy. accuracy/quality, reliability, autonomy production-standard Standard practice; see AgentTuning (Zeng et al., 2023) and FireAct (Chen et al., 2023)

Part V — Reliability & evaluation

16. Reliability, structured generation & evaluation-as-optimization

Making agents dependable is itself an optimization target. Constrained/structured decoding guarantees parseable output; guardrails and validation catch failures; automatic prompt optimizers (DSPy/OPRO/GEPA) tune the system against an eval; observability and offline replay turn production into a feedback signal. fak’s policy-in-the-kernel is the default-deny floor; trajectory replay lives in internal/turnbench.

27 methods.

Method What it is & how it optimizes Optimizes Maturity Representative reference
Agent observability & tracing
LLM observability, span tracing, OpenTelemetry GenAI		 Instrumentation that captures structured traces (prompts, tool calls, latencies, tokens, costs, scores) of every step in an LLM/agent run for debugging and evaluation. Emits nested spans per LLM/tool/retriever call with inputs, outputs, and metadata to a tracing backend, enabling replay, regression eval, and cost/latency attribution. reliability, cost/tokens, latency production-standard LangSmith; Langfuse; Arize Phoenix; OpenTelemetry GenAI semantic conventions, 2023-2025
Automatic Prompt Engineer
APE		 A method that automatically generates and selects instruction prompts by treating prompt engineering as a black-box optimization problem. An LLM proposes candidate instructions from input-output demonstrations, scores each by execution accuracy on held-out data, and iteratively resamples around the best candidates. accuracy/quality research Zhou et al., ‘Large Language Models Are Human-Level Prompt Engineers’, 2022
Chain-of-Verification
CoVe		 A method to reduce hallucination where the model drafts an answer, plans verification questions, answers them independently, and revises the answer accordingly. Independently answering self-generated verification questions (without conditioning on the original answer) catches inconsistencies that are then folded into a corrected final response.
↔ also: Prompting & reasoning strategies; Self-improvement & feedback loops at inference		 accuracy/quality, reliability emerging Dhuliawala et al., 2023, ‘Chain-of-Verification Reduces Hallucination in Large Language Models’
Constrained / grammar-guided decoding
structured decoding, JSON-schema-constrained decoding, grammar-constrained decoding		 A decoding-time technique that restricts the model’s next-token choices to only those allowed by a formal grammar or finite-state machine, guaranteeing syntactically valid output (e.g. valid JSON, valid SQL, a regex-matching string). At each step it masks the logits of any token that would violate the grammar/FSM, so sampling can only produce a parse-valid continuation.
↔ also: Tool use & function calling optimization		 reliability, accuracy/quality, latency production-standard Willard & Louf, ‘Efficient Guided Generation for LLMs’ (Outlines), 2023; Geng et al. (GCD), 2023
DSPy
declarative self-improving pipelines, programming-not-prompting, DSP		 A framework that expresses LLM pipelines as typed modules with signatures and uses optimizers (teleprompters) to compile prompts/weights automatically against a metric. Treats the pipeline as a program and runs optimizers (e.g. bootstrap few-shot, MIPRO) that search instructions and demonstrations to maximize a developer-supplied metric on a trainset. accuracy/quality, reliability production-standard Khattab et al., ‘DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines’, 2023
Eval-harness-driven optimization
benchmark-driven development, SWE-bench / WebArena / GAIA / tau-bench harnesses		 Using standardized agent benchmarks with executable verifiers as the objective that an agent system is iteratively tuned against. Runs the agent inside a reproducible harness whose pass/fail verifier (unit tests, task completion checks) provides an objective score that guides scaffold, prompt, and tool changes. accuracy/quality, reliability, autonomy production-standard Jimenez et al., ‘SWE-bench’, 2023; Zhou et al., ‘WebArena’, 2023; Mialon et al., ‘GAIA’, 2023; Yao et al., ‘tau-bench’, 2024
EvoPrompt (evolutionary prompt optimization) Discrete prompt optimization that evolves a population of prompts via LLM-driven mutation/crossover guided by dev-set scores. Genetic operators implemented as LLM rewrites plus selection on a held-out metric search the prompt space without gradients, a peer of PromptBreeder/GEPA. accuracy/quality research Guo et al., 2023 (EvoPrompt / Connecting LLMs with Evolutionary Algorithms)
GEPA
Genetic-Pareto prompt optimization, reflective prompt evolution		 A prompt optimizer that uses reflective natural-language mutation plus Pareto-based selection to evolve prompts sample-efficiently. Reflects on full execution traces to propose targeted prompt edits and maintains a Pareto frontier of candidates across instances, often matching RL with far fewer rollouts. accuracy/quality, cost/tokens emerging Agrawal et al., ‘GEPA: Reflective Prompt Evolution Can Outperform RL’, 2025
Guardrails / I-O safety filtering
input-output guardrails, Llama Guard, NeMo Guardrails		 Policy-enforcing classifiers and programmable rails placed around an LLM/agent to screen inputs and outputs for unsafe, off-topic, or policy-violating content and to enforce conversational flow. Dedicated guard models or rule/colang flows classify each turn and block, rewrite, or redirect responses that violate the configured policy before they reach the user or downstream tools. reliability production-standard Inan et al., ‘Llama Guard’, 2023; NVIDIA NeMo Guardrails (Rebedea et al., 2023)
JSON mode / structured outputs API
JSON mode, function-calling schema enforcement, response_format json_schema		 A provider-side feature (OpenAI, Anthropic, etc.) that guarantees the model’s response conforms to a supplied JSON Schema or tool/function signature. The provider applies constrained decoding and/or schema validation against the supplied schema during generation so the returned object always parses. reliability, accuracy/quality production-standard OpenAI Structured Outputs (2024); Anthropic tool use / Claude API
LLM-as-a-judge
LLM evaluator, model-graded evaluation, AI feedback evaluator		 Using a strong LLM as an automatic evaluator that scores or pairwise-ranks candidate outputs against a rubric or reference. Prompts an evaluator model with the task, rubric, and candidate(s) to emit a graded verdict, enabling cheap scalable evaluation and reward signals (with known position/verbosity biases to mitigate).
↔ also: Self-improvement & feedback loops at inference		 accuracy/quality, reliability production-standard Zheng et al., ‘Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena’, 2023
Lookahead / coherence-checking decoding for structured output (jsonformer / token-masking schema decoding) Filling only the value slots of a known JSON/structured schema by generating data tokens and emitting the fixed scaffolding deterministically. The schema’s literal structure is emitted programmatically and the model is only asked to produce typed values under a token mask, guaranteeing valid structure with fewer tokens. reliability, cost/tokens, latency production-standard Jsonformer (Singh, 2023); token-masking structured decoding
MIPRO
MIPROv2, Multi-prompt Instruction Proposal Optimizer		 A DSPy optimizer that jointly optimizes instructions and few-shot demonstrations for multi-stage LLM programs. Bayesian/surrogate search proposes candidate instructions (grounded in data and program structure) and demo sets, evaluating combinations against the metric to pick the best configuration. accuracy/quality emerging Opsahl-Ong et al., ‘Optimizing Instructions and Demonstrations for Multi-Stage LM Programs’, 2024
OPRO
Optimization by PROmpting		 Uses an LLM as a general-purpose optimizer where the optimization trajectory (prior prompts and their scores) is given in natural language and the model proposes the next candidate. Iteratively feeds the LLM a meta-prompt containing previously tried solutions sorted by score, asking it to generate a new, higher-scoring solution (e.g. a better instruction). accuracy/quality research Yang et al., ‘Large Language Models as Optimizers’ (OPRO), 2023
Outlines
regex/JSON-schema guided generation		 An open-source library that compiles a regex, JSON Schema, or context-free grammar into an FSM index over the tokenizer to drive structured generation. Precomputes, per FSM state, the set of allowed next tokens so logit masking is an O(1) lookup, making constrained decoding nearly free at serving time. reliability, latency, accuracy/quality production-standard Willard & Louf, 2023 (dottxt-ai/outlines)
Output validation & auto-repair
self-healing parsing, reask / retry-on-validation-failure, Guardrails AI		 A wrapper layer that validates model output against a schema/type/assertions and, on failure, automatically re-prompts the model with the validation error to obtain a corrected output. Runs validators (Pydantic types, regex, custom checks) on the output and feeds any failures back as a corrective ‘reask’ until validation passes or a retry budget is exhausted. reliability, accuracy/quality production-standard Guardrails AI (Rajpal et al.); Instructor (Liu); LangChain OutputFixingParser
PromptBreeder
self-referential prompt evolution		 An evolutionary, self-referential method that evolves a population of task-prompts and the mutation-prompts that modify them. Runs a genetic algorithm where an LLM mutates task-prompts using mutation-prompts that are themselves evolved, selecting on fitness measured against a task metric. accuracy/quality research Fernando et al., ‘Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution’, 2023
Regression / canary eval gating in CI
eval-driven CI, prompt regression tests, golden-set gating		 Running a curated evaluation suite automatically on every prompt/model/agent change and blocking deploys that regress the metric. Executes a deterministic eval set (with assertions or LLM-judge scores) in continuous integration and fails the pipeline if scores drop below a baseline threshold. reliability, accuracy/quality production-standard promptfoo; OpenAI Evals; industry eval-driven-development practice, 2023-2025
Selective prediction / abstention
reject option, I-don’t-know routing, selective answering		 Letting the model decline to answer (or escalate) when its confidence is below a threshold rather than emitting a likely-wrong answer. Thresholds an uncertainty/confidence score to choose between answering and abstaining/deferring, trading coverage for higher accuracy on answered cases. reliability, accuracy/quality emerging Ren et al., selective prediction for LLMs, 2023; Cole et al., ‘Selectively Answering Ambiguous Questions’, 2023
Self-verification / generate-then-verify
verifier models, self-checking, answer verification		 Pairing a generator with a separate verification step (model or executable check) that accepts, ranks, or rejects candidate outputs. Generates candidates then runs a learned or rule-based verifier (e.g. backward reasoning, test execution) to select the most likely-correct answer or trigger a retry. accuracy/quality, reliability emerging Weng et al., ‘Large Language Models are Better Reasoners with Self-Verification’, 2022; Cobbe et al. verifiers, 2021
SelfCheckGPT (sampling-based hallucination detection) A zero-resource hallucination detector that samples multiple responses and flags statements that are not consistent across the samples. Consistency of a claim across stochastic samples (via NLI/QA/n-gram scoring) estimates factuality without external knowledge or token-probabilities. reliability, accuracy/quality emerging Manakul et al., 2023 (SelfCheckGPT)
Semantic entropy / lexical-invariant uncertainty for abstention Estimating hallucination by clustering sampled answers into meaning-equivalence classes and measuring entropy over the clusters rather than tokens. Bidirectional-entailment clustering collapses paraphrases so entropy reflects semantic uncertainty, giving a calibrated confound-free hallucination signal. reliability, accuracy/quality emerging Kuhn et al., 2023; Farquhar et al., 2024 (Semantic Entropy, Nature)
Speculative / constrained tool-call repair at the boundary
function-call validation, tool-argument schema enforcement, arg auto-repair		 Validating an agent’s emitted tool/function call against the tool’s declared schema and repairing or rejecting malformed calls before execution. Parses the tool call, checks argument names/types against the tool signature, and either constrains generation to the schema or re-asks the model to fix invalid arguments. reliability, autonomy production-standard OpenAI/Anthropic tool-use schema enforcement; Instructor/Guardrails tool validation, 2023-2025
TextGrad
textual gradients, automatic differentiation via text		 A framework that optimizes components of compound LLM systems using natural-language ‘gradients’ (textual feedback) backpropagated through the system. An LLM produces critique-as-gradient for each variable (prompt, solution, code) and a textual analog of gradient descent applies those critiques to improve the variable. accuracy/quality emerging Yuksekgonul et al., ‘TextGrad: Automatic Differentiation via Text’, 2024
Trajectory replay / offline policy evaluation
off-policy evaluation, OPE, counterfactual evaluation		 Scoring or comparing candidate agent policies against previously recorded interaction traces instead of re-executing live, to estimate which policy is better cheaply. Replays a logged trajectory deterministically and re-derives each candidate policy’s decisions/score against the recorded environment responses, turning a product of live runs into a sum over one recorded run. cost/tokens, reliability, accuracy/quality emerging Precup et al., off-policy evaluation literature; applied LLM-agent trajectory-replay practice, 2024-2025
Uncertainty estimation / confidence calibration
confidence elicitation, semantic entropy, calibration		 Methods that estimate how likely an LLM’s output is to be correct, used to flag low-confidence answers. Derives a confidence signal from token logprobs, verbalized confidence, sampled-answer agreement, or semantic-entropy over clustered samples, and calibrates it against observed accuracy. reliability emerging Kuhn et al., ‘Semantic Uncertainty’, 2023; Kadavath et al., ‘Language Models (Mostly) Know What They Know’, 2022
XGrammar
context-free-grammar constrained decoding		 A high-performance constrained-decoding engine for general context-free grammars integrated into serving stacks (vLLM, SGLang, MLC). Splits tokens into context-independent vs context-dependent, precomputes an adaptive token mask cache, and overlaps mask generation with GPU compute to make CFG-constrained decoding near-zero-overhead. reliability, latency, throughput production-standard Dong et al., ‘XGrammar: Flexible and Efficient Structured Generation’, 2024

Alphabetical master index

All 370 methods, A→Z, with their home family — for lookup and grep.

Method Family Maturity
A-MEM (agentic memory / Zettelkasten-style) 5. Context-window management & compression research
Activation Beacon (sliding context condensation) 5. Context-window management & compression research
AdaPlanner 10. Planning algorithms & world models for agents research
ADaPT (As-Needed Decomposition and Planning) 10. Planning algorithms & world models for agents research
Adapter / Prefix / Prompt Tuning (PEFT family) 15. Training & adaptation for agentic capability production-standard
Adaptive / early-stopping sampling 2. Test-time compute scaling & search over reasoning emerging
Adaptive-RAG (query-complexity-routed retrieval depth) 7. Retrieval & knowledge augmentation emerging
Agent observability & tracing 16. Reliability, structured generation & evaluation-as-optimization production-standard
Agent-as-Tool 11. Multi-agent orchestration patterns production-standard
Agentic RAG 7. Retrieval & knowledge augmentation emerging
Agentic RL with Environment Interaction 15. Training & adaptation for agentic capability emerging
AgentTuning 15. Training & adaptation for agentic capability research
Algorithm of Thoughts 1. Prompting & reasoning strategies research
Analogical prompting 1. Prompting & reasoning strategies emerging
Associative graph-traversal recall (HippoRAG-style) 6. Agent memory architectures emerging
Astute RAG / knowledge-conflict resolution 7. Retrieval & knowledge augmentation emerging
Attention sinks / StreamingLLM 5. Context-window management & compression production-standard
AutoGen conversable-agent framework (programmable multi-agent conversation) 11. Multi-agent orchestration patterns production-standard
AutoGPT-style Autonomous Loop 11. Multi-agent orchestration patterns emerging
Automatic Chain-of-Thought 1. Prompting & reasoning strategies research
Automatic Prompt Engineer 16. Reliability, structured generation & evaluation-as-optimization research
AutoMix (self-verification mixing) 13. Model routing, cascades & cost optimization research
AWQ 14. Inference / serving optimizations beneath agents production-standard
Batch / off-peak request scheduling 13. Model routing, cascades & cost optimization production-standard
Best-of-N sampling 2. Test-time compute scaling & search over reasoning production-standard
Best-of-N with reward/verifier model 3. Self-improvement & feedback loops at inference production-standard
Blackboard Architecture 11. Multi-agent orchestration patterns emerging
Budget forcing / long-thinking (s1-style) 2. Test-time compute scaling & search over reasoning emerging
Cache-augmented generation (CAG) 13. Model routing, cascades & cost optimization emerging
Cache-aware load balancing / routing 8. KV cache & prefix reuse emerging
CacheBlend (segment-level / non-prefix KV reuse) 8. KV cache & prefix reuse emerging
Cascade / multi-tier KV inference (CascadeAttention) 8. KV cache & prefix reuse emerging
Cascade threshold / abstention tuning 13. Model routing, cascades & cost optimization emerging
Chain-of-Note (CoN) 7. Retrieval & knowledge augmentation research
Chain-of-Thought prompting 1. Prompting & reasoning strategies production-standard
Chain-of-Thought without prompting (CoT-decoding) 1. Prompting & reasoning strategies research
Chain-of-Verification 16. Reliability, structured generation & evaluation-as-optimization emerging
ChatDev (Communicative Software Agents) 11. Multi-agent orchestration patterns research
ChatEval-style role-diverse multi-agent evaluation (referee-team) 11. Multi-agent orchestration patterns research
ChunkAttention / prefix-aware fused attention 8. KV cache & prefix reuse research
Chunked prefill 14. Inference / serving optimizations beneath agents production-standard
Code as Policies 10. Planning algorithms & world models for agents research
Code generation for API composition (programmatic tool use) 9. Tool use & function calling optimization emerging
Code-as-Action (CodeAct) 9. Tool use & function calling optimization emerging
Cognitive memory typing (working / episodic / semantic / procedural) 6. Agent memory architectures emerging
ColBERT Late-Interaction Retrieval 7. Retrieval & knowledge augmentation emerging
Complexity-based prompting 1. Prompting & reasoning strategies research
Compute-optimal test-time scaling 2. Test-time compute scaling & search over reasoning emerging
Constitutional AI (critique-and-revise) 15. Training & adaptation for agentic capability production-standard
Constrained / grammar-guided decoding 16. Reliability, structured generation & evaluation-as-optimization production-standard
Context caching for agent / tool-loop reuse 8. KV cache & prefix reuse emerging
Context offloading via structured note-taking to a scratchpad tool (Anthropic ‘think’ tool) 12. Context engineering & agent-design patterns production-standard
Context-aware / Context-Aware Decoding (CAD, PMI context amplification) 4. Decoding-time control research
Contextual Compression / Retrieved-Context Pruning 5. Context-window management & compression emerging
Contextual Retrieval 7. Retrieval & knowledge augmentation emerging
Continuous / in-flight batching 14. Inference / serving optimizations beneath agents production-standard
Contract-Net / Auction Allocation 11. Multi-agent orchestration patterns research
Contrastive Decoding (expert vs amateur) 4. Decoding-time control research
Conversation summarization / compaction 5. Context-window management & compression production-standard
Conversational summary-buffer memory 6. Agent memory architectures production-standard
Corrective RAG 7. Retrieval & knowledge augmentation research
CRITIC (tool-augmented self-correction) 3. Self-improvement & feedback loops at inference emerging
Cross-Encoder Re-ranking 7. Retrieval & knowledge augmentation production-standard
Cross-layer KV sharing (YOCO / CLA) 8. KV cache & prefix reuse emerging
Cross-Model Mixture / Model Diversity Ensemble 11. Multi-agent orchestration patterns research
CUDA graphs 14. Inference / serving optimizations beneath agents production-standard
DAPO / Dr.GRPO (decoupled-clip and bias-corrected GRPO variants) 15. Training & adaptation for agentic capability emerging
Deduplication / redundancy elimination in context 5. Context-window management & compression emerging
Demonstration / few-shot tool-call exemplars 9. Tool use & function calling optimization production-standard
Dense Passage Retrieval 7. Retrieval & knowledge augmentation production-standard
Describe, Explain, Plan and Select (DEPS) 10. Planning algorithms & world models for agents research
Difficulty / complexity-based routing 13. Model routing, cascades & cost optimization production-standard
Direct Preference Optimization 15. Training & adaptation for agentic capability production-standard
Disaggregated KV transfer / KV-aware disaggregation 8. KV cache & prefix reuse production-standard
Disaggregated prefill/decode 14. Inference / serving optimizations beneath agents production-standard
Distributed / global KV cache pool 8. KV cache & prefix reuse emerging
DMC / Dynamic Memory Compression (learned KV merging at decode) 5. Context-window management & compression research
DoLa (Decoding by Contrasting Layers) 4. Decoding-time control research
DoRA (weight-decomposed low-rank adaptation) 15. Training & adaptation for agentic capability emerging
DSPy 16. Reliability, structured generation & evaluation-as-optimization production-standard
DuoAttention (retrieval vs streaming head split) 8. KV cache & prefix reuse emerging
Dynamic / Self-Organizing Agent Topology 11. Multi-agent orchestration patterns research
Dynamic replanning 10. Planning algorithms & world models for agents production-standard
EAGLE 14. Inference / serving optimizations beneath agents emerging
Editable core / persona memory blocks 6. Agent memory architectures production-standard
Embedding/predictive router (zero-shot model selection) 13. Model routing, cascades & cost optimization emerging
Emotion Prompting (EmotionPrompt) 1. Prompting & reasoning strategies research
Entity / profile memory 6. Agent memory architectures production-standard
Eval-harness-driven optimization 16. Reliability, structured generation & evaluation-as-optimization production-standard
EvoPrompt (evolutionary prompt optimization) 16. Reliability, structured generation & evaluation-as-optimization research
Experiential memory / case-based experience reuse 6. Agent memory architectures emerging
Expert parallelism 14. Inference / serving optimizations beneath agents production-standard
Faithful Chain-of-Thought 1. Prompting & reasoning strategies research
FastGen (adaptive KV compression) 8. KV cache & prefix reuse research
Few-shot prompting 1. Prompting & reasoning strategies production-standard
FireAct 15. Training & adaptation for agentic capability research
FLARE (Forward-Looking Active Retrieval) 7. Retrieval & knowledge augmentation research
FlashAttention 14. Inference / serving optimizations beneath agents production-standard
FlashDecoding / FlashDecoding++ 14. Inference / serving optimizations beneath agents production-standard
Forced / directed tool choice 9. Tool use & function calling optimization production-standard
Forest-of-Thoughts / ensemble-of-trees 2. Test-time compute scaling & search over reasoning research
FP8 quantization 14. Inference / serving optimizations beneath agents production-standard
FrugalGPT LLM cascade 13. Model routing, cascades & cost optimization research
Generate-then-Read / GenRead (LLM-as-retriever) 7. Retrieval & knowledge augmentation research
Generative / parametric memory (memory tokens & soft prompts) 5. Context-window management & compression research
Generative / RAG-free memory via Cache-Augmented Generation distinction — RecurrentGPT (language-based recurrence) 6. Agent memory architectures research
Generative-agent reflection-and-planning memory 10. Planning algorithms & world models for agents research
Generator-Critic (Actor-Critic / Reviewer) 11. Multi-agent orchestration patterns production-standard
GEPA 16. Reliability, structured generation & evaluation-as-optimization emerging
GGUF / llama.cpp k-quants 14. Inference / serving optimizations beneath agents production-standard
Goal decomposition / task decomposition 10. Planning algorithms & world models for agents production-standard
Gorilla / API-aware retriever finetuning 9. Tool use & function calling optimization research
GPTQ 14. Inference / serving optimizations beneath agents production-standard
Graph of Thoughts 2. Test-time compute scaling & search over reasoning research
Graph-based agent memory 6. Agent memory architectures production-standard
GraphRAG 7. Retrieval & knowledge augmentation emerging
Group Chat (Conversational Multi-Agent) 11. Multi-agent orchestration patterns production-standard
Group Relative Policy Optimization 15. Training & adaptation for agentic capability emerging
Grouped-Query Attention 8. KV cache & prefix reuse production-standard
Guardrails / I-O safety filtering 16. Reliability, structured generation & evaluation-as-optimization production-standard
Guidance / constrained generation interleaving (template-guided + acceleration) 4. Decoding-time control production-standard
Hadamard / incoherence-processed quantization (QuIP and QuIP#) 14. Inference / serving optimizations beneath agents research
Handoff 11. Multi-agent orchestration patterns production-standard
Heavy KV merging (CaM / KVMerger) 8. KV cache & prefix reuse research
Hierarchical / Recursive Agents 11. Multi-agent orchestration patterns production-standard
Hierarchical / recursive summarization 5. Context-window management & compression production-standard
Hierarchical Task Network planning 10. Planning algorithms & world models for agents production-standard
Hybrid LLM routing (cost-aware quality threshold) 13. Model routing, cascades & cost optimization research
Hybrid Retrieval (sparse + dense) 7. Retrieval & knowledge augmentation production-standard
Hydra / multi-token Medusa successors and SpecInfer tree speculation 14. Inference / serving optimizations beneath agents emerging
Hydragen (shared-prefix attention decomposition) 8. KV cache & prefix reuse research
Hypothetical Document Embeddings 7. Retrieval & knowledge augmentation emerging
Identity Preference Optimization 15. Training & adaptation for agentic capability emerging
Importance scoring / memory prioritization 6. Agent memory architectures emerging
In-Context / Retrieval-Augmented Long-Context (kNN-LM family) 7. Retrieval & knowledge augmentation research
In-context autoencoding / 500x context compression (ICAE — explicit), and prompt-distillation 5. Context-window management & compression research
Instruction Tuning for Tool Use 15. Training & adaptation for agentic capability production-standard
IRCoT (Interleaving Retrieval with Chain-of-Thought) 7. Retrieval & knowledge augmentation research
Irrelevance / no-call detection 9. Tool use & function calling optimization production-standard
Iterative / Online DPO 15. Training & adaptation for agentic capability emerging
Iterative refinement / generate-evaluate-refine loop 3. Self-improvement & feedback loops at inference production-standard
JSON mode / structured outputs API 16. Reliability, structured generation & evaluation-as-optimization production-standard
Kahneman-Tversky Optimization 15. Training & adaptation for agentic capability emerging
Knowledge Distillation 15. Training & adaptation for agentic capability production-standard
KV cache deduplication / content-addressed block sharing 8. KV cache & prefix reuse production-standard
KV cache offloading (CPU / disk tiering) 8. KV cache & prefix reuse production-standard
KV cache quantization (KIVI) 8. KV cache & prefix reuse emerging
KV cache reuse across requests via offload + load (CacheGen / LMCache streaming load) 14. Inference / serving optimizations beneath agents emerging
Late Chunking 7. Retrieval & knowledge augmentation emerging
LATS (Language Agent Tree Search) 10. Planning algorithms & world models for agents research
Least-to-Most / Decomposed Prompting 2. Test-time compute scaling & search over reasoning research
Lifting text/inline tool calls at the boundary 9. Tool use & function calling optimization production-standard
LLM ensembling for cost/quality 13. Model routing, cascades & cost optimization research
LLM+P (LLM + Classical Planner) 10. Planning algorithms & world models for agents research
LLM-as-a-judge 16. Reliability, structured generation & evaluation-as-optimization production-standard
LLM-as-Compiler / LLMCompiler parallel tool orchestration 9. Tool use & function calling optimization emerging
LLM-as-judge debate / multi-agent evaluation 3. Self-improvement & feedback loops at inference emerging
LLM-as-Judge Orchestration 11. Multi-agent orchestration patterns production-standard
LLM-DP (LLM Dynamic Planner) 10. Planning algorithms & world models for agents research
LLMLingua 5. Context-window management & compression emerging
LLMLingua-2 5. Context-window management & compression emerging
Long chain-of-thought RL reasoning (o1/R1-style) 2. Test-time compute scaling & search over reasoning production-standard
LongLLMLingua 5. Context-window management & compression emerging
Lookahead / coherence-checking decoding for structured output (jsonformer / token-masking schema decoding) 16. Reliability, structured generation & evaluation-as-optimization production-standard
Lookahead / rollout-guided decoding 2. Test-time compute scaling & search over reasoning research
Lookahead decoding 14. Inference / serving optimizations beneath agents emerging
Lost-in-the-middle mitigation via reordering 5. Context-window management & compression production-standard
Low-Rank Adaptation 15. Training & adaptation for agentic capability production-standard
Maieutic prompting 1. Prompting & reasoning strategies research
Map-Reduce over Agents 11. Multi-agent orchestration patterns production-standard
Maximal Marginal Relevance retrieval 7. Retrieval & knowledge augmentation production-standard
Medusa 14. Inference / serving optimizations beneath agents emerging
Mem0 (extraction-and-update memory layer) 6. Agent memory architectures production-standard
MemGPT (paged/virtual context memory) 5. Context-window management & compression production-standard
Memoized / deduplicated tool execution caching 9. Tool use & function calling optimization production-standard
Memory consolidation 6. Agent memory architectures emerging
Memory forgetting / decay 6. Agent memory architectures emerging
Memory retrieval re-ranking / relevance gating 6. Agent memory architectures emerging
Memory writeback policy 6. Agent memory architectures emerging
MemoryBank with Ebbinghaus forgetting curve 6. Agent memory architectures research
MetaGPT / SOP-Encoded Pipeline 11. Multi-agent orchestration patterns research
Min-p sampling 4. Decoding-time control emerging
MInference (dynamic sparse prefill attention) 8. KV cache & prefix reuse emerging
MIPRO 16. Reliability, structured generation & evaluation-as-optimization emerging
Mixture-of-Agents (for cost/quality) 11. Multi-agent orchestration patterns research
Mixture-of-Thought / model-internal effort routing (reasoning-effort parameter) 13. Model routing, cascades & cost optimization production-standard
Model cascade / quantization-tier serving 13. Model routing, cascades & cost optimization emerging
Model Context Protocol (MCP) / tool standardization 9. Tool use & function calling optimization production-standard
MoE routing (sparse gating / top-k) 14. Inference / serving optimizations beneath agents production-standard
Monte Carlo Tree Search for LLM planning 10. Planning algorithms & world models for agents research
Monte Carlo Tree Self-refine / AlphaLLM-style self-improvement 10. Planning algorithms & world models for agents research
Multi-agent debate 11. Multi-agent orchestration patterns emerging
Multi-head Latent Attention (MLA) 8. KV cache & prefix reuse production-standard
Multi-LLM optimal assignment / portfolio routing 13. Model routing, cascades & cost optimization research
Multi-Query Retrieval 7. Retrieval & knowledge augmentation production-standard
Odds Ratio Preference Optimization 15. Training & adaptation for agentic capability emerging
OPRO 16. Reliability, structured generation & evaluation-as-optimization research
Orchestrator-Worker (Manager-Workers) 11. Multi-agent orchestration patterns production-standard
Outcome Reward Model reranking 2. Test-time compute scaling & search over reasoning production-standard
Outcome-Reward Reinforcement Learning for Reasoning 15. Training & adaptation for agentic capability emerging
Outlines 16. Reliability, structured generation & evaluation-as-optimization production-standard
Output validation & auto-repair 16. Reliability, structured generation & evaluation-as-optimization production-standard
Output-length control / token budgeting 13. Model routing, cascades & cost optimization production-standard
PagedAttention 8. KV cache & prefix reuse production-standard
Parallel tool calling 9. Tool use & function calling optimization production-standard
Parallelization (Sectioning & Voting) 11. Multi-agent orchestration patterns production-standard
Pipeline parallelism 14. Inference / serving optimizations beneath agents production-standard
Plan caching / plan templates 10. Planning algorithms & world models for agents emerging
Plan-and-Solve Prompting 10. Planning algorithms & world models for agents emerging
Planner-Executor (Plan-and-Execute) 10. Planning algorithms & world models for agents production-standard
Procedural memory / learned skills writeback 6. Agent memory architectures emerging
Process Reward Model guided refinement 3. Self-improvement & feedback loops at inference emerging
Process Supervision / Process Reward Models 15. Training & adaptation for agentic capability emerging
ProgPrompt (programmatic planning) 10. Planning algorithms & world models for agents research
Program-Aided Language models 1. Prompting & reasoning strategies emerging
Program-of-Thoughts 1. Prompting & reasoning strategies emerging
Prompt Cache (modular position-independent KV) 8. KV cache & prefix reuse emerging
Prompt Lookup Decoding / n-gram speculative decoding 14. Inference / serving optimizations beneath agents production-standard
Prompt-cache-friendly prompt ordering (stable prefix / append-only design) 12. Context engineering & agent-design patterns production-standard
Prompt-Chaining (Sequential Pipeline) 11. Multi-agent orchestration patterns production-standard
PromptBreeder 16. Reliability, structured generation & evaluation-as-optimization research
Provider prompt caching (OpenAI / Anthropic / Gemini) 8. KV cache & prefix reuse production-standard
Provider/API cost routing (cheapest-provider arbitrage) 13. Model routing, cascades & cost optimization production-standard
PyramidKV / PyramidInfer (layer-pyramidal KV budget) 8. KV cache & prefix reuse research
Quantization-aware / low-bit serving (W4A16, QLoRA-style) 14. Inference / serving optimizations beneath agents emerging
Quantized Low-Rank Adaptation 15. Training & adaptation for agentic capability production-standard
QuaRot / SpinQuant (rotation-based outlier-free quantization) 14. Inference / serving optimizations beneath agents emerging
Query Decomposition 7. Retrieval & knowledge augmentation production-standard
Query Expansion 7. Retrieval & knowledge augmentation production-standard
Query Rewriting 7. Retrieval & knowledge augmentation production-standard
Quest (query-aware KV page selection) 8. KV cache & prefix reuse research
Quiet-STaR 2. Test-time compute scaling & search over reasoning research
Rank Responses to Align Human Feedback 15. Training & adaptation for agentic capability research
RankGPT / LLM listwise reranking 7. Retrieval & knowledge augmentation emerging
RAP (Reasoning via Planning) 10. Planning algorithms & world models for agents research
RAPTOR 5. Context-window management & compression emerging
ReAct 9. Tool use & function calling optimization production-standard
Reasoning Distillation 15. Training & adaptation for agentic capability emerging
Reasoning-budget control / token-budget-aware reasoning 13. Model routing, cascades & cost optimization emerging
Reciprocal Rank Fusion 7. Retrieval & knowledge augmentation production-standard
Recitation / re-stating goals to combat goal drift (todo.md recitation) 12. Context engineering & agent-design patterns emerging
Recurrent / Memorizing-Transformer memory 6. Agent memory architectures research
Recurrent / segment-level memory transformers 5. Context-window management & compression research
Recursive Criticism and Improvement 3. Self-improvement & feedback loops at inference emerging
Reflection / Verbal-Feedback Self-Improvement Tuning 15. Training & adaptation for agentic capability emerging
Reflection / verifier-in-the-loop agent reflection 3. Self-improvement & feedback loops at inference production-standard
Reflective / metacognitive prompting 3. Self-improvement & feedback loops at inference research
Reflexion (verbal self-reflection memory) 3. Self-improvement & feedback loops at inference emerging
Reflexion with self-generated unit tests 3. Self-improvement & feedback loops at inference emerging
Reflexion-for-planning successor — AdaPlanner closed-loop refinement (explicit in-plan + out-of-plan) 10. Planning algorithms & world models for agents research
Regression / canary eval gating in CI 16. Reliability, structured generation & evaluation-as-optimization production-standard
Reinforced Self-Training 15. Training & adaptation for agentic capability research
Reinforcement learning for tool use 9. Tool use & function calling optimization research
Reinforcement Learning from Human Feedback 15. Training & adaptation for agentic capability production-standard
Repeated sampling / coverage scaling 2. Test-time compute scaling & search over reasoning emerging
Rephrase and Respond 1. Prompting & reasoning strategies research
REPLUG 7. Retrieval & knowledge augmentation research
ReST-MCTS* / self-training with tree-search-generated rationales 15. Training & adaptation for agentic capability research
Retrieval Routing 7. Retrieval & knowledge augmentation emerging
Retrieval-Augmented Fine-Tuning 7. Retrieval & knowledge augmentation research
Retrieval-Augmented Generation 7. Retrieval & knowledge augmentation production-standard
Retrieval-augmented memory (retrieval-over-memory / RAG) 6. Agent memory architectures production-standard
Retrieval-based context selection 5. Context-window management & compression production-standard
Retrieval-head-aware KV eviction (RazorAttention / retrieval heads) 8. KV cache & prefix reuse research
Retrieval-scored memory (recency-importance-relevance) 6. Agent memory architectures emerging
Retry with backoff / circuit breaking for flaky tools 9. Tool use & function calling optimization production-standard
Reward-Ranked / Conditioned Fine-Tuning 15. Training & adaptation for agentic capability research
ReWOO (Reasoning WithOut Observation) 10. Planning algorithms & world models for agents emerging
Ring Attention / Striped Attention / Blockwise context parallel (explicit) 14. Inference / serving optimizations beneath agents emerging
RLEF / execution-feedback RL for code & tool agents 15. Training & adaptation for agentic capability emerging
Role Specialization (Persona / Expert Agents) 11. Multi-agent orchestration patterns production-standard
Role-Playing Cooperative Agents (Instructor-Assistant) 11. Multi-agent orchestration patterns research
RouteLLM (learned LLM router) 13. Model routing, cascades & cost optimization production-standard
Routing / Dispatch 11. Multi-agent orchestration patterns production-standard
Sandboxed code-action execution 9. Tool use & function calling optimization production-standard
SayCan (grounded affordance planning) 10. Planning algorithms & world models for agents research
Scratchpad reasoning 5. Context-window management & compression research
Selective Context 5. Context-window management & compression emerging
Selective prediction / abstention 16. Reliability, structured generation & evaluation-as-optimization emerging
Self-Ask 1. Prompting & reasoning strategies emerging
Self-Collaboration / Single-Model Multi-Persona 11. Multi-agent orchestration patterns research
Self-Consistency as feedback / verifier-guided refinement 3. Self-improvement & feedback loops at inference research
Self-consistency confidence (logit/entropy-weighted) aggregation 2. Test-time compute scaling & search over reasoning research
Self-Consistency early termination by entropy / agreement (ESC, Adaptive-Consistency adaptive sampling) 2. Test-time compute scaling & search over reasoning emerging
Self-consistency over tool-augmented / program reasoning 2. Test-time compute scaling & search over reasoning emerging
Self-consistency with adaptive sampling 13. Model routing, cascades & cost optimization research
Self-Contrast (multi-perspective divergent self-checking) 3. Self-improvement & feedback loops at inference research
Self-Correction with learned corrector (Self-Correct) 3. Self-improvement & feedback loops at inference research
Self-critique / self-correction 3. Self-improvement & feedback loops at inference production-standard
Self-Debugging 3. Self-improvement & feedback loops at inference emerging
Self-Discover 3. Self-improvement & feedback loops at inference research
Self-editing / learning-to-memorize agents 6. Agent memory architectures research
Self-evaluation / self-assessment 3. Self-improvement & feedback loops at inference emerging
Self-Instruct 15. Training & adaptation for agentic capability production-standard
Self-Play preference / SPIN and SPPO self-play fine-tuning 15. Training & adaptation for agentic capability research
Self-Querying Retrieval (metadata-filtered retrieval) 7. Retrieval & knowledge augmentation production-standard
Self-RAG 7. Retrieval & knowledge augmentation research
Self-Refine 3. Self-improvement & feedback loops at inference production-standard
Self-Rewarding / LLM-as-Judge Self-Training 15. Training & adaptation for agentic capability research
Self-speculative decoding 14. Inference / serving optimizations beneath agents emerging
Self-Taught Reasoner (inference-time bootstrapping) 15. Training & adaptation for agentic capability research
Self-verification 3. Self-improvement & feedback loops at inference emerging
Self-verification / generate-then-verify 16. Reliability, structured generation & evaluation-as-optimization emerging
Self-Verification / self-evaluation selection 2. Test-time compute scaling & search over reasoning emerging
SelfCheckGPT (sampling-based hallucination detection) 16. Reliability, structured generation & evaluation-as-optimization emerging
Semantic caching (GPTCache) 13. Model routing, cascades & cost optimization production-standard
Semantic chunking 5. Context-window management & compression production-standard
Semantic entropy / lexical-invariant uncertainty for abstention 16. Reliability, structured generation & evaluation-as-optimization emerging
Sentence-Window / Parent-Document Retrieval 7. Retrieval & knowledge augmentation production-standard
Sequence / context parallelism 14. Inference / serving optimizations beneath agents production-standard
Sequence packing / sample packing for training & prefill efficiency 14. Inference / serving optimizations beneath agents production-standard
Shared / collective multi-agent memory 6. Agent memory architectures research
Shared Long-Term Memory / Knowledge Base Coordination 11. Multi-agent orchestration patterns emerging
SimPO (reference-free simple preference optimization) 15. Training & adaptation for agentic capability emerging
Skeleton-of-Thought 2. Test-time compute scaling & search over reasoning research
Sliding-window / segment context processing 5. Context-window management & compression production-standard
Small-model-first escalation 13. Model routing, cascades & cost optimization production-standard
SmoothQuant 14. Inference / serving optimizations beneath agents production-standard
Spectrum / GaLore (memory-efficient full-rank training) 15. Training & adaptation for agentic capability emerging
Speculative / constrained tool-call repair at the boundary 16. Reliability, structured generation & evaluation-as-optimization production-standard
Speculative / predictive tool prefetching 9. Tool use & function calling optimization research
Speculative cascades 13. Model routing, cascades & cost optimization research
Speculative decoding (draft model) 14. Inference / serving optimizations beneath agents production-standard
Speculative reasoning / draft-then-verify reasoning skeletons (Skeleton-of-Thought parallel decode) 2. Test-time compute scaling & search over reasoning research
Step-Back prompting 1. Prompting & reasoning strategies emerging
Step-level Beam Search over reasoning 2. Test-time compute scaling & search over reasoning emerging
Stigmergic Coordination (Environment-Mediated) 11. Multi-agent orchestration patterns research
StreamingLLM (attention sinks) 8. KV cache & prefix reuse production-standard
Structured / running state notes 5. Context-window management & compression production-standard
Structured pruning / sparsity 14. Inference / serving optimizations beneath agents emerging
Sub-agent / context isolation (sub-context offload) 5. Context-window management & compression production-standard
Sub-agent context isolation / orchestrator-with-clean-subcontexts (Claude Code / Cognition pattern) 12. Context engineering & agent-design patterns production-standard
Supervised Fine-Tuning on Agent Trajectories 15. Training & adaptation for agentic capability production-standard
System 2 Attention (S2A) 1. Prompting & reasoning strategies research
Tab-CoT / Structured (tabular) reasoning 1. Prompting & reasoning strategies research
Take a Deep Breath / optimized meta-prompts (OPRO-discovered instructions) 1. Prompting & reasoning strategies research
Tensor parallelism 14. Inference / serving optimizations beneath agents production-standard
TextGrad 16. Reliability, structured generation & evaluation-as-optimization emerging
Thread of Thought (ThoT) 1. Prompting & reasoning strategies research
Token / context pruning by saliency 5. Context-window management & compression research
Tool masking / dynamic gating 9. Tool use & function calling optimization production-standard
Tool masking over tool removal for stable cache (logit-masked action space) 12. Context engineering & agent-design patterns emerging
Tool pruning / fewer-tools curation 9. Tool use & function calling optimization production-standard
Tool Retrieval / RAG-based tool selection 9. Tool use & function calling optimization production-standard
Tool schema / documentation design 9. Tool use & function calling optimization production-standard
Tool-call validation / argument schema enforcement 9. Tool use & function calling optimization production-standard
Tool-result / observation truncation and windowing 5. Context-window management & compression production-standard
Tool-result compression / truncation 9. Tool use & function calling optimization production-standard
Tool-use benchmarking / capability gating 9. Tool use & function calling optimization production-standard
Tool/retrieval gating for cost 13. Model routing, cascades & cost optimization emerging
ToolChain* 10. Planning algorithms & world models for agents research
Toolformer 9. Tool use & function calling optimization research
ToolkenGPT (tools as learned tokens) 9. Tool use & function calling optimization research
ToolLLM / ToolBench (large-scale tool-use instruction tuning) 9. Tool use & function calling optimization research
Trajectory replay / offline policy evaluation 16. Reliability, structured generation & evaluation-as-optimization emerging
Tree of Thoughts 2. Test-time compute scaling & search over reasoning research
Tree-of-Mixed-Thought / Thought-of-Search and learned planning programs 10. Planning algorithms & world models for agents research
Tree-of-Thoughts agent search successor — ToolChain* / A-guided tool-action search (explicit A) 10. Planning algorithms & world models for agents research
Tree-Planner 10. Planning algorithms & world models for agents research
Typed multi-view memory 6. Agent memory architectures research
Uncertainty estimation / confidence calibration 16. Reliability, structured generation & evaluation-as-optimization emerging
Universal Self-Consistency 2. Test-time compute scaling & search over reasoning emerging
Verifier-gated cascade / answer verification deferral 13. Model routing, cascades & cost optimization emerging
Verifier-guided / reward-guided decoding 2. Test-time compute scaling & search over reasoning emerging
Voyager skill library 6. Agent memory architectures research
Weight offloading / heterogeneous inference 14. Inference / serving optimizations beneath agents production-standard
Weighted Self-Consistency 2. Test-time compute scaling & search over reasoning emerging
World-model / look-ahead simulation 10. Planning algorithms & world models for agents research
XGrammar 16. Reliability, structured generation & evaluation-as-optimization production-standard
Zero-shot Chain-of-Thought 1. Prompting & reasoning strategies production-standard

Scope & honesty

Sources & related reading (repo cross-links)

Last updated: 2026-06-23.