The compounding benefits of a saved call

Who this is for: anyone reasoning about what fak is actually worth on a long agent run, and why the worth is larger than the headline “turns saved” number — without inventing a number to say so. Prerequisite: a rough grasp of the loop ladder and the O(1) context economics. You will come away able to state the two compounding structures precisely, name the seam each rides on, and say exactly where the claim is measured and where it is modeled.

Short version. fak’s shipped accounting (internal/turnbench, Net) takes one integer — turns_saved — and prices it four ways: tokens, dollars, latency. That is honest but it under-models the benefit in two specific ways, and both are the question the user posed.

  1. One saved call discharges across orthogonal budgets, not one budget in four currencies. An avoided tool call does not “cost −X tokens.” It simultaneously returns local CPU (the in-process adjudication or spawn it never ran), GPU/prefill FLOPs (the forward pass that never happened), a context-window slot (the result that never entered the window), and wall-clock (the round-trip the loop never blocked on). These are separate accounts with separate ceilings. The flat Net collapses three of them into the token price; the true discharge is a vector, and the binding constraint on a given run is whichever account is scarcest — usually not dollars.

  2. The returned budget buys more horizon, and that is multiplicative. Context and dollars are finite per session. The number of useful calls a session can make before it must compact, reset, or stop is budget / effective_cost_per_call. fak pushes on both ends of that ratio — it shrinks the denominator (each call is cheaper) and refills the numerator (containment and reuse hand budget back to the pool). Numerator-up and denominator-down compound: the horizon gain is the product, not the sum, of the per-call savings and the budget-recovery. This is the “longer-horizon work, and faster work at the same horizon” the question names, made precise.

The rest of this note builds both claims from the seams and fences them hard. The one-line thesis: fak’s value is not a discount on a turn; it is a discount on a turn paid out of four budgets, reinvested into horizon.

1. The flat model fak ships, and exactly what it leaves on the table

The turn-tax harness is the honest core. For a replayed trace it classifies every call the kernel saw (ClassBreakdown in internal/turnbench/turnbench.go): a grammar repair, a vDSO local serve (pure / content-cache / static), a quarantine, a deny, or a plain pass. The turns the baseline pays and fak does not is turns_saved = grammar + vdso (ClassBreakdown.turnsSaved), split honestly into forced turns the baseline demonstrably re-issues (a duplicate read, an aliased call that errors and is re-prompted) and elision turns for optional calls a stronger model could have skipped (forcedTurns / elisionTurns). That split is the project’s anti-overclaim discipline and this note keeps it.

Then netFor prices it (internal/turnbench/turnbench.go):

func netFor(turns int, cm CostModel) Net {
    return Net{
        TurnsSaved:     turns,
        TokensSaved:    turns * cm.tokensPerTurn(),
        DollarsSaved:   float64(turns) * cm.dollarsPerTurn(),
        LatencySavedMs: float64(turns) * cm.ModelTurnLatencyMs,
    }
}

Read that closely. It is one count, turns, multiplied by one per-turn constant in each unit. The structure it encodes is “a saved turn is worth tokensPerTurn tokens and dollarsPerTurn dollars and ModelTurnLatencyMs ms,” and those three are the same saving expressed in three currencies — you would never add them, because dollars are tokens at a price and latency is turns at a round-trip. The model is a scalar saving with three exchange rates.

That is correct as far as it goes, and it is deliberately conservative. What it cannot express is the two things this note is about:

Both omissions are safe — Net under-claims, which is the right direction. But “the benefit is larger than Net says, and here is the structure of the part it omits” is a true and useful statement, and it is what follows.

2. Claim 1 — a saved call is a vector, not a scalar

A tool call, when it actually runs, draws down four distinct resources. Avoid it, or serve it locally, or contain its result, and you credit each of the four — but by different amounts from different seams, and against different ceilings. Here is the discharge, one row per account.

account what a run call draws what fak’s lever returns the seam measured or modeled
local CPU the adjudication cost + (in a hooked harness) a process spawn per gate an in-process decide instead of a spawn; a vDSO hit instead of a dispatch path Adjudicator.Adjudicate, vdso.Lookup at kernel.go Submit; baseline bench.MeasureSpawnedBaseline measured (M3: ~362 ns decide; ~2,849× vs spawned hook)
GPU / prefill a forward pass over the call’s prompt + the new result’s tokens, attention O(L²) in context length the forward pass that never runs (elided/served call) + the prefill never paid for a result that never enters context Session.Prefill / kvcache.go; result-elision via vdso; ultra-long-context floor geometry.go measured for the reuse arms (B/C 2.4–2.7× vs tuned, at T=8/16 on SmolLM2; the 50×5 headline run is still pending); modeled for the ultra-long floor (geometry, no model)
context window one result’s tokens permanently occupy a window slot for the rest of the session a quarantined/paged-out result costs a sub-2KB pointer, not its full body; an O(1) view holds the window flat ctxmmu.Admit (page-out stub), ctxplan.Optimize (bounded view) measured as a rate (pollution rate, resident-token compression on real transcripts); the window→horizon link is modeled
wall-clock the loop blocks on a model round-trip (~seconds) before it can act on the result a locally-served or elided call returns without the round-trip ModelTurnLatencyMs in the cost model; vDSO serve path modeled (a knobbed round-trip constant, never a wall-clock measurement)

The load-bearing word is orthogonal. These are not one saving in four units. They are four different budgets, each with its own ceiling, and a single saved call credits all four at once:

The honest consequence: the binding benefit of fak on a given run is whichever of these four accounts is scarcest, and it is rarely dollars. A long local agent on a laptop is context- and wall-clock-bound; a fleet on a rented GPU is prefill-bound; a hooked CI harness is CPU-bound on the gate itself. The flat Net reports a dollar figure for all three, which is exactly the wrong axis for each. Reporting the vector — even with three of its four entries modeled — tells a reader which lever matters for their bottleneck.

Why this is not just “four benefits” — the discharge is from one event

The subtle part, and the reason “compounding” is the right word and not “list”: all four credits come from the same single adjudication decision. The kernel decides once — at Submit, before any engine or network — whether these bytes may enter the model’s attention. That one verdict is what simultaneously (a) skips the spawn, (b) skips the forward pass, (c) keeps the slot out of the window, and (d) skips the round-trip. This is the inference-front-end lens restated as economics: one decision, enforced once, discharges across every downstream budget the decision would have committed. A serving engine that sees only the wire bytes has already lost three of the four accounts — it cannot decline to spawn, cannot decline to prefill before the prompt arrives, cannot keep a result out of a window it does not own. The kernel can, because it is upstream of all four.

3. Claim 2 — the horizon multiplier (the part the question is really about)

Now the compounding. Define, for one session under a binding budget B (pick the scarce account from §2 — for a long agent it is almost always the context window, secondarily dollars):

effective_horizon  =  B  /  effective_cost_per_call

effective_horizon is the number of progress-making calls the session can still make before it hits the wall and must compact, reset, or stop. “Effective” excludes the calls that buy nothing — a re-issued duplicate read, an aliased retry, a turn spent re-reading context that fell out of the window. Those are exactly the calls fak’s turn-tax levers delete. So fak moves both terms of the ratio, and they multiply.

Denominator — each call costs less. Every lever in §2 lowers effective_cost_per_call: a vDSO hit costs a pointer instead of a forward pass; a grammar repair costs a re-store instead of a re-prompt round-trip; an O(1) view costs a bounded prefill instead of an O(L²) re-prefill. Call the factor d < 1 (cost multiplier per call after fak).

Numerator — budget is returned to the pool, not just spent more slowly. This is the move Net cannot see. Containment and reuse give budget back:

What governs how much budget comes back is which spans the planner keeps resident, and there is a natural prior here worth naming. The load-bearing context in a long agent run tends to sit at the two ends: the oldest spans (the system framing, the active goal, the original constraints) and the newest (the last tool result, the current sub-task) — while the middle (resolved sub-goals, stale reads, superseded results) is the reclaimable part. The planner already favors the new end (recency in internal/ctxplan/forecast.go, a ramp that rises toward the latest step) and pins the known anchors, but the old end is only a binary pin, not a graded prior. An experimental primacy term (default off; Weights.Primacy, the old-end twin of recency) makes the prior symmetric. One honest finding falls out immediately and is pinned in the tests: a linear recency + primacy sum is flat, not U-shaped (primacy = 1 − recency, so at equal weights every step scores the same) — a true “remove the middle” dip requires asymmetric weights or a convex positional transform. So the lever is real but narrow, and it is exactly the kind of r-raising change that must be measured against the fence, not shipped on faith: run fak ctxplanbench --primacy 0.2 against the baseline and compare with fak horizon-recovery — a recovery-ratio gain that raises the fault rate (or merely shrinks the bounded set, or displaces served faults into refused ones) is rejected. On a 12-session real-transcript run the prior passes that gate, modestly: recovery ratio +0.3% with the fault rate down 1.5% (39.3%→38.5%, 366 fewer misses), the bounded set held, and zero faults displaced to refused — the old-end spans it kept were genuinely referenced again, so the small budget it reclaimed did not cost recall. A mild, honest win, consistent with the planner already capturing most of the middle-recovery through recency and pins; the larger convex-U variant and a multi-turn fault window are the named next steps before any stronger claim.

Call the budget-recovery factor r > 1 (the effective B is r·B over the session because spent budget keeps coming back).

Then:

effective_horizon_fak     r·B / (d · c)        r
------------------------ = ------------- = --------- = r / d        ( > r, and  > 1/d )
effective_horizon_naive     B / c              d

The horizon gain is r/d — the product of budget-recovery and per-call-cheapening, not their sum. A modest d = 0.7 (each call 30% cheaper) and a modest r = 1.5 (budget effectively recovered half-again over the session) is not a 1.8× horizon (1 + 0.5 + 0.3); it is 1.5 / 0.7 ≈ 2.1×. The two levers reinforce because cheaper calls spend the recovered budget more slowly, and recovered budget gives the cheaper calls more runway — each makes the other worth more.

This is the precise statement of the user’s two intuitions:

And the compounding is why you cannot get this by optimizing one lever in isolation. A pure caching layer moves d and leaves r = 1. A pure compaction tool moves r and leaves d = 1. fak’s bet — the loops-all-the-way-down assembly — is that one kernel holding the adjudication decision moves both, and the value is their product.

The honest fence on Claim 2

r/d is a model, and it inherits every caveat the O(1) economics doc carries, plus one of its own:

4. Where the compounding does and does not hold

The model is sharpest where the scarce budget is real and the levers are sound. It is weakest — and must be fenced — where either fails.

It holds strongly when:

It weakens or inverts when:

It is structurally unavailable to a shared-slot engine. The numerator lever (r > 1) needs per-agent KV ownership — the ability to evict a span from the middle of one agent’s run and re-RoPE the survivors bit-exact. A PagedAttention/RadixAttention pool shares cells across requests and cannot do this without forking the pool. So the compounding is not a tuning fak happens to have; it is downstream of the one structural property (per-agent addressable KV) that the inference-front-end lens names as the un-commoditized moat.

5. What this changes about how to report fak’s value

Three concrete shifts, each honest:

  1. Report the saving as a vector, not a dollar figure. Net is a scalar with three exchange rates; the real saving is four accounts with four ceilings. The smallest honest upgrade is to surface the local-CPU account Net omits entirely (the boundary tax is already measured) and to label which account is binding for the run’s profile. A reader on a laptop cares about context and wall-clock; a reader on a GPU fleet cares about prefill. One dollar number serves neither.

  2. Frame the long-run benefit as horizon, not total spend. “fak saved $X over the session” is the weakest true thing you can say, because the harness cache may already own most of it. “fak let the session make N more effective calls before it had to reset” is the strong true thing, and it is the thing a long-horizon agent author is actually budgeting against. The seam that would measure N is the agentic lifecycle KPIs — specifically the session-layer KPIs (resume warmth, promotion rate, core-image size) that today are guarded, not standing.

  3. Keep the multiplier as structure, ship the inputs as numbers. The product r/d is the right mental model and the wrong headline. Publish d’s measured parts (boundary tax, turns_saved, B/C reuse) and r’s measured inputs (pollution rate, resident-token compression, demand-page fault rate) — each a real event — and let the reader compose the multiplier for their workload. This is the same discipline that corrected the webbench number from “measured 9.7×” to an honest modeled floor: the structure is real, the single number would be invented.

6. The one-paragraph version

A tool call draws on four separate budgets — local CPU, GPU prefill, context window, wall-clock — and fak’s single upstream adjudication decision can decline to spend any of them, so one avoided or cheapened call pays back from all four accounts at once, against whichever ceiling is actually binding (rarely dollars). That saving then compounds, because the budget it returns to a finite session — chiefly context window, via containment and bit-exact eviction — extends how many effective calls the session can still make: effective_horizon = budget / effective_cost_per_call, and fak pushes the denominator down (cheaper calls) while pushing the numerator up (recovered budget), so the horizon gain is their product r/d, not their sum. The per-call discharge is largely measured; the horizon multiplier is a model whose inputs are measured but whose single number is deliberately not published, because quoting it soundly needs a task-success eval that the budget you reclaimed did not cost an answer. The shipped Net accounting is a conservative scalar that under-models both effects — which is the safe direction, and the gap this note names.

Reproduce / read next

# the flat Net this note extends — the measured turn-tax and its four-way price
go test ./internal/turnbench/...
fak turntax --suite turntax-airline     # the safety floor: 1 injection->0, 1 destructive->0

# the denominator's measured inputs
fak benchmarks run kernel-latency       # the local-CPU account (boundary tax)
python tools/ctxcost.py crossover        # the context account's O(1) economics

# the numerator's measured inputs (r's grounding — operand + fence, never their product)
fak ctxplanbench --heaviest 25 --out cpb.json   # measure budget recovery over 25 real sessions
fak horizon-recovery cpb.json                    # the recovery ratio AND its fault-rate fence, co-located; no r printed
python tools/session_audit.py            # is the harness cache already harvesting the dollar saving?