The prediction-calibration contract

Every module that emits a projection — a router’s cache-hit-rate estimate, a planner’s residency forecast, a resume cache’s cold/warm posture call, a dojo lever’s claimed saving — eventually has to answer one question: did reality match the projection, and if not, which way did it miss? fak already practices the discipline that answers it well, in two places:

The rule both keep is one reusable correctness discipline: back-test a projection against telemetry the predictor did not author before defaulting it on, and never silently over-claim. The gap this page closes is that the rule is locked inside those packages — there is no domain-free primitive another agent fleet can call. This page is that primitive, written as an engine-free contract. It is G5 of the agent-programming-grammar epic (#1213), the calibration sibling of net-true-value (a value claim), the observer-effect contract (a cost number), and the agent-routing schema (a routing decision). The verb that walks this contract over a (prediction, measurement, eval-fn) tuple is dos calibrate; its home is the installed DOS package, and it is not yet (see the honest fences). fak’s internal/dojo and internal/resume are the reference implementations — the offline witnesses that the contract below is real, not a wish.

The contract

A calibration is three inputs and one verdict. None of the inputs mentions a fak package — the decision is the shape, taken with no model in the loop.

Prediction — the declared theory, before reality is consulted

Prediction {
  metric           : string    // what is being predicted (a hit rate, a residency, a saving)
  claimed          : number    // the value the theory asserts
  unit             : string    // the metric's unit (fraction, tokens, seconds, USD, …)
  basis            : string    // the provenance of the claim (how it was derived) — review context
  lower_is_better  : bool       // the metric's DIRECTION: false (default) = higher is better
  intentional_floor: bool       // the claim is a GUARD reality must not breach, not an estimate
}

lower_is_better names the metric’s polarity so the verdict can tell the worse side of the claim from the better side — the field that makes the conservative-bias direction explicit (see the direction rule). intentional_floor marks a claim that is a guard the predictor defends (a false_warm_rate that must stay 0.0), not a best-guess central tendency — a floor is scored by its breach, never recalibrated up to its empirical rate.

Measurement — the witnessed ground truth the predictor did not author

Measurement {
  realized   : number       // the value reality delivered
  provenance : Provenance    // CLOSED: "WITNESSED" | "OBSERVED" — whose number it is
  source     : string        // where the ground truth came from (a usage record, a meter)
  measured   : bool          // false when no ground truth existed (scores INSUFFICIENT)
  sample     : int           // how many boundaries/turns stand behind `realized`
}

The measurement is evidence-bound: realized is ground truth lifted from a source the predictor did not write — the provider’s own billed usage records, a meter fak controls. provenance keeps every number honest about whose it is — a WITNESSED value is one fak authored and controls; an OBSERVED value is relayed from an upstream party (the model provider) and fak does not control it. A calibration over a measurement the predictor authored is not a calibration; it is the projection grading itself. measured: false (or a sample below the band’s floor) is the fail-closed INSUFFICIENT case, never a scored zero.

EvalBand — the closed tolerance the verdict reads

EvalBand {
  calibrated_max : number   // a normalized residual at or under this is CALIBRATED (default 0.10)
  min_sample     : int       // fewer measured boundaries than this scores INSUFFICIENT (fail-closed)
}

The band is the eval-fn made data: a residual within calibrated_max of the claim is calibrated; a corpus thinner than min_sample is too thin to score and is reported INSUFFICIENT rather than rounded up. The default calibrated_max of 0.10 (within 10% of the claim) is the dojo’s conservative DefaultCalibBand.

Verdict — the closed, validatable output

Calibrating a Prediction against a Measurement under an EvalBand yields exactly one verdict from a closed set. The decision is a finite switch — an out-of-set token cannot be produced, and a validator decides membership without a live service:

verdict when meaning
CALIBRATED the normalized residual is within calibrated_max reality met the claim within tolerance
OVER_CLAIM reality landed on the worse-than-claim side, beyond the band the theory promised more than reality delivered — the harmful direction
UNDER_CLAIM reality landed on the better-than-claim side, beyond the band the theory under-promised; a margin reality beat (free headroom uncredited)
INSUFFICIENT no ground truth, or a corpus thinner than min_sample there is no admissible evidence to score the claim — reported, never rounded up

The normalized residual is the relative error |realized − claimed| / |claimed|, capped so a claim near zero that reality refutes cannot dominate with an unbounded ratio; a claim of “nothing” is scored by the absolute residual instead (the exact calibErr the reference implementation uses). The machine-checkable schema publishes this verdict set as a closed JSON Schema enum (Draft 2020-12), so any orchestrator validates a verdict with an off-the-shelf validator, no fak engine present.

The conservative-bias direction

OVER_CLAIM and UNDER_CLAIM are not symmetric — which side is the over-claim is the load-bearing, direction-aware rule, and it is explicit by design. For a higher-is-better metric (the default — a hit rate, an accuracy, a saving), realized below the claim is the worse side: billed reality delivered less than the theory promised, so it scores OVER_CLAIM. For a lower_is_better metric (a false_warm_rate, a latency), the polarity flips: realized above the claim is the worse side. This is the exact worseThanClaim rule the reference implementation applies, surfaced here as a named contract so the conservative bias — over-claiming is the direction that must never hide — is part of the vocabulary, not an implementation detail. An intentional_floor claim is scored on its breach (the worse side only), so “recalibrating a floor up to its empirical rate” can never erase the guard.

The three properties (the acceptance, made checkable)

This contract is portable and honest because it holds three properties an external caller can check without fak’s engine — the issue’s three acceptance criteria, each bound to a witness:

  1. The verdict vocabulary is closed and validatable. The four verdicts above are the whole set; an out-of-set token is UNCLASSIFIED and refused, never coerced to a pass — the same fail-closed posture dos check-reason keeps for a refusal token. Membership is decided by a finite switch, not a lookup against a live service. The prediction-calibration.json schema publishes the enum so the claim is machine-checkable, not asserted; the reference implementation’s verdict constants (internal/dojo/dojo.go) are the same closed set, offline-witnessed by go test ./internal/dojo/.
  2. An over-claim is surfaced, never hidden; the conservative-bias direction is explicit. OVER_CLAIM is a first-class verdict with its own decision arm, and the direction rule names which side is the over-claim for each metric polarity. The dojo’s fold surfaces every over-claim as an advisory line and its FoldCalibrable folds a floor by its breach, so a loop optimising the calibration can never “gain” by recalibrating an over-claim away — the warm-hit bimodal lesson is the worked case: a fixed 0.17 claim that real corpora refuted scored OVER_CLAIM, and the fix was to recalibrate the claim down, never to hide the verdict.
  3. It is generic enough for a non-fak projection. Nothing in the three inputs is fak-shaped. A router’s hit-rate estimate calibrates as Prediction{metric:"cache_hit_rate", claimed:0.94} against a Measurement{realized:0.91, source:"router_telemetry"}; a planner’s residency forecast as Prediction{metric:"resident_tokens", claimed:48000} against the meter’s witnessed count. The worked round-trip below is on disk as fixtures, validated against the schema with a stock validator.

The fail-closed floor

The single rule that keeps a calibration honest under thin evidence: a corpus that cannot conclusively score the claim is reported INSUFFICIENT, never rounded up to CALIBRATED. Two cases collapse to it — measured: false (no ground truth existed, the dojo’s UNMEASURED) and a sample below min_sample (the corpus is too thin to trust a central tendency, the discipline internal/resume keeps by excluding an ambiguous partial re-serve from its accuracy denominator rather than scoring it on a guess, and internal/dojocal keeps with a DefaultMinSample floor before a recalibration is trusted). The warm-hit bimodal note is the canonical reason this rule matters: a claim recalibrated 0.17 → 0.0 to the conservative lower bound — because over-claiming is the harmful direction, so the floor is the value that never over-claims. INSUFFICIENT is the calibration analogue of the verification ladder’s INDETERMINATE: absence of an affirmative score escalates to “not yet measured,” it does not pass.

The round-trip, as data

The contract’s whole claim is that (prediction, measurement, eval-fn) → verdict is data, not narration. The fixtures under fixtures/ are the on-disk witness — each validates against prediction-calibration.json with a stock Draft 2020-12 validator, no fak engine present:

A reviewer reads the three inputs and the verdict; no model and no fak engine are needed to confirm the calibration. fak’s internal/dojo scores the same prediction against the same measurement and produces the same verdict — offline, witnessed by go test ./internal/dojo/, the reference implementation’s proof that the contract holds.

Reference implementation and witness

Contract element Reference stick Status
Prediction / Measurement / EvalBand dojo.Prediction, dojo.Outcome, dojo.CalibBand (internal/dojo/dojo.go) [SHIPPED]
Closed Verdict vocabulary dojo.VerdictCalibrated / VerdictOverClaim / VerdictUnderClaim / VerdictUnmeasured constants + the Score switch [SHIPPED]
Direction-aware conservative bias dojo.worseThanClaim (polarity rule) + FloorRespectErr (floor breach) [SHIPPED]
Fail-closed INSUFFICIENT Score → UNMEASURED on !Measured; resume.Backtest ambiguous-exclusion; dojocal.DefaultMinSample [SHIPPED]
Projection back-tested vs witnessed telemetry resume.Backtest (projection vs provider’s own cache_read records) → BacktestReport [SHIPPED]
Offline determinism witness go test ./internal/dojo/, go test ./internal/resume/ (no model in the loop) [SHIPPED]
Machine-checkable verdict enum prediction-calibration.json (Draft 2020-12) [SHIPPED]
Portable dos calibrate verb the installed DOS package not yet

Honest fences

Cross-references