Ceilings — ‘the speed of light in a vacuum’, per facet
Each facet’s c is the best that is possible for anyone, not the best fak or any competitor has achieved. Three kinds:
- physical — a derived limit (bandwidth roofline, information content).
- definitional — the metric’s own bound (zero attack success, zero lost work).
- lower-bound — the best system known; the true ceiling is at least this and probably higher. Cells against one are capped at
RELATIVISTIC.
Token economy
For the same delivered work, how much of the input-token bill does it remove?
- metric — fraction of input-token cost eliminated versus re-sending the full history uncached every turn (fraction, higher-is-better)
- c —
1(definitional) - derivation — 1.0 = every token that has been transmitted once is never paid for again – you pay for each distinct token exactly once. That is the information-theoretic floor of the bill: a system cannot charge you for less than the distinct content it must transmit. No prefix-reuse, caching, or compaction scheme can exceed it.
- caveat — Cells in the wrapper shape use pure_tax against the bare agent instead: there the alternative is the reference point, the absolute elimination fraction cancels in the ratio, and mediation can only subtract.
Raw speed
How fast do tokens actually come out, against the hardware’s physical limit?
- metric — sustained single-stream decode throughput as a fraction of the host’s memory-bandwidth roofline (fraction, higher-is-better)
- c —
1(physical) - derivation — Autoregressive decode at batch 1 is memory-bandwidth-bound: producing one token requires streaming the entire weight set from memory once. The roofline is therefore bandwidth / bytes-per-weight-pass. On the primary bench node (Apple M3 Pro, 150 GB/s) with Qwen2.5-1.5B Q8_0 resident at ~1.54 GB that is ~97.4 tok/s. No decoder can exceed it without changing the arithmetic itself – quantizing further, speculating, or batching. Normalizing to the roofline makes the facet hardware-independent.
- caveat — Cells in the gateway shape use pure_tax against the served engine: when fak fronts vLLM or SGLang, the engine IS the ceiling for that deployment and mediation can only take a cut.
Session longevity
How long can one session keep going before context forces you to start over?
- metric — share of the model’s context window that must stay resident for the session to keep going (fraction of window, lower-is-better)
- c —
0(definitional) - derivation — 0 = an ideal manager retains only a sufficient statistic of the work so far, so the resident share tends to zero as the session grows without bound. The floor is 0 by definition: no system needs to hold context it can reconstruct on demand. The opposite pole, 1.0, is where a client that manages nothing sits – it holds everything and then stops.
- caveat — The ideal is unreachable in practice; what the score measures is the gap to it, not arrival at it.
Run integrity
When you are not watching, does the run end in a state you can trust and resume?
- metric — fraction of unattended runs that end in a true, inspectable state – no false completion, no silent death (fraction, higher-is-better)
- c —
1(definitional) - derivation — 1.0 = every run that reports done is done, and every run that dies leaves a state you can inspect and resume. The bound is definitional: a run cannot end more truthfully than truthfully. This is distinct from task success – a run that correctly reports failure counts as integrity, because the operator learns the truth.
Injection control
Can hostile text inside the data the agent reads make it act for the attacker?
- metric — attack success rate under an indirect-prompt-injection suite, at unchanged benign utility (fraction, lower-is-better)
- c —
0(definitional) - derivation — 0 = no instruction embedded in data the agent reads ever produces an attacker-chosen action. Definitional floor. It matters that this floor is REACHED today by more than one system: the published formal-isolation defenses (CaMeL at ASR 0, MELON at 0.0-2.4%) are already there, which is why fak’s own 0/38 scores as parity rather than as a lead.
- caveat — At unchanged benign utility is load-bearing: a system that refuses everything also scores 0 and is useless. fak’s benign/under-attack utility on the same suite is NEEDS_KEY – unmeasured – so every cell on this facet is fenced.
Observability
After the fact, from one artifact, can you reconstruct why it did what it did?
- metric — fraction of consequential decision classes reconstructable after the fact from a single artifact, with no live attach (fraction, higher-is-better)
- c —
1(definitional) - derivation — The five classes that change a run’s outcome are: model traffic, cache reuse, context compaction, tool-policy verdicts, and recovery decisions. 1.0 = all five are recoverable post-hoc from one artifact by someone who was not present. The bound is definitional over the enumerated classes.
- caveat — A five-class decomposition is coarse. It is the granularity at which both sides can be scored without inventing detail, and no system would sit exactly at 1.0 under a finer one – which is part of why cells here are capped by their provenance.
Portability
Once you adopt it, what can you still swap out – model, backend, agent, host?
- metric — fraction of the model x backend support grid that actually runs, rather than refusing or falling back (fraction, higher-is-better)
- c —
1(definitional) - derivation — 1.0 = every model family you might want runs on every backend you might have. The bound is definitional over the declared grid (14 model families x 4 backends = 56 cells in
internal/covmatrix). Adopting any layer can only narrow this relative to what you already run, which is why cells here are pure_tax. - caveat — An honestly FENCED cell – the accelerated path refusing rather than returning a wrong answer – is good engineering and bad portability at the same time. This facet scores the second thing only.
docs/HARDWARE-MATRIX.mdscores the first, and grades it A.
Steady-state ops
After you have adopted it, what does it cost every month to keep it running?
- metric — recurring engineer-hours per month to keep it running, after adoption is complete (hours/month, lower-is-better)
- c —
0(definitional) - derivation — 0 = it never needs attention again. Definitional floor. This is deliberately disjoint from the adoption tax: the tax is one-time hours to first durable value, this is the bill that arrives every month afterwards. Counting both is not double-counting – they are different money, and a layer that is cheap to install and expensive to keep is a real and common failure mode.
- caveat — Because c = 0, the denominator is -N: any system whose recurring cost is more than double the incumbent’s falls straight to the display floor. That insensitivity is intended. Doubling somebody’s monthly operational burden is off the map regardless of by how much.