Native implementation benchmark contracts
Status: INCOMPLETE. This is the machine-readable starting spine for requiring every fak-native implementation to be compared with the strongest practical alternatives. It deliberately does not turn missing measurements into claims.
Run:
fak native-benchmarks
fak native-benchmarks --json
fak native-benchmarks --check # non-zero until every declared contract has a witness
Comparison rule
Each native capability must declare one shared-workload contract with:
- the fak-native arm;
- the tuned no-feature or incumbent baseline (never a deliberately weak baseline);
- the next-best external implementation known for that capability;
- an explicit inventory of every fak first-class integration that supplies an equivalent capability, with validation that each inventory entry has a measured
fak + integrationarm besidefak + nativeon the same workload; - quality/correctness, latency, token/resource use, and total-cost metrics; and
- a reproducible witness before any performance claim is promoted.
An integration arm is additive to the next-best arm, not a substitute for it. If the best external implementation is also a first-class integration, one arm may carry both facts in its provenance, but the report must still name the integration explicitly.
The command discovers every production Go leaf directly beneath internal/ and requires an explicit disposition for each one: capability, multi_capability, or infrastructure. Unclassified leaves remain missing; discovery never promotes a package to covered by inference. Capability and multi-capability leaves name every benchmark contract they contain, while infrastructure leaves carry a reason and need no performance comparison. This makes classification debt machine-visible without pretending that one package equals one capability. The registry starts with the two examples called out by the operator:
| Native capability | Native location | tuned baseline | next-best comparison | required outcomes |
|---|---|---|---|---|
| tool filtering | internal/gateway/mcp_defer.go |
all schemas, provider cache enabled | retrieval-based selection (ToolRAG class) | task success, tool recall, input tokens, TTFT, total cost |
| context compression | internal/headroom/native.go |
full history, provider cache enabled | LongLLMLingua | task success, retained-fact recall, input tokens, latency, total cost |
| prefix KV reuse | internal/radixkv/radixkv.go |
prefix caching disabled | SGLang RadixAttention, plus fak + llm-d |
output equivalence, prefix hit rate, TTFT, throughput, KV bytes, total cost |
| timeout-phase attribution | internal/timeoutphase/timeoutphase.go |
one timeout bucket | OpenTelemetry spans, Datadog APM, AWS X-Ray | phase precision/recall, drops, latency, CPU/RSS/network/storage, total cost |
| KV-memory budget modeling | internal/kvbudget/kvbudget.go |
full-MHA closed form | vLLM memory profiler, SGLang memory pool, NVIDIA GenAI-Perf | bytes/token and peak-allocation error, fit/concurrency, latency/throughput, GPU/host memory, total cost |
| failure-signature resume backoff | internal/resumebackoff/resumebackoff.go |
immediate resume | Kubernetes CrashLoopBackOff, systemd RestartSec, AWS Step Functions retry | schedule equivalence, storm prevention/recovery, restarts, CPU/RSS/network, total cost |
| retry-attempt budgeting | internal/attemptbudget/attemptbudget.go |
unlimited retries | Envoy retry budget, gRPC retry policy, AWS SDK adaptive retry | retry/stop equivalence, recovery/amplification, latency, requests, CPU/RSS/network, total cost |
| cache observability | internal/cacheobs/cacheobs.go |
no telemetry | Prometheus, OpenTelemetry metrics, Datadog DogStatsD | counter/ratio equivalence, drops/cardinality, latency, CPU/RSS/network/storage, total cost |
| cache-cost accounting | internal/cacheprice/cacheprice.go |
charge full prompt | AWS, Google Cloud, and Azure pricing calculators | admission-token equivalence, billed-unit error, latency, bytes, RSS, total cost |
| tool-call rate limiting | internal/ratelimit/ratelimit.go |
no limiter | Envoy local rate limit, Kong rate limiting, Redis-cell | decision equivalence, overshoot, latency/throughput, state/network bytes, RSS, total cost |
| engine-cache invalidation | internal/enginecache/enginecache.go |
no invalidation | standalone vLLM, standalone SGLang, LMCache, plus separate fak + vLLM and fak + SGLang arms |
poisoned-reuse prevention, invalidated objects, latency, control requests/bytes, RSS, total cost |
| tool-result caching | internal/vdso/vdso.go |
uncached optimized upstream | Redis client-side/server-assisted cache and Momento Cache | output equivalence, hit rate, latency, upstream calls, RSS, total cost |
| context-memory management | internal/ctxmmu/mmu.go |
retain full history | Letta, plus separate fak + mem0, fak + Letta, fak + Zep/Graphiti, and fak + LangMem arms |
task success, write precision, retained-fact recall, tokens, latency, RSS, total cost |
| policy adjudication | internal/adjudicator/decide.go |
direct allow/deny lookup | OPA/Rego and Cedar | verdict equivalence, policy coverage, latency, throughput, RSS, total cost |
| model routing | internal/modelroute/modelroute.go |
fixed strongest model | RouteLLM, plus separate fak + LiteLLM, fak + OpenRouter, and fak + Portkey arms |
task success, route quality, latency, tokens, RSS, total cost |
| tokenization | internal/tokenizer/tokenizer.go |
exhaustive adjacent-pair BPE | llama.cpp tokenizer, plus fak + Hugging Face tokenizers |
exact token IDs, decode round-trip, throughput, latency, RSS, initialization, total cost |
| native agent harness | internal/agent/loop.go |
OpenCode CLI (direct tool execution) | OpenAI Codex CLI, Cursor Agent, Claude Code, plus separate fak + OpenCode, fak + OpenAI Codex, fak + Cursor, and fak + Claude Code arms |
task success, turns, hit rate, input tokens, output tokens, latency, peak RSS bytes, total cost |
All currently have missing witnesses, which is why --check fails. The registry is in internal/nativebench; new native capabilities must be added there with their alternatives before their benchmark obligation can be considered covered.
Scope still to enumerate
The benchmark-governance packages (internal/bench* and internal/nativebench) are explicitly classified as infrastructure with per-leaf reasons. This is not a blanket naming rule: each entry is enumerated and validation still leaves every newly discovered package unclassified by default.
This spine does not yet prove repository-wide coverage. Leaf discovery is exhaustive at the package boundary and the disposition schema is now enforced, but most leaves are still explicitly unclassified. The authoritative completion work is to classify every discovered leaf, split every multi-capability leaf into contracts, map equivalent first-class integrations, and attach benchmark witnesses. Until the unclassified count reaches zero and every contract passes, the broad claim “all native implementations are benchmarked against next best alternatives” remains not yet.
| Deadline-aware EDF admission + predicted-miss shedding | internal/deadlineadmit | native; FIFO/no shedding; Mooncake; vLLM; SGLang; fak + vLLM; fak + SGLang | local contract, external witnesses #6135 | INCOMPLETE |
| GitHub API mutation reserve + hourly call estimator | internal/mutationbudget | native; direct/no reserve; Octokit; gh api; Envoy global rate limit | local contract, external witnesses #6136 | INCOMPLETE |
| Worker dispatch-to-heartbeat histogram + p50/p95 | internal/launchlatency | native; raw/no summary; Prometheus; OpenTelemetry; Datadog | local contract, external witnesses #6137 | INCOMPLETE |
| Compute-region taxonomy + live-lease collision admission | internal/computeadmit | native; no admission; Kubernetes; Slurm; Ray; AWS Batch | local contract, external witnesses #6139 | INCOMPLETE |
| Shared lane/tree collision admission | internal/laneadmit | native; geometry-only; fak + DOS arbitrate; GitHub Actions; Kubernetes Lease; etcd | local contract, external/integration witnesses #6144 | INCOMPLETE |
| Issue near-duplicate detection with existing-issue pointers | internal/issuededup | native; exact title; fak + GitHub; GitHub; Linear; Jira; sentence-transformers | local contract, external witnesses #6228 | INCOMPLETE |
| Guard-journal typed action routing | internal/guardroute | native; count threshold; fak + DOS; OPA; Cedar; Drools; Alertmanager | local contract, external witnesses #6349 | INCOMPLETE |
| Host test-execution route selection | internal/testroute | native; GOOS-only; fak + GitHub Actions; Go; WSL; GitHub Actions; Bazel | local contract, external witnesses #6353 | INCOMPLETE |
| Budgeted issue-close batch planning with rollback | internal/closebatch | native; fixed chunks; fak + GitHub; gh; GraphQL; Jira; Linear | local contract, external witnesses #6364 | INCOMPLETE |
| Multi-signal answer-degeneration detection | internal/answershape | native; exact lines; fak + OpenAI; fak + Anthropic; llama.cpp; vLLM; Transformers; NeMo | local contract, external witnesses #6370 | INCOMPLETE |
| Reverse-dependency affected-test selection | internal/affectedtests | native; changed only; fak + Go test; Bazel; Pants; Nx; Gradle | local contract, external witnesses #6371 | INCOMPLETE |
| Dependency blast radius intersected with leases and issues | internal/blastradius | native; direct tree; fak + DOS; Bazel; Pants; Nx; Kubernetes | local contract, external witnesses #6375 | INCOMPLETE |
| Shared execution-surface region admission | internal/regionadmit | native; geometry-only; fak + DOS arbitrate; fak + Git-ref leases; Kubernetes Lease; etcd; GitHub Actions | local contract, external/integration witnesses #6149 | INCOMPLETE |
| Prefix-cache budget/reuse sweep + ROI knee | internal/cachesweep | native; no cache; libCacheSim; Caffeine; Redis/Valkey | local contract, external witnesses #6146 | INCOMPLETE |
| Warm-cache budget planning + value-ranked scheduling | internal/vcachegov | native; demand-only; fak + LMCache; fak + Mooncake; fak + NIXL; vLLM APC; SGLang HiCache | local contract, external/integration witnesses #6152 | INCOMPLETE |
| Cross-provenance cache reuse-divergence detection | internal/cachewitness | native; raw counters; fak + Prometheus; fak + OpenTelemetry; Prometheus rules; Datadog | local contract, external/integration witnesses #6155 | INCOMPLETE |
| Trailing-window cache reuse regression gate | internal/cachevalueledger | native; raw JSONL; fak + Prometheus; fak + OpenTelemetry; Prometheus rules; Datadog | local contract, external/integration witnesses #6157 | INCOMPLETE |
| Concentration-weighted shared cache-budget allocation | internal/vcachecal | native; equal-share; volume-proportional; fak + LMCache; fak + Mooncake; vLLM; SGLang | local contract, external/integration witnesses #6159 | INCOMPLETE |
| Dedicated prefix-cache warming and readback accounting | internal/vcachewarm | native; demand-only; fak + Anthropic; fak + Gemini; fak + OpenAI; fak + LMCache; fak + Mooncake; vLLM; SGLang | local contract, external/integration witnesses #6162 | INCOMPLETE |
| Provider-cache economics and realized-value fold | internal/vcacheobserve | native; raw usage; fak + Prometheus; fak + OpenTelemetry; Anthropic; OpenAI; Datadog; LangSmith | local contract, external/integration witnesses #6164 | INCOMPLETE |
| Bounded fsynced provider-cache snapshots and tolerant replay | internal/vcachesnapshot | native; append-only JSONL; fak + Prometheus; fak + OpenTelemetry; SQLite WAL; Prometheus TSDB; ClickHouse | local contract, external/integration witnesses #6165 | INCOMPLETE |
| Codex session token-counter extraction and content sanitization | internal/vcacheextract | native; raw JSONL pass-through; fak + OpenTelemetry; fak + Prometheus; jq; Vector VRL; Fluent Bit | local contract, external/integration witnesses #6169 | INCOMPLETE |
| Cache-warmth context-elision honesty lint | internal/vcacheqa | native; tuned non-test text scan; go/analysis; Semgrep; CodeQL; golangci-lint custom analyzer | local contract, external witnesses #6171 | INCOMPLETE |
| Provider-cache telemetry reconciliation and confirmed-only cost booking | internal/vcachestar | native; trust manifest; fak + Anthropic; fak + OpenAI; fak + Gemini; fak + Prometheus; fak + OpenTelemetry; Prometheus rules; Datadog; LangSmith | local contract, integration/external witnesses #6173 | INCOMPLETE |
| Default-on cache readiness across cold-path, usefulness, and provenance evidence | internal/vcachescore | native; usefulness threshold; fak + Prometheus; fak + OpenTelemetry; OPA/Rego; Prometheus rules; Datadog; LangSmith | local contract, integration/external witnesses #6178 | INCOMPLETE |
| Net-true, strawman, and not-yet claim grading | internal/claimcheck | native; witness-presence heuristic; fak + Prometheus; fak + OpenTelemetry; OPA/Rego; OpenAI Evals; LangSmith; Braintrust; DeepEval | local contract, integration/external witnesses #6180 | INCOMPLETE |
| Go structural expression search with metavariable bindings | internal/astquery | native; literal search; Semgrep; ast-grep; Comby; gogrep | local contract, external witnesses #6182 | INCOMPLETE |
| Syntactic multi-file Go call graph and forward/reverse path traversal | internal/codegraph | native; go/ast direct scan; x/tools callgraph; gopls; guru; CodeQL; SCIP Go | local contract, external witnesses #6205 | INCOMPLETE |
| Indexed literal code search with exact line verification | internal/trigram | native; linear scan; ripgrep; git grep; Zoekt; livegrep; Sourcegraph Search | local contract, external witnesses #6184 | INCOMPLETE |
| Standalone Go syntax validation with all-error locations | internal/codelint | native; go/parser first-error; go test; gofmt; go vet; staticcheck; golangci-lint; gopls | local contract, external witnesses #6187 | INCOMPLETE |
| Multi-provider token-usage class normalization and reconciliation | internal/canon | native; totals only; fak + OpenAI; fak + Anthropic; fak + local; fak + OpenTelemetry; provider SDKs; LiteLLM; OTel GenAI; LangSmith | local contract, integration/external witnesses #6189 | INCOMPLETE |
| Self-contained compiled ABI and adjudication conformance | internal/conformance | native; schema equality; OPA test; Conftest; JSON Schema/OpenAPI; Pact; Cedar | local contract, external witnesses #6191 | INCOMPLETE |
| Model/backend/precision support classification and stale-declaration reporting | internal/covmatrix | native; static table; fak + CUDA; fak + Metal; fak + Vulkan; vLLM; explicitly selected benchmark/reference: llama.cpp; Optimum; ONNX Runtime; TensorRT-LLM | local contract, external witnesses #6196 | INCOMPLETE |
| Native agentic loop with in-kernel vDSO tool caching, write barrier, and deterministic state | internal/agent | native; OpenCode CLI direct; OpenAI Codex CLI; Cursor Agent; Claude Code; fak + OpenCode; fak + OpenAI Codex; fak + Cursor; fak + Claude Code | local contract, external witnesses #10720 | INCOMPLETE |