collectives: the MPI reduce/allreduce/bcast family, mapped honestly

fak has collective-shaped surfaces in two different rank spaces, and the single biggest overclaim risk in the MPI-shaped epic (#639) is conflating them. This doc is the canonical map every other collective-shaped child links: one row per MPI collective primitive → its real fak symbol → which layer it lives in, so the rank-space distinction is written down once instead of inferred from terse inline comments. It is part of the MPI-shaped message-passing epic (#639).

Honesty caveat (read first). fak borrows the structure of MPI collectives and the vocabulary, never an MPI/HPC number. There are two distinct rank spaces:

MPI is the design lens and vocabulary that tells us where these boundaries are and what to call them. It is not a claim that fak is MPI or inherits any HPC throughput, latency, message-rate, or wire-protocol property.

The map

MPI primitive fak symbol Layer
MPI_Reduce / MPI_Allreduce (the general fold) modelroute.Combine over the Reduce* set — ReduceFirst / ReduceVote / ReduceBestOf / ReduceAllReduce / ReduceConcat AGENT (deterministic on structure only)
MPI_Allreduce (the named all-reduce) modelroute.ReduceAllReduce — weighted mean of the members’ scalar outputs AGENT (scalars, not tensors)
The live fan-out that produces the votes Combine folds gateway.dispatchEnsemble — N independently-adjudicated Kernel.Syscall calls in member order (#597) AGENT (each member crosses the default-deny floor)
MPI_Bcast (the broadcast bound) abi.ShareScope — ScopeAgent / ScopeFleet / ScopeTenant AGENT (authorizes visibility, moves no bytes)
MPI_Allreduce (real cross-process sum) model.DistComm.AllReduceSum TENSOR (real cross-process HOST float32)
MPI_Allgather (real cross-process gather) model.DistComm.AllGather TENSOR (real cross-process HOST float32)

The AGENT layer — non-bit-exact, scope-bounded

This layer folds and bounds agent outputs. It is deterministic on structure (the routing decision, the reduce order, the scope partition), never on the member text/scalar a non-bit-exact engine produced.

modelroute.Combine + the Reduce* set ≈ MPI_Reduce / MPI_Allreduce

Combine(reduce, votes) (internal/modelroute/modelroute.go:559) is the ensemble reduce: it folds many members’ outputs into one Result under a CLOSED, additive set of reductions —

It is pure and deterministic: every tie is broken by a stable key, and the caller MUST pass votes in Plan.Members order, so the same votes always fold the same way. That is the MPI analogue’s load-bearing honesty: like MPI_Reduce, the fold is deterministic on its inputs — but the inputs (member answers) come from non-bit-exact engines, so determinism is pinned to the decision and its reduce, never to end-to-end answer reproducibility.

gateway.dispatchEnsemble — the fan-out that produces the votes

Combine is the pure fold; the live dispatch that PRODUCES the votes is dispatchEnsemble (internal/gateway/gateway.go:1142, issue #597). It runs each member as its OWN independently-adjudicated kernel call — carrying that member’s model in abi.ToolCall.Engine — gathers the ALLOWED members’ outputs in Plan.Members order, and folds them with modelroute.Combine. The MPI-shaped invariant it honors: an ensemble expands to N independently-adjudicated Kernel.Submit calls, never one fan-out that bypasses the default-deny floor. A member bound for a REMOTE model still crosses the residency/policy gate and is denied for a tenant/sensitive payload; on a full wipeout (every member refused) it fails closed, surfacing the last refusal verdict rather than a silent empty success.

abi.ShareScope ≈ MPI_Bcast — but it is the broadcast bound, not a broadcast

ShareScope (internal/abi/types.go:93) is the CLOSED, additive isolation scope a shared Ref carries:

ShareScope meaning broadcast analogue
ScopeAgent private to one agent (the fail-closed default) not broadcast — rank-private
ScopeFleet shareable across the fleet’s trusted partition the fleet broadcast bound
ScopeTenant shareable within a tenant boundary the tenant broadcast bound

The default ScopeAgent is fail-closed (private): a value becomes visible to a wider audience only when its scope is explicitly widened. This is the bound on a broadcast, not the broadcast itself — widening a scope authorizes a later share, it does not transport data. A one-sided write can never widen sharing past its ShareScope (an Accumulate into a ScopeFleet window cannot publish at ScopeTenant); the default stays (TaintTainted, ScopeAgent).

The TENSOR layer — real cross-process HOST float32 (NOT NCCL, NOT multi-GPU)

model.DistComm (internal/model/dist_collective.go) is the first REAL cross-process collective on fak: a coordinator-rooted process group where each rank holds only its own part, performing AllReduceSum / AllGather over a real wire, proven byte-identical to the in-process default. Its ranks are tensor-parallel host-float32 shards of ONE model — a completely different rank space from the agent layer above.

This layer carries the load-bearing disclaimer for the whole epic. Quoted verbatim from the package, this is the rank space that must be held apart from the agent layer:

internal/model/dist_collective.go (HONESTY), verbatim:

This is a cross-PROCESS collective over HOST float32 — it is NOT multi-GPU and is NOT NCCL. “Multi-GPU” stays unclaimable until a non-cpu-ref compute.CollectiveBackend (the NCCL/RCCL device backend) all-reduces a DEVICE tensor across 2 GPUs and matches cpu-ref on the GPU server. DistComm proves the distributed architecture above the device line; the device line is the next, GPU-node rung. Following the repo’s own TCPTransport precedent, the gate runs the ranks as goroutines over a loopback socket — a genuine cross-process send, verifiable on one box.

The all_reduce caveat — the same disclaimer at the agent layer

The agent-layer ReduceAllReduce borrows the distributed-systems name but is a scalar reduce, not a tensor one. Quoted verbatim from the package (internal/modelroute/modelroute.go:221):

modelroute.ReduceAllReduce, verbatim:

ReduceAllReduce numerically aggregates the members’ SCALAR outputs into their weighted mean — the map-reduce / all-reduce form for numeric answers (a score, a count, a probability). It is NOT a tensor all-reduce: outputs that do not parse as a float are an error, not a silent guess. (Name borrows the distributed-systems term for the scalar reduce family; the scope is scalars.)

This is the borrow-the-term / disclaim-the-scope template the whole epic copies: take the MPI word, then state exactly what scope it does and does not cover.

What this is NOT

See also