Managed worker worktrees

Managed worker worktrees provide filesystem and build isolation for concurrent autonomous coding agents. A git worktree is an additional linked working tree checked out from the same repository; in fak, each worker receives its own private working directory checked out at a detached HEAD (a specific commit hash rather than a branch name). This architecture allows parallel workers to edit, build, and test code simultaneously without race conditions, while preserving the repository’s strict single-trunk discipline.

This guide provides the complete operator reference and runbook for discovering defaults, configuring environment variables, driving lifecycle operations, and recovering from crashes. See also the CLI reference, AGENTS.md for shared-trunk rules, CONTRIBUTING.md for the contributor workflow, and WIP inventory for tracking uncommitted work across checkouts.

Architecture and core principles

Concurrent execution on a shared repository creates three major bottlenecks when multiple workers execute in the same working tree (#1334 / #1333):

  1. Shared git index lock: Simultaneous git commands collide on .git/index.lock.
  2. Build cache collisions: Shared Go build caches (GOCACHE) cause intermediate compilation artifacts from one worker to turn another worker’s builds red.
  3. Dirty working tree cross-talk: In-flight uncommitted edits from one session leak into diffs and status checks of another session.

Managed worker worktrees eliminate these issues while adhering to the single-source-of-truth trunk law (OFF_TRUNK):

Portable defaults discovery

Operators and automation scripts can inspect the active worktree configuration without modifying any state using the defaults sub-command:

fak worktree worker defaults
fak worktree worker defaults --json

The plain-text output displays human-readable paths:

schema: fak.worktree.defaults.v1
repo_root: /path/to/repo
worker_worktree_root: /path/to/worker-worktrees
root_source: environment: FLEET_WORKER_WORKTREE_ROOT
default_lease_identity_basis: lane_key_timestamp
supported_env_overrides: FLEET_WORKER_WORKTREE_ROOT

With --json, it emits machine-readable JSON adhering to the fak.worktree.defaults.v1 schema:

{
  "schema": "fak.worktree.defaults.v1",
  "repo_root": "/path/to/repo",
  "worker_worktree_root": "/path/to/worker-worktrees",
  "root_source": "environment: FLEET_WORKER_WORKTREE_ROOT",
  "default_lease_identity_basis": "lane_key_timestamp",
  "supported_env_overrides": ["FLEET_WORKER_WORKTREE_ROOT"]
}

Zero-mutation guarantee: The defaults command is strictly read-only. It performs no disk writes, git operations, or ref mutations.

Storage roots and environment configuration

Worker root resolution

Managed worktrees live outside the repository working tree so they never appear in git status or interfere with uncommitted file tracking. The parent directory for all worker worktrees is resolved in order:

  1. Environment override: The FLEET_WORKER_WORKTREE_ROOT environment variable if set and non-empty (root_source: "environment: FLEET_WORKER_WORKTREE_ROOT").
  2. Windows OS fallback: %LOCALAPPDATA%\Fleet\worker-worktrees if LOCALAPPDATA is defined (root_source: "os_fallback: LOCALAPPDATA").
  3. Non-Windows OS fallback: $TMPDIR/Fleet/worker-worktrees or /tmp/Fleet/worker-worktrees via os.TempDir() (root_source: "os_fallback: temp_dir").

Directory naming convention

Each worktree directory follows the deterministic naming scheme:

fak-worker-wt-<lane>-<hashed-key>

Build isolation environment variables

When a worker process executes inside a managed worktree, its child environment is populated with isolated paths:

Variable Value Purpose
GOCACHE <worktree>/.gocache Private Go build cache; prevents cross-worker compiler pollution.
GOTMPDIR <worktree>/.gotmp Private temporary directory for compiler operations.
DISPATCH_WORKSPACE <worktree> Repoints tools to the isolated workspace root.
FLEET_WORKER_WORKTREE_DIR <worktree> Identifies the active worktree directory to child processes.

Disposable build directories (.gocache and .gotmp) are created by EnsureBuildDirs during preparation, recreated upon reuse, and purged upon reap.

Lifecycle operations runbook

The managed worker worktree lifecycle follows an orderly state progression:

[PREPARE] -> [WORK & TEST] -> [LAND] -> [REAP]
    |                             |
    +-----> (on crash) ---------> [RECOVER]

1. prepare — Create or lease a detached worktree

Prepares an isolated worktree directory pinned at trunk HEAD (or an explicit commit SHA), stamped with ownership metadata.

fak worktree worker prepare --lane <lane> --key <key> [flags]

Flags

Behavior and output

2. list — Inspect inventory and status evidence

Enumerates active managed worktrees and audits their lifecycle state:

fak worktree worker list
fak worktree worker list --json

Flags

Output details

3. land — Apply worktree diff back to main

Applies the worktree’s diff-since-base back onto the main trunk as a single verified commit.

fak worktree worker land --worktree <dir> [flags]

Flags

Landing mechanics and safety guarantees

  1. Local recovery ref anchor: Before touching the trunk ref, the candidate commit is anchored locally at refs/fak/worker-land/<worktree-name>/<candidate-sha>. If the process terminates during landing, the commit is not lost.
  2. Isolated index: Staging and commit construction execute in a throwaway index (GIT_INDEX_FILE), avoiding contention on the shared .git/index.
  3. Compare-and-swap (CAS): Trunk HEAD is updated using an atomic CAS ref update. If a peer landed in the gap, CAS fails and retries (up to 5 attempts).
  4. Readback verification: After committing, LandReadbackVerify confirms trunk HEAD contains the worker’s intended paths (LAND_READBACK_MISMATCH refusal if missing).
  5. Symptom witness: Fix commits (fix(...)) must include a test that reproduces the failure on the parent commit and passes on the fix.

4. reap — Clean removal and bulk sweeps

Releases worktrees that are no longer needed. Supports single-worktree targeted mode and bulk cold sweeps:

fak worktree worker reap --worktree <dir> [--superseded-by <sha>] [--max-wait <duration>]
fak worktree worker reap --all-cold [--apply] [--age-floor-min <min>] [--even-if-unlanded]

Flags

Bulk sweep safety invariants

5. gc — Owner-stamped leak garbage collection

Performs leak garbage collection targeting worktrees abandoned by crashed processes:

fak worktree worker gc [--max-age <duration>]
fak worktree worker gc --apply [--max-age <duration>]

Flags

Dual qualification rule

A worktree is eligible for gc removal only when both conditions are proven:

  1. Owner process is dead: The PID recorded in the owner stamp no longer exists.
  2. Lane lease is released: The lease oracle confirms the stamped lease is inactive.

6. publish and recover — Remote publication and crash recovery

Provides durability against local host loss and tools for resuming interrupted lands.

Remote publication

Publishes a scrubbed snapshot of local worktree states to a remote Git ref:

fak worktree worker publish --remote origin --dry-run
fak worktree worker publish --remote origin --apply

Crash recovery inventory

When a worker crashes after commit-tree but before or during trunk CAS, the candidate commit remains anchored under refs/fak/worker-land/<worktree>/<candidate-sha>. To inspect and recover:

fak worktree worker recover
fak worktree worker recover --remote origin --fetch

Recovery candidate states

State Meaning Recommended action
LOCAL_ONLY Candidate exists locally; protected against process crash. Inspect with git show <ref>; re-run land or cherry-pick.
REPLICATED Candidate exists locally and in verified remote mirror. Safe against host loss; re-run land or cherry-pick.
REMOTE_ONLY Found on remote mirror but missing locally (fresh clone). Restore local ref via printed command, then inspect/land.
LANDED Git history proves the candidate is already in trunk HEAD. Safe to clean up.

Guarded recovery cleanup

Local recovery refs can be pruned once landed:

fak worktree worker recover --cleanup refs/fak/worker-land/<worktree>/<sha>
fak worktree worker recover --cleanup refs/fak/worker-land/<worktree>/<sha> --force

Remote recovery refs require ancestry verification:

fak worktree worker recover --remote origin \
  --cleanup-remote refs/fak/worker-land/<worktree>/<sha> \
  --worktree-name <worktree>

fak worktree worker recover --remote origin \
  --cleanup-remote refs/fak/worker-land/<worktree>/<sha> \
  --worktree-name <worktree> --apply

Cleaning a peer’s remote recovery ref additionally requires --allow-peer.

Sub-commands summary

Sub-command Purpose Default mode Primary receipt / schema
defaults Discover resolved roots and supported env overrides Read-only fak.worktree.defaults.v1
prepare Create/lease detached worktree with isolated env Mutating worktreePrepareOut
list Enumerate active worktrees and lifecycle states Read-only fak-worker-worktree-lifecycle/1
land Apply worktree diff onto trunk as a verified commit Mutating workerworktree.Result
reap Release single worktree or perform bulk cold sweep Dry-run (--all-cold) worktreeColdReapOut
gc Collect dead-owner, released-lease worktrees Dry-run workerworktree.GCReport
publish Publish scrubbed host lifecycle snapshot to remote Dry-run SnapshotPublishResult
recover Enumerate recovery candidates and clean up landed refs Read-only worktreeWorkerRecoverOut