felhom.eu/documentation/audits/SPIKE-controller-escrow-2026-07-13.md

SPIKE — Controller-driven escrow ceremony: daemon-context invocation mechanics (2026-07-13)

Empirical validation of every mechanism the planned controller-driven escrow ceremony (controller → agent local API → sudo self-invocation → ceremony → R returned once) depends on, run BEFORE the production spec is written. All five mechanisms under test: GO. No production code was written; the deployed agent binary was exercised as-is.

Hygiene note, stated up front: this spike deliberately violated the ceremony runbook's "never pipe the output" rule (RUNBOOK-escrow-ceremony.md §Do NOT) — R was captured on a pipe, programmatically parsed, and round-trip-consumed. This happened ONLY on the non-production drill VM; every captured R is a throwaway test-box secret and every capture file was shredded (P8). No R value, blob bytes, or fingerprint+R pair appears in this document.

1. Target + baselines

Item	Recorded
Environment	Drill VM (qm 300 drill-day0 on felhom-pve, nested PVE) — running at spike start, LAN 192.168.0.152, root key-SSH (the DRILL-day0-take2 access path, reused verbatim); left running at spike end
Box state at start	The DRILL-day0-take2-2026-07-12 end state: host demo-vm-felhom-2f4b00, escrowed, offsite round-trip proven
Deployed agent	felhom-agent 0.87.0 (service active, non-root felhom-agent uid 999)
escrow.pbs_storage_id	felhom-pbs (agent.json; no secret values read)
PBS key file	/etc/pve/priv/storage/felhom-pbs.enc present (255 B, 0600 root:www-data) + .pw
age	/usr/bin/age 1.2.1
Hub reachability	https://hub.felhom.eu/ → 302 from the target
Staged restic password	ABSENT (/var/lib/felhom-agent/escrow-stage/ empty — the take-two ceremony wiped it). Drill target → proceed per the P0 gate; consequence: the P5 blob carries an empty restic_pw_sha256 (verified hub-side, §2.5)
sudo / ptmx	sudo 1.9.16p2; /dev/ptmx 0666
Probes run	P0–P8 all ran, none skipped (drill target → full plan incl. P5 --upload)
Repo baselines	felhom-agent main @ adf7882f7dd6 v0.87.0 (untouched); felhom.eu docs-only commit

Demo-host blast radius: nothing on 192.168.0.162 (host level) or the live demo escrow was touched; demo-felhom-01's hub escrow row still timestamps 2026-07-09 14:16:16 after the spike (§2.5).

2. SQ verdicts

2.1 SQ1 — PTY allocation under NO controlling terminal (the load-bearing result): WORKS

The full ceremony — including the PBS key change-passphrase re-key that pty_linux.go drives over a manually allocated /dev/ptmx pty — succeeded from a daemon-equivalent context (systemd transient unit, service uid/gid, no TTY anywhere in the chain), 3/3 runs:

systemd-run --wait --pipe --collect --uid=felhom-agent --gid=felhom-agent \
  /usr/bin/sudo -n /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json \
  --selftest=escrow-create --offline

Every run: exit 0; R banner + a plausible EFF-wordlist R on stdout (one dash-joined token, 10 words / 69 chars, ~129 bits R printed); blob: 383 bytes … key fingerprint f2:87:…:f7:8e · posture zero_knowledge; the self-verify: the blob unwraps back to the key with R line present (self-verify itself exercises a SECOND pty round for the unwrap re-key — both directions work no-TTY); identity escrow: 450 bytes … self-verify OK. The pty path needs no controlling terminal at all: Setsid + Setctty on the slave gives the CHILD its controlling terminal regardless of the parent having none.

stderr per run: only the two slog INFO lines (+wg_private_key, creating zero-knowledge recovery-code escrow with field names, never values) + systemd-run framing. No secret material on stderr.

2.2 sudoers exact-argv refusal matrix (the spike's red-proof): 5/5 REFUSED

Temp drop-in /etc/sudoers.d/zz-felhom-escrow-spike (0440 root:root, visudo -cf-gated), exactly two fixed-argv lines (recorded verbatim):

felhom-agent ALL=(root) NOPASSWD: /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --offline
felhom-agent ALL=(root) NOPASSWD: /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --upload

Each probe run as the felhom-agent user via sudo -n (daemon shape):

#	Altered argv	Result
(a)	value altered: --selftest=escrow-consume --offline	REFUSED — exit 1, sudo: a password is required, ceremony never spawned
(b)	extra flag appended: … --offline --paperkey	REFUSED — same
(c)	argv reordered: --selftest… --config … --offline	REFUSED — same
(d)	--config omitted	REFUSED — same
(e)	different config path: --config /tmp/evil.json	REFUSED — same

The exact allowed argv was accepted (that is P3/P5 themselves), and after the P8 removal the same argv is refused again (post-cleanup re-check: sudo: a password is required).

Consequences confirmed: sudoers matches the argument vector byte-for-byte — the production caller must pin argv ORDER exactly as written in the sudoers line. Extra trap recorded: Go's flag package accepts both -flag and --flag spellings, but sudoers only matches the literal form in the line — the caller must emit the exact ---spelled argv, never normalize.

2.3 R pipe-capture fidelity + round-trip: PROVEN

From one P3 run's captured stdout, entirely on the box (R never left it): R parsed programmatically (the first single-token line after the banner box's └ edge); fingerprint parsed from the blob: line; the --offline base64 decoded to a root-0600 scratch blob (383 B). Then, as root directly:

FELHOM_RECOVERY_CODE='<captured R>' felhom-agent --config /etc/felhom-agent/agent.json \
  --selftest=escrow-consume --blob <scratch> --fingerprint f2:87:… --keydest <0600 scratch>

→ exit 0, [OK] recovered key installed … (fingerprint-gated, 0600) — the recovered key is 255 bytes, the live key file's exact size. The R that crossed a pipe is the R that opens the blob — the exact property the controller endpoint relies on. Both scratch files shredded immediately (keydest right after the consume, blob at P8).

Banner-parse brittleness (motivates --output=json): R is identifiable only positionally (after Unicode box-drawing lines) as "the indented single-token line"; the fingerprint sits inside a ·-separated human line; the offline copy is "the long base64-shaped line". Every one of these breaks on any cosmetic banner change. A machine mode is not optional for production.

2.4 Ceremony under sudo -n env_reset: CLEAN

With env_reset stripping the environment (including FELHOM_AGENT_CONFIG — which is why the sudoers line pins --config explicitly): config discovery worked from the pinned path; escrow.pbs_storage_id resolved; the live WG key was auto-captured (identity bundle: +wg_private_key, identity=true in the banner); the hub upload leg (P5) authenticated and completed. Nothing in the ceremony depends on inherited environment.

2.5 Full --upload path (P5, drill VM only): WORKS, hub-verified

Same systemd-run shape with the --upload line: exit 0, 2.35 s wall, uploaded the opaque blob(s) to the hub (host record); the hub cannot open them. Hub-side proof (read-only sqlite query of host_escrow on the k3s node's Longhorn mount): demo-vm-felhom-2f4b00 | blob 383 B | restic_pw_sha256 EMPTY | updated_at 2026-07-13 15:06:50Z — the timestamp is the P5 run to the second. The empty hash is correct: no staged secret existed (P0), so nothing was sealed for auto-confirm to match. demo-felhom-01's row unchanged.

2.6 Timings (→ the production job budget)

Run	Mode	Exit	Wall (SSH-side)	Service runtime (systemd)
P3-1	--offline	0	2.28 s	2.26 s
P3-2	--offline	0	2.30 s	—
P3-3	--offline	0	2.37 s	—
P5	--upload	0	2.35 s	2.33 s

min/median/max (offline): 2.28 / 2.30 / 2.37 s; upload adds ≈ nothing on LAN (budget for WAN upload latency anyway). Memory peak of the ceremony process: ~264 MiB (systemd accounting) — the job endpoint spawns a second full agent process; worth remembering on small hosts.

2.7 P6 — staged-dir permission interplay: NO STRANDING

/var/lib/felhom-agent/escrow-stage/ is 0700 felhom-agent:felhom-agent. Root created a 0600 root:root probe file (spike-probe, deliberately NOT the real staged filename) inside it; the felhom-agent user unlinked it cleanly (unlink permission comes from the directory, which the daemon owns). The production root-ceremony-wipes / daemon-re-stages cycle has no permission trap.

2.8 P7 — R hygiene sweep: ZERO leaks

For each of the 4 captured Rs (3× P3 + P5): the full R and its first two words grepped across the ENTIRE journal (journalctl, incl. -u felhom-agent), auth.log*, and /var/lib/felhom-agent/ → 0 hits total. sudo logs the argv only (verified lines carry the fixed flags, no secrets); journald never sees R.

3. Recommendations for the implementation spec

1. Production sudoers line — a single fixed-argv --upload variant (the --offline line was probe-only), as a new alias in configs/felhom-agent.sudoers, refusal-matrix-validated shape:

   Cmnd_Alias FELHOM_ESCROW = \
     /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --upload
   

   The caller must exec exactly this argv (order + -- spelling pinned; §2.2). --config stays pinned explicitly: env_reset strips FELHOM_AGENT_CONFIG, and the pin closes alternate-config injection (probe (e)).

2. --output=json machine mode — grounded in §2.3's brittleness: a SINGLE JSON object on stdout ({recovery_code, key_fingerprint, entropy_bits, blob_bytes, identity_blob_bytes, restic_pw_sealed, uploaded}), banner suppressed, everything human-facing to stderr. All human printing already lives in runSelftestEscrowCreate (the CLI shell), not in escrow.Create — the variant is a thin switch in main.go, no library change.

3. Job + one-shot in-memory R-claim endpoint (the netstorage job pattern — mandatory anyway, the agentapi client timeout is 15 s): measured ceremony ≈ 2.4 s incl. upload, so a poll interval 2 s, job timeout 60 s is a ≥25× margin over LAN reality while absorbing WAN upload latency. R held in memory only, single-flight mutex (one ceremony at a time per host), one-shot claim (read once → wiped), TTL ~10 min; recovery story: "R unclaimed → ceremony void → a re-run supersedes" — which §2.5 shows is exactly how the blob store behaves (a new upload replaces the row; the superseded R opens only pre-existing history).

4. Capability-manifest caveat — CORRECTED by this spike. The task premise ("manifest.go probes EXECUTE their argv") is FALSE at live source: internal/capability/manifest.go probes are list-mode (sudo -n -l <argv>, "never executing" — its own header, L3–4/L46–47), verified live here (sudo -n -l on the escrow line: exit 0, echoes the grant, no ceremony ran). The escrow line can therefore get a NORMAL probe entry safely — recommend adding a standard escrow-ceremony capability row so degradation is visible on the hub like every other grant, no special representation needed.

5. Ruling F1 (2026-07-13), carried: R transiting the Cloudflare tunnel is an accepted, documented risk (same trust class as the claim code / login password). The ceremony spec MUST carry a threat-model paragraph documenting this acceptance explicitly.

6. Minor spec inputs: exit codes observed/confirmed in source — 2 = usage/config error, 1 = operational failure, 0 = success; the ceremony process peaks ~264 MiB (§2.6); stderr is log-clean (§2.1) so the job runner may capture it for diagnostics without an R filter, but stdout must be treated as secret-bearing until parsed + wiped.

4. Cleanup checklist (P8)

• /etc/sudoers.d/zz-felhom-escrow-spike removed; visudo -c on the remaining set: parsed OK
• Post-removal red-check: the previously allowed argv is refused again
• All scratch files shredded (spike-p3-run{1,2,3}.{out,err}, spike-p5-run.{out,err}, spike-p4-blob; spike-p4-keydest shredded immediately after the P4 consume; P6 spike-probe unlinked; P2 temp files removed)
• Drill VM left in its prior power state (running — the take-two end state, as found)
• Demo host / live demo escrow untouched (demo-felhom-01 hub escrow row still 2026-07-09; no ceremony was run outside the drill VM) — the demo-path §P8 check is otherwise N/A on the drill target

5. Observations (out of scope — documented, NOT acted on)

• The drill box's escrow blob was superseded by this spike (deliberately, P5): the take-two paper R (2026-07-12 22:46) no longer opens the CURRENT hub blob (it still opens the superseded one for pre-existing history, per the R-supersede rule). The new blob has an empty restic_pw_sha256. The drill guest's controller showed no reaction within ~6 min of the upload (no escrow log lines); whether an escrowed controller re-warns on a later ACK carrying a hash-less blob is version behavior worth one glance at the next drill reset — the box is re-drill scratch either way.
• The hub-side verification path used here (read-only sqlite3 query of host_escrow via the node's Longhorn mount) is a useful CC-side check pattern for hub state that the password-gated UI otherwise hides.
• systemd-run reports the transient unit's resource envelope for free (runtime, CPU, memory peak) — handy shape for the job runner's own accounting/logging.
• The spike's sudo journal lines confirm sudo logs the FULL argv of both accepted and refused invocations — with fixed-argv design this is secrets-free by construction and doubles as an audit trail of ceremony invocations.