# SPIKE — Controller-driven escrow ceremony: daemon-context invocation mechanics (2026-07-13)

Empirical validation of every mechanism the planned controller-driven escrow ceremony
(controller → agent local API → `sudo` self-invocation → ceremony → R returned once) depends on,
run BEFORE the production spec is written. **All five mechanisms under test: GO.** No production
code was written; the deployed agent binary was exercised as-is.

> Hygiene note, stated up front: this spike deliberately violated the ceremony runbook's
> "never pipe the output" rule (RUNBOOK-escrow-ceremony.md §Do NOT) — R was captured on a pipe,
> programmatically parsed, and round-trip-consumed. This happened ONLY on the non-production
> drill VM; every captured R is a throwaway test-box secret and every capture file was shredded
> (P8). No R value, blob bytes, or fingerprint+R pair appears in this document.

## 1. Target + baselines

| Item | Recorded |
|---|---|
| Environment | **Drill VM** (qm 300 `drill-day0` on felhom-pve, nested PVE) — running at spike start, LAN `192.168.0.152`, root key-SSH (the DRILL-day0-take2 access path, reused verbatim); left running at spike end |
| Box state at start | The DRILL-day0-take2-2026-07-12 end state: host `demo-vm-felhom-2f4b00`, escrowed, offsite round-trip proven |
| Deployed agent | **felhom-agent 0.87.0** (service active, non-root `felhom-agent` uid 999) |
| `escrow.pbs_storage_id` | `felhom-pbs` (agent.json; no secret values read) |
| PBS key file | `/etc/pve/priv/storage/felhom-pbs.enc` present (255 B, 0600 root:www-data) + `.pw` |
| `age` | `/usr/bin/age` 1.2.1 |
| Hub reachability | `https://hub.felhom.eu/` → 302 from the target |
| Staged restic password | **ABSENT** (`/var/lib/felhom-agent/escrow-stage/` empty — the take-two ceremony wiped it). Drill target → proceed per the P0 gate; consequence: the P5 blob carries an **empty `restic_pw_sha256`** (verified hub-side, §2.5) |
| sudo / ptmx | sudo 1.9.16p2; `/dev/ptmx` 0666 |
| Probes run | **P0–P8 all ran, none skipped** (drill target → full plan incl. P5 `--upload`) |
| Repo baselines | felhom-agent `main` @ `adf7882f7dd6` v0.87.0 (untouched); felhom.eu docs-only commit |

Demo-host blast radius: nothing on 192.168.0.162 (host level) or the live demo escrow was touched;
`demo-felhom-01`'s hub escrow row still timestamps `2026-07-09 14:16:16` after the spike (§2.5).

## 2. SQ verdicts

### 2.1 SQ1 — PTY allocation under NO controlling terminal (the load-bearing result): **WORKS**

The full ceremony — including the PBS `key change-passphrase` re-key that `pty_linux.go`
drives over a manually allocated `/dev/ptmx` pty — succeeded from a daemon-equivalent context
(systemd transient unit, service uid/gid, no TTY anywhere in the chain), 3/3 runs:

```
systemd-run --wait --pipe --collect --uid=felhom-agent --gid=felhom-agent \
  /usr/bin/sudo -n /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json \
  --selftest=escrow-create --offline
```

Every run: exit 0; R banner + a plausible EFF-wordlist R on stdout (one dash-joined token,
10 words / 69 chars, `~129 bits R` printed); `blob: 383 bytes … key fingerprint f2:87:…:f7:8e ·
posture zero_knowledge`; the `self-verify: the blob unwraps back to the key with R` line present
(self-verify itself exercises a SECOND pty round for the unwrap re-key — both directions work
no-TTY); `identity escrow: 450 bytes … self-verify OK`. The pty path needs no controlling
terminal at all: `Setsid` + `Setctty` on the slave gives the CHILD its controlling terminal
regardless of the parent having none.

stderr per run: only the two slog INFO lines (`+wg_private_key`, `creating zero-knowledge
recovery-code escrow` with field names, never values) + systemd-run framing. **No secret
material on stderr.**

### 2.2 sudoers exact-argv refusal matrix (the spike's red-proof): **5/5 REFUSED**

Temp drop-in `/etc/sudoers.d/zz-felhom-escrow-spike` (0440 root:root, `visudo -cf`-gated),
exactly two fixed-argv lines (recorded verbatim):

```
felhom-agent ALL=(root) NOPASSWD: /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --offline
felhom-agent ALL=(root) NOPASSWD: /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --upload
```

Each probe run as the `felhom-agent` user via `sudo -n` (daemon shape):

| # | Altered argv | Result |
|---|---|---|
| (a) | value altered: `--selftest=escrow-consume --offline` | **REFUSED** — exit 1, `sudo: a password is required`, ceremony never spawned |
| (b) | extra flag appended: `… --offline --paperkey` | **REFUSED** — same |
| (c) | argv reordered: `--selftest… --config … --offline` | **REFUSED** — same |
| (d) | `--config` omitted | **REFUSED** — same |
| (e) | different config path: `--config /tmp/evil.json` | **REFUSED** — same |

The exact allowed argv was accepted (that is P3/P5 themselves), and after the P8 removal the
same argv is refused again (post-cleanup re-check: `sudo: a password is required`).

**Consequences confirmed:** sudoers matches the argument vector byte-for-byte — the production
caller must pin argv ORDER exactly as written in the sudoers line. Extra trap recorded: Go's
`flag` package accepts both `-flag` and `--flag` spellings, but sudoers only matches the literal
form in the line — the caller must emit the exact `--`-spelled argv, never normalize.

### 2.3 R pipe-capture fidelity + round-trip: **PROVEN**

From one P3 run's captured stdout, entirely on the box (R never left it):
R parsed programmatically (the first single-token line after the banner box's `└` edge);
fingerprint parsed from the `blob:` line; the `--offline` base64 decoded to a root-0600 scratch
blob (383 B). Then, as root directly:

```
FELHOM_RECOVERY_CODE='<captured R>' felhom-agent --config /etc/felhom-agent/agent.json \
  --selftest=escrow-consume --blob <scratch> --fingerprint f2:87:… --keydest <0600 scratch>
```

→ exit 0, `[OK] recovered key installed … (fingerprint-gated, 0600)` — the recovered key is
255 bytes, the live key file's exact size. **The R that crossed a pipe is the R that opens the
blob** — the exact property the controller endpoint relies on. Both scratch files shredded
immediately (keydest right after the consume, blob at P8).

**Banner-parse brittleness (motivates `--output=json`):** R is identifiable only positionally
(after Unicode box-drawing lines) as "the indented single-token line"; the fingerprint sits
inside a `·`-separated human line; the offline copy is "the long base64-shaped line". Every one
of these breaks on any cosmetic banner change. A machine mode is not optional for production.

### 2.4 Ceremony under `sudo -n` env_reset: **CLEAN**

With env_reset stripping the environment (including `FELHOM_AGENT_CONFIG` — which is why the
sudoers line pins `--config` explicitly): config discovery worked from the pinned path;
`escrow.pbs_storage_id` resolved; the live WG key was auto-captured (`identity bundle:
+wg_private_key`, `identity=true` in the banner); the hub upload leg (P5) authenticated and
completed. Nothing in the ceremony depends on inherited environment.

### 2.5 Full `--upload` path (P5, drill VM only): **WORKS**, hub-verified

Same systemd-run shape with the `--upload` line: exit 0, 2.35 s wall,
`uploaded the opaque blob(s) to the hub (host record); the hub cannot open them`.
Hub-side proof (read-only sqlite query of `host_escrow` on the k3s node's Longhorn mount):
`demo-vm-felhom-2f4b00 | blob 383 B | restic_pw_sha256 EMPTY | updated_at 2026-07-13 15:06:50Z`
— the timestamp is the P5 run to the second. The empty hash is correct: no staged secret existed
(P0), so nothing was sealed for auto-confirm to match. `demo-felhom-01`'s row unchanged.

### 2.6 Timings (→ the production job budget)

| Run | Mode | Exit | Wall (SSH-side) | Service runtime (systemd) |
|---|---|---|---|---|
| P3-1 | `--offline` | 0 | 2.28 s | 2.26 s |
| P3-2 | `--offline` | 0 | 2.30 s | — |
| P3-3 | `--offline` | 0 | 2.37 s | — |
| P5 | `--upload` | 0 | **2.35 s** | 2.33 s |

min/median/max (offline): 2.28 / 2.30 / 2.37 s; upload adds ≈ nothing on LAN (budget for WAN
upload latency anyway). Memory peak of the ceremony process: **~264 MiB** (systemd accounting) —
the job endpoint spawns a second full agent process; worth remembering on small hosts.

### 2.7 P6 — staged-dir permission interplay: **NO STRANDING**

`/var/lib/felhom-agent/escrow-stage/` is 0700 `felhom-agent:felhom-agent`. Root created a 0600
root:root probe file (`spike-probe`, deliberately NOT the real staged filename) inside it; the
`felhom-agent` user unlinked it cleanly (unlink permission comes from the directory, which the
daemon owns). The production root-ceremony-wipes / daemon-re-stages cycle has no permission trap.

### 2.8 P7 — R hygiene sweep: **ZERO leaks**

For each of the 4 captured Rs (3× P3 + P5): the full R and its first two words grepped across
the ENTIRE journal (`journalctl`, incl. `-u felhom-agent`), `auth.log*`, and
`/var/lib/felhom-agent/` → **0 hits total**. sudo logs the argv only (verified lines carry the
fixed flags, no secrets); journald never sees R.

## 3. Recommendations for the implementation spec

1. **Production sudoers line** — a single fixed-argv `--upload` variant (the `--offline` line was
   probe-only), as a new alias in `configs/felhom-agent.sudoers`, refusal-matrix-validated shape:

   ```
   Cmnd_Alias FELHOM_ESCROW = \
     /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json --selftest=escrow-create --upload
   ```

   The caller must exec exactly this argv (order + `--` spelling pinned; §2.2). `--config` stays
   pinned explicitly: env_reset strips `FELHOM_AGENT_CONFIG`, and the pin closes alternate-config
   injection (probe (e)).

2. **`--output=json` machine mode** — grounded in §2.3's brittleness: a SINGLE JSON object on
   stdout (`{recovery_code, key_fingerprint, entropy_bits, blob_bytes, identity_blob_bytes,
   restic_pw_sealed, uploaded}`), banner suppressed, everything human-facing to stderr. All
   human printing already lives in `runSelftestEscrowCreate` (the CLI shell), not in
   `escrow.Create` — the variant is a thin switch in main.go, no library change.

3. **Job + one-shot in-memory R-claim endpoint** (the netstorage job pattern — mandatory anyway,
   the agentapi client timeout is 15 s): measured ceremony ≈ 2.4 s incl. upload, so a
   **poll interval 2 s, job timeout 60 s** is a ≥25× margin over LAN reality while absorbing WAN
   upload latency. R held in memory only, single-flight mutex (one ceremony at a time per host),
   one-shot claim (read once → wiped), **TTL ~10 min**; recovery story: "R unclaimed → ceremony
   void → a re-run supersedes" — which §2.5 shows is exactly how the blob store behaves (a new
   upload replaces the row; the superseded R opens only pre-existing history).

4. **Capability-manifest caveat — CORRECTED by this spike.** The task premise ("manifest.go
   probes EXECUTE their argv") is FALSE at live source: `internal/capability/manifest.go`
   probes are **list-mode** (`sudo -n -l <argv>`, "never executing" — its own header, L3–4/L46–47),
   verified live here (`sudo -n -l` on the escrow line: exit 0, echoes the grant, no ceremony
   ran). **The escrow line can therefore get a NORMAL probe entry safely** — recommend adding a
   standard `escrow-ceremony` capability row so degradation is visible on the hub like every
   other grant, no special representation needed.

5. **Ruling F1 (2026-07-13), carried:** R transiting the Cloudflare tunnel is an accepted,
   documented risk (same trust class as the claim code / login password). The ceremony spec MUST
   carry a threat-model paragraph documenting this acceptance explicitly.

6. Minor spec inputs: exit codes observed/confirmed in source — 2 = usage/config error, 1 =
   operational failure, 0 = success; the ceremony process peaks ~264 MiB (§2.6); stderr is
   log-clean (§2.1) so the job runner may capture it for diagnostics without an R filter, but
   stdout must be treated as secret-bearing until parsed + wiped.

## 4. Cleanup checklist (P8)

- [x] `/etc/sudoers.d/zz-felhom-escrow-spike` removed; `visudo -c` on the remaining set: parsed OK
- [x] Post-removal red-check: the previously allowed argv is refused again
- [x] All scratch files shredded (`spike-p3-run{1,2,3}.{out,err}`, `spike-p5-run.{out,err}`,
      `spike-p4-blob`; `spike-p4-keydest` shredded immediately after the P4 consume;
      P6 `spike-probe` unlinked; P2 temp files removed)
- [x] Drill VM left in its prior power state (running — the take-two end state, as found)
- [x] Demo host / live demo escrow untouched (`demo-felhom-01` hub escrow row still 2026-07-09;
      no ceremony was run outside the drill VM) — the demo-path §P8 check is otherwise N/A on
      the drill target

## 5. Observations (out of scope — documented, NOT acted on)

- **The drill box's escrow blob was superseded by this spike** (deliberately, P5): the take-two
  paper R (2026-07-12 22:46) no longer opens the CURRENT hub blob (it still opens the superseded
  one for pre-existing history, per the R-supersede rule). The new blob has an empty
  `restic_pw_sha256`. The drill guest's controller showed **no reaction within ~6 min** of the
  upload (no escrow log lines); whether an `escrowed` controller re-warns on a later ACK carrying
  a hash-less blob is version behavior worth one glance at the next drill reset — the box is
  re-drill scratch either way.
- The hub-side verification path used here (read-only `sqlite3` query of `host_escrow` via the
  node's Longhorn mount) is a useful CC-side check pattern for hub state that the
  password-gated UI otherwise hides.
- systemd-run reports the transient unit's resource envelope for free (runtime, CPU, memory
  peak) — handy shape for the job runner's own accounting/logging.
- The spike's sudo journal lines confirm sudo logs the FULL argv of both accepted and refused
  invocations — with fixed-argv design this is secrets-free by construction and doubles as an
  audit trail of ceremony invocations.