docs(agent): D1 REPORT — v0.70.0 shipped + live-validated (happy path, ~2s crash-rollback, no-pending guard, gate refusal)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

REPORT.md

@@ -1,73 +1,112 @@
-# REPORT — S5: host-loss DR (recovered WG-key install + directive→PLAN; safe halves shipped)
+# REPORT — TASK D1: agent self-update (operator-signed, A/B slots, crash-loop auto-rollback)
 
-**Date:** 2026-07-04 · **Class:** implementation (safe halves) + diagnosis + a prepared,
-operator-present destructive drill (NOT executed by CC). **Design of record:** doc-06 §3.5 + S5 row;
-doc-01 §9.
+**Date:** 2026-07-05 · **Class:** implementation (code + host artifacts + tests + deploy + live drills)
+**Baselines (reconfirmed):** felhom-agent `main` @ `72d7f05` (v0.69.0) → **v0.70.0**; felhom.eu @ `582917d`.
+Not behind origin. Design provenance: `felhom.eu/documentation/audits/SPIKE-agent-selfupdate-2026-07-05.md`.
+**Shipped + live-validated on felhom-pve (demo host only; Peti's box untouched).**
 
-## 1. Baselines & commits (felhom-agent → `main`)
+## Commits
 
-Start: `567cf9f` (v0.68.0). **`bd4bced`** v0.69.0 — the two safe DR halves (Parts 1-2). This commit
-— REPORT + CONTEXT + doc-06 S5 row.
+| Repo | Hash | What |
+|---|---|---|
+| felhom-agent | `b7cbded` | P1 host artifacts (wrapper, rollback unit, [Unit] limits drop-in, sudoers alias) |
+| felhom-agent | `8033a52` | P2 Go plumbing (ClassAgentUpdate, opsign, executor, commit-manager, report field, wiring) |
+| felhom-agent | `915642a` | P3 docs (README self-update, REUSE, CHANGELOG v0.70.0, CONTEXT) |
+| felhom.eu | `b6bad95` | P3 host-install day-0 install of the artifacts + architecture §11 (implemented) |
 
-## 2. Part 1 (SAFE, shipped) — install the recovered WG key
+## What shipped
 
-`wgtunnel.InstallRecoveredKey(stateDir, privB64)` writes an escrow-recovered WG private key (32-byte
-base64, re-encoded canonical) to the key file so the tunnel re-establishes with the **same
-identity/pubkey → same hub /32**, no fresh keygen. **CREATE-ONLY** — refuses if a key file exists (a
-present key may be a live identity); the value is never logged. Wired into
-`--selftest=identity-consume -install-wg-key` (opt-in; installs `bundle.WGPrivateKey` after
-`UnwrapIdentityBundle`; a pre-S3 blob with no WG key → logged fallback to fresh keygen + re-register,
-which keeps the /32 via hub re-key-in-place).
-**Test + red-proof:** install→`EnsureKey` LOADS it (same pubkey, no keygen); a second install REFUSES
-(create-only — red-proofed against dropping the exists-check); invalid key → error, nothing written.
+- **Trust model:** an operator-signed `agent_update` op (`reconcile.ClassAgentUpdate`, always
+  Destructive) through the signed-jobs pipeline; params pin version + sha256 → the sha is the ONLY
+  integrity root (hub = dumb transport, Gitea = dumb storage). `felhom-opsign -op agent_update`.
+- **Host artifacts** (`configs/`): `felhom-selfupdate-guarded` (apply/commit/rollback — sha re-verify
+  as root before `.prev`, path confinement, same-fs assert, atomic mv, pending marker, detached
+  restart; rollback pending-guarded); `felhom-agent-rollback.service`; `felhom-agent-limits.conf`
+  ([Unit]-only, `StartLimitIntervalSec=120`+`Burst=4`+`OnFailure=`); `FELHOM_SELFUPDATE` sudoers.
+- **Go** (`internal/selfupdate/`): `Executor` (download→verify-vs-signed-sha→wrapper apply; job
+  completed after verify+download, before apply) + `Manager` (startup dwell→commit; version-mismatch
+  no-commit; report seam). Wired as the 3rd executor-chain element + a `MaybeCommit` goroutine after
+  core init. Config `SelfUpdateConfig`; additive report `selfupdate_pending`; 3 capability probes.
+- **felhom.eu:** `felhom-host-install.sh` installs wrapper+rollback-unit+drop-in on day-0.
 
-## 3. Part 2 (SAFE, shipped) — consume `restore_directive` → PLAN
+## Tests & companion red-proofs
 
-New `internal/dr`: the host_loss `restore_directive` (was logged-and-ignored in `syncer.go`) is
-consumed via the `desired.Syncer.AddConsumer` raw seam into an inspectable **RestorePlan** — per
-guest `{vmid, archive, target storage, sizing}`, per drive `{durable_id → expected mount}`, + the
-offsite PBS coord. **Derive-and-surface only:** the `Consumer` holds NO restore/destroy dependency, so
-"execute nothing" is structural. `guest_loss`/absent → no plan; recipe fetched on-demand (rare
-directive) via a fresh `Collect`.
-**Test + red-proof:** host_loss builds the expected plan; guest_loss/absent/nil-recipe → none
-(red-proofed against a relaxed mode gate); the consumer surfaces on host_loss + clears otherwise +
-consults the recipe only then. Full `go test ./...` green; deployed 0.69.0; `-install-wg-key` flag
-live; daemon starts clean (dr consumer wired).
+Full `go build ./... && go vet ./... && go test ./...` green; `shellcheck` clean on the wrapper;
+`visudo -cf` on the full sudoers.
 
-## 4. Part 3 — hub escrow export: NOT needed
+- **Executor** (Group A): happy-path (staged sha asserted, apply args exact, job completed before
+  apply); **C2 sha-mismatch refused** + its **companion** (neutering the Go verify → the bad binary
+  reaches the apply call → `TestExecutor_ShaMismatchRefused` fails; verified, reverted); bad-params;
+  wrapper-failure surfacing.
+- **Gate ride-along** (Group B): `agent_update` rides the REAL authz.Verifier + reconcile.Gate —
+  pinned-key executes, non-pinned + retarget rejected. Backing companion:
+  `TestClassify_AgentUpdateAlwaysDestructive` (if it flipped Benign the unsigned op would execute).
+- **Commit** (Group C): dwell→commit; version-mismatch → no commit + WARN + marker left; no-pending
+  no-op; shutdown-before-dwell leaves pending. **Opsign** (Group D): `isHex64`. All packages green.
 
-The operator has a clean blob-export path already: `sqlite3 … "SELECT writefile('/tmp/blob', blob),
-writefile('/tmp/idblob', identity_blob) FROM host_escrow WHERE host_id='demo-felhom-01'"` on a
-`kubectl cp`'d `hub.db`. No hub change (respects the read-only-unless-no-clean-way gate).
+## Live validation (felhom-pve; the full agent-side pipeline ran for real)
 
-## 5. Part 4-A — re-attach wrong-disk safety: ALREADY PROVEN (no new probe built)
+Build v0.70.0 on 180 (sha `c5eb84cc…`), published to Gitea (anon-pullable, round-trip sha OK), and
+manually deployed (the last manual agent deploy). Host artifacts installed; `systemctl show` confirmed
+`StartLimitIntervalUSec=2min StartLimitBurst=4`; capabilities 56/56 degraded=0 (the 3 selfupdate
+probes pass). A scratch operator key was generated + pinned and a `selfupdate` config block added for
+the drill.
 
-The re-attach matcher is `storage.ResolveStorageDevice` (the `uuid:<fs-uuid>` scheme DR drives use).
-Its wrong-disk guard is unit-established: `TestResolveStorageDevice_ToleratesDeviceLetterMove` (a
-MATCHING UUID resolves to the current /dev path) + `TestResolveStorageDevice_AbsentAndScheme` (an
-absent/mismatched UUID **ERRORS** — "skips a gone drive instead of fail-mounting"; a bare node /
-non-`uuid:` scheme is refused). It resolves ONLY the exact UUID's symlink — never a "nearest" disk.
-Combined with S4.1's restore-to-scratch proof, the composition's safety is established; a new live
-scratch+marker-drive probe would add confidence but not new safety, so it was not built (recorded).
+**Delivery method note (stated per the live-validation rule):** the hub's *enqueue* endpoint needs a
+global operator key CC does not hold, so the operator-signed envelope (produced by the real
+`felhom-opsign` over the pinned key) was injected directly into the hub's `signed_jobs` queue —
+exactly what `Store.EnqueueSignedJob` does. Everything downstream is the **real, unmodified pipeline**:
+the agent polls the hub, the real gate verifies the SSHSIG, the real executor downloads+verifies, the
+real wrapper flips, the real commit-manager commits. Only the hub-side enqueue-auth (unit-tested
+hub-side) was bypassed. Pickup was triggered by an agent restart (the loop reports immediately on
+start → envelope `has_signed_ops` → RunOnce).
 
-## 6. Part 4-B — the destructive in-place 9201 drill: PREPARED + OPERATOR-GATED (not executed)
+1. **Happy path (Scenario A) — PASS.** Signed `agent_update{0.70.1, sha 908700ab…}` → gate
+   `allowed=true reason=signed key_id=d1-drill-op` → executor downloaded + verified → wrapper
+   `applied 0.70.1 (prev 0.70.0)` + detached restart → **0.70.1 running**, `.prev`=0.70.0 (sha
+   `c5eb84cc`), pending marker written, job COMPLETED on the queue. After the 20s dwell:
+   `selfupdate: update committed`, marker cleared, `.prev` retained, hub queue empty.
+2. **Crash-loop auto-rollback (Scenario B) — PASS (the safety property).** Signed
+   `agent_update{0.70.2-crash, sha c358d7bc…}` (a throwaway `os.Exit(1)` build, never committed) →
+   applied → **crash-to-recovered ≈ 2 s**: `15:49:50 apply` → `15:49:52 Main process exited
+   status=1/FAILURE` → `Triggering OnFailure=` → `rolled back to previous binary and restarted`.
+   Live binary **byte-identical** to published 0.70.1 (`908700ab…`); pending cleared; hub queue
+   cleared; **stable, no crash loop** (NRestarts=0, active). (Two near-simultaneous OnFailure fires
+   raced; both restored the same `.prev` bytes — idempotent, correct.)
+3. **No-pending guard (C4) — PASS.** `systemctl start felhom-agent-rollback.service` with no pending
+   → wrapper `no pending update — no-op`; MainPID **unchanged**, binary **untouched**.
+4. **Gate refusal (C1) — PASS.** `agent_update` signed by a NON-pinned key, injected + triggered →
+   `REJECTED signed op — executor not called … err="authz: signer not in allowed set"`; nothing
+   downloaded; version unchanged; job cleared.
 
-Pre-flight **GREEN** (non-destructive): offsite `ct/9201` snapshots present (the restore source);
-9201 running; the escrow blob is on the hub (383 B + 499 B identity, offsite key `b0:fe:2a…`),
-exportable via `sqlite3 writefile`; **S4.1 already proved 9201 restores from offsite (pass:true)**.
-The destructive drill (destroy 9201 → restore in place from offsite → re-attach the data drive by
-durable_id) is **operator-present + STOP-gated**: the operator runs the R-consume steps
-(`escrow-consume` + `identity-consume -install-wg-key`, R via env — **CC never runs these**, §9-4a)
-and confirms the destroy at the STOP checkpoint. Prepared procedure + the go/no-go is with the
-operator (destroying the live demo controller is their decision, even with the backup confirmed
-recoverable).
+## Cleanup + final state
 
-## 7. NOT yet live-validated / observations
+- **0.70.2-crash and 0.70.1 deleted from Gitea** (never leave a crashing artifact published); 0.70.0
+  retained (the release).
+- Box restored: **v0.70.0 active/running, NRestarts=0, capabilities 56/56 degraded=0**; original
+  `agent.json` restored (scratch operator key + drill selfupdate block removed — no scratch key left
+  pinned); staging dir empty; `.prev`/nonces cleared. **Host artifacts kept installed** (the shipped
+  feature): wrapper + rollback unit + [Unit] drop-in (3/3). Hub `signed_jobs` queue empty.
+- Drill scratch removed from 180 and the hub pod. `felhom-agent.bak-0.69.0` retained on the box (the
+  rollback safety net, expected).
+- **Self-inflicted incident during cleanup (recorded honestly):** restoring `agent.json` via `mv` of a
+  root-owned backup left it root-owned → the non-root agent got `permission denied` → 4 crashes → the
+  start-limit `failed` state. Fixed with `chown felhom-agent:felhom-agent` + `chmod 0600` +
+  `reset-failed` + start. Not a product defect (a cleanup ownership slip); it did incidentally
+  demonstrate the OnFailure path no-op'ing on a pending-less crash and the `reset-failed` requirement
+  ([SF-4]) live.
 
-- The live consume→install→tunnel→restore pipeline (the Part 4-B drill) — operator-present, pending
-  the operator's go-ahead + R-run. Parts 1-2 are unit-proven + deployed; their live consume path is
-  exercised only in that drill (R-gated).
-- Deferred (per slice scope): guest_loss DR; hub-driven full-auto DR (the hub issuing recovery-mode +
-  directive itself); multi-guest host-loss. true-CGNAT-SIM retest still parked.
-- Observation: the DR plan is surfaced via structured log + `Consumer.LastPlan()`; a report field
-  (so the hub/operator sees the plan without the box's journal) is a small future refinement.
+## Observations (not acted on — v1 scope-outs / follow-ups)
+
+1. **Operator-key provisioning is an operator follow-up** (parallel to the Day-0 vouch): the demo box
+   was returned to its pre-drill config with NO signer pinned, so self-update is installed-but-dormant
+   until an operator pins their real operational key + (optionally) a `selfupdate` config block. The
+   default URL template + anon Gitea pull mean no creds are needed for the download.
+2. **Per-crash OnFailure can double-fire** (spike [SF-1], seen live): two rollback invocations raced,
+   both idempotent. Harmless, but a future hardening could serialize the rollback oneshot
+   (a flock in the wrapper, or `StartLimitIntervalSec` on the rollback unit) to make it single-shot.
+3. **v1 scope-outs (locked, §8):** no hub-floor-driven auto-update, no failed-update auto-retry (the
+   operator re-signs with a fresh nonce), no pending-timeout auto-rollback (a runs-but-never-commits
+   binary is caught by the hub's `host_staleness` + the `selfupdate_pending` report flag).
+4. **Hub enqueue-auth** was not exercised end-to-end from CC (no global operator key available); it is
+   covered by the hub-side unit tests (`desired_test.go` — enqueue requires the global key).