# CONTEXT — felhom-agent working state

> Snapshot of the current state + open threads. Authoritative history lives in `CHANGELOG.md` (top
> entry = current); the end-of-task detail lives in `REPORT.md`.

## R-199 (v0.125.0) — links 6–8 of the recovery chain, assembled and walked

`POST /escrow/recover-offsite-password` (pinned local API, `withGuest`): the controller supplies the
customer's recovery code, the agent fetches THIS host's own sealed blob from the hub
(`hub.Client.FetchIdentityEscrow` → `GET /hosts/{id}/escrow`, hub >= v0.94.0, self-scoped by the
per-host key), unseals it via `escrow.OffsiteKeyRecoverer`, and returns **only** the offsite restic
repository password plus its sha256.

**Rules that must not erode:**
- **Only that field.** Not the tunnel token, not the PBS token, not the WG key — the controller is a
  trust tier down and needs none of them. Narrowing cost nothing and is not recoverable later.
- **The unseal stays in the agent.** `age` is an agent runtime dependency (`/usr/bin/age` — hardcoded,
  no config override; 1.2.1 on demo-felhom) and is deliberately absent from the controller image.
- **R:** in memory for one call, cleared on the success path AND every failure path, never on disk,
  never in argv, never logged at any level including inside an error, never echoed. Verified live: 0
  log lines, 0 files, 0 leftover `felhom-idesc-*` dirs, with a positive control proving the search worked.
- **Three distinct outcomes**, not one generic failure: no blob (404), a bundle that opens but predates
  the field (409 — pre-fork-4, cannot be retro-fitted), a code that does not open it (400 — fail-closed
  at age's KDF, nothing written).
- **The wiring is pinned by an AST walk** (`cmd/felhom-agent/escrow_recover_wiring_test.go`):
  `main` → `runDaemon` → `buildLocalAPIServer`, an `escrow.OffsiteKeyRecoverer` constructed there, the
  `Options.EscrowRecovery` field present, and the fetcher calling the DAEMON's own `hubClient` (the
  self-scoping that makes cross-host retrieval impossible is a property of WHICH key is used).
  Links 6 and 7 were two of this project's six built-but-never-wired instances.

**Proven live on demo-felhom 2026-08-04:** recovered sha256 == on-disk sha256 == the hub's stored hash.
A wrong code five minutes earlier failed closed. **The chain stops at link 8** — nothing installs a
recovered password, reopens a repository, or restores a file.

**§8.6, fixed while here:** `runSelftestIdentityConsume`'s success line used to recite
"tunnel_token + pbs_token", which became a misstatement when v0.77.0 sealed the repository password
into the same bundle — anyone reading it would conclude the password was not there. It now names what
THIS bundle carried and what it did not.

## Current

- **2026-08-03 — v0.123.0 (R-185): a tier the box cannot READ now says so.** The agent's token had
  `FelhomAgentStore` on `local`, `local-lvm`, `felhom-pbs` and **not** on `felhom-backup` — the
  storage both demo boxes configure as `local_backup_target`. That storage answered `{"data":[]}`
  through the token while root listed three archives, and `pickForThisRun` skipped it as *"no settled
  archive yet"* — **which is what a brand-new tier reports**, so the host tier was never
  restore-testable and nothing said so.
  - **The permission question is asked directly**, because unlike the listing it has a definite
    answer: `Client.Permissions` reads `/access/permissions?path=/storage/<target>` **as the agent's
    own token**, and `storeGrantStatuses` emits one `capability.Status` per configured tier. It
    composes AROUND the sudo prober, the way `poolReadStatus` already does — an API read does not
    belong inside a sudo-policy probe. `Status`'s wire shape is untouched, so the hub's critical
    degraded alert applies with **no hub change**.
  - **MEASURED FIRST, and the obvious reading is wrong:** an ungranted path answers neither empty nor
    403 — it carries the privileges INHERITED from the box-wide `/` grant
    (`Sys.Audit, SDN.Use, Datastore.Audit`). Checking path-presence, or `Datastore.Audit`, reports a
    blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**; re-measure before ever
    changing that constant (`storeGrantRequiredPriv`, red-proved).
  - **The probed set comes from `BackupTiers()`, never a fixed list** — a hardcoded probe list is the
    defect reproduced inside the fix. Critical, EXCEPT the `local` fallback target (reported, but it
    does not page). It never consults content, so it cannot alarm on a newborn tier; it never reports
    ok when it could not ask.
  - **LIVE:** degraded observed on the still-blind box (hub emailed `agent_capability_degraded`) →
    grant applied on **both** demo boxes → token lists 3 and 4 archives → `ok=70 total=70 degraded=0`
    and `degraded → ok` at the hub → **the host tier became a due-check candidate for the first time**,
    correctly picking the 08-02 archive (08-03 had not settled 24 h).
  - **The installer's real defect was NOT `PVE_STORAGES`** — see `felhom.eu` CONTEXT S-22: Case A
    grants, the Scenario-F reuse arm did not. Fixed in installer **1.24.0** with a gate.

- **2026-08-03 — v0.122.0 (R-189 · R-188 · R-186): three signals that lied about their own work.**
  None touches data; all three cost attention, which every other signal depends on.
  - **R-189 — a passing restore-test no longer vanishes on a restart.** `restore_tests[]` came only
    from the in-memory `backup.Store` (*"lost on restart; the cadence re-populates"* — true under a
    timer, FALSE since R-86, because the agent will not re-test a proven archive). **Observed live:**
    a 14.5 GB offsite PASS at 15:25:14, agent restarted 2 m 43 s later, hub logged `0 restore-tests`
    twice. `RestoreTestState` now stores `tier` + `verified` beside the archive (v3 shape; v1/v2
    still read, and a record missing archive-or-tier is NOT reported), exposes
    `ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it — **one entry per tier,
    newest by `TestedAt` wins**, so a fresh failure beats a stored success and a tier never appears
    twice. Wiring pinned by an AST test: the method this replaces (`Snapshot`) claimed a
    "host-report gauge" in its doc comment and had **no caller** for weeks.
  - **ONLY SUCCESSES ARE PERSISTED, and the reason is now in the code:** a success *suppresses*
    future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less
    tested than it believes); a failure *causes* future work and heals itself at the next evaluation.
  - **R-188 — the release stopped emailing false failures.** Only the tag PUSH moved (build → tag
    locally → publish → push tag): the push is what wakes CI, and a tag visible before its package
    made the gate correctly fail a correct release ~half the time. The old order's invariant is now
    asserted directly — `check-published-versions.py` refuses a **published version with no tag**, as
    a bounded, printed probe (the package listing api is still 401 without a token, re-measured).
  - **R-186 — a released binary is verifiable.** `-trimpath -buildvcs=false`: same source → same
    bytes whether or not the tag exists. Measured. `publish-agent.sh`'s fallback also forced
    `CGO_ENABLED=0` and built a **74 KB different** binary for the same version — both paths now
    identical. The verification command is in `CLAUDE.md`.

- **2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock.** The ticker is
  now only the **evaluation interval**; a tier is **DUE** when its newest archive that has settled for
  `settle` (default 24 h) **has not been proven**. Daily tier → proved daily on yesterday's archive;
  weekly tier → weekly on its own; newborn → UNKNOWN. **The trap, so it is not reintroduced:** the
  literal reading of R-86 — *"due when the newest archive is ≥ 24 h old"* — is NEVER true on a daily
  tier (a new archive resets the age before it reaches the lag), so it switches restore-testing off
  where it matters most. Red-proved at 0 runs over 5 simulated days.
  - **The state now records WHICH archive was proven**, not just when a tier passed. A pre-R-86 file
    keeps its time (ordering survives) and yields no proven archive → each tier is due once after the
    upgrade, deliberately.
  - **The old cadence key:** `restore_test_cadence_seconds` is DEPRECATED. Negative still DISABLES
    (verbatim); a positive value now seeds the **settle lag** and the daemon WARNs once at start-up
    naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
    (default 24 h). It is NOT carried into the evaluation interval.
  - **6 h is bounded from both ends:** measured evaluation cost (local 18 ms, PBS-over-WAN 392 ms,
    both 430 ms) says cost is irrelevant; the ceiling is that a FAILING tier stays due, so the
    evaluation interval is also its retry interval for a multi-GB restore.
  - The due-check now runs **before** the heavy-operation gate is taken (a frequent poll must not be
    able to make a starting backup record a failure — F-A1), and the candidate picker skips archives
    failing `archivePlausiblyComplete` (a phantom would be due forever and fail forever).
  - New read-only `--selftest=restore-test-due` prints the per-tier verdict + its cost.
  - **v0.121.1 — a quiet evaluation is AUDIBLE.** "Nothing is due" is now the NORMAL outcome, and at
    DEBUG it was silent: an empty journal would have been equally consistent with a healthy loop and
    a dead goroutine (standing rule 3 — the shape the R-88 watcher was retired for). A not-due
    evaluation logs ONE INFO line naming every tier's verdict; an unlistable tier reads `UNKNOWN`
    with its error in that same line.
  - **PROVEN LIVE 2026-08-03 on demo-felhom:** due-triggered offsite restore-test of a 14.5 GB
    encrypted PBS archive — restored, booted, verified, scratch destroyed, **635 s**; the state then
    named that archive, a second evaluation ran nothing, and an agent restart ran nothing.
  - **R-185 (filed, NOT fixed here):** on demo-felhom the agent token has no ACL on
    `/storage/felhom-backup`, so its content listing comes back EMPTY (root sees 3 archives) — the
    host tier has never been restore-testable there, and the due-check cannot distinguish that from
    a newborn tier.

- **2026-07-28 — v0.107.0: F-REBOOT fixed — a guest rebooted mid-backup now comes back by itself.**
  New `internal/localapi/guestpower.go`: a 60 s watchdog that starts a guest which is `onboot:1`,
  stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let
  `RecoverStaleLockedGuests` miss campaign fault 11 — that recovery acts only on a **stale vzdump
  lock** (fault 11's guest was unlocked) and runs **once at agent startup** (fault 11's guest went
  down while the agent was already up). `onboot` is the deliberate-stop discriminator and is *not*
  invented here: it is already what `stalelock.go` uses for this decision, it is 0 on scratch/golden
  guests, and it is what `pve-guests` consults at host boot — so the agent agrees with the platform
  instead of keeping a second private definition of "should be running". Retry bounded at 3
  (1m/2m/4m) then escalates **once**; an unbounded silent retry loop is the over-correction here.
  Live on demo-hp: **120 s unattended** recovery vs the incident's **587 s** with a human; Scenario B
  proven (an `onboot:0` guest left stopped throughout). Detail: `REPORT.md`.

- **2026-07-28 — F-LEAK took THREE attempts; v0.108.0 and v0.110.0 are the corrections.** The cause is
  structural: `FelhomAgentGuest` is granted at `/pool/felhom` and a guest joins that pool only when its
  restore **completes**, so a *failed* restore-test leaves a pool-less guest out of reach (403).
  **(1) v0.107.0 pool adoption — REFUTED LIVE:** `PUT /pools/{pool}` also requires `VM.Allocate` on the
  VM being added, so membership cannot bootstrap its own authority; removed in **v0.108.0**.
  **(2) host-install v1.21.0 per-path `/vms/990000..990009` ACLs — works, but exactly ONCE per slot:**
  PVE's destroy calls `AccessControl::remove_vm_access` (`API2/LXC.pm:906`) which deletes every ACL at
  `/vms/<vmid>` (`AccessControl.pm:1898`) — **the grant is consumed by the op it authorises**. Caught by
  counting ACL rows after the fix, not by reasoning. **(3) v0.110.0 SHIPPED —
  `Privileged.DestroyScratchLXC`, the FOURTH root-fenced exception** (was exactly three: keyctl
  `pct create`, USB mount/fstab, SMART/sensors). Band enforced in **sudoers literally**
  (`pct destroy 99000[0-9] --purge`) + re-checked in code + journal provenance at the caller; none is
  consumed by use. API destroy still tried FIRST; band ACLs stay provisioned so the common case needs no
  privileged call. **Ships with a sudoers change — deploy `configs/felhom-agent.sudoers` WITH the
  binary.** Live: token 403 on a stranded scratch → fenced path removed the guest and all 3 LVs; sudo
  PERMITS the band and REFUSES `9201`/`9100`/`9999`/`990010`/`1`, and refuses `pct start 990000` too.

- **2026-07-28 — v0.109.0: the guest-power watchdog got the observable it shipped without.** A
  self-correction: v0.107.0's watchdog logged only at startup and when it *acted*, so on a healthy box
  its health could be read only from **absence** — F-OBS's exact shape, shipped in the same session
  F-OBS was fixed in the controller. Now an INFO summary every 10th sweep carrying
  `sweeps_since_boot`/`guests_evaluated`/`currently_stopped`. An **aborted** sweep (unproven
  ownership) does not count, or the heartbeat would claim liveness for a watchdog examining nothing.

- **2026-07-28 — v0.106.0: F-CRIT-2 fixed — a failed backup no longer looks like a fresh one.**
  `NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, NEWEST) as a successful
  backup, so the tier reported fresh, went **not due**, and was never retried — 7 days of silence on
  the real 168h cadence, invisible to both the R-88 breaker (defers only DUE tiers) and the hub
  deadline monitor (reads the same freshness). Now only *plausibly complete* entries count, via a
  measured floor `minPlausibleArchiveBytes` = 1 MiB; undecidable ⇒ not counted.
  **Size is the only tier-agnostic discriminator** — `verification` and `encrypted` are absent on
  every local (dir) archive and on a good PBS snapshot until verify-new catches up, so gating on
  either would reject 100% of local backups and cause fleet-wide backup THRASH. Floor measured:
  smallest real backup on the fleet is 612,397,450 B, so 1 MiB leaves 584x headroom (asserted by a
  test). Rejections logged at WARN once per volid. Re-tested live by replaying campaign fault 2 on
  demo-hp — both directions, incl. a no-thrash window with 91 scheduler ticks as the positive
  observable. Deployed on both boxes. Detail: `REPORT.md`.
  **Also established:** server-side prune does NOT count phantoms toward `keep-last` (dry-run kept
  2 real + the phantom) ⇒ **no retention/data-loss bug** — but it never removes them either, so they
  accumulate. Filed as R-99 (LOW).

- **2026-07-25 — v0.95.0 (additive): SMART coverage fixes (spike B+A) + device model.** Union-path
  drives (USB/registry) now get SMART via `storage.SmartReader.SMARTForBacking` wired into the localapi
  `/disks` union (localapi `Smart` seam); `smartDeviceFor` resolves dm/LVM to the whole disk via
  `/sys/block/<dm>/slaves` (recursive, skips >1-disk); the builtin `local` dir on the LVM root gets a
  **SMART-only** device from its containing filesystem (never touches backing/durable_id — the
  removable-safety guard in build() stays intact); `SmartSummary.ModelName` captured from smartctl. The
  watchdog `Known` path stays enrich-free. Consumed by controller v0.171.0. Source of WHERE:
  `felhom.eu/documentation/audits/SPIKE-smart-coverage-2026-07-25.md`.
- **2026-07-24 — v0.94.0 (additive): SMART serialized into /disks.** `localapi.DiskInfo` gains
  `Smart *hub.SmartSummary` (omitempty), copied from the target's already-computed Observe-time
  enrichment when `Health != ""` — no new smartctl load, no endpoint, no sudoers/MinAgent change. The
  controller v0.169.0 renders a "Lemezek állapota" card + 6h degradation alert from it; old controllers
  ignore it. **NOTE: at the remote-site vacation window the agent is DOWN (localapi binds .162 → fails),
  so live /disks-from-real-agent validation is deferred — the field is unit-proven; publish only.**
- **2026-07-22 — v0.93.0 is the FLEET AGENT.** Built, published (sha `a68b2ff73200622e…`),
  Day-0-manifest-vouched (MinAgent also 0.93.0, operator-ruled) and deployed to BOTH boxes
  (`demo-felhom-8363b5` + `demo-hp-bb76ea`, the latter over G1 break-glass — still no key baked);
  clean-restart 5/5 on both, `.bak-0.92.1` retained. Discharges the onboarding runbook §A5
  ceremony gate. Record: `felhom.eu/documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md`.
  **The bullet below ("agent is DOWN … deployed 0.90.0") is SUPERSEDED history** — vmbr0 was made
  static .162 on 2026-07-20 (F1 mitigation) and the agent has been up since; kept for the record.

- **2026-07-20 — REMOTE SITE until ~2026-08-02; the agent is DOWN there and cannot self-recover.**
  felhom-pve moved off the home LAN; `ssh felhom-pve` = tailnet `100.70.170.35` (direct, ~37 ms). The
  host is on DHCP and holds `192.168.0.147`, so `localapi`'s literal `192.168.0.162` bind fails with
  `bind: cannot assign requested address` — the daemon exits ~1.1 s after start, systemd gave up after
  4 retries, and a manual restart reproduces it exactly. Deployed binary is **0.90.0**. Fix needs
  `listen_addr` in `/etc/felhom-agent/agent.json` **and** the guest bootstrap endpoint (plus the
  pinned leaf's SAN) → **Viktor GO**; re-pinning to another literal just re-breaks on the next lease.
  Also re-observed each start: `pbs: cannot read token secret … /etc/pve/priv/storage/felhom-pbs.pw:
  permission denied` (R-39-adjacent). Evidence + ranked findings:
  `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`

- **v0.90.0** (2026-07-17) — **agent train: guest RAM resize (R-24) + fast-tick (R-28); LIVE on BOTH
  demo hosts (felhom-pve + nested demo-vm-felhom-4846bc).** MinAgent coupling: felhom-controller
  v0.143.0 gates its resize UI on this agent. (1) **R-24 guest RAM resize (controller-direct)** —
  self-scoped `GET`/`POST /guest/memory` (`internal/localapi/guestmemory.go`); the AGENT enforces
  every bound fresh per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)) and
  applies via PVE `SetConfig` — **live cgroup apply, no reboot** (Phase-0 PROVEN on the nested box; the
  break-glass access path + the proof are in `~/.claude/.../nested-vm-access-breakglass.md`). Verify-
  after-apply re-reads maxmem before claiming success. New narrow `MemoryOps` seam (GuestAPI untouched);
  memory only. (2) **R-28 fast-tick** (`internal/fasttick/`) — while any desired-state item is unapplied
  (esp. the pre-tunnel WG-registration window a hub poke can't reach) pulse the shared out-of-band
  trigger every 30 s, self-disarm on convergence; four cached sources (desired-gen==0, reconcile
  Planned−Pending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational). Seams:
  `reconcile.Engine.LastResult()` + `wgtunnel.Manager.TunnelConvergence()` (cached — no per-tick exec).
  (3) **Guests-0/0** REFUTED live: the 0/0 was the pre-provision window (guest not yet created), not a
  pool-membership bug; the fast-tick shortens that window. **OPEN (operator GO):** publish 0.90.0 +
  hub Day-0 manifest vouch + MinAgent-floor raise to 0.90.0 (password-gated UI; the safety gate — both
  agents on 0.90.0 — is satisfied and the coupling is proven live via the version header). See REPORT.md.

- **v0.89.0** (2026-07-16) — **agent train: three bundled agent-plane items; built + published to
  Gitea (sha256 `3969fd91…`); paired with hub 0.59.0 (LIVE).** (1) **pbsdr self-grant (R-22)** —
  closes the F4 self-deadlock: a 403 on the token-auth `StorageEntry` pre-check now self-grants via
  the root wrapper + re-reads instead of aborting before the grant (the demo's `felhom-offsite`
  case). (2) **escrow config live-reload** — `/escrow/preflight`'s `pbs_storage_id` row now reads
  the live agent.json (late-bound `CurrentPBSStorageID`) so a pbsdr-seeded id flips green with no
  restart. (3) **agent-plane poke listener (Direction-2a)** — `internal/poke`: contentless UDP poke
  bound to the box WG /32 (port **51822**), leading-edge debounced, fires the hub-loop out-of-band
  trigger for an immediate desired-state cycle; enabled with `wg_tunnel.enabled`; first slice of
  R-13. Red-proofs for all three (run-fail-revert). **ALL THREE LIVE LEGS PROVEN on the demo
  (2026-07-17), demo now LIVE on 0.89.0:** Scenario 4 floor-driven A/B train 0.88→0.89 (operator
  signed+enqueued the `agent_update` op — the vouch+floor alone does NOT trigger it; committed, no
  rollback); Scenario 1 R-22 self-heal (marker aside + ACLs revoked → `pre-check 403 … self-granting
  (R-22)` → `converged state=adopted` in ~3 s, ACLs restored, offsite active); Scenario 3 poke→tick
  ~31 ms ep0→box + immediate report cycle (save→tick ≈ ~0.45 s). Details: REPORT.md.

- **v0.88.0** (2026-07-13 eve) — **controller-driven escrow ceremony (agent half), LIVE on demo
  host + drill VM (63/63 capabilities both).** `--output=json` machine mode (text mode
  byte-identical; extraction into `escrowCeremony()`); the ONE fixed argv
  (`escrow.CeremonyArgs()` — shared by the localapi exec + the `escrow-ceremony` capability
  (Critical, pbs_dr-gated EXPLICIT) + the new `FELHOM_ESCROW` sudoers alias, three-way
  pin-tested); localapi job endpoints (`POST /escrow/ceremony` single-flight 60 s,
  status, ONE-SHOT claim → 410, 10-min TTL → `unclaimed_void`, `GET /escrow/preflight`).
  R in-memory ONLY (never the job struct — snapshot-hygiene-tested; restart loses it safely).
  Live-proven on drill endpoint-exact: stage → preflight all-green (live FELHOM_ESCROW
  list-probe) → job ~4 s → hub blob `restic_pw_sha256` covering (repaired the spike's hash-less
  blob) → claim 200 once → 410. Coupled: controller v0.127.0 (MinAgent 0.88.0 for the wizard).
  **OPEN: publish 0.88.0 + Day-0 manifest vouch (operator) at the next train; deployed hosts got
  direct deploys.** Details: REPORT.md + felhom.eu RUNBOOK-escrow-ceremony.md (F1 threat model).

- **v0.87.0** (2026-07-13) — **SystemDisks device-mapper walk (IA finding 2, MEDIUM):
  legacy-boot hosts get a working drive wizard.** Operator ruling (approved 2026-07-13,
  verbatim): *resolve device-mapper/raid parents — for the root filesystem's backing block
  device, walk `/sys/block/<dev>/slaves` recursively down to physical disks; those, plus any
  ESP holder when present, are system. Disks outside that set become wizard candidates (still
  subject to the existing data-bearing guards). The all-system fail-safe remains ONLY for walk
  failure — it returns to being the error case, not the legacy-boot common case.* Implemented as
  `physicalDisksOf`/`walkSlaves` + `HostReader.BlockSlaves` (one seam method); per-branch
  conservatism (any unresolvable slave → ok=false → unchanged all-system path); signature test
  `TestSystemDisks_WalkTopologies` (root-backing disk ALWAYS system — never weaken). §3 spike
  transcripts: drill (legacy) dm-1→sda3→sda; felhom-pve (EFI+LVM) ESP+walk agree on sda →
  byte-identical regression. §13.2 wizard leg COMPLETE (offered → enrolled → formatted →
  torn down, boxes as found) + Day-0 manifest vouched to 0.87.0 (operator). The leg also
  surfaced two CONTROLLER bugs (fixed same-day: v0.126.3 claimed-box wizard CSRF, v0.126.4
  502-through-CF + native-alert ban).

- **v0.83.0** (2026-07-11, LIVE on felhom-pve; NOT published — Peti stays 0.81.0) — **observability
  pass** (pairs with controller v0.116.1 + hub v0.46.0). `applog.New` → `(logger, *Ring)`: slog
  fan-out, journald at the configured level, ~1000-entry ring FIXED at DEBUG. `GET /debug/logs`
  (local API, token-authed; the controller Debug page's Ügynök tab) + request-level DEBUG
  middleware. Heartbeat log-pull: envelope `log_tail_requested` → next heartbeat ships `log_tail`
  (128 KB, consume-once; failed push re-armed by the next envelope; `operator log pull served` INFO
  on fulfillment). Gap-fill sweep: netverify phase/verdict lines (job start, trigger outcome,
  /proc/mounts verdict, journal bytes, classification code, rollback outcome, durations), netmount
  unit steps, signedjobs op-received (class/host/expiry — never signatures) + fetch duration,
  selfupdate invariants + download sha/duration, disks outcome INFOs, controller-swap pre-pull +
  health verdicts, desired/loop per-exchange DEBUG. Logging conventions:
  `felhom.eu/documentation/runbooks/logging-conventions.md`. OPEN: the hub-side live pull awaits
  the operator's button click (hub UI password-gated); pre-existing lanresolver
  permission-denied WARN on /var/lib/felhom-agent/guests noted in REPORT.

- **v0.77.0** (2026-07-09) — **fork-4: escrow the offsite restic repo password under R.** `IdentityBundle`
  gains `ResticRepoPassword` (rides the existing age-under-R `WrapIdentityBundle` path — validated by the
  custody spike `febdc56`). New `POST /escrow/stage-secret` (`withGuest`) transiently stages the
  controller-pushed password (0600, never logged), which the `--selftest=escrow-create` ceremony
  auto-injects into the bundle and then wipes. `AttachResticPassword`/`StagedResticPasswordPath`/
  `WipeStagedResticPassword` added. Pairs with controller v0.105.0 (push + atomicity gate + DR inject +
  `DRResticCoord`). **NOT yet live-validated** — the supervised escrow ceremony is operator-run.

- **v0.76.0** (2026-07-08, LIVE on felhom-pve + **PUBLISHED sha `9828c5f7…f50b`** — THE Day-0
  manifest bump target; **0.75.0 superseded unpublished**) — **GL-5b / G12: restore-test
  full-fidelity**. Params derive from the ARCHIVE's embedded config (`drRestoreOverrides`, same as
  DR — the old live-source-config path verified the wrong object AND dropped storage mpN per PVE's
  all-or-nothing rule; deleted with `bindMountOverrides`/`archiveVMID`). NEW mount-parity assert
  (restored mpN vs archive; miss/mispath/undersize/extra = FAIL naming the delta) + `MountParity`/
  `MountInventory` on the wire record (additive). Live-proven: scratch 990000 ← 6.5GB 9201 archive,
  parity ok, inventory mp0 200G+mp1 50G+2 throwaways, **3m4s local tier** (cheaper than feared);
  rotated-out archive volid → clean up-front refusal (nice failure mode). bringup.go untouched.

- **v0.75.0** (2026-07-08, LIVE on felhom-pve) — **GL-5 / go-live G8: guest-loss DR bring-up
  actually restores** (closes the v0.74.0 OPEN item + SPIKE-dr-bindmount-source §8). DR passes the
  COMPLETE explicit restore param set derived from the archive's embedded config (NEW
  `Client.ExtractArchiveConfig`, 200 under the scoped token) — **two live-discovered PVE rules: mpN
  params need an explicit rootfs, AND unlisted mountpoints are silently DROPPED** (first run booted
  without mp0/mp1!) — storage mpN passed through, structural mp8/mp9 → throwaways, then step 4d
  swaps the REAL binds in via the host runner (root pct; new `EngineOptions.HostRunner`+`StateDir`
  seam) and deletes the unusedN residue. Scratch-DR live-proven end-to-end (9310 from a real 9201
  archive: mp0 200G + mp1 50G + real binds + no residue + clean teardown). Provision = nil
  overrides (regression-tested). NOTE: published/vouch-pending agent is 0.74.0 — publish 0.75.0
  before/with the manifest bump. OBSERVATION: the DR selftest hardcodes KeepMAC=true — a scratch DR
  while the SOURCE guest is live briefly duplicates its MAC on the bridge (pre-existing; fine for
  supervised runs, worth a -keep-mac flag someday). Full customer-data DR drill = GL-6/S5 family.

- **2026-07-07 — v0.74.0 Gitea-PUBLISHED (RUNBOOK GL-1)** — the LIVE felhom-pve binary's exact
  bytes, sha256 `1ec3f58842edce1e…76af05`, anon-fetch-verified. This supersedes/closes every
  standing "publish 0.6x + Day-0 vouch" OPEN item below (0.64→0.73 were never published; 0.74.0
  is the vouch target). Golden 0.103.0 published in the same run (felhom.eu execution record
  `documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`). **Day-0 manifest vouch = operator
  step** (agent 0.74.0 / golden 0.103.0).

- **v0.74.0** (2026-07-07) — **campaign-2 R2 CLOSED; the mislabelled "R1" was a symptom** (LIVE on
  felhom-pve). Pool membership is what lets the pool-scoped token reach a guest; `pct restore --pool`
  sets it only at CREATE, so a restore-over-existing dropped 9201 from the `felhom` pool → no
  `VM.Audit` → restore-test's *existing* `bindMountOverrides` never ran → "mp8 … only possible for
  root". Fix: `Client.PoolAddVMID` + bring-up re-asserts membership post-restore (warn-not-fail).
  Role/ACL + `bindMountOverrides` untouched (both correct). **Live restore-test PASSED for the first
  time** once the pool was healed (Part A one-liner): read config → neutralize 2 binds → restore →
  boot+running → clean teardown, 4m35s. B3 (scratch-teardown 403) confirmed a cascade — no code.
  OPEN: DR `bring-up -mode dr` bind-override gap (spike `SPIKE-dr-bindmount-source-2026-07-07.md`:
  small known-constant override reusing `bindMountOverrides`; mp8/mp9 are structural constants).

- **v0.73.0** (2026-07-06) — **F2 mount-role fallback CLOSED** (LIVE on felhom-pve). `roleForMountPath`
  gained a mount-table fallback (Impl-2b style): a bind-mounted RAW enrolled user-data drive is not a PVE
  storage, so it fail-safe'd to `system` and the eject/decommission gates 403'd EVERY user-data drive
  (campaign F2, `where=/mnt/teszt_enroll role=system`). Device-keyed classification + whole-disk containment
  (`storage.SameWholeDisk`); Observe-error keeps the fail-safe BEFORE the fallback. Only `roleForMountPath`
  touched. Live-proven full lifecycle on teszt_enroll (eject/decommission 200, no-rebind across restart,
  end==pre). OPEN follow-up: the `deviceRole`/`roleForMountPath` unification refactor (deferred).

- **v0.72.0** (2026-07-05) — **OOB operator access (merged E1+H1)** — TASK H1, provenance both
  `SPIKE-{felhom-sshd,oob-wg-operator-peer}-2026-07-05`. Operator `/32` RENDERED into wg-felhom
  AllowedIPs (survives self-heal, [OF-1]); dedicated `internal/felhomsshd` (port claim + config
  render→sshd -t→reload + operator authorized_keys + heal + oob heartbeat stanza); static
  `inet felhom_oob` belt (agent mutates SET ELEMENTS ONLY); `configs/felhom-sshd.service` (NO
  RuntimeDirectory [SF-1]) + `felhom-oob.nft` + `felhom-op.sudoers`; `FELHOM_SSHD`+`FELHOM_OOB`
  grants; `oob.enabled` DEFAULT FALSE. Live on felhom-pve (8822, belt filled, operator SSH as
  felhom-op with scoped sudo); hub v0.35.0. Rollback `.bak-0.71.0`. 5 live-found bugs fixed (port
  path, self-listen flip-flop, nil-block lockout, reachable-via-dial, operator-configured source).
- **v0.71.0** (2026-07-05) — **management-plane break-glass: privsep-dir watchdog + mgmt_plane
  health** — TASK G1 (prereq for felhom-sshd/H1), provenance `SPIKE-felhom-sshd-2026-07-05` §8.
  Host artifacts (`configs/felhom-privsep.tmpfiles` + `felhom-mgmt-watchdog.{sh,service,timer}`) make
  `/run/sshd` boot-persistent AND auto-heal it every ~60s **agent-independently** (heals with the
  agent stopped — proven live: `/run/sshd` removed → restored in 30.0s, `:22` back, no login).
  `internal/mgmtplane` reports the additive `mgmt_plane` heartbeat stanza; hub v0.34.1 raises
  `mgmt_plane_healed`. **NO unit declares `RuntimeDirectory=`** (the incident cause). H1 may now
  assume `/run/sshd` is guaranteed present. Live on felhom-pve; rollback `.bak-0.70.0`.
- **v0.70.0** (2026-07-05) — **agent self-update (operator-signed A/B slots + crash-loop
  auto-rollback)** — TASK D1, provenance `SPIKE-agent-selfupdate-2026-07-05`. An operator-signed
  `agent_update` op (version+sha256, sha is the only integrity root) rides the signed-jobs gate;
  `internal/selfupdate.Executor` downloads+verifies+hands to `felhom-selfupdate-guarded apply` (root
  re-verify → A/B atomic flip → pending marker → detached restart); the new binary commits after a
  60s dwell; a crash-looping binary is auto-reverted by `OnFailure=felhom-agent-rollback.service`
  (first-crash trigger [SF-1]) with the tuned `[Unit]` start-limit (120s/4) as backstop. Host
  artifacts + sudoers `FELHOM_SELFUPDATE` + `felhom-host-install.sh` day-0 install + report field
  `selfupdate_pending`. Green tests + companions. **LIVE-VALIDATED on felhom-pve (2026-07-05): all 4
  drills PASS** — happy path (0.70.0→0.70.1 signed op → download+verify+flip+commit), crash-rollback
  (0.70.2-crash → OnFailure → **~2s crash-to-recovered**, byte-identical revert, no loop), no-pending
  guard, gate refusal (non-pinned key). Full agent-side pipeline ran real (envelope injected into the
  hub `signed_jobs` queue — CC lacks the hub global operator key; hub enqueue-auth is hub-unit-tested).
  Box restored to canonical **v0.70.0** (host artifacts KEPT installed; scratch operator key REMOVED —
  self-update dormant until an operator pins a real key, a Day-0-vouch-style follow-up). Rollback
  `felhom-agent.bak-0.69.0`. OPEN (v1 scope-outs): no hub-floor auto-update, no failed-update
  auto-retry, no pending-timeout auto-rollback; per-crash OnFailure can double-fire (idempotent —
  future: serialize the rollback oneshot). Detail: REPORT.md.

- **v0.69.0** (2026-07-04, live on felhom-pve) — **S5: host-loss DR — safe halves shipped**.
  **Part 1** `wgtunnel.InstallRecoveredKey` — writes an escrow-recovered WG privkey (create-only,
  refuse-overwrite) so the tunnel re-establishes with the SAME identity/pubkey (same /32), no keygen;
  wired into `--selftest=identity-consume -install-wg-key` (opt-in; pre-S3 blob → logged fresh-keygen
  fallback). **Part 2** new `internal/dr` — consumes the host_loss `restore_directive` (was
  logged-ignored) into an inspectable RestorePlan via AddConsumer: per-guest {vmid,archive,target,
  sizing} + per-drive {durable_id→mount} + offsite PBS coord; DERIVE-AND-SURFACE only (Consumer has
  no restore/destroy dep — execute-nothing is structural). Tests + red-proofs (WG create-only; plan
  mode-gate). **Part 3** hub escrow-GET NOT needed (operator exports the blob via `sqlite3 writefile`
  on a cp'd hub.db). **Part 4-A** re-attach wrong-disk safety already unit-proven
  (`ResolveStorageDevice`: match resolves, absent/mismatch ERRORS, non-uuid scheme refused — never a
  near disk). **Part 4-B (destructive in-place 9201 restore) PREPARED + OPERATOR-GATED, NOT executed**
  — pre-flight green (offsite ct/9201 restorable per S4.1); the operator runs the R-consume steps +
  confirms the destroy (§9-4a: CC never runs a consume/R command — see [[operator-present-one-time-secrets]]).
  OPEN: the operator-run 4-B drill; guest_loss DR; hub-driven full-auto DR. Rollback
  felhom-agent.bak-0.68.0. Detail: REPORT.md + doc-06 §3.5/S5.
- **v0.68.0** (2026-07-04, live on felhom-pve) — **S4.1: unattended offsite restore-test**.
  **Tier-aware restore-task deadline:** `RestoreTestSpec.RestoreTaskTimeout` (0→10m default) from
  `config.RestoreTestPBSRestoreTimeoutSeconds` (accessor default **120m**), set only when
  `SourceTier=="pbs"` (`main.restoreTaskTimeout`); local tier UNCHANGED. Fixes the WAN restore being
  killed at 10m → mid-restore teardown → leaked scratch. **Teardown "VM.Allocate" follow-up =
  PHANTOM (diagnosed, not blind-fixed):** ran the restore-test on the AGENT-TOKEN path sourcing the
  offsite (pbs) backup → `pass:true verified:boot+running`, teardown succeeded (`torn down
  vmid=990000`, no 403), scratch band clean. The earlier 403 was the 10m-timeout consequence (guest
  not yet pool-associated); the scratch is restored INTO `/pool/felhom` (ACL already grants
  VM.Allocate) so teardown is authorized once the restore completes. **No ACL/host-install change.**
  OPEN: publish 0.68.0 + Day-0 vouch; Tier-1/Tier-2 split for offsite-as-default; S5 DR consume.
  Rollback `felhom-agent.bak-0.67.0`. Detail: REPORT.md.
- **v0.66.0 + v0.67.0** (2026-07-04, live on felhom-pve) — **S4: PBS over the tunnel**. **v0.66.0**:
  wgtunnel **v4-pin** (renderConf writes the resolved A LITERAL, never DNS/AAAA; `Resolver` seam,
  lowest addr; cached → steady-state zero-DNS/zero-exec) + **re-resolve watchdog** (`Manager.Watchdog`,
  loop-only; handshake stale > `stale_after_seconds`=180 → re-resolve → IP-changed re-render+restart)
  + FELHOM_WG **Critical** flips (conf-install/enable/restart/handshake-read). **v0.67.0**:
  **namespace-aware PBS client** (Config.Namespace → `Snapshots ?ns=`, `Verify ns=`; root-ns
  unchanged) — the operator-approved fix after Phase-1 showed the ns-unaware datastore-root 403s a
  per-tenant token. **Live Scenario-D (all green):** real vzdump of 9201 → **ciphertext** in ns
  `demo-felhom-01` over the tunnel; ns-scoped verify=ok under the box's own `felhom@pbs!demo-felhom-01`
  token; WARN gone; restore round-tripped (decrypt with box-born key → boot → teardown).
  **Confirmed tenant ACL (felhom-hetzner):** `DatastoreBackup` on `/datastore/felhom-offsite/<ns>`
  (NOT `/ns/<ns>`) to BOTH user `felhom@pbs` AND token (privsep=intersection; cross-ns 403);
  DatastoreBackup can't prune (safety). **FINDINGS:** retarget field is `local_backup_target` (not
  `backup_target`); retarget REVERTED to `local` (controller backs up ~every 30 min → single-target
  offsite = near-continuous 20-min uploads; needs Tier-1/Tier-2 split); restore-test scheduler needs
  a WAN restore deadline + scratch-band `VM.Allocate` before it runs offsite unattended. **OPEN:**
  escrow-create (OPERATOR-PRESENT, new R); publish 0.66/0.67 + Day-0 vouch; S5 DR consume. Rollback:
  `felhom-agent.bak-0.65.0`/`.bak-0.66.0`. Detail: REPORT.md + doc-06 §3.4/§4.2 + runbook §4a/§4b.
- **v0.65.0** (2026-07-04, live on felhom-pve) — **S3.1 offsite-tunnel client MTU 1420 → 1280**:
  resolves `06 §4.3`'s OPEN DECISION left by the CGNAT smoke test. 1420 **silently black-holed bulk
  TCP** on sub-~1480 paths (mobile ~1400, DS-Lite ~1452) — handshake+ping healthy, PBS TLS page
  (and at S4 the backup itself) drops. New `const clientMTU = 1280` (RFC 8200 IPv6-minimum floor;
  outer 1340 v4 / 1360 v6 fits every realistic path), **permanent + fleet-wide + family-agnostic**.
  **Client-only by construction** — interface MTU caps box→PBS, advertised MSS caps PBS→box, so the
  endpoint's `wg0` is untouched (zero live-endpoint risk). Golden pins exact `MTU = 1280`
  (red-proofed vs a 1420 flip); no wire/JSON change. Live: agent re-rendered on restart (hash-gated
  apply), conf + live iface both 1280, PBS page loads at 1280 (no regression on wired). OPEN:
  true-CGNAT-SIM retest (low risk); publish 0.65.0 + Day-0 vouch (operator); S4 PBS-over-tunnel.
  Rollback: `felhom-agent.bak-0.64.0` on the box. The v4-pin (§4.2 determinism) is a separate,
  optional future note — NOT needed for MTU correctness.
- **v0.64.0** (2026-07-04, live on felhom-pve) — **S3 offsite WG tunnel**: new `internal/wgtunnel`
  (keygen 0600/0700, marker-gated one-shot registration, agent-managed `wg-quick@wg-felhom` from
  the hub's desired-state `wireguard` block via the new `desired.Syncer.AddConsumer` seam,
  revoked-stays-revoked teardown, report stanza) + `FELHOM_WG` sudoers/capabilities +
  `IdentityBundle.WGPrivateKey` escrow auto-inject. **`wg_tunnel.enabled` DEFAULTS FALSE** (safety
  gate — rollout to Peti's box is a no-op until the production endpoint exists; enabled explicitly
  on felhom-pve only). Live: tunnel to ep0.felhom.eu:443 up 3 s after enable (PBS page through
  10.77.0.1:8007), reboot-persistent, revocation drill clean, 30-min keepalive soak. GOTCHAS:
  hub envelope poll_interval_seconds (hub-side const 900 s) silently overrides agent poll_seconds
  on cycle 1; `wg show <if> dump` leaks the PRIVATE key (forbidden everywhere — sudoers only
  grants `latest-handshakes`). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix);
  publish 0.64.0 to Gitea + Day-0 vouch (operator); S4 points PBS at the tunnel.
- **configs: build-golden.sh v2.0.0** (2026-07-03, @ `ceca355`; no agent version change) — **drill
  findings B5 + B1 FIXED** (`DRILL-golden-098-2026-07-03.md`): the controller tag is a MANDATORY
  argument (the default rotted twice — a fresh install booted a pre-floor controller, forcing the
  guide's manual D.1b update) and the golden now bakes a `felhom-controller-bootstrap.path` unit
  (controller deploys the moment the back-half hot-plugs the bootstrap mount — no reboot; installer
  v1.9.1's reboot is a redundant belt, kept). **Golden 0.98.3** baked on the drill VM, clean-room
  validated (bake integrity → isolated hot-plug proof → local-golden Day-0 → published-artifact
  Day-0), published (sha256 b9a02ef1…fd01) + operator-vouched — Day-0 manifest now vouches
  **agent 0.63.0 + golden 0.98.3** (the v0.63.0 vouch follow-up below is DONE). Fresh installs land
  current and self-manage. NEW operator follow-up (SECURITY): the customer-config `git.token` has
  Gitea package-WRITE rights — scope down + rotate (evidence-doc observation O1).
- **v0.63.0** (2026-07-03, live on felhom-pve + Gitea-published sha256 b4a89c81…) — **drill findings
  B3 + B2 FIXED** (`DRILL-day0-cleanroom-2026-07-03.md`): `TokenStore.Lookup` reloads the append-only
  store once on a miss (cross-process coherence with the one-shot provisioner — no more fresh-install
  `/controller/swap` 401 / manual restart; size short-circuit bounds the cost; behind the
  `TokenAuthority` seam) + `guesthook.InstallSnippet` issues a fenced `mkdir -p /var/lib/vz/snippets`
  first (fresh boxes lacked the dir → the self-heal hook silently never installed). Sudoers gained
  exactly that one grant — **ship sudoers WITH the binary** (done on felhom-pve). Red-proofed both;
  Scenario-E method: compiled test suite run ON felhom-pve + live channel-health hit-path.
  **OPERATOR FOLLOW-UP: bump the hub Day-0 manifest to agent 0.63.0** — until then fresh installs get
  0.62.0 and the guide's D.1b restart-first step still applies (narrowed to "< v0.63.0" in the guide).
- **v0.62.0** (2026-07-03) — **audit A1 RESOLVED**: the stale-lock reaper's scan is now
  pool-intersected (`staleLockController.Guests()` = `ListLXC` ∩ `Client.Pool("felhom")` members),
  fail-safe skip on pool-read failure; `pve:pool-read` capability (non-critical) + `--selftest`
  "pool read" line. Companion host-install **v1.9.0** adds `Pool.Audit` to `FelhomAgentGuest` —
  **deploy order on any box: rescope ACL first, then this agent.** Per
  `SPIKE-a1-pool-membership-read-2026-07-03.md`; red-proofed tests in stalelock_pool_test.go.
- **2026-07-03 — CLAUDE.md refreshed**: version narrative removed (state lives HERE + CHANGELOG top), layout completed (all 17 internal packages + cmd/felhom-opsign); deploy runbook now in the `felhom-build-deploy` skill (`felhom.eu/skills/`).
- **2026-07-03 — `REUSE.md` exists at the repo root** (canonical helpers / format-safety guards / traps / seams, code-verified); maintenance rule active: update it in the same commit that changes a shared helper.
- **v0.61.0** (2026-07-03) — blast-radius audit fixes **B1 + D1 + D2 + D3** from
  `felhom.eu/documentation/audits/AUDIT-blast-radius-hostroot-localapi-2026-07-02.md`: random temp
  staging for root-installed scripts (+ sudoers/manifest glob updates), mkfs-wrapper member/RO
  re-checks (validated by `scripts/mkfs-guarded-harness.sh`), classifyClaim empty-lsblk fail-safe,
  and the blank-format anti-retarget (durable-id-bound, AGENT-001's benign-branch twin).
- Deployed on demo host `felhom-pve` (node `demo-felhom`), non-root `felhom-agent` service user,
  pool-scoped token (`felhom` pool).

## Open threads

- Deferred audit items (housekeeping/design, all INFO): C1 (controller-swap version floor), C2 (NAS
  server allowlist), A2 (gate journal cross-check), B2–B5, E1/E2.
- Drive-enrollment leftovers: (a) `runStorageInit` slow-device detached-format polling; (b) Impl-3
  shared-box operator format gate.
- BUNDLE leftover: non-root agent can't read the PBS key; migration must preserve cert/key/tokens.
- Not run (needs a supervised session): the destructive D1/D3 live proofs (real mkfs on a crafted
  member; a live /dev re-enumeration race during a real format).
