felhom-agent/CONTEXT.md

CONTEXT — felhom-agent working state

Snapshot of the current state + open threads. Authoritative history lives in CHANGELOG.md (top entry = current); the end-of-task detail lives in REPORT.md.

R-199 (v0.125.0) — links 6–8 of the recovery chain, assembled and walked

POST /escrow/recover-offsite-password (pinned local API, withGuest): the controller supplies the customer's recovery code, the agent fetches THIS host's own sealed blob from the hub (hub.Client.FetchIdentityEscrow → GET /hosts/{id}/escrow, hub >= v0.94.0, self-scoped by the per-host key), unseals it via escrow.OffsiteKeyRecoverer, and returns only the offsite restic repository password plus its sha256.

Rules that must not erode:

• Only that field. Not the tunnel token, not the PBS token, not the WG key — the controller is a trust tier down and needs none of them. Narrowing cost nothing and is not recoverable later.
• The unseal stays in the agent. age is an agent runtime dependency (/usr/bin/age — hardcoded, no config override; 1.2.1 on demo-felhom) and is deliberately absent from the controller image.
• R: in memory for one call, cleared on the success path AND every failure path, never on disk, never in argv, never logged at any level including inside an error, never echoed. Verified live: 0 log lines, 0 files, 0 leftover felhom-idesc-* dirs, with a positive control proving the search worked.
• Three distinct outcomes, not one generic failure: no blob (404), a bundle that opens but predates the field (409 — pre-fork-4, cannot be retro-fitted), a code that does not open it (400 — fail-closed at age's KDF, nothing written).
• The wiring is pinned by an AST walk (cmd/felhom-agent/escrow_recover_wiring_test.go): main → runDaemon → buildLocalAPIServer, an escrow.OffsiteKeyRecoverer constructed there, the Options.EscrowRecovery field present, and the fetcher calling the DAEMON's own hubClient (the self-scoping that makes cross-host retrieval impossible is a property of WHICH key is used). Links 6 and 7 were two of this project's six built-but-never-wired instances.

Proven live on demo-felhom 2026-08-04: recovered sha256 == on-disk sha256 == the hub's stored hash. A wrong code five minutes earlier failed closed. The chain stops at link 8 — nothing installs a recovered password, reopens a repository, or restores a file.

§8.6, fixed while here: runSelftestIdentityConsume's success line used to recite "tunnel_token + pbs_token", which became a misstatement when v0.77.0 sealed the repository password into the same bundle — anyone reading it would conclude the password was not there. It now names what THIS bundle carried and what it did not.

Current

• 2026-08-03 — v0.123.0 (R-185): a tier the box cannot READ now says so. The agent's token had FelhomAgentStore on local, local-lvm, felhom-pbs and not on felhom-backup — the storage both demo boxes configure as local_backup_target. That storage answered {"data":[]} through the token while root listed three archives, and pickForThisRun skipped it as "no settled archive yet" — which is what a brand-new tier reports, so the host tier was never restore-testable and nothing said so.

  • The permission question is asked directly, because unlike the listing it has a definite answer: Client.Permissions reads /access/permissions?path=/storage/<target> as the agent's own token, and storeGrantStatuses emits one capability.Status per configured tier. It composes AROUND the sudo prober, the way poolReadStatus already does — an API read does not belong inside a sudo-policy probe. Status's wire shape is untouched, so the hub's critical degraded alert applies with no hub change.
  • MEASURED FIRST, and the obvious reading is wrong: an ungranted path answers neither empty nor 403 — it carries the privileges INHERITED from the box-wide / grant (Sys.Audit, SDN.Use, Datastore.Audit). Checking path-presence, or Datastore.Audit, reports a blinded storage HEALTHY. The probe tests Datastore.AllocateSpace; re-measure before ever changing that constant (storeGrantRequiredPriv, red-proved).
  • The probed set comes from BackupTiers(), never a fixed list — a hardcoded probe list is the defect reproduced inside the fix. Critical, EXCEPT the local fallback target (reported, but it does not page). It never consults content, so it cannot alarm on a newborn tier; it never reports ok when it could not ask.
  • LIVE: degraded observed on the still-blind box (hub emailed agent_capability_degraded) → grant applied on both demo boxes → token lists 3 and 4 archives → ok=70 total=70 degraded=0 and degraded → ok at the hub → the host tier became a due-check candidate for the first time, correctly picking the 08-02 archive (08-03 had not settled 24 h).
  • The installer's real defect was NOT PVE_STORAGES — see felhom.eu CONTEXT S-22: Case A grants, the Scenario-F reuse arm did not. Fixed in installer 1.24.0 with a gate.

• 2026-08-03 — v0.122.0 (R-189 · R-188 · R-186): three signals that lied about their own work. None touches data; all three cost attention, which every other signal depends on.

  • R-189 — a passing restore-test no longer vanishes on a restart. restore_tests[] came only from the in-memory backup.Store ("lost on restart; the cadence re-populates" — true under a timer, FALSE since R-86, because the agent will not re-test a proven archive). Observed live: a 14.5 GB offsite PASS at 15:25:14, agent restarted 2 m 43 s later, hub logged 0 restore-tests twice. RestoreTestState now stores tier + verified beside the archive (v3 shape; v1/v2 still read, and a record missing archive-or-tier is NOT reported), exposes ProvenRestoreTests, and Collector.SetProvenRestoreTests merges it — one entry per tier, newest by TestedAt wins, so a fresh failure beats a stored success and a tier never appears twice. Wiring pinned by an AST test: the method this replaces (Snapshot) claimed a "host-report gauge" in its doc comment and had no caller for weeks.
  • ONLY SUCCESSES ARE PERSISTED, and the reason is now in the code: a success suppresses future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it believes); a failure causes future work and heals itself at the next evaluation.
  • R-188 — the release stopped emailing false failures. Only the tag PUSH moved (build → tag locally → publish → push tag): the push is what wakes CI, and a tag visible before its package made the gate correctly fail a correct release ~half the time. The old order's invariant is now asserted directly — check-published-versions.py refuses a published version with no tag, as a bounded, printed probe (the package listing api is still 401 without a token, re-measured).
  • R-186 — a released binary is verifiable. -trimpath -buildvcs=false: same source → same bytes whether or not the tag exists. Measured. publish-agent.sh's fallback also forced CGO_ENABLED=0 and built a 74 KB different binary for the same version — both paths now identical. The verification command is in CLAUDE.md.

• 2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock. The ticker is now only the evaluation interval; a tier is DUE when its newest archive that has settled for settle (default 24 h) has not been proven. Daily tier → proved daily on yesterday's archive; weekly tier → weekly on its own; newborn → UNKNOWN. The trap, so it is not reintroduced: the literal reading of R-86 — "due when the newest archive is ≥ 24 h old" — is NEVER true on a daily tier (a new archive resets the age before it reaches the lag), so it switches restore-testing off where it matters most. Red-proved at 0 runs over 5 simulated days.

  • The state now records WHICH archive was proven, not just when a tier passed. A pre-R-86 file keeps its time (ordering survives) and yields no proven archive → each tier is due once after the upgrade, deliberately.
  • The old cadence key: restore_test_cadence_seconds is DEPRECATED. Negative still DISABLES (verbatim); a positive value now seeds the settle lag and the daemon WARNs once at start-up naming restore_test_eval_interval_seconds (default 6 h) and restore_test_settle_seconds (default 24 h). It is NOT carried into the evaluation interval.
  • 6 h is bounded from both ends: measured evaluation cost (local 18 ms, PBS-over-WAN 392 ms, both 430 ms) says cost is irrelevant; the ceiling is that a FAILING tier stays due, so the evaluation interval is also its retry interval for a multi-GB restore.
  • The due-check now runs before the heavy-operation gate is taken (a frequent poll must not be able to make a starting backup record a failure — F-A1), and the candidate picker skips archives failing archivePlausiblyComplete (a phantom would be due forever and fail forever).
  • New read-only --selftest=restore-test-due prints the per-tier verdict + its cost.
  • v0.121.1 — a quiet evaluation is AUDIBLE. "Nothing is due" is now the NORMAL outcome, and at DEBUG it was silent: an empty journal would have been equally consistent with a healthy loop and a dead goroutine (standing rule 3 — the shape the R-88 watcher was retired for). A not-due evaluation logs ONE INFO line naming every tier's verdict; an unlistable tier reads UNKNOWN with its error in that same line.
  • PROVEN LIVE 2026-08-03 on demo-felhom: due-triggered offsite restore-test of a 14.5 GB encrypted PBS archive — restored, booted, verified, scratch destroyed, 635 s; the state then named that archive, a second evaluation ran nothing, and an agent restart ran nothing.
  • R-185 (filed, NOT fixed here): on demo-felhom the agent token has no ACL on /storage/felhom-backup, so its content listing comes back EMPTY (root sees 3 archives) — the host tier has never been restore-testable there, and the due-check cannot distinguish that from a newborn tier.

• 2026-07-28 — v0.107.0: F-REBOOT fixed — a guest rebooted mid-backup now comes back by itself. New internal/localapi/guestpower.go: a 60 s watchdog that starts a guest which is onboot:1, stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let RecoverStaleLockedGuests miss campaign fault 11 — that recovery acts only on a stale vzdump lock (fault 11's guest was unlocked) and runs once at agent startup (fault 11's guest went down while the agent was already up). onboot is the deliberate-stop discriminator and is not invented here: it is already what stalelock.go uses for this decision, it is 0 on scratch/golden guests, and it is what pve-guests consults at host boot — so the agent agrees with the platform instead of keeping a second private definition of "should be running". Retry bounded at 3 (1m/2m/4m) then escalates once; an unbounded silent retry loop is the over-correction here. Live on demo-hp: 120 s unattended recovery vs the incident's 587 s with a human; Scenario B proven (an onboot:0 guest left stopped throughout). Detail: REPORT.md.

• 2026-07-28 — F-LEAK took THREE attempts; v0.108.0 and v0.110.0 are the corrections. The cause is structural: FelhomAgentGuest is granted at /pool/felhom and a guest joins that pool only when its restore completes, so a failed restore-test leaves a pool-less guest out of reach (403). (1) v0.107.0 pool adoption — REFUTED LIVE: PUT /pools/{pool} also requires VM.Allocate on the VM being added, so membership cannot bootstrap its own authority; removed in v0.108.0. (2) host-install v1.21.0 per-path /vms/990000..990009 ACLs — works, but exactly ONCE per slot: PVE's destroy calls AccessControl::remove_vm_access (API2/LXC.pm:906) which deletes every ACL at /vms/<vmid> (AccessControl.pm:1898) — the grant is consumed by the op it authorises. Caught by counting ACL rows after the fix, not by reasoning. (3) v0.110.0 SHIPPED — Privileged.DestroyScratchLXC, the FOURTH root-fenced exception (was exactly three: keyctl pct create, USB mount/fstab, SMART/sensors). Band enforced in sudoers literally (pct destroy 99000[0-9] --purge) + re-checked in code + journal provenance at the caller; none is consumed by use. API destroy still tried FIRST; band ACLs stay provisioned so the common case needs no privileged call. Ships with a sudoers change — deploy configs/felhom-agent.sudoers WITH the binary. Live: token 403 on a stranded scratch → fenced path removed the guest and all 3 LVs; sudo PERMITS the band and REFUSES 9201/9100/9999/990010/1, and refuses pct start 990000 too.

• 2026-07-28 — v0.109.0: the guest-power watchdog got the observable it shipped without. A self-correction: v0.107.0's watchdog logged only at startup and when it acted, so on a healthy box its health could be read only from absence — F-OBS's exact shape, shipped in the same session F-OBS was fixed in the controller. Now an INFO summary every 10th sweep carrying sweeps_since_boot/guests_evaluated/currently_stopped. An aborted sweep (unproven ownership) does not count, or the heartbeat would claim liveness for a watchdog examining nothing.

• 2026-07-28 — v0.106.0: F-CRIT-2 fixed — a failed backup no longer looks like a fresh one. NewestArchiveTime counted an aborted PBS upload (1 byte, manifest-less, NEWEST) as a successful backup, so the tier reported fresh, went not due, and was never retried — 7 days of silence on the real 168h cadence, invisible to both the R-88 breaker (defers only DUE tiers) and the hub deadline monitor (reads the same freshness). Now only plausibly complete entries count, via a measured floor minPlausibleArchiveBytes = 1 MiB; undecidable ⇒ not counted. Size is the only tier-agnostic discriminator — verification and encrypted are absent on every local (dir) archive and on a good PBS snapshot until verify-new catches up, so gating on either would reject 100% of local backups and cause fleet-wide backup THRASH. Floor measured: smallest real backup on the fleet is 612,397,450 B, so 1 MiB leaves 584x headroom (asserted by a test). Rejections logged at WARN once per volid. Re-tested live by replaying campaign fault 2 on demo-hp — both directions, incl. a no-thrash window with 91 scheduler ticks as the positive observable. Deployed on both boxes. Detail: REPORT.md. Also established: server-side prune does NOT count phantoms toward keep-last (dry-run kept 2 real + the phantom) ⇒ no retention/data-loss bug — but it never removes them either, so they accumulate. Filed as R-99 (LOW).

• 2026-07-25 — v0.95.0 (additive): SMART coverage fixes (spike B+A) + device model. Union-path drives (USB/registry) now get SMART via storage.SmartReader.SMARTForBacking wired into the localapi /disks union (localapi Smart seam); smartDeviceFor resolves dm/LVM to the whole disk via /sys/block/<dm>/slaves (recursive, skips >1-disk); the builtin local dir on the LVM root gets a SMART-only device from its containing filesystem (never touches backing/durable_id — the removable-safety guard in build() stays intact); SmartSummary.ModelName captured from smartctl. The watchdog Known path stays enrich-free. Consumed by controller v0.171.0. Source of WHERE: felhom.eu/documentation/audits/SPIKE-smart-coverage-2026-07-25.md.

• 2026-07-24 — v0.94.0 (additive): SMART serialized into /disks. localapi.DiskInfo gains Smart *hub.SmartSummary (omitempty), copied from the target's already-computed Observe-time enrichment when Health != "" — no new smartctl load, no endpoint, no sudoers/MinAgent change. The controller v0.169.0 renders a "Lemezek állapota" card + 6h degradation alert from it; old controllers ignore it. NOTE: at the remote-site vacation window the agent is DOWN (localapi binds .162 → fails), so live /disks-from-real-agent validation is deferred — the field is unit-proven; publish only.

• 2026-07-22 — v0.93.0 is the FLEET AGENT. Built, published (sha a68b2ff73200622e…), Day-0-manifest-vouched (MinAgent also 0.93.0, operator-ruled) and deployed to BOTH boxes (demo-felhom-8363b5 + demo-hp-bb76ea, the latter over G1 break-glass — still no key baked); clean-restart 5/5 on both, .bak-0.92.1 retained. Discharges the onboarding runbook §A5 ceremony gate. Record: felhom.eu/documentation/pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md. The bullet below ("agent is DOWN … deployed 0.90.0") is SUPERSEDED history — vmbr0 was made static .162 on 2026-07-20 (F1 mitigation) and the agent has been up since; kept for the record.

• 2026-07-20 — REMOTE SITE until ~2026-08-02; the agent is DOWN there and cannot self-recover. felhom-pve moved off the home LAN; ssh felhom-pve = tailnet 100.70.170.35 (direct, ~37 ms). The host is on DHCP and holds 192.168.0.147, so localapi's literal 192.168.0.162 bind fails with bind: cannot assign requested address — the daemon exits ~1.1 s after start, systemd gave up after 4 retries, and a manual restart reproduces it exactly. Deployed binary is 0.90.0. Fix needs listen_addr in /etc/felhom-agent/agent.json and the guest bootstrap endpoint (plus the pinned leaf's SAN) → Viktor GO; re-pinning to another literal just re-breaks on the next lease. Also re-observed each start: pbs: cannot read token secret … /etc/pve/priv/storage/felhom-pbs.pw: permission denied (R-39-adjacent). Evidence + ranked findings: felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md

• v0.90.0 (2026-07-17) — agent train: guest RAM resize (R-24) + fast-tick (R-28); LIVE on BOTH demo hosts (felhom-pve + nested demo-vm-felhom-4846bc). MinAgent coupling: felhom-controller v0.143.0 gates its resize UI on this agent. (1) R-24 guest RAM resize (controller-direct) — self-scoped GET/POST /guest/memory (internal/localapi/guestmemory.go); the AGENT enforces every bound fresh per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)) and applies via PVE SetConfig — live cgroup apply, no reboot (Phase-0 PROVEN on the nested box; the break-glass access path + the proof are in ~/.claude/.../nested-vm-access-breakglass.md). Verify- after-apply re-reads maxmem before claiming success. New narrow MemoryOps seam (GuestAPI untouched); memory only. (2) R-28 fast-tick (internal/fasttick/) — while any desired-state item is unapplied (esp. the pre-tunnel WG-registration window a hub poke can't reach) pulse the shared out-of-band trigger every 30 s, self-disarm on convergence; four cached sources (desired-gen==0, reconcile Planned−Pending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational). Seams: reconcile.Engine.LastResult() + wgtunnel.Manager.TunnelConvergence() (cached — no per-tick exec). (3) Guests-0/0 REFUTED live: the 0/0 was the pre-provision window (guest not yet created), not a pool-membership bug; the fast-tick shortens that window. OPEN (operator GO): publish 0.90.0 + hub Day-0 manifest vouch + MinAgent-floor raise to 0.90.0 (password-gated UI; the safety gate — both agents on 0.90.0 — is satisfied and the coupling is proven live via the version header). See REPORT.md.

• v0.89.0 (2026-07-16) — agent train: three bundled agent-plane items; built + published to Gitea (sha256 3969fd91…); paired with hub 0.59.0 (LIVE). (1) pbsdr self-grant (R-22) — closes the F4 self-deadlock: a 403 on the token-auth StorageEntry pre-check now self-grants via the root wrapper + re-reads instead of aborting before the grant (the demo's felhom-offsite case). (2) escrow config live-reload — /escrow/preflight's pbs_storage_id row now reads the live agent.json (late-bound CurrentPBSStorageID) so a pbsdr-seeded id flips green with no restart. (3) agent-plane poke listener (Direction-2a) — internal/poke: contentless UDP poke bound to the box WG /32 (port 51822), leading-edge debounced, fires the hub-loop out-of-band trigger for an immediate desired-state cycle; enabled with wg_tunnel.enabled; first slice of R-13. Red-proofs for all three (run-fail-revert). ALL THREE LIVE LEGS PROVEN on the demo (2026-07-17), demo now LIVE on 0.89.0: Scenario 4 floor-driven A/B train 0.88→0.89 (operator signed+enqueued the agent_update op — the vouch+floor alone does NOT trigger it; committed, no rollback); Scenario 1 R-22 self-heal (marker aside + ACLs revoked → pre-check 403 … self-granting (R-22) → converged state=adopted in ~3 s, ACLs restored, offsite active); Scenario 3 poke→tick ~31 ms ep0→box + immediate report cycle (save→tick ≈ ~0.45 s). Details: REPORT.md.

• v0.88.0 (2026-07-13 eve) — controller-driven escrow ceremony (agent half), LIVE on demo host + drill VM (63/63 capabilities both). --output=json machine mode (text mode byte-identical; extraction into escrowCeremony()); the ONE fixed argv (escrow.CeremonyArgs() — shared by the localapi exec + the escrow-ceremony capability (Critical, pbs_dr-gated EXPLICIT) + the new FELHOM_ESCROW sudoers alias, three-way pin-tested); localapi job endpoints (POST /escrow/ceremony single-flight 60 s, status, ONE-SHOT claim → 410, 10-min TTL → unclaimed_void, GET /escrow/preflight). R in-memory ONLY (never the job struct — snapshot-hygiene-tested; restart loses it safely). Live-proven on drill endpoint-exact: stage → preflight all-green (live FELHOM_ESCROW list-probe) → job ~4 s → hub blob restic_pw_sha256 covering (repaired the spike's hash-less blob) → claim 200 once → 410. Coupled: controller v0.127.0 (MinAgent 0.88.0 for the wizard). OPEN: publish 0.88.0 + Day-0 manifest vouch (operator) at the next train; deployed hosts got direct deploys. Details: REPORT.md + felhom.eu RUNBOOK-escrow-ceremony.md (F1 threat model).

• v0.87.0 (2026-07-13) — SystemDisks device-mapper walk (IA finding 2, MEDIUM): legacy-boot hosts get a working drive wizard. Operator ruling (approved 2026-07-13, verbatim): resolve device-mapper/raid parents — for the root filesystem's backing block device, walk /sys/block/<dev>/slaves recursively down to physical disks; those, plus any ESP holder when present, are system. Disks outside that set become wizard candidates (still subject to the existing data-bearing guards). The all-system fail-safe remains ONLY for walk failure — it returns to being the error case, not the legacy-boot common case. Implemented as physicalDisksOf/walkSlaves + HostReader.BlockSlaves (one seam method); per-branch conservatism (any unresolvable slave → ok=false → unchanged all-system path); signature test TestSystemDisks_WalkTopologies (root-backing disk ALWAYS system — never weaken). §3 spike transcripts: drill (legacy) dm-1→sda3→sda; felhom-pve (EFI+LVM) ESP+walk agree on sda → byte-identical regression. §13.2 wizard leg COMPLETE (offered → enrolled → formatted → torn down, boxes as found) + Day-0 manifest vouched to 0.87.0 (operator). The leg also surfaced two CONTROLLER bugs (fixed same-day: v0.126.3 claimed-box wizard CSRF, v0.126.4 502-through-CF + native-alert ban).

• v0.83.0 (2026-07-11, LIVE on felhom-pve; NOT published — Peti stays 0.81.0) — observability pass (pairs with controller v0.116.1 + hub v0.46.0). applog.New → (logger, *Ring): slog fan-out, journald at the configured level, ~1000-entry ring FIXED at DEBUG. GET /debug/logs (local API, token-authed; the controller Debug page's Ügynök tab) + request-level DEBUG middleware. Heartbeat log-pull: envelope log_tail_requested → next heartbeat ships log_tail (128 KB, consume-once; failed push re-armed by the next envelope; operator log pull served INFO on fulfillment). Gap-fill sweep: netverify phase/verdict lines (job start, trigger outcome, /proc/mounts verdict, journal bytes, classification code, rollback outcome, durations), netmount unit steps, signedjobs op-received (class/host/expiry — never signatures) + fetch duration, selfupdate invariants + download sha/duration, disks outcome INFOs, controller-swap pre-pull + health verdicts, desired/loop per-exchange DEBUG. Logging conventions: felhom.eu/documentation/runbooks/logging-conventions.md. OPEN: the hub-side live pull awaits the operator's button click (hub UI password-gated); pre-existing lanresolver permission-denied WARN on /var/lib/felhom-agent/guests noted in REPORT.

• v0.77.0 (2026-07-09) — fork-4: escrow the offsite restic repo password under R. IdentityBundle gains ResticRepoPassword (rides the existing age-under-R WrapIdentityBundle path — validated by the custody spike febdc56). New POST /escrow/stage-secret (withGuest) transiently stages the controller-pushed password (0600, never logged), which the --selftest=escrow-create ceremony auto-injects into the bundle and then wipes. AttachResticPassword/StagedResticPasswordPath/ WipeStagedResticPassword added. Pairs with controller v0.105.0 (push + atomicity gate + DR inject + DRResticCoord). NOT yet live-validated — the supervised escrow ceremony is operator-run.

• v0.76.0 (2026-07-08, LIVE on felhom-pve + PUBLISHED sha 9828c5f7…f50b — THE Day-0 manifest bump target; 0.75.0 superseded unpublished) — GL-5b / G12: restore-test full-fidelity. Params derive from the ARCHIVE's embedded config (drRestoreOverrides, same as DR — the old live-source-config path verified the wrong object AND dropped storage mpN per PVE's all-or-nothing rule; deleted with bindMountOverrides/archiveVMID). NEW mount-parity assert (restored mpN vs archive; miss/mispath/undersize/extra = FAIL naming the delta) + MountParity/ MountInventory on the wire record (additive). Live-proven: scratch 990000 ← 6.5GB 9201 archive, parity ok, inventory mp0 200G+mp1 50G+2 throwaways, 3m4s local tier (cheaper than feared); rotated-out archive volid → clean up-front refusal (nice failure mode). bringup.go untouched.

• v0.75.0 (2026-07-08, LIVE on felhom-pve) — GL-5 / go-live G8: guest-loss DR bring-up actually restores (closes the v0.74.0 OPEN item + SPIKE-dr-bindmount-source §8). DR passes the COMPLETE explicit restore param set derived from the archive's embedded config (NEW Client.ExtractArchiveConfig, 200 under the scoped token) — two live-discovered PVE rules: mpN params need an explicit rootfs, AND unlisted mountpoints are silently DROPPED (first run booted without mp0/mp1!) — storage mpN passed through, structural mp8/mp9 → throwaways, then step 4d swaps the REAL binds in via the host runner (root pct; new EngineOptions.HostRunner+StateDir seam) and deletes the unusedN residue. Scratch-DR live-proven end-to-end (9310 from a real 9201 archive: mp0 200G + mp1 50G + real binds + no residue + clean teardown). Provision = nil overrides (regression-tested). NOTE: published/vouch-pending agent is 0.74.0 — publish 0.75.0 before/with the manifest bump. OBSERVATION: the DR selftest hardcodes KeepMAC=true — a scratch DR while the SOURCE guest is live briefly duplicates its MAC on the bridge (pre-existing; fine for supervised runs, worth a -keep-mac flag someday). Full customer-data DR drill = GL-6/S5 family.

• 2026-07-07 — v0.74.0 Gitea-PUBLISHED (RUNBOOK GL-1) — the LIVE felhom-pve binary's exact bytes, sha256 1ec3f58842edce1e…76af05, anon-fetch-verified. This supersedes/closes every standing "publish 0.6x + Day-0 vouch" OPEN item below (0.64→0.73 were never published; 0.74.0 is the vouch target). Golden 0.103.0 published in the same run (felhom.eu execution record documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md). Day-0 manifest vouch = operator step (agent 0.74.0 / golden 0.103.0).

• v0.74.0 (2026-07-07) — campaign-2 R2 CLOSED; the mislabelled "R1" was a symptom (LIVE on felhom-pve). Pool membership is what lets the pool-scoped token reach a guest; pct restore --pool sets it only at CREATE, so a restore-over-existing dropped 9201 from the felhom pool → no VM.Audit → restore-test's existing bindMountOverrides never ran → "mp8 … only possible for root". Fix: Client.PoolAddVMID + bring-up re-asserts membership post-restore (warn-not-fail). Role/ACL + bindMountOverrides untouched (both correct). Live restore-test PASSED for the first time once the pool was healed (Part A one-liner): read config → neutralize 2 binds → restore → boot+running → clean teardown, 4m35s. B3 (scratch-teardown 403) confirmed a cascade — no code. OPEN: DR bring-up -mode dr bind-override gap (spike SPIKE-dr-bindmount-source-2026-07-07.md: small known-constant override reusing bindMountOverrides; mp8/mp9 are structural constants).

• v0.73.0 (2026-07-06) — F2 mount-role fallback CLOSED (LIVE on felhom-pve). roleForMountPath gained a mount-table fallback (Impl-2b style): a bind-mounted RAW enrolled user-data drive is not a PVE storage, so it fail-safe'd to system and the eject/decommission gates 403'd EVERY user-data drive (campaign F2, where=/mnt/teszt_enroll role=system). Device-keyed classification + whole-disk containment (storage.SameWholeDisk); Observe-error keeps the fail-safe BEFORE the fallback. Only roleForMountPath touched. Live-proven full lifecycle on teszt_enroll (eject/decommission 200, no-rebind across restart, end==pre). OPEN follow-up: the deviceRole/roleForMountPath unification refactor (deferred).

• v0.72.0 (2026-07-05) — OOB operator access (merged E1+H1) — TASK H1, provenance both SPIKE-{felhom-sshd,oob-wg-operator-peer}-2026-07-05. Operator /32 RENDERED into wg-felhom AllowedIPs (survives self-heal, [OF-1]); dedicated internal/felhomsshd (port claim + config render→sshd -t→reload + operator authorized_keys + heal + oob heartbeat stanza); static inet felhom_oob belt (agent mutates SET ELEMENTS ONLY); configs/felhom-sshd.service (NO RuntimeDirectory [SF-1]) + felhom-oob.nft + felhom-op.sudoers; FELHOM_SSHD+FELHOM_OOB grants; oob.enabled DEFAULT FALSE. Live on felhom-pve (8822, belt filled, operator SSH as felhom-op with scoped sudo); hub v0.35.0. Rollback .bak-0.71.0. 5 live-found bugs fixed (port path, self-listen flip-flop, nil-block lockout, reachable-via-dial, operator-configured source).

• v0.71.0 (2026-07-05) — management-plane break-glass: privsep-dir watchdog + mgmt_plane health — TASK G1 (prereq for felhom-sshd/H1), provenance SPIKE-felhom-sshd-2026-07-05 §8. Host artifacts (configs/felhom-privsep.tmpfiles + felhom-mgmt-watchdog.{sh,service,timer}) make /run/sshd boot-persistent AND auto-heal it every ~60s agent-independently (heals with the agent stopped — proven live: /run/sshd removed → restored in 30.0s, :22 back, no login). internal/mgmtplane reports the additive mgmt_plane heartbeat stanza; hub v0.34.1 raises mgmt_plane_healed. NO unit declares RuntimeDirectory= (the incident cause). H1 may now assume /run/sshd is guaranteed present. Live on felhom-pve; rollback .bak-0.70.0.

• v0.70.0 (2026-07-05) — agent self-update (operator-signed A/B slots + crash-loop auto-rollback) — TASK D1, provenance SPIKE-agent-selfupdate-2026-07-05. An operator-signed agent_update op (version+sha256, sha is the only integrity root) rides the signed-jobs gate; internal/selfupdate.Executor downloads+verifies+hands to felhom-selfupdate-guarded apply (root re-verify → A/B atomic flip → pending marker → detached restart); the new binary commits after a 60s dwell; a crash-looping binary is auto-reverted by OnFailure=felhom-agent-rollback.service (first-crash trigger [SF-1]) with the tuned [Unit] start-limit (120s/4) as backstop. Host artifacts + sudoers FELHOM_SELFUPDATE + felhom-host-install.sh day-0 install + report field selfupdate_pending. Green tests + companions. LIVE-VALIDATED on felhom-pve (2026-07-05): all 4 drills PASS — happy path (0.70.0→0.70.1 signed op → download+verify+flip+commit), crash-rollback (0.70.2-crash → OnFailure → ~2s crash-to-recovered, byte-identical revert, no loop), no-pending guard, gate refusal (non-pinned key). Full agent-side pipeline ran real (envelope injected into the hub signed_jobs queue — CC lacks the hub global operator key; hub enqueue-auth is hub-unit-tested). Box restored to canonical v0.70.0 (host artifacts KEPT installed; scratch operator key REMOVED — self-update dormant until an operator pins a real key, a Day-0-vouch-style follow-up). Rollback felhom-agent.bak-0.69.0. OPEN (v1 scope-outs): no hub-floor auto-update, no failed-update auto-retry, no pending-timeout auto-rollback; per-crash OnFailure can double-fire (idempotent — future: serialize the rollback oneshot). Detail: REPORT.md.

• v0.69.0 (2026-07-04, live on felhom-pve) — S5: host-loss DR — safe halves shipped. Part 1 wgtunnel.InstallRecoveredKey — writes an escrow-recovered WG privkey (create-only, refuse-overwrite) so the tunnel re-establishes with the SAME identity/pubkey (same /32), no keygen; wired into --selftest=identity-consume -install-wg-key (opt-in; pre-S3 blob → logged fresh-keygen fallback). Part 2 new internal/dr — consumes the host_loss restore_directive (was logged-ignored) into an inspectable RestorePlan via AddConsumer: per-guest {vmid,archive,target, sizing} + per-drive {durable_id→mount} + offsite PBS coord; DERIVE-AND-SURFACE only (Consumer has no restore/destroy dep — execute-nothing is structural). Tests + red-proofs (WG create-only; plan mode-gate). Part 3 hub escrow-GET NOT needed (operator exports the blob via sqlite3 writefile on a cp'd hub.db). Part 4-A re-attach wrong-disk safety already unit-proven (ResolveStorageDevice: match resolves, absent/mismatch ERRORS, non-uuid scheme refused — never a near disk). Part 4-B (destructive in-place 9201 restore) PREPARED + OPERATOR-GATED, NOT executed — pre-flight green (offsite ct/9201 restorable per S4.1); the operator runs the R-consume steps + confirms the destroy (§9-4a: CC never runs a consume/R command — see operator-present-one-time-secrets). OPEN: the operator-run 4-B drill; guest_loss DR; hub-driven full-auto DR. Rollback felhom-agent.bak-0.68.0. Detail: REPORT.md + doc-06 §3.5/S5.

• v0.68.0 (2026-07-04, live on felhom-pve) — S4.1: unattended offsite restore-test. Tier-aware restore-task deadline: RestoreTestSpec.RestoreTaskTimeout (0→10m default) from config.RestoreTestPBSRestoreTimeoutSeconds (accessor default 120m), set only when SourceTier=="pbs" (main.restoreTaskTimeout); local tier UNCHANGED. Fixes the WAN restore being killed at 10m → mid-restore teardown → leaked scratch. Teardown "VM.Allocate" follow-up = PHANTOM (diagnosed, not blind-fixed): ran the restore-test on the AGENT-TOKEN path sourcing the offsite (pbs) backup → pass:true verified:boot+running, teardown succeeded (torn down vmid=990000, no 403), scratch band clean. The earlier 403 was the 10m-timeout consequence (guest not yet pool-associated); the scratch is restored INTO /pool/felhom (ACL already grants VM.Allocate) so teardown is authorized once the restore completes. No ACL/host-install change. OPEN: publish 0.68.0 + Day-0 vouch; Tier-1/Tier-2 split for offsite-as-default; S5 DR consume. Rollback felhom-agent.bak-0.67.0. Detail: REPORT.md.

• v0.66.0 + v0.67.0 (2026-07-04, live on felhom-pve) — S4: PBS over the tunnel. v0.66.0: wgtunnel v4-pin (renderConf writes the resolved A LITERAL, never DNS/AAAA; Resolver seam, lowest addr; cached → steady-state zero-DNS/zero-exec) + re-resolve watchdog (Manager.Watchdog, loop-only; handshake stale > stale_after_seconds=180 → re-resolve → IP-changed re-render+restart)

  • FELHOM_WG Critical flips (conf-install/enable/restart/handshake-read). v0.67.0: namespace-aware PBS client (Config.Namespace → Snapshots ?ns=, Verify ns=; root-ns unchanged) — the operator-approved fix after Phase-1 showed the ns-unaware datastore-root 403s a per-tenant token. Live Scenario-D (all green): real vzdump of 9201 → ciphertext in ns demo-felhom-01 over the tunnel; ns-scoped verify=ok under the box's own felhom@pbs!demo-felhom-01 token; WARN gone; restore round-tripped (decrypt with box-born key → boot → teardown). Confirmed tenant ACL (felhom-hetzner): DatastoreBackup on /datastore/felhom-offsite/<ns> (NOT /ns/<ns>) to BOTH user felhom@pbs AND token (privsep=intersection; cross-ns 403); DatastoreBackup can't prune (safety). FINDINGS: retarget field is local_backup_target (not backup_target); retarget REVERTED to local (controller backs up ~every 30 min → single-target offsite = near-continuous 20-min uploads; needs Tier-1/Tier-2 split); restore-test scheduler needs a WAN restore deadline + scratch-band VM.Allocate before it runs offsite unattended. OPEN: escrow-create (OPERATOR-PRESENT, new R); publish 0.66/0.67 + Day-0 vouch; S5 DR consume. Rollback: felhom-agent.bak-0.65.0/.bak-0.66.0. Detail: REPORT.md + doc-06 §3.4/§4.2 + runbook §4a/§4b.

• v0.65.0 (2026-07-04, live on felhom-pve) — S3.1 offsite-tunnel client MTU 1420 → 1280: resolves 06 §4.3's OPEN DECISION left by the CGNAT smoke test. 1420 silently black-holed bulk TCP on sub-~1480 paths (mobile ~1400, DS-Lite ~1452) — handshake+ping healthy, PBS TLS page (and at S4 the backup itself) drops. New const clientMTU = 1280 (RFC 8200 IPv6-minimum floor; outer 1340 v4 / 1360 v6 fits every realistic path), permanent + fleet-wide + family-agnostic. Client-only by construction — interface MTU caps box→PBS, advertised MSS caps PBS→box, so the endpoint's wg0 is untouched (zero live-endpoint risk). Golden pins exact MTU = 1280 (red-proofed vs a 1420 flip); no wire/JSON change. Live: agent re-rendered on restart (hash-gated apply), conf + live iface both 1280, PBS page loads at 1280 (no regression on wired). OPEN: true-CGNAT-SIM retest (low risk); publish 0.65.0 + Day-0 vouch (operator); S4 PBS-over-tunnel. Rollback: felhom-agent.bak-0.64.0 on the box. The v4-pin (§4.2 determinism) is a separate, optional future note — NOT needed for MTU correctness.

• v0.64.0 (2026-07-04, live on felhom-pve) — S3 offsite WG tunnel: new internal/wgtunnel (keygen 0600/0700, marker-gated one-shot registration, agent-managed wg-quick@wg-felhom from the hub's desired-state wireguard block via the new desired.Syncer.AddConsumer seam, revoked-stays-revoked teardown, report stanza) + FELHOM_WG sudoers/capabilities + IdentityBundle.WGPrivateKey escrow auto-inject. wg_tunnel.enabled DEFAULTS FALSE (safety gate — rollout to Peti's box is a no-op until the production endpoint exists; enabled explicitly on felhom-pve only). Live: tunnel to ep0.felhom.eu:443 up 3 s after enable (PBS page through 10.77.0.1:8007), reboot-persistent, revocation drill clean, 30-min keepalive soak. GOTCHAS: hub envelope poll_interval_seconds (hub-side const 900 s) silently overrides agent poll_seconds on cycle 1; wg show <if> dump leaks the PRIVATE key (forbidden everywhere — sudoers only grants latest-handshakes). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix); publish 0.64.0 to Gitea + Day-0 vouch (operator); S4 points PBS at the tunnel.

• configs: build-golden.sh v2.0.0 (2026-07-03, @ ceca355; no agent version change) — drill findings B5 + B1 FIXED (DRILL-golden-098-2026-07-03.md): the controller tag is a MANDATORY argument (the default rotted twice — a fresh install booted a pre-floor controller, forcing the guide's manual D.1b update) and the golden now bakes a felhom-controller-bootstrap.path unit (controller deploys the moment the back-half hot-plugs the bootstrap mount — no reboot; installer v1.9.1's reboot is a redundant belt, kept). Golden 0.98.3 baked on the drill VM, clean-room validated (bake integrity → isolated hot-plug proof → local-golden Day-0 → published-artifact Day-0), published (sha256 b9a02ef1…fd01) + operator-vouched — Day-0 manifest now vouches agent 0.63.0 + golden 0.98.3 (the v0.63.0 vouch follow-up below is DONE). Fresh installs land current and self-manage. NEW operator follow-up (SECURITY): the customer-config git.token has Gitea package-WRITE rights — scope down + rotate (evidence-doc observation O1).

• v0.63.0 (2026-07-03, live on felhom-pve + Gitea-published sha256 b4a89c81…) — drill findings B3 + B2 FIXED (DRILL-day0-cleanroom-2026-07-03.md): TokenStore.Lookup reloads the append-only store once on a miss (cross-process coherence with the one-shot provisioner — no more fresh-install /controller/swap 401 / manual restart; size short-circuit bounds the cost; behind the TokenAuthority seam) + guesthook.InstallSnippet issues a fenced mkdir -p /var/lib/vz/snippets first (fresh boxes lacked the dir → the self-heal hook silently never installed). Sudoers gained exactly that one grant — ship sudoers WITH the binary (done on felhom-pve). Red-proofed both; Scenario-E method: compiled test suite run ON felhom-pve + live channel-health hit-path. OPERATOR FOLLOW-UP: bump the hub Day-0 manifest to agent 0.63.0 — until then fresh installs get 0.62.0 and the guide's D.1b restart-first step still applies (narrowed to "< v0.63.0" in the guide).

• v0.62.0 (2026-07-03) — audit A1 RESOLVED: the stale-lock reaper's scan is now pool-intersected (staleLockController.Guests() = ListLXC ∩ Client.Pool("felhom") members), fail-safe skip on pool-read failure; pve:pool-read capability (non-critical) + --selftest "pool read" line. Companion host-install v1.9.0 adds Pool.Audit to FelhomAgentGuest — deploy order on any box: rescope ACL first, then this agent. Per SPIKE-a1-pool-membership-read-2026-07-03.md; red-proofed tests in stalelock_pool_test.go.

• 2026-07-03 — CLAUDE.md refreshed: version narrative removed (state lives HERE + CHANGELOG top), layout completed (all 17 internal packages + cmd/felhom-opsign); deploy runbook now in the felhom-build-deploy skill (felhom.eu/skills/).

• 2026-07-03 — REUSE.md exists at the repo root (canonical helpers / format-safety guards / traps / seams, code-verified); maintenance rule active: update it in the same commit that changes a shared helper.

• v0.61.0 (2026-07-03) — blast-radius audit fixes B1 + D1 + D2 + D3 from felhom.eu/documentation/audits/AUDIT-blast-radius-hostroot-localapi-2026-07-02.md: random temp staging for root-installed scripts (+ sudoers/manifest glob updates), mkfs-wrapper member/RO re-checks (validated by scripts/mkfs-guarded-harness.sh), classifyClaim empty-lsblk fail-safe, and the blank-format anti-retarget (durable-id-bound, AGENT-001's benign-branch twin).

• Deployed on demo host felhom-pve (node demo-felhom), non-root felhom-agent service user, pool-scoped token (felhom pool).

Open threads

• Deferred audit items (housekeeping/design, all INFO): C1 (controller-swap version floor), C2 (NAS server allowlist), A2 (gate journal cross-check), B2–B5, E1/E2.
• Drive-enrollment leftovers: (a) runStorageInit slow-device detached-format polling; (b) Impl-3 shared-box operator format gate.
• BUNDLE leftover: non-root agent can't read the PBS key; migration must preserve cert/key/tokens.
• Not run (needs a supervised session): the destructive D1/D3 live proofs (real mkfs on a crafted member; a live /dev re-enumeration race during a real format).