felhom-agent/CONTEXT.md

CONTEXT — felhom-agent working state

Snapshot of the current state + open threads. Authoritative history lives in CHANGELOG.md (top entry = current); the end-of-task detail lives in REPORT.md.

Current

• v0.69.0 (2026-07-04, live on felhom-pve) — S5: host-loss DR — safe halves shipped. Part 1 wgtunnel.InstallRecoveredKey — writes an escrow-recovered WG privkey (create-only, refuse-overwrite) so the tunnel re-establishes with the SAME identity/pubkey (same /32), no keygen; wired into --selftest=identity-consume -install-wg-key (opt-in; pre-S3 blob → logged fresh-keygen fallback). Part 2 new internal/dr — consumes the host_loss restore_directive (was logged-ignored) into an inspectable RestorePlan via AddConsumer: per-guest {vmid,archive,target, sizing} + per-drive {durable_id→mount} + offsite PBS coord; DERIVE-AND-SURFACE only (Consumer has no restore/destroy dep — execute-nothing is structural). Tests + red-proofs (WG create-only; plan mode-gate). Part 3 hub escrow-GET NOT needed (operator exports the blob via sqlite3 writefile on a cp'd hub.db). Part 4-A re-attach wrong-disk safety already unit-proven (ResolveStorageDevice: match resolves, absent/mismatch ERRORS, non-uuid scheme refused — never a near disk). Part 4-B (destructive in-place 9201 restore) PREPARED + OPERATOR-GATED, NOT executed — pre-flight green (offsite ct/9201 restorable per S4.1); the operator runs the R-consume steps + confirms the destroy (§9-4a: CC never runs a consume/R command — see operator-present-one-time-secrets). OPEN: the operator-run 4-B drill; guest_loss DR; hub-driven full-auto DR. Rollback felhom-agent.bak-0.68.0. Detail: REPORT.md + doc-06 §3.5/S5.
• v0.68.0 (2026-07-04, live on felhom-pve) — S4.1: unattended offsite restore-test. Tier-aware restore-task deadline: RestoreTestSpec.RestoreTaskTimeout (0→10m default) from config.RestoreTestPBSRestoreTimeoutSeconds (accessor default 120m), set only when SourceTier=="pbs" (main.restoreTaskTimeout); local tier UNCHANGED. Fixes the WAN restore being killed at 10m → mid-restore teardown → leaked scratch. Teardown "VM.Allocate" follow-up = PHANTOM (diagnosed, not blind-fixed): ran the restore-test on the AGENT-TOKEN path sourcing the offsite (pbs) backup → pass:true verified:boot+running, teardown succeeded (torn down vmid=990000, no 403), scratch band clean. The earlier 403 was the 10m-timeout consequence (guest not yet pool-associated); the scratch is restored INTO /pool/felhom (ACL already grants VM.Allocate) so teardown is authorized once the restore completes. No ACL/host-install change. OPEN: publish 0.68.0 + Day-0 vouch; Tier-1/Tier-2 split for offsite-as-default; S5 DR consume. Rollback felhom-agent.bak-0.67.0. Detail: REPORT.md.
• v0.66.0 + v0.67.0 (2026-07-04, live on felhom-pve) — S4: PBS over the tunnel. v0.66.0: wgtunnel v4-pin (renderConf writes the resolved A LITERAL, never DNS/AAAA; Resolver seam, lowest addr; cached → steady-state zero-DNS/zero-exec) + re-resolve watchdog (Manager.Watchdog, loop-only; handshake stale > stale_after_seconds=180 → re-resolve → IP-changed re-render+restart)
  • FELHOM_WG Critical flips (conf-install/enable/restart/handshake-read). v0.67.0: namespace-aware PBS client (Config.Namespace → Snapshots ?ns=, Verify ns=; root-ns unchanged) — the operator-approved fix after Phase-1 showed the ns-unaware datastore-root 403s a per-tenant token. Live Scenario-D (all green): real vzdump of 9201 → ciphertext in ns demo-felhom-01 over the tunnel; ns-scoped verify=ok under the box's own felhom@pbs!demo-felhom-01 token; WARN gone; restore round-tripped (decrypt with box-born key → boot → teardown). Confirmed tenant ACL (felhom-hetzner): DatastoreBackup on /datastore/felhom-offsite/<ns> (NOT /ns/<ns>) to BOTH user felhom@pbs AND token (privsep=intersection; cross-ns 403); DatastoreBackup can't prune (safety). FINDINGS: retarget field is local_backup_target (not backup_target); retarget REVERTED to local (controller backs up ~every 30 min → single-target offsite = near-continuous 20-min uploads; needs Tier-1/Tier-2 split); restore-test scheduler needs a WAN restore deadline + scratch-band VM.Allocate before it runs offsite unattended. OPEN: escrow-create (OPERATOR-PRESENT, new R); publish 0.66/0.67 + Day-0 vouch; S5 DR consume. Rollback: felhom-agent.bak-0.65.0/.bak-0.66.0. Detail: REPORT.md + doc-06 §3.4/§4.2 + runbook §4a/§4b.
• v0.65.0 (2026-07-04, live on felhom-pve) — S3.1 offsite-tunnel client MTU 1420 → 1280: resolves 06 §4.3's OPEN DECISION left by the CGNAT smoke test. 1420 silently black-holed bulk TCP on sub-~1480 paths (mobile ~1400, DS-Lite ~1452) — handshake+ping healthy, PBS TLS page (and at S4 the backup itself) drops. New const clientMTU = 1280 (RFC 8200 IPv6-minimum floor; outer 1340 v4 / 1360 v6 fits every realistic path), permanent + fleet-wide + family-agnostic. Client-only by construction — interface MTU caps box→PBS, advertised MSS caps PBS→box, so the endpoint's wg0 is untouched (zero live-endpoint risk). Golden pins exact MTU = 1280 (red-proofed vs a 1420 flip); no wire/JSON change. Live: agent re-rendered on restart (hash-gated apply), conf + live iface both 1280, PBS page loads at 1280 (no regression on wired). OPEN: true-CGNAT-SIM retest (low risk); publish 0.65.0 + Day-0 vouch (operator); S4 PBS-over-tunnel. Rollback: felhom-agent.bak-0.64.0 on the box. The v4-pin (§4.2 determinism) is a separate, optional future note — NOT needed for MTU correctness.
• v0.64.0 (2026-07-04, live on felhom-pve) — S3 offsite WG tunnel: new internal/wgtunnel (keygen 0600/0700, marker-gated one-shot registration, agent-managed wg-quick@wg-felhom from the hub's desired-state wireguard block via the new desired.Syncer.AddConsumer seam, revoked-stays-revoked teardown, report stanza) + FELHOM_WG sudoers/capabilities + IdentityBundle.WGPrivateKey escrow auto-inject. wg_tunnel.enabled DEFAULTS FALSE (safety gate — rollout to Peti's box is a no-op until the production endpoint exists; enabled explicitly on felhom-pve only). Live: tunnel to ep0.felhom.eu:443 up 3 s after enable (PBS page through 10.77.0.1:8007), reboot-persistent, revocation drill clean, 30-min keepalive soak. GOTCHAS: hub envelope poll_interval_seconds (hub-side const 900 s) silently overrides agent poll_seconds on cycle 1; wg show <if> dump leaks the PRIVATE key (forbidden everywhere — sudoers only grants latest-handshakes). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix); publish 0.64.0 to Gitea + Day-0 vouch (operator); S4 points PBS at the tunnel.
• configs: build-golden.sh v2.0.0 (2026-07-03, @ ceca355; no agent version change) — drill findings B5 + B1 FIXED (DRILL-golden-098-2026-07-03.md): the controller tag is a MANDATORY argument (the default rotted twice — a fresh install booted a pre-floor controller, forcing the guide's manual D.1b update) and the golden now bakes a felhom-controller-bootstrap.path unit (controller deploys the moment the back-half hot-plugs the bootstrap mount — no reboot; installer v1.9.1's reboot is a redundant belt, kept). Golden 0.98.3 baked on the drill VM, clean-room validated (bake integrity → isolated hot-plug proof → local-golden Day-0 → published-artifact Day-0), published (sha256 b9a02ef1…fd01) + operator-vouched — Day-0 manifest now vouches agent 0.63.0 + golden 0.98.3 (the v0.63.0 vouch follow-up below is DONE). Fresh installs land current and self-manage. NEW operator follow-up (SECURITY): the customer-config git.token has Gitea package-WRITE rights — scope down + rotate (evidence-doc observation O1).
• v0.63.0 (2026-07-03, live on felhom-pve + Gitea-published sha256 b4a89c81…) — drill findings B3 + B2 FIXED (DRILL-day0-cleanroom-2026-07-03.md): TokenStore.Lookup reloads the append-only store once on a miss (cross-process coherence with the one-shot provisioner — no more fresh-install /controller/swap 401 / manual restart; size short-circuit bounds the cost; behind the TokenAuthority seam) + guesthook.InstallSnippet issues a fenced mkdir -p /var/lib/vz/snippets first (fresh boxes lacked the dir → the self-heal hook silently never installed). Sudoers gained exactly that one grant — ship sudoers WITH the binary (done on felhom-pve). Red-proofed both; Scenario-E method: compiled test suite run ON felhom-pve + live channel-health hit-path. OPERATOR FOLLOW-UP: bump the hub Day-0 manifest to agent 0.63.0 — until then fresh installs get 0.62.0 and the guide's D.1b restart-first step still applies (narrowed to "< v0.63.0" in the guide).
• v0.62.0 (2026-07-03) — audit A1 RESOLVED: the stale-lock reaper's scan is now pool-intersected (staleLockController.Guests() = ListLXC ∩ Client.Pool("felhom") members), fail-safe skip on pool-read failure; pve:pool-read capability (non-critical) + --selftest "pool read" line. Companion host-install v1.9.0 adds Pool.Audit to FelhomAgentGuest — deploy order on any box: rescope ACL first, then this agent. Per SPIKE-a1-pool-membership-read-2026-07-03.md; red-proofed tests in stalelock_pool_test.go.
• 2026-07-03 — CLAUDE.md refreshed: version narrative removed (state lives HERE + CHANGELOG top), layout completed (all 17 internal packages + cmd/felhom-opsign); deploy runbook now in the felhom-build-deploy skill (felhom.eu/skills/).
• 2026-07-03 — REUSE.md exists at the repo root (canonical helpers / format-safety guards / traps / seams, code-verified); maintenance rule active: update it in the same commit that changes a shared helper.
• v0.61.0 (2026-07-03) — blast-radius audit fixes B1 + D1 + D2 + D3 from felhom.eu/documentation/audits/AUDIT-blast-radius-hostroot-localapi-2026-07-02.md: random temp staging for root-installed scripts (+ sudoers/manifest glob updates), mkfs-wrapper member/RO re-checks (validated by scripts/mkfs-guarded-harness.sh), classifyClaim empty-lsblk fail-safe, and the blank-format anti-retarget (durable-id-bound, AGENT-001's benign-branch twin).
• Deployed on demo host felhom-pve (node demo-felhom), non-root felhom-agent service user, pool-scoped token (felhom pool).

Open threads

• Deferred audit items (housekeeping/design, all INFO): C1 (controller-swap version floor), C2 (NAS server allowlist), A2 (gate journal cross-check), B2–B5, E1/E2.
• Drive-enrollment leftovers: (a) runStorageInit slow-device detached-format polling; (b) Impl-3 shared-box operator format gate.
• BUNDLE leftover: non-root agent can't read the PBS key; migration must preserve cert/key/tokens.
• Not run (needs a supervised session): the destructive D1/D3 live proofs (real mkfs on a crafted member; a live /dev re-enumeration race during a real format).