felhom.eu/CONTEXT.md

# CONTEXT — felhom.eu working state

> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.

- **2026-07-12 — CAMPAIGN-3 Task A SHIPPED: agent v0.85.0 boot/recovery plane + appliance self-heal.**
  Fixes F12 (CRITICAL boot ordering cycle — templates drop network-online, `MigrateNetworkUnits`
  repairs installed units), F11/F10/F9 (reassert: fstype-driven classify, reset-failed+rearm, per-unit
  verdict, + re-arm-on-guest-blind matrix-correction), F2/F1 (zero residue: reset-failed + rmdir), the
  guest-hook rc255 belt (recover+timeout, wrapper no longer execs), and the appliance node self-heal
  (`internal/selfheal`: host networking recovery, gated on `deployment_mode` — host-install `--mode`
  templates it; byo remedy unreachable). 3 sudoers grants added (reset-failed, rmdir, start networking).
  **Live matrix on demo felhom-pve: host reboot ×2 = 0 ordering-cycle lines both boots (F12 dead);
  appliance self-heal recovered networking on attempt 1; F10 reset-failed+rearm sweep loud; zero
  residue; guest-reboot heals.** **NEW finding F13 (HIGH, deferred):** active nfs4 under mp8 can fail
  PVE rbind (rc255); NAS-down-at-boot → apps `Created` (fix-3). Peti's box (agent 0.81) unaffected
  until his rollout — carries the F12 coin flip if it has a NAS share (interim drop-in in the agent
  REPORT). host-install v1.14.0; scripts CHANGELOG. Details: `documentation/controller/network-storage-nas.md`.
- **2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; host recovered, RCA closed.** Unattended chaos campaign
  22:09→04:27 vs demo box (ctrl 0.117.0/agent 0.84.0): data plane solid (refusals/verify/rollback/
  restore byte-identical/EIO instant/stub badge + deploy-409 live-validated). The NAS automount
  plane produced 2 CRITICAL + 3 HIGH: **F12 CRITICAL (= the overnight host loss, RCA'd next
  morning): the agent automount template's `After/Wants=network-online.target` creates a boot
  ordering cycle — systemd deletes an arbitrary job to break it; boot -1 sacrificed networking
  (host dark 7h, hardware exonerated), boot 0 sacrificed the automount (NAS dead). Every host
  boot with an enrolled share = coin flip until the template drops that ordering (`_netdev` on
  the .mount suffices).** F10 CRITICAL (start-limit never re-armed, once blocked guest start),
  F11/F9 HIGH (post-start reassert "skip-active" skips the healing automount restart; agent-restart
  sweep silent) → 4 NAS apps dead-at-boot on 3/3 guest reboots; F7 HIGH (in-place dump write →
  0-byte tar replaced last good backup during outage window). Fix-6 numbers: ring horizon 55min
  idle / **6.5min under load**, wiped on every restart. Box left in defined state (6 wave apps
  healthy, temps removed, NAS baseline-identical). Full ledger/evidence:
  `documentation/audits/CAMPAIGN-3-2026-07-11.md` + `180:~/campaign3/`.
- **2026-07-11 — HUB v0.47.0 UI REORGANIZATION SHIPPED** (CHANGELOG v0.47.0; commits `9f29bf3` →
  `0daddcd` + docs). Five deliverables: **(1)** `.data-table td a:not(.btn)` button-contrast fix;
  **(2)** customer page = **8 hash tabs** (`#tab=…`, sticky summary strip, Events error badge,
  no-JS graceful degradation — panels hide only via a JS-added body class); **(3)** shared
  `host_detail_body` sub-template rendered by `/hosts/{id}` AND the new per-customer **Host tab**
  (a list by design; `store.ListHostsByCustomer` + `hostDetailData` builder); **(4)** **stale host
  removal** — `GET /hosts/{id}/delete-impact` (counts/booleans only) + `POST /hosts/{id}/delete`
  behind type-to-confirm; ONLINE → 409 always (no override), escrow needs an explicit checkbox
  (`ErrHostEscrowPresent`, tx never starts), one-tx cascade incl. the bound wg peer (wgsync's 5-min
  declarative push converges the endpoint; log bundles die by `scope_id == host_id` only);
  **(5)** **/offsite multi-endpoint management UI** — all `wg_endpoints` rows as cards +
  add/edit/delete with 409 guards (peers-in-subnet pins subnet + delete). **Deferral (explicit):
  peer allocation, the wgsync reconciler push and the desired-state merge stay lowest-endpoint-id
  (`GetWGEndpoint` untouched); the future arc is a `wg_peers.endpoint_id` migration + per-endpoint
  allocation/reconciler/desired-state.** Five red-proofs ran (online gate, escrow ack, bundle
  scope, endpoint-delete guard, subnet-change guard). *Resolved:* the earlier ⚠️ about `146d165`
  sweeping the Part-4 WIP — the sweep caught `hosts.go` mid-red-proof (escrow ack bypassed);
  `068427a` restored the gate; the arc is now complete and author-reviewed. **Stale-doc note:** the
  workspace-root CLAUDE.md's hub deploy wording predates GitOps — deploys are manifest-tag bumps +
  deliberate ArgoCD sync, never `kubectl set image`.

- **2026-07-11 — RCA FIXES 1+2+4 SHIPPED + Q1c GREEN (agent v0.84.0 + controller v0.117.0 LIVE on
  demo).** Reboot survival: automatic since agent 0.84.0, live-proven 2026-07-11 21:20 (`pct reboot
  9201` under operator GO — the hook re-armed + verified nas-media DURING the start task, CWA
  auto-started healthy, health ok first cycle, no badge). Agent: `ReassertNetworkMounts` (hook
  post-start + startup; stop+enable-now on existing sudoers verbs — NO sudoers widening needed;
  never in periodic paths). Controller: statfs consuming-namespace classifier → probe
  `not_network_fs` (stub can never verify), deploy-time stub refusal (409; idle autofs deploys
  fine), distinct stub badge (wins over unreachable; unreachable line byte-identical), deploy view
  selects stored HDD_PATH (RCA fix 4). MinAgent unchanged 0.81.0. All red-proofs recorded in repo
  REPORTs. Feature doc: `documentation/controller/network-storage-nas.md` §Reboot survival; audit §7
  follow-up added. Queued: RCA fixes 3/5/6; Peti publish train now carries 0.84 + 0.117.

- **2026-07-11 — NAS/CWA RCA (diagnostic, no fixes):** `documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md`.
  The nas-media re-add verified legitimately (new host automounts DO propagate into running
  guests); a second `pct reboot 9201` at 15:39 severed the trigger → guest-side stub; CWA
  deployed correctly to the NAS (app.yaml + binds + data all nas-media) but is DOWN since 15:39
  (docker can't recreate the bind source; perms luckily blocked any system-drive writes; zero data
  loss). UI "felhom-usb" = display bug (deploy.html selects by IsDefault, never stored HDD_PATH);
  hub device-ID WARN was CORRECT and the only guard that fired; NAS badge masks because it asks
  the agent (host ns). 6 prioritized fix tasks in the audit §5. **Remediation §6 PERFORMED on
  operator GO (~20:14–20:20 CEST): automount-unit restart re-planted the trigger in the running
  guest instantly (fix-1 mechanism live-proven), CWA restarted healthy on the NAS library,
  uid-1000 write OK, controller health back to status=ok at 20:18. Demo fully healed.**

- **2026-07-11 — OBSERVABILITY PASS (hub v0.46.0 LIVE Synced/Healthy; agent v0.83.0 + controller
  v0.116.1 on demo).** Remote pull-only log access to both box components' always-DEBUG capture
  rings. Hub: `log_bundle_requests`/`log_bundles` store (gzip, newest-3, **72 h TTL** on the 60 s
  sweep), the **secret gate fail-closed** before storing (`blocked: possible secret`; `[REDACTED]`/
  checksums pass), report-ACK `controller_log_requested` + heartbeat-envelope `log_tail_requested`
  channels (consume-once; pre-0.83 agents stay visibly pending), host-detail Diagnostics section
  (Request controller/agent logs + View/Download; the hosts read-only invariant amended: these two
  forms are the only actions). Conventions codified:
  `documentation/runbooks/logging-conventions.md` + CLAUDE.md one-liners in all three repos.
  **OPEN: operator clicks both request buttons on hub.felhom.eu/hosts/<demo> to close the live
  round-trip** (UI password-gated; the flow is fully API-test-proven). Hub bearer key rotation
  still DUE (pre-existing).

- **2026-07-11 — PUBLISH-TRAIN RULES CODIFIED** (`documentation/runbooks/publish-train-rules.md`):
  1. manifest before floor; 2. the manifest screen carries the LIVE DB floor
  (`hub_settings.min_controller_version` overrides env, acts immediately) — save the floor field
  LAST; 3. MinAgent gate — **now HUB-ENFORCED PER-BOX (hub v0.45.0, `store.ResolveManagedFloor`),
  the manual fleet check retired**; 4. box-level backstop = the controller's agent-capability gate
  (v0.114.0 route probe → v0.115.0 version compare). (Floor-UI separation + per-box gating SHIPPED
  in hub v0.45.0 — see the backlog-clear entry above; this rules-doc note supersedes the original
  "hub untouched / option-3 deferred" wording.)

- **2026-07-11 — NAS/coupling backlog CLEARED (hub v0.45.0 LIVE + agent v0.82.0/ctrl v0.115.0 on demo).**
  Composite A–F. Hub v0.45.0 (Synced/Healthy): floor-UI separation + effective-floor source line
  (DB `hub_settings` vs env, `store.ResolveGlobalFloor`) + type-to-confirm w/ live below-floor count;
  per-box MinAgent conditional floor (`store.ResolveManagedFloor` HOLDS the controller floor for a box
  whose `hosts.agent_version` < the manifest's MinAgent, flags it on the Hosts dashboard) — manual
  fleet check retired (publish-train-rules.md rule 3). THE one comparator extracted to
  `internal/semver`. Agent v0.82.0 = `X-Felhom-Agent-Version` header (demo-only, Peti stays 0.81 =
  probe fallback); ctrl v0.115.0 = version-aware Supports + DSM-validated NFS guidance. **DSM spike**
  (real DSM 7.2 via virtual-dsm): consumer recipes validated E2E, hardlink caveat closed for Synology
  (record: documentation/audits/SPIKE-nas-dsm-2026-07-11.md); QNAP caveat stands. **Q1c (supervised
  reboot) FAILED**: a NAS automount trigger does NOT survive a guest reboot (empty-dir; agent lacks a
  network-mount reassert) — fix spec'd at documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md.
  NOT published (agent 0.82 demo-only). Record: felhom.eu/REPORT.md + each repo's REPORT.

- **2026-07-11 — PUBLISH TRAIN 0.81/0.113 COMPLETE — PETI LIVE on agent 0.81.0 + controller 0.113.0.**
  Record: `documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md`. Agent 0.81.0 published
  (sha `914bb3b5…30df0b`, live-bytes, anon-verified) + golden 0.113.0 baked/published (sha
  `66de5e64…82054c`, drill VM, virgin-reverted); manifest + signed `agent_update`
  (key-id `felhom-op-1`) by Viktor → Peti 0.79.0→0.81.0 clean (no rollback); **floor auto-confirm
  proof CAPTURED** (10:41 0.112.0 → 10:42 0.113.0, anonymous pull proven on HIS box) — KEY LEARNING:
  the hub UI manifest screen ALSO sets the GLOBAL floor as a DB `hub_settings.min_controller_version`
  row that OVERRIDES the env and acts immediately (this is how Peti reached 0.112 and now 0.113
  before the GitOps env bump `16390d0`); a transient forbidden skew (ctrl 0.113 + agent 0.79,
  ~10 min) was benign only because zero NAS shares existed — next train: save the floor field LAST.
  OPEN: Peti journal-group one-liner (E), temp Git Sync creds deletion (G1), **hub bearer key
  rotation now DUE** (committed in manifests/hub.yaml + screenshot-exposed this session).

- **2026-07-11 — NAS verify-before-commit SHIPPED (agent v0.81.0 + controller v0.113.0 LIVE on demo;
  host-install v1.13.0).** The spike's production task landed same-day: add = agent verify (retry=0
  unit, journal-classified, auto-rollback) → controller uid-1000 write probe (--netprobe re-exec) →
  register LAST; full rollback on any failure; §3.2 Hungarian error map (nfs_export merged);
  storage_network.html redesigned (SMB-first, Route-A guidance); orphan rows; systemd-journal group
  (NO sudoers). Live A–E validated vs an isolated sim NAS, Route A proven in production (alien-uid
  1060 server-side). Feature doc: `documentation/controller/network-storage-nas.md`. NOT published
  (no Gitea/Day-0 bump; Peti pending with the usermod one-liner). Record:
  `felhom-controller/REPORT.md`.

- **2026-07-11 — SPIKE: NAS verify mechanisms + NFS squash matrix — ALL FIVE QUESTIONS ANSWERED, no
  blocker.** Record: `documentation/audits/SPIKE-nas-verify-2026-07-11.md`. Q1 in-guest access DOES
  wake the host automount (make-or-break PASS; container bind-mount too); Q2 controller drops to uid
  1000 and writes (setpriv proof; SETUID/SETGID caps present); Q3 **Route A ACCEPTED-with-caveats**
  (`all_squash` to ANY server uid = full rw+hardlink for the guest app; guest sees nobody; chown
  EPERM-clean, chmod persists; June's `anonuid=1000` WRONG case pinned = anonuid without all_squash
  is a no-op for non-root); Q4 error taxonomy captured (NFSv4 makes no-export vs not-permitted
  INDISTINGUISHABLE; systemd 90 s start-timeout is the real ceiling; **`retry=0` → production unit**:
  dead-NAS access 91 s→3.8 s; verify must NOT ride the 15 s agentapi timeout → own path/status-poll,
  ~120 s budget); Q5 plain-user SMB PASS incl. hardlink. Teardown verified to pre-counts. Production
  TASK is the operator side's next move.

- **2026-07-11 — PBS DR tier SLICE 2 SHIPPED (agent v0.80.0): the apply-bridge — demo
  ADOPTION-PROVEN live + PUBLISHED; the supervised Peti phase PENDING.** Full chain fired on the
  demo: hub enable (storage-id `felhom-offsite`) → tenantsync provisioned ns `demo-felhom` on ep0
  → agent ADOPTED the manual entry (tenancy entry-owned, "entry wins"; staged secret asserted
  UNCONSUMED; DR green; `escrow.pbs_storage_id` seeded). Laws red-proof-verified: set-only (K
  destruction guard), secret-on-stdin, verify-pin-before-consume, loud consumed-failed. Part 0
  evidence: write path needs the §4b dual-grant exactly (403 `Datastore.AllocateSpace` → re-assert
  → token vzdump OK); demo grants restored. Live finding fixed: escrow seed = in-place write
  (/etc/felhom-agent dir is root-owned; rename impossible for the non-root agent). 0.80.0
  published (`f2ba62ca…f06d2`, == deployed bytes). **NEXT = RUNBOOK-peti-pbsdr-2026-07-11.md**
  (Peti box prep: sudoers+wrapper+wg_tunnel enable — no tunnel/OOB on his box today; Viktor opsign
  0.80.0; hub enable → FRESH path, K born, degradation clears; ceremony → auto-confirm → first
  offsite run = the arc closes). Detail: felhom-agent/REPORT.md + CHANGELOG v0.80.0.

- **2026-07-10 — PBS DR tier SLICE 1 SHIPPED+LIVE (hub v0.44.0): ep0 tenantsync surface + hub
  provisioning.** `felhom-tenantsync` v1.0.0 installed on ep0 (second forced-command key, peersync
  untouched — hashes asserted; smoke caught + fixed the client `{"data":[…]}` json wrap; orphan
  `root@pam!spike` cleaned). Hub: `internal/tenantsync` (pinned-SSH twin of wgsync; errors never
  embed stdout), config-form "PBS DR tier (ep0)" section → WG-peer check (fail-closed) → provision →
  `host_pbs_secrets` consume-once (agent fetches `POST /api/v1/hosts/{id}/pbs/consume-token` —
  note: /hosts/, not the task's /host/) → `pbs_dr` descriptor into HOST desired_json + gen bump;
  idempotent re-save; Re-issue button (F4 twin). Red-proofs: consume-once / fail-closed /
  idempotency all mutation-proven. **NOT live-provisioned — slice 2 (agent bridge) + supervised
  Peti enable pending.** Runbook §10 addendum in offsite-endpoint.md. Commits ce6a566 + 32ca1f6.

- **2026-07-10 — SPIKE: PBS DR tier auto-provisioning SLICE 0 — all three mechanisms SETTLED.** Record:
  `documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md`. (1) PVE token storage-create **impossible**
  (create/modify/delete all check `/storage` root) → **sudoers vector proven** (pinned-id alias, cycle green as
  the agent user, unpinned denied; secret must ride a stdin-wrapper, never sudo argv). (2) ep0 tenancy op-set +
  one-time-secret-over-SSH-stdout **proven live on ep0** (🛑 Viktor-approved; read/write cross-tenant 403s);
  current peersync channel can't carry it → slice 1 ships a second forced-command surface (`felhom-tenantsync`).
  (3) `pvesm add pbs --encryption-key autogen` births K at `PBSEncKeyPath` + vzdump lands ciphertext (doc-06
  property) + escrow-create finds K. **⚠ `pvesm remove` DELETES K** → re-apply must be set-only. All spike
  artifacts torn down; flags: orphan `root@pam!spike` on ep0 scratch; demo's §4b step-3 storage grants absent.
  Epic slice map in the record §5.

- **2026-07-10 — new-box restore round-trip VALIDATED + old-box archive RETIRED.** Record:
  `documentation/audits/VALIDATION-newbox-restore-2026-07-10.md`. Both toggled apps restored from the
  production pool box via the real endpoint — **9/9 sha256-identical** (immich 44MB db-dump exact,
  `d78466fcbf…4526b`, agreeing with the campaign's independent round). ~10s/app. **Viktor's decision:
  RETIRE the old archive** — he deletes `u629193-sub1` in the old project's console (the PBS-storage-1 box
  itself stays for dev). C2 fixed same day as controller v0.110.0 (crash-lock self-heal, live-proven).
  Offsite arc: build-complete + restore-proven both directions; remaining = the Peti onboarding runbook.

- **2026-07-10 — offsite arc overnight soak (unattended) — PASS + 1 HIGH finding.** Record:
  `documentation/audits/CAMPAIGN-offsite-overnight-2026-07-10.md`. F4 re-issue proved LIVE (reset invalidates
  the old password); restore round-trip 9/9 byte-identical new-box↔USB (immich db-dump exact → old-box
  archive `u629193*` safe to retire, Viktor's decision); single-flight + crash-restart (zero consume, escrow
  untouched) PASS. **FINDING C2 (HIGH, OPEN):** a controller crash MID-PRUNE leaves a stale restic exclusive
  lock the controller can't self-heal → offbox runs fail until manual `restic unlock` (fix candidate:
  `ensureOffboxRepo` auto-unlocks a self-owned stale lock). Track B (live quota-crossing) + hub-orchestrated
  Track A blocked — the hub operator config surface is session/CSRF-gated (unattended can't reach it) → Peti
  runbook. Demo restored to start-state. One expected `backup_failed` email (the crash test).

- **2026-07-10 — SLICE 4 soft-quota + offsite health SHIPPED+LIVE (hub v0.41.0 + controller v0.109.1) —
  the offsite arc is BUILD-COMPLETE.** Controller: `quota_gb` rides the descriptor into the target (hash
  includes it → quota changes re-apply via key-auth-first, no consume); `repo_size_bytes` from restic
  stats; pre-run gate (≥100% refuses NEW backups Hungarian-loudly, **prune still runs** — red-proofed;
  restore never gated; ≥80% warns); /backups usage bar. Hub: `monitor.OffsiteChecker` (fill 90/95 of
  quota + `offsite_stale` >48h for enabled+escrowed only — red-proofed; nil-safe on old reports) +
  **manual-only Freeze/Unfreeze lever** (readonly on the exactly-1 labelled sub-account, SSH preserved).
  **Live:** key-auth-first proven against the real box (the deploy's own re-apply); that re-apply exposed
  + fixed v0.109.1 (re-apply must preserve EscrowState + runtime status — it demoted the escrowed demo);
  demo repaired (manual confirm + run: escrowed, 45 MB / 50 GB, checker healthy 0 events, usage bar
  renders). Freeze + 100%-refusal NOT live-exercised (STOP; red-proofs cover). NEXT: Peti onboarding
  runbook, NEW-box restore round-trip → old-box retirement.

- **2026-07-09 — SLICE 3 hub-verified escrow auto-confirm SHIPPED+LIVE (hub v0.40.0 + controller v0.108.0 +
  agent v0.79.0).** Replaces the operator-trust confirm with a verified fact: ceremony hashes the sealed
  repo password (agent, `restic_pw_sha256,omitempty` on the escrow upload — contract test moved in lockstep)
  → hub stores it (additive `host_escrow` migration, NULL on legacy rows) + serves
  `escrow:{identity_blob_present,restic_pw_sha256,created_at}` in the report ACK → controller
  (`report.EscrowAutoConfirmer`) flips pending→escrowed ONLY on hash match with its local repo password.
  **Red-proofed core: blob-presence alone never confirms** (stale blob = false custody claim); mismatch
  warns once per hash naming the ceremony; never un-confirms; flip wipes the agent-staged secret. Hashers
  pinned by the same cross-repo vector. Manual confirm = deprecated fallback (legacy blobs — demo included,
  hash-NULL verified live; the escrowed demo produced zero confirm activity = Scenario E live). Full happy
  chain proves out at the next real enable (Peti). NEXT: SLICE 4 soft-quota, Peti runbook.

- **2026-07-09 — offsite hardening bundle SHIPPED+LIVE (hub v0.39.0 + controller v0.107.0 + agent v0.78.0).**
  The e2e's sharp edges closed before slices 3–4: **F4** "Re-issue offsite credentials" (explicit operator
  recovery for consumed-password dead-ends — exactly-1 label scope, fresh one-time secret, ConfigVersion
  bump; `hetznerapi.ResetBoxPassword` added for dedicated); **F2** host-key scan retry (~60s ladder — first
  save survives fresh-subaccount DNS lag); **F5** submit-disable + in-flight notice; **key-auth-first bridge**
  (working installed key → re-pin+reconfigure WITHOUT consuming; fingerprint verify never bypassed; kills the
  stale-descriptor consume-404 loop); **staged-secret wipe** (agent `DELETE /escrow/stage-secret`; controller
  calls it on every flip to `escrowed`; the e2e's lingering staged file wiped live through the new endpoint,
  idempotency proven). All red-proofed. F4 reset NOT exercised live (demo credential untouchable — throwaway
  customer under supervision when needed). NEXT: SLICE 3 auto-confirm, SLICE 4 soft-quota, Peti runbook.

- **2026-07-09 — offsite provisioning VALIDATED LIVE end-to-end (hub v0.38.1 + controller v0.106.1) — PASS.**
  Full loop through the real pipeline on `demo-felhom` against the new dedicated-project pool box (611714 /
  `u629488`, project 15282031): UI save → sub-account 269008 provisioned → descriptor v6 + one-time password →
  controller config-refresh (≤15 min) → apply-bridge (verify-pin → consume-once → pinned `ssh-copy-id` →
  `EscrowState=pending`) → run-gate refusal proven → escrow confirm (Option A: the repo password was already
  escrowed — `WriteOffboxSecrets` preserves it) → first run: **restic repo + 2 snapshots on the new box,
  key-only**. Two live bugs found+fixed mid-run: **F1** hub v0.38.1 (`applyOffsite` on `context.WithoutCancel` —
  a re-click canceled the provision between create and password-store, stranding the sub-account) and **F3**
  controller v0.106.1 (`ssh-copy-id -s` needs `~/.ssh` to exist in the container — died AFTER consuming the
  password). OPEN: F2 (fresh sub-account DNS lags → first-save scan 502s; retry-with-backoff), F4 (hub has no
  password-reset path — recovery today is delete-subaccount + re-save), F5 (no save spinner → re-click bait).
  Phase-0 wiring: `manifests/hub.yaml` now injects `HETZNER_TOKEN`/`HETZNER_POOL_BOX_ID` from
  Secret/storagebox; scoped-token check = list boxes with it (must see ONLY 611714). Old-box offsite kept as
  frozen archive until a NEW-box restore round-trip. Record:
  `documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`. NEXT: SLICE 3 (auto-confirm),
  SLICE 4 (soft-quota), F2/F4/F5, Peti onboarding runbook.

- **2026-07-09 — offsite provisioning SLICE 2 (hub v0.38.0 + controller v0.106.0).** The controller apply-bridge:
  on startup it reconciles the hub-served `offsite:` descriptor into a key-only offbox target
  (`controller/internal/offsiteapply.Bridge`) — **verify-pin the box host key against the hub-captured
  `host_fingerprint` (no blind TOFU)** → consume the one-time password (single-use) → `sshpass ssh-copy-id -s -f`
  install → configure offbox → `EscrowState="pending"` → persist a descriptor-hash marker. Idempotent +
  fail-safe; both red-proofs green. Hub v0.38.0 adds `Descriptor.HostFingerprint` captured via an
  `x/crypto/ssh` keyscan (fail-closed). **NOT yet live-applied** — supervised end-to-end (hub provisions on
  the new pool box → controller apply) is the next runbook, gated on the hub's new scoped `HETZNER_TOKEN`.
  NEXT: SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota).

- **2026-07-09 — hub offsite provisioning SLICE 1 (hub v0.37.0).** The hub can now provision the offsite tier
  on operator enable: `internal/hetznerapi` (typed client, base **api.hetzner.com/v1**, `CloudAPI` interface +
  exported `Fake`, `WaitAction`), `internal/offsite` (`Provisioner.ProvisionOffsite` — idempotent by label
  `felhom-customer=<id>`, shared sub-account / dedicated box, generates the transient password, builds the
  NON-SECRET descriptor, **fail-closed**), `one_time_secrets` store (single-use `Save/ConsumeOneTimeSecret`),
  `POST /offsite/consume-password/{id}` (customer-API-key auth, once), config-form Offsite section →
  `applyOffsite` (502 + no-save on error) → descriptor in ConfigJSON → ConfigVersion bump. Token from
  `HETZNER_TOKEN` (out-of-band); password NEVER in ConfigJSON/logs. Unit-tested vs a faked Cloud API (+ a
  fail-closed red-proof). **NOT yet live-provisioned** — awaiting the **dedicated-project scoped token** (the
  current token can delete ep0). NEXT: SLICE 2 (controller apply-bridge consuming the descriptor + one-time
  password), SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota). Deployed via GitOps.

- **2026-07-09 — Hetzner API provisioning spike (offsite tier).** Both offsite models are API-provisionable
  (FIT). **#1 correction: the Storage Box API is `https://api.hetzner.com/v1`, NOT `api.hetzner.cloud/v1`**
  (Cloud API 404s for storage boxes); same Bearer token. Proven live on `spike-*` (torn down): sub-account
  create/reset/readonly/delete + a billable bx11 box create/rescale(`change_type`)/delete. Create takes a
  **password (no ssh-key param)** → transient-password→`ssh-copy-id -s -f`→restic-reach→`reset_subaccount_password`
  (transient pw invalidated, key survives) = key-only steady state. RepoPath `/home/<repo>` (chroot). Quota:
  dedicated=box type/`change_type` (hard); shared=no native quota → Felhom soft-quota + `update_access_settings{readonly}`
  freeze. Idempotency = **label_selector** (box names not unique). Rate 3600/h. **SECURITY: the token is
  project-wide unscopeable Read+Write — it can also touch ep0 (the prod offsite server) + 3 SSH keys →
  move customer boxes to a DEDICATED Hetzner project before the hub holds a write token.** Record:
  `documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md`. NEXT: the hub-provisioning impl.

- **2026-07-09 — fork-4 custody VALIDATED end-to-end live (supervised).** The full custody cycle proven on
  the demo (agent v0.77.0 + controller v0.105.0): enable → controller pushes the repo password to the agent →
  **real escrow ceremony** (`--selftest=escrow-create`, K + restic-pw under a fresh R, uploaded to the hub,
  staged secret wiped, K byte-unchanged) → **atomicity proof:** a run **skipped** before `confirm-escrow` and
  **proceeded** after (2 apps, 2 snapshots) — the gate opens exactly on confirm. **DR money shot:** the restic
  password recovered from the **production hub blob** (`host_escrow.identity_blob` for `demo-felhom-01`, via
  `identity-consume` with R) is byte-identical to the live password (`sha 24c44f49…`) and **decrypted the real
  box repo**; the real `inject-password`→`restore` endpoints reproduced a 45 MB db-dump byte-perfect. Demo
  offbox left **escrowed** (un-paused); Viktor holds R out-of-band; no secret in any log; live `repo_password`
  unchanged. Record: `documentation/audits/VALIDATION-fork4-custody-2026-07-09.md`. **NEXT:** hub-provisioning +
  soft-quota spec (hub-verified auto-confirm replacing operator-trust; both storage models); per-customer
  onboarding runbook; the `identity-consume` OK-message fix (omits the recovered restic field).

- **2026-07-09 — offbox v0.104.0 VALIDATED live against the Storage Box (supervised).** Closed the offsite
  arc: controller `:0.104.0`@`908e4b9` ran the real `/backup/offbox/*` endpoints against a fresh box
  **sub-account** (`u629193-sub1`, RepoPath `/home/felhom-demo-repo`). The capstone proof — **immich**
  (undeployed, recovery unit only on the USB drive, the exact case v0.103.0 mis-resolved to `systemDataPath`)
  → discovery found the USB unit → `ok`, 1 snapshot; 45 MB db-dump restore sha256-matched. No-silent-success
  confirmed on the live box: 0/N → `error` + `backup_failed` operator alert; partial → `ok` +
  Hungarian `LastWarning` on `/backups`. Persists across restart. **Forks:** (1) direct topology + (2)
  sub-account-per-customer (RepoPath `/home/<name>`, `ssh-copy-id -p 23 -s -f` — console SSH-key field still
  broken, F3) confirmed; (3) PBS/WG tier deferred; **(4) custody surface RECORDED, escrow design OPEN** —
  the per-customer restic repo password + SSH key are 0600 in the controller data dir, **NOT independently
  recoverable** (ride DR only via the PBS whole-CT) → the load-bearing follow-up before real reliance.
  **KEPT** as a standing demo-offsite (€4.06/mo). Record: `documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md`.

- **2026-07-09 — TASK GL-7 SHIPPED: customer-page passphrase hardening + install-command generator
  (hub v0.36.0, `844fbfa7`; deploy `02c748eb`; LIVE, ArgoCD Synced/Healthy)** — go-live G7/G9/G10/G11.
  Two coupled, security-first changes to the operator customer page (`customer_unified.html` +
  `configs.go`). **(1) Passphrase hardening:** the per-customer retrieval passphrase was cleartext
  twice (visible `#retrieval-pw` + baked into the Option-3 debug curl `X-Retrieval-Password:` header);
  now masked-by-default with reveal (`toggleSecret`) + copy (`copySecret`), value in `data-secret`
  (existing model), and the Option-3 command carries a `<YOUR-RETRIEVAL-PASSWORD>` placeholder — no
  secret in any copyable command. (Zero-secret-in-DOM fetch = noted future follow-up, NOT scoped.)
  **(2) Install-command generator:** the three hard-coded `<code>` blocks became a client-side builder
  (vanilla JS, no framework/CDN/network) emitting ONLY real host-install v1.12.0 flags, download-then-
  run, byo-requires-caps enforced client-side, CustomerID prefilled from the server, JS-off static
  fallback; the seven dangerous/operator-only flags (`--force`/`--rotate-recovery`/`--enable-oob`/
  `--remove-golden`/`--uninstall`/`--adopt-pool`/`--rescope-acl`) are never offered as controls. Tests
  `TestTemplates_PassphraseHardened` + `TestTemplates_InstallGenerator` (green; passphrase test
  red-proofs). **Tester agreement written** (`documentation/pilot/PETI-tester-agreement.md`): BYO
  trust boundary, honest limitations (no offsite backup, F2 removable-drive, F7 pool-reassert, F6
  `:53`-free), exit rights, Peti box facts (80c/128 GB → caps 12/32768, `sajatfelhom.hu` re-point),
  byo install command, onboarding sequence. GO-LIVE-PACKAGE: GL-7 ✔; G9/G10/G11 → done. **One open
  pilot question (owned by the agreement §6):** Peti's local backup TARGET — does his box have a
  second disk/pool? → sets `--acl-storages` at onboarding. Deploy gotcha: first build was stale
  (local commits not pushed before `build.sh` pulls origin) — pushed then rebuilt. **NEXT: Peti day-0
  onboarding (operator-run).**
- **2026-07-08 — TASK GL-8 SHIPPED: BYO coexistence hardening (host-install v1.12.0, `18a556a5`)** —
  the GL-6 findings F3/F6/F1. **F3:** the populated-host leaf guard fires only when a `felhom_guests`
  guest exists (not `pct list`) → a fresh byo install on a host running only the OWNER's guests needs
  no `--allow-new-leaf`; the pin protection for real Felhom guests is unchanged. **F6:** byo preflight
  REFUSES a foreign process on `:53` with remediation and mutates NOTHING (Viktor's ruling: refuse+
  instruct only — the dnsmasq analog of the break-glass no-touch rule; appliance untouched). **F1:**
  uninstall purges the config `.bak*` siblings (a GL-6 residue held a live hub api_key). F6
  live-proven on felhom-pve (byo preflight refuses at :53 — the exact GL-6 scenario, read-only die).
  Harness 36/36 (+GL8-F3/F6/F1 static, GL8-F1b behavioural, GL8-F6 live) + red-proofs RP-F3/F6/F1;
  GL-2/GL-4 regression green; shellcheck clean. Shipped BEFORE GL-7 so Peti's command needs no
  workaround. **F2/F7 remain open (non-blocking). NEXT: GL-7.**
- **2026-07-08 — TASK GL-5 SHIPPED: DR bring-up bind overrides (agent v0.75.0, LIVE + scratch-DR
  validated)** — go-live G8; docs-only here (GO-LIVE-PACKAGE G8/G6/G1 flips + decision log; the
  code + full report live in felhom-agent `b3446213`). Guest-loss DR restore WORKS now: the two
  structural binds (mp8/mp9) are overridden with throwaways at restore and swapped for the REAL
  binds pre-start (root pct via the new Engine HostRunner seam), displaced unusedN deleted. TWO
  live-discovered PVE rules beyond the spike: explicit-params restore needs an explicit rootfs AND
  silently DROPS unlisted mountpoints — the full param set now derives from the archive's embedded
  config (`ExtractArchiveConfig`, 200 under the scoped token). Live proof: scratch 9310 from a real
  9201 archive — mp0 200G + mp1 50G restored WITH content (7m23s), real binds, zero residue, clean
  teardown; the same op failed outright on v0.74.0. OPERATOR: **publish agent 0.75.0 + bump the
  manifest to it** (vouching 0.74.0 ships broken DR). Follow-ups: restore-test boot-verifies
  without mp0/mp1 (same PVE rule, reverse direction — candidate reuse of drRestoreOverrides);
  DR-selftest KeepMAC duplicates a live source's MAC briefly; no host-loss DR runbook exists yet
  (GL-6/S5 writes it).
- **2026-07-08 — TASK GL-4 SHIPPED: uninstaller gap-closure + operator-key day-0 fold
  (host-install v1.11.0)** — go-live G6 + the G1 key-pin follow-up; **awaiting GL-6** for the real
  (non-dry) teardown + armed-pin install. Uninstall now removes the self-update artifacts (4b4:
  guarded wrapper, .prev/.new.* slots, rollback unit, limits drop-in — derived from
  configs/felhom-selfupdate-guarded), unmounts enrolled drives under /mnt/felhom-drives (plain
  umount ONLY, busy = warn+guidance, root-bind guarded), and ends with a KEPT-vs-WIPED statement
  (drives/PBS/hub record/escrow/vaulted recovery credential live on; guest-only mode prints the
  vmid's bind-store drives). Key-pin: OPERATOR_KEY_* constants (EMPTY until the operator ceremony)
  + `--operator-pubkey-file` (validated at argv, comment=key_id required) → authz.signers written
  at step 6 per the agent SignerKey schema; **no-keys-resolved KEEPS preserved signers** (never
  un-pin a manually-pinned box); verify reports armed/dormant (dormant = WARN). Harness 28/28 on
  felhom-pve incl. the NEW GL4 H-U full-uninstall DRY transcript vs live 9201; red-proofs RP-1..3
  green. **GO-LIVE-PACKAGE.md was ABSENT AGAIN** (spec said the operator attaches it; not present
  in the repo or workspace) — G6/G1 status recorded here, doc still pending its operator commit.
  Key CEREMONY (real keypairs + felhom-pve pin) = operator step.
- **2026-07-07 — RUNBOOK GL-1 EXECUTED: agent 0.74.0 + golden 0.103.0 PUBLISHED** — go-live G1
  (partial): the published-artifact chain is current. Agent = the LIVE felhom-pve bytes (sha
  `1ec3f588…76af05`, provenance preserved, publish-agent.sh round-trip verified); golden baked
  clean-room in the drill VM (build-golden.sh v2.0.0 @ ced60ddf, controller 0.103.0 baked, all
  integrity gates green, sha `8481e8a1…6026e`, 583MB) and published HTTP 201. Both artifacts
  fetch-verified **ANONYMOUSLY** (Gate 3c finding: generic packages are world-readable → G3
  git.token read-only rotation CANNOT break fetches; but the INSTALLER dies on an empty token —
  script-side requirement, not Gitea's). Drill VM restored to `virgin`; evidence
  `~/drill/bake-0.103.0.log` on 180 + `documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`.
  **OPERATOR NEXT: bump the hub Day-0 manifest** to agent 0.74.0 / golden 0.103.0 (the four values
  are in the execution record) — until then fresh installs land 0.63.0/0.98.3. Also recorded:
  operator-key pin follow-up (fold into GL-4); GO-LIVE-PACKAGE.md still not in the repo (G1 status
  recorded here instead).
- **2026-07-07 — TASK GL-2 SHIPPED: `--mode appliance|byo` install profile (host-install v1.10.0 +
  harness)** — go-live items G2/G4/G5 implemented, **awaiting the supervised GL-6 drill** for live
  provision/uninstall + C7 verify-drift. DECISIONS: the flag is MANDATORY (no default; the old
  `--mode provision` value retired with a specific die, `dr` stub kept); break-glass gated at its
  CALL SITE (byo never touches root@pam); byo requires `--cores`+`--memory`, refuses
  `--enable-oob`/`--rotate-recovery`/non-9.x-PVE/missing `--acl-storages`; disclosure + typed-hostname
  ack; **the config write's `lan_resolver` default flips to OFF in byo** (the spec assumed
  default-off — it was default-ON; appliance unchanged) + post-write asserts refuse
  lan_resolver/wg_tunnel/oob=true; pool+ACL verify asserts in BOTH modes (R2 lesson);
  `--preflight-only` (no state, PASS/FAIL verdict, proven live on felhom-pve). NEW
  `scripts/hostinstall-mode-harness.sh` (static C1–C4 + invariants everywhere; PVE tier: C5 +
  A/B dry transcripts) — 16/16 PASS on felhom-pve incl. C5 live; red-proofs RP-1..3 run→fail→revert.
  FINDING: demo-felhom's controller.yaml has EMPTY git.username/git.token → any step-5
  (re)install for that customer dies until the operator sets real Gitea read creds (relates to the
  standing scope-down+rotate follow-up). `documentation/pilot/GO-LIVE-PACKAGE.md` is NOT in the
  repo and no operator copy was provided — the G2/G4/G5 status flip is recorded here instead;
  add the doc when the operator supplies it. Next: **GL-6 supervised drill** (appliance + byo
  end-to-end, uninstall/re-provision, C7, the ack prompt interactively).
- **2026-07-05 — TASK H1 SHIPPED: OOB operator access (hub v0.35.0 + installer + endpoint; agent
  v0.72.0)** — merged E1+H1. Hub: operator OOB peer (`store/wg_operator.go`, `PUT/GET
  /admin/wg/operator-peer` global key) + `oob_peer_ip`/`oob_operator_ssh_key` in the desired-state
  wireguard block + `monitor/host_oob` (oob_degraded/recovered). Installer: `--enable-oob`/`install_oob`
  (static felhom-sshd + belt). Doc 06 §4.5/§4.6 amended (forwarding per-pair allow-listed, box↔box
  drop explicit, oob health stanza); endpoint runbook §9. Live-validated: operator→box SSH as
  felhom-op with scoped sudo; OF-1 /32 self-heal survival; coexistence (:22 PID unchanged); belt
  drops LAN + box↔box; PBS unaffected. Operator key = hub report_api_key.
- **2026-07-05 — TASK G1 SHIPPED: management-plane break-glass (hub v0.34.1 + installer; agent
  v0.71.0)** — prereq for felhom-sshd (H1). Hub: `store.host_recovery` vault (per-host root@pam
  console password; `PUT /hosts/{id}/recovery-credential` self-scoped + `GET /admin/hosts/{id}/
  recovery-credential` global-only, secret never logged) + `host_mgmtplane` monitor raising
  `mgmt_plane_healed`. Installer: `step_break_glass` (generate+set+vault root@pam, `--rotate-recovery`)
  + `install_mgmt_watchdog` (the 3 host artifacts, RuntimeDirectory-guarded). Live-validated:
  agent-independent `/run/sshd` auto-heal in 30.0s; warning fired end-to-end; break-glass
  vault→retrieve→PVE-ticket(200). felhom-pve root@pam is now the vaulted value.
- **2026-07-04 — S3 SHIPPED: agent WG tunnel (felhom-agent v0.64.0; docs-only here)** — the
  doc-06 §3.3 handshake is now END-TO-END LIVE on demo: agent keygen → `POST /hosts/{id}/wg` →
  desired-state `wireguard` block → agent-managed `wg-quick@wg-felhom` → PBS page over
  10.77.0.1:8007 in 3 s. **`wg_tunnel.enabled` defaults FALSE agent-side** (rollouts to other
  boxes are no-ops until the production endpoint exists). Revocation semantics completed (§3.5):
  removed peer → agent disables + keeps its marker + NEVER re-registers; operator re-adds via the
  pubkey the heartbeat report now carries. Escrow: `IdentityBundle.WGPrivateKey` auto-injected on
  new escrow-create runs (pre-S3 blobs can't be retrofitted — honest fallback = fresh-key
  re-register, /32 kept). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix, deferred
  non-blocking); S4 = PBS over the tunnel; hub-side S6 = tunnel-health alerting on the new report
  stanza. Detail: felhom-agent CHANGELOG v0.64.0 + REPORT.
- **2026-07-04 — S2 SHIPPED: box-facing WG registration + wireguard desired-state block + /offsite UI (hub v0.33.0)** —
  doc-06 row S2. DECISIONS: the WG assignment is **hub-owned state, merged into served
  desired-state at READ time** — stored `desired_json` stays a pure operator blob; admin PUT with
  a top-level `wireguard` key → 400 (no copy-paste clobber); **re-key-in-place keeps the /32**
  (stable tunnel addressing across rotation/DR); one bound peer per host (partial unique index);
  generation bumps ONLY on real change (register/re-key/adopt/bound-delete — idempotent paths
  move nothing). `POST /hosts/{id}/wg` self-scoped per-host key; NEW golden
  `desired-state-wireguard.golden.json` = the S3 cross-repo contract (byte-identical agent copy
  rule). peersync **v1.0.1** (strip out of process substitution — the S1 exit-swallow finding;
  sandbox red-proofed) redeployed to the dev endpoint. Live-proven on the REAL demo-felhom-01
  record via the real per-host key: gen 2→3, served block = golden shape, endpoint `wg show`
  carried the peer, v0.63.0 agent tolerated the unknown key (fetch clean, guests untouched),
  then removed → gen 4, all baselines restored. `/offsite` page unit-proven (live view is
  operator-password-gated — eyeball after next login). Next: **S3** (agent keygen +
  registration + `wg-felhom` service + escrow join + CGNAT smoke — risky/supervised).
- **2026-07-04 — S1 SHIPPED: offsite endpoint live + hub peer-sync (hub v0.32.1)** — the doc-06
  roadmap's first slice, done-criterion met live. Peer-sync design point RESOLVED: **hub pushes
  over SSH** to a forced-command `felhom-peersync` script (pull rejected: weakens immediate
  revocation; HTTPS API rejected: new binary + third public port). Endpoint = the dev
  `felhom-hetzner` box, provisioned by the NEW runbook `documentation/runbooks/offsite-endpoint.md`
  (WG on **443/udp**, subnet 10.77.0.0/24, PBS datastore `felhom-offsite` at `/srv/pbs-felhom`,
  nftables drop-all + wg0-only 8007, `ep0.felhom.eu` grey-cloud). Hub: `wg_endpoints`/`wg_peers` +
  /32 allocator + `/admin/wg/*` (global-key, pubkey-in-body) + `internal/wgsync` (FixedHostKey pin
  + **pinned-type HostKeyAlgorithms — live-caught bug**: multi-hostkey sshd presented ECDSA vs the
  ed25519 pin; fixed v0.32.1 + regression test) + full-list declarative reconciler (5-min drift
  repair). Live: add/remove→wg show, malformed→state-untouched, reboot→persist+converge, client
  tunnel via ep0:443→PBS login page, public 8007 closed. INCIDENT handled: first WG server key
  leaked via `wg show dump` into the session log → rotated immediately (warning now in the
  runbook). OPERATOR follow-up: **fix the AAAA record** (currently `…7aa1::`, must be `…7aa1::1`).
  Next: **S2** (hub schema host-join + `wireguard` block on WireDesiredState + box-facing
  registration + UI).
- **2026-07-03 — OFFSITE-CONNECTIVITY DESIGN-OF-RECORD WRITTEN (architecture Part 06)** —
  `documentation/architecture/06-offsite-connectivity.md`: records the six settled decisions
  (plain WG / host-side agent-managed `wg-felhom` as the agent-managed-unit PILOT / one shared
  hub-driven endpoint VM running WG+PBS, NO agent on it / hub = source of truth, `wireguard`
  block rides `WireDesiredState` on the existing DesiredGeneration channel / one datastore +
  per-customer namespaces / PBS ON the VM, relay-through-DooPlex REJECTED as non-scaling),
  the Day-0 join handshake, robustness set (WG-roaming-not-DynDNS, DNS re-resolve watchdog, MTU
  1420, per-/32 isolation, tunnel-health via storage-target model), trust conformance (outbound
  only; operator-inbound + game-ingress out of scope), the honest ledger (CGNAT still unmeasured
  — mobile-hotspot smoke closure; peer-sync push-vs-pull = slice-1 design point), and the S1–S6
  slice roadmap (**MVP = S1 endpoint + S2 hub schema/desired-state + S3 agent WG service, then S4
  PBS-over-tunnel**; S5 merges with DR-completeness). day0-install backlog line now cites spike +
  design doc. Docs-only; slices are separate future tasks.
- **2026-07-03 — OFFSITE-BACKUP TRANSPORT DECIDED (connectivity spike, GO at the lowest rung)** —
  plain WireGuard, host-side, box-dials-out to a small public endpoint; **no Headscale/DERP needed
  for backup** (separable fleet layer, adoptable later). Live-proven on demo-felhom → throwaway
  Hetzner: 11.4-min idle hold, PBS API through the tunnel with the exact-cert pin intact
  (positive + negative), real 2 GiB worst-case backup at 4.26 MiB/s (= the home uplink; tunnel
  overhead ~5%), UDP 51820 *and* 443 both pass. P0 honesty: the operator's line is plain-NAT
  (fixed public IP, One Hungary), NOT CGNAT, and has zero IPv6 — CGNAT confirmation deferred to
  Peti's VM 110 (second ISP). P7 surprise: kernel WG *works* inside the unprivileged guest (no TUN
  needed) — host placement stands on architecture (backups originate host-side; cloudflared
  precedent), not infeasibility. Evidence + architecture recommendation (endpoint VM shapes, key
  custody via IdentityBundle escrow, per-peer /32 isolation):
  `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`. **This unblocks the
  DR-completeness work: next task = production connectivity spec** (endpoint VM, agent-managed WG
  service, hub peer registry, escrow join). Throwaway Hetzner box parked powered-off (secrets
  shredded); demo-felhom verified residue-free.
- **2026-07-03 — 4th skill: `felhom-app-catalog`** (catalog authoring workflow; points at
  app-catalog REUSE.md §1–2, no duplicated tables; auto-discovered by `install_skills.py`, 4/4
  found fresh-session). Proven by finalizing SparkyFitness live on demo (image-inspected
  healthchecks, probe-container naming rule discovered + REUSE row added, UI-applied, controller
  probe healthy). App-count drift recorded: website says "45+", per-category spans sum 49, real
  catalog 53 — website fix is a separate task.
- **2026-07-03 — GOLDEN 0.98.3 LIVE + VOUCHED; guide D.1b RETIRED (drill B1+B5 FIXED)** —
  `build-golden.sh` v2.0.0 (mandatory controller tag + baked bootstrap `.path` unit); golden baked
  with controller 0.98.3, clean-room validated (drill VM) through all four scenarios incl. the
  published-artifact `--force-gitea-golden` path, published + operator-vouched (Day-0 manifest now:
  agent 0.63.0 + golden 0.98.3). Fresh installs land CURRENT and self-manage — D.1b is a one-line
  check; the prior "vouch agent 0.63.0" operator follow-up is DONE. Evidence:
  `documentation/audits/DRILL-golden-098-2026-07-03.md`. NEW operator follow-ups: (1) SECURITY —
  the customer-config `git.token` is a Gitea admin token with package-WRITE (proven during the
  bake); scope down + rotate. (2) optionally raise the global controller floor to 0.98.3.
- **2026-07-03 — drill findings B2+B3 FIXED (agent v0.63.0, live on felhom-pve + published)** —
  token reload-on-miss (no more fresh-install 401/manual restart) + guesthook snippets-dir mkdir.
  Guide D.1b restart step narrowed to "agent < v0.63.0"; drill ledger updated. **OPERATOR: vouch
  agent 0.63.0 in the Day-0 manifest UI** (until then fresh installs fetch 0.62.0).
- **2026-07-03 — GO-LIVE PACKAGE SHIPPED (host-install v1.9.1 + day0-install runbook + clean-room drill)** —
  `documentation/runbooks/day0-install.md` is the operator Day-0 guide, **every command executed
  verbatim** on a throwaway nested PVE 9.2.2 (build server): install → verify → app deploy →
  uninstall → **zero-residue diff** → reinstall. v1.9.1 = residue-free uninstall (agent config w/
  hub api_key, shared-parent unit/script/mount, mkfs wrapper, hook snippet, dnsmasq snippets) +
  post-provision guest reboot (without it the controller never deploys on slower hardware) + header
  sync. Evidence + go/no-go for Peti (**GO**, 3 install-day items):
  `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md`. Key OQ-2 finding: golden bakes a
  pre-floor controller 0.85.1 → **fresh installs need the one-time D.1b update** (agent restart →
  update button → 0.98.3); follow-ups recorded: rebuild golden ≥0.86.0 + re-vouch (operator), agent
  `mkdir -p` for the guesthook snippet dir, fresh-install local-API 401 (pre-existing), hub has no
  host-delete path. Drill VM parked virgin-snapshotted on 180 (`~/drill/`).
- **2026-07-03 — A1 CLOSED (host-install v1.9.0 + agent v0.62.0)** — `Pool.Audit` added to
  `FelhomAgentGuest`; the agent's stale-lock reaper now intersects its scan with
  `GET /pools/felhom` membership (fail-safe skip on read failure). Spike:
  `documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. Deploy order on any box:
  rescope ACL first, then the agent. STILL PENDING (operator): rotate + de-git the 3 committed
  secrets in `manifests/felhom.secret.yaml` (spike appendix; REUSE §2 errata added).
- **2026-07-03 — hub v0.31.0 LIVE** (critical-severity ingest fix + UI badges; ArgoCD Synced/Healthy;
  live probe: critical event stored + operator email delivered).
- **2026-07-03 — Felhom skills exist**: `skills/{felhom-build-deploy,felhom-ui-design,felhom-testing}`,
  installed via `scripts/install_skills.py` — **junction mode** (repo edits live immediately; no
  re-install needed after editing). Fresh-session discovery verified.
- **2026-07-03 — CLAUDE.md refreshed** (all four repos): no version-pinned state in CLAUDE.md —
  current state lives here / CHANGELOG / REUSE.md.

- **2026-07-03 — `REUSE.md` exists at the repo root** (hub helpers / website conventions / gate scripts / manifest patterns, code-verified); maintenance rule active: update it in the same commit that changes a shared helper. `scripts/reuse_refs_check.py` machine-checks its citations across all four felhom repos.