felhom.eu/CONTEXT.md

# CONTEXT — felhom.eu working state

> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.

- **2026-07-08 — TASK GL-5 SHIPPED: DR bring-up bind overrides (agent v0.75.0, LIVE + scratch-DR
  validated)** — go-live G8; docs-only here (GO-LIVE-PACKAGE G8/G6/G1 flips + decision log; the
  code + full report live in felhom-agent `b3446213`). Guest-loss DR restore WORKS now: the two
  structural binds (mp8/mp9) are overridden with throwaways at restore and swapped for the REAL
  binds pre-start (root pct via the new Engine HostRunner seam), displaced unusedN deleted. TWO
  live-discovered PVE rules beyond the spike: explicit-params restore needs an explicit rootfs AND
  silently DROPS unlisted mountpoints — the full param set now derives from the archive's embedded
  config (`ExtractArchiveConfig`, 200 under the scoped token). Live proof: scratch 9310 from a real
  9201 archive — mp0 200G + mp1 50G restored WITH content (7m23s), real binds, zero residue, clean
  teardown; the same op failed outright on v0.74.0. OPERATOR: **publish agent 0.75.0 + bump the
  manifest to it** (vouching 0.74.0 ships broken DR). Follow-ups: restore-test boot-verifies
  without mp0/mp1 (same PVE rule, reverse direction — candidate reuse of drRestoreOverrides);
  DR-selftest KeepMAC duplicates a live source's MAC briefly; no host-loss DR runbook exists yet
  (GL-6/S5 writes it).
- **2026-07-08 — TASK GL-4 SHIPPED: uninstaller gap-closure + operator-key day-0 fold
  (host-install v1.11.0)** — go-live G6 + the G1 key-pin follow-up; **awaiting GL-6** for the real
  (non-dry) teardown + armed-pin install. Uninstall now removes the self-update artifacts (4b4:
  guarded wrapper, .prev/.new.* slots, rollback unit, limits drop-in — derived from
  configs/felhom-selfupdate-guarded), unmounts enrolled drives under /mnt/felhom-drives (plain
  umount ONLY, busy = warn+guidance, root-bind guarded), and ends with a KEPT-vs-WIPED statement
  (drives/PBS/hub record/escrow/vaulted recovery credential live on; guest-only mode prints the
  vmid's bind-store drives). Key-pin: OPERATOR_KEY_* constants (EMPTY until the operator ceremony)
  + `--operator-pubkey-file` (validated at argv, comment=key_id required) → authz.signers written
  at step 6 per the agent SignerKey schema; **no-keys-resolved KEEPS preserved signers** (never
  un-pin a manually-pinned box); verify reports armed/dormant (dormant = WARN). Harness 28/28 on
  felhom-pve incl. the NEW GL4 H-U full-uninstall DRY transcript vs live 9201; red-proofs RP-1..3
  green. **GO-LIVE-PACKAGE.md was ABSENT AGAIN** (spec said the operator attaches it; not present
  in the repo or workspace) — G6/G1 status recorded here, doc still pending its operator commit.
  Key CEREMONY (real keypairs + felhom-pve pin) = operator step.
- **2026-07-07 — RUNBOOK GL-1 EXECUTED: agent 0.74.0 + golden 0.103.0 PUBLISHED** — go-live G1
  (partial): the published-artifact chain is current. Agent = the LIVE felhom-pve bytes (sha
  `1ec3f588…76af05`, provenance preserved, publish-agent.sh round-trip verified); golden baked
  clean-room in the drill VM (build-golden.sh v2.0.0 @ ced60ddf, controller 0.103.0 baked, all
  integrity gates green, sha `8481e8a1…6026e`, 583MB) and published HTTP 201. Both artifacts
  fetch-verified **ANONYMOUSLY** (Gate 3c finding: generic packages are world-readable → G3
  git.token read-only rotation CANNOT break fetches; but the INSTALLER dies on an empty token —
  script-side requirement, not Gitea's). Drill VM restored to `virgin`; evidence
  `~/drill/bake-0.103.0.log` on 180 + `documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md`.
  **OPERATOR NEXT: bump the hub Day-0 manifest** to agent 0.74.0 / golden 0.103.0 (the four values
  are in the execution record) — until then fresh installs land 0.63.0/0.98.3. Also recorded:
  operator-key pin follow-up (fold into GL-4); GO-LIVE-PACKAGE.md still not in the repo (G1 status
  recorded here instead).
- **2026-07-07 — TASK GL-2 SHIPPED: `--mode appliance|byo` install profile (host-install v1.10.0 +
  harness)** — go-live items G2/G4/G5 implemented, **awaiting the supervised GL-6 drill** for live
  provision/uninstall + C7 verify-drift. DECISIONS: the flag is MANDATORY (no default; the old
  `--mode provision` value retired with a specific die, `dr` stub kept); break-glass gated at its
  CALL SITE (byo never touches root@pam); byo requires `--cores`+`--memory`, refuses
  `--enable-oob`/`--rotate-recovery`/non-9.x-PVE/missing `--acl-storages`; disclosure + typed-hostname
  ack; **the config write's `lan_resolver` default flips to OFF in byo** (the spec assumed
  default-off — it was default-ON; appliance unchanged) + post-write asserts refuse
  lan_resolver/wg_tunnel/oob=true; pool+ACL verify asserts in BOTH modes (R2 lesson);
  `--preflight-only` (no state, PASS/FAIL verdict, proven live on felhom-pve). NEW
  `scripts/hostinstall-mode-harness.sh` (static C1–C4 + invariants everywhere; PVE tier: C5 +
  A/B dry transcripts) — 16/16 PASS on felhom-pve incl. C5 live; red-proofs RP-1..3 run→fail→revert.
  FINDING: demo-felhom's controller.yaml has EMPTY git.username/git.token → any step-5
  (re)install for that customer dies until the operator sets real Gitea read creds (relates to the
  standing scope-down+rotate follow-up). `documentation/pilot/GO-LIVE-PACKAGE.md` is NOT in the
  repo and no operator copy was provided — the G2/G4/G5 status flip is recorded here instead;
  add the doc when the operator supplies it. Next: **GL-6 supervised drill** (appliance + byo
  end-to-end, uninstall/re-provision, C7, the ack prompt interactively).
- **2026-07-05 — TASK H1 SHIPPED: OOB operator access (hub v0.35.0 + installer + endpoint; agent
  v0.72.0)** — merged E1+H1. Hub: operator OOB peer (`store/wg_operator.go`, `PUT/GET
  /admin/wg/operator-peer` global key) + `oob_peer_ip`/`oob_operator_ssh_key` in the desired-state
  wireguard block + `monitor/host_oob` (oob_degraded/recovered). Installer: `--enable-oob`/`install_oob`
  (static felhom-sshd + belt). Doc 06 §4.5/§4.6 amended (forwarding per-pair allow-listed, box↔box
  drop explicit, oob health stanza); endpoint runbook §9. Live-validated: operator→box SSH as
  felhom-op with scoped sudo; OF-1 /32 self-heal survival; coexistence (:22 PID unchanged); belt
  drops LAN + box↔box; PBS unaffected. Operator key = hub report_api_key.
- **2026-07-05 — TASK G1 SHIPPED: management-plane break-glass (hub v0.34.1 + installer; agent
  v0.71.0)** — prereq for felhom-sshd (H1). Hub: `store.host_recovery` vault (per-host root@pam
  console password; `PUT /hosts/{id}/recovery-credential` self-scoped + `GET /admin/hosts/{id}/
  recovery-credential` global-only, secret never logged) + `host_mgmtplane` monitor raising
  `mgmt_plane_healed`. Installer: `step_break_glass` (generate+set+vault root@pam, `--rotate-recovery`)
  + `install_mgmt_watchdog` (the 3 host artifacts, RuntimeDirectory-guarded). Live-validated:
  agent-independent `/run/sshd` auto-heal in 30.0s; warning fired end-to-end; break-glass
  vault→retrieve→PVE-ticket(200). felhom-pve root@pam is now the vaulted value.
- **2026-07-04 — S3 SHIPPED: agent WG tunnel (felhom-agent v0.64.0; docs-only here)** — the
  doc-06 §3.3 handshake is now END-TO-END LIVE on demo: agent keygen → `POST /hosts/{id}/wg` →
  desired-state `wireguard` block → agent-managed `wg-quick@wg-felhom` → PBS page over
  10.77.0.1:8007 in 3 s. **`wg_tunnel.enabled` defaults FALSE agent-side** (rollouts to other
  boxes are no-ops until the production endpoint exists). Revocation semantics completed (§3.5):
  removed peer → agent disables + keeps its marker + NEVER re-registers; operator re-adds via the
  pubkey the heartbeat report now carries. Escrow: `IdentityBundle.WGPrivateKey` auto-injected on
  new escrow-create runs (pre-S3 blobs can't be retrofitted — honest fallback = fresh-key
  re-register, /32 kept). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix, deferred
  non-blocking); S4 = PBS over the tunnel; hub-side S6 = tunnel-health alerting on the new report
  stanza. Detail: felhom-agent CHANGELOG v0.64.0 + REPORT.
- **2026-07-04 — S2 SHIPPED: box-facing WG registration + wireguard desired-state block + /offsite UI (hub v0.33.0)** —
  doc-06 row S2. DECISIONS: the WG assignment is **hub-owned state, merged into served
  desired-state at READ time** — stored `desired_json` stays a pure operator blob; admin PUT with
  a top-level `wireguard` key → 400 (no copy-paste clobber); **re-key-in-place keeps the /32**
  (stable tunnel addressing across rotation/DR); one bound peer per host (partial unique index);
  generation bumps ONLY on real change (register/re-key/adopt/bound-delete — idempotent paths
  move nothing). `POST /hosts/{id}/wg` self-scoped per-host key; NEW golden
  `desired-state-wireguard.golden.json` = the S3 cross-repo contract (byte-identical agent copy
  rule). peersync **v1.0.1** (strip out of process substitution — the S1 exit-swallow finding;
  sandbox red-proofed) redeployed to the dev endpoint. Live-proven on the REAL demo-felhom-01
  record via the real per-host key: gen 2→3, served block = golden shape, endpoint `wg show`
  carried the peer, v0.63.0 agent tolerated the unknown key (fetch clean, guests untouched),
  then removed → gen 4, all baselines restored. `/offsite` page unit-proven (live view is
  operator-password-gated — eyeball after next login). Next: **S3** (agent keygen +
  registration + `wg-felhom` service + escrow join + CGNAT smoke — risky/supervised).
- **2026-07-04 — S1 SHIPPED: offsite endpoint live + hub peer-sync (hub v0.32.1)** — the doc-06
  roadmap's first slice, done-criterion met live. Peer-sync design point RESOLVED: **hub pushes
  over SSH** to a forced-command `felhom-peersync` script (pull rejected: weakens immediate
  revocation; HTTPS API rejected: new binary + third public port). Endpoint = the dev
  `felhom-hetzner` box, provisioned by the NEW runbook `documentation/runbooks/offsite-endpoint.md`
  (WG on **443/udp**, subnet 10.77.0.0/24, PBS datastore `felhom-offsite` at `/srv/pbs-felhom`,
  nftables drop-all + wg0-only 8007, `ep0.felhom.eu` grey-cloud). Hub: `wg_endpoints`/`wg_peers` +
  /32 allocator + `/admin/wg/*` (global-key, pubkey-in-body) + `internal/wgsync` (FixedHostKey pin
  + **pinned-type HostKeyAlgorithms — live-caught bug**: multi-hostkey sshd presented ECDSA vs the
  ed25519 pin; fixed v0.32.1 + regression test) + full-list declarative reconciler (5-min drift
  repair). Live: add/remove→wg show, malformed→state-untouched, reboot→persist+converge, client
  tunnel via ep0:443→PBS login page, public 8007 closed. INCIDENT handled: first WG server key
  leaked via `wg show dump` into the session log → rotated immediately (warning now in the
  runbook). OPERATOR follow-up: **fix the AAAA record** (currently `…7aa1::`, must be `…7aa1::1`).
  Next: **S2** (hub schema host-join + `wireguard` block on WireDesiredState + box-facing
  registration + UI).
- **2026-07-03 — OFFSITE-CONNECTIVITY DESIGN-OF-RECORD WRITTEN (architecture Part 06)** —
  `documentation/architecture/06-offsite-connectivity.md`: records the six settled decisions
  (plain WG / host-side agent-managed `wg-felhom` as the agent-managed-unit PILOT / one shared
  hub-driven endpoint VM running WG+PBS, NO agent on it / hub = source of truth, `wireguard`
  block rides `WireDesiredState` on the existing DesiredGeneration channel / one datastore +
  per-customer namespaces / PBS ON the VM, relay-through-DooPlex REJECTED as non-scaling),
  the Day-0 join handshake, robustness set (WG-roaming-not-DynDNS, DNS re-resolve watchdog, MTU
  1420, per-/32 isolation, tunnel-health via storage-target model), trust conformance (outbound
  only; operator-inbound + game-ingress out of scope), the honest ledger (CGNAT still unmeasured
  — mobile-hotspot smoke closure; peer-sync push-vs-pull = slice-1 design point), and the S1–S6
  slice roadmap (**MVP = S1 endpoint + S2 hub schema/desired-state + S3 agent WG service, then S4
  PBS-over-tunnel**; S5 merges with DR-completeness). day0-install backlog line now cites spike +
  design doc. Docs-only; slices are separate future tasks.
- **2026-07-03 — OFFSITE-BACKUP TRANSPORT DECIDED (connectivity spike, GO at the lowest rung)** —
  plain WireGuard, host-side, box-dials-out to a small public endpoint; **no Headscale/DERP needed
  for backup** (separable fleet layer, adoptable later). Live-proven on demo-felhom → throwaway
  Hetzner: 11.4-min idle hold, PBS API through the tunnel with the exact-cert pin intact
  (positive + negative), real 2 GiB worst-case backup at 4.26 MiB/s (= the home uplink; tunnel
  overhead ~5%), UDP 51820 *and* 443 both pass. P0 honesty: the operator's line is plain-NAT
  (fixed public IP, One Hungary), NOT CGNAT, and has zero IPv6 — CGNAT confirmation deferred to
  Peti's VM 110 (second ISP). P7 surprise: kernel WG *works* inside the unprivileged guest (no TUN
  needed) — host placement stands on architecture (backups originate host-side; cloudflared
  precedent), not infeasibility. Evidence + architecture recommendation (endpoint VM shapes, key
  custody via IdentityBundle escrow, per-peer /32 isolation):
  `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`. **This unblocks the
  DR-completeness work: next task = production connectivity spec** (endpoint VM, agent-managed WG
  service, hub peer registry, escrow join). Throwaway Hetzner box parked powered-off (secrets
  shredded); demo-felhom verified residue-free.
- **2026-07-03 — 4th skill: `felhom-app-catalog`** (catalog authoring workflow; points at
  app-catalog REUSE.md §1–2, no duplicated tables; auto-discovered by `install_skills.py`, 4/4
  found fresh-session). Proven by finalizing SparkyFitness live on demo (image-inspected
  healthchecks, probe-container naming rule discovered + REUSE row added, UI-applied, controller
  probe healthy). App-count drift recorded: website says "45+", per-category spans sum 49, real
  catalog 53 — website fix is a separate task.
- **2026-07-03 — GOLDEN 0.98.3 LIVE + VOUCHED; guide D.1b RETIRED (drill B1+B5 FIXED)** —
  `build-golden.sh` v2.0.0 (mandatory controller tag + baked bootstrap `.path` unit); golden baked
  with controller 0.98.3, clean-room validated (drill VM) through all four scenarios incl. the
  published-artifact `--force-gitea-golden` path, published + operator-vouched (Day-0 manifest now:
  agent 0.63.0 + golden 0.98.3). Fresh installs land CURRENT and self-manage — D.1b is a one-line
  check; the prior "vouch agent 0.63.0" operator follow-up is DONE. Evidence:
  `documentation/audits/DRILL-golden-098-2026-07-03.md`. NEW operator follow-ups: (1) SECURITY —
  the customer-config `git.token` is a Gitea admin token with package-WRITE (proven during the
  bake); scope down + rotate. (2) optionally raise the global controller floor to 0.98.3.
- **2026-07-03 — drill findings B2+B3 FIXED (agent v0.63.0, live on felhom-pve + published)** —
  token reload-on-miss (no more fresh-install 401/manual restart) + guesthook snippets-dir mkdir.
  Guide D.1b restart step narrowed to "agent < v0.63.0"; drill ledger updated. **OPERATOR: vouch
  agent 0.63.0 in the Day-0 manifest UI** (until then fresh installs fetch 0.62.0).
- **2026-07-03 — GO-LIVE PACKAGE SHIPPED (host-install v1.9.1 + day0-install runbook + clean-room drill)** —
  `documentation/runbooks/day0-install.md` is the operator Day-0 guide, **every command executed
  verbatim** on a throwaway nested PVE 9.2.2 (build server): install → verify → app deploy →
  uninstall → **zero-residue diff** → reinstall. v1.9.1 = residue-free uninstall (agent config w/
  hub api_key, shared-parent unit/script/mount, mkfs wrapper, hook snippet, dnsmasq snippets) +
  post-provision guest reboot (without it the controller never deploys on slower hardware) + header
  sync. Evidence + go/no-go for Peti (**GO**, 3 install-day items):
  `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md`. Key OQ-2 finding: golden bakes a
  pre-floor controller 0.85.1 → **fresh installs need the one-time D.1b update** (agent restart →
  update button → 0.98.3); follow-ups recorded: rebuild golden ≥0.86.0 + re-vouch (operator), agent
  `mkdir -p` for the guesthook snippet dir, fresh-install local-API 401 (pre-existing), hub has no
  host-delete path. Drill VM parked virgin-snapshotted on 180 (`~/drill/`).
- **2026-07-03 — A1 CLOSED (host-install v1.9.0 + agent v0.62.0)** — `Pool.Audit` added to
  `FelhomAgentGuest`; the agent's stale-lock reaper now intersects its scan with
  `GET /pools/felhom` membership (fail-safe skip on read failure). Spike:
  `documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. Deploy order on any box:
  rescope ACL first, then the agent. STILL PENDING (operator): rotate + de-git the 3 committed
  secrets in `manifests/felhom.secret.yaml` (spike appendix; REUSE §2 errata added).
- **2026-07-03 — hub v0.31.0 LIVE** (critical-severity ingest fix + UI badges; ArgoCD Synced/Healthy;
  live probe: critical event stored + operator email delivered).
- **2026-07-03 — Felhom skills exist**: `skills/{felhom-build-deploy,felhom-ui-design,felhom-testing}`,
  installed via `scripts/install_skills.py` — **junction mode** (repo edits live immediately; no
  re-install needed after editing). Fresh-session discovery verified.
- **2026-07-03 — CLAUDE.md refreshed** (all four repos): no version-pinned state in CLAUDE.md —
  current state lives here / CHANGELOG / REUSE.md.

- **2026-07-03 — `REUSE.md` exists at the repo root** (hub helpers / website conventions / gate scripts / manifest patterns, code-verified); maintenance rule active: update it in the same commit that changes a shared helper. `scripts/reuse_refs_check.py` machine-checks its citations across all four felhom repos.