felhom.eu/CONTEXT.md

CONTEXT — felhom.eu working state

Created with the REUSE.md rollout (2026-07-03). Authoritative history: hub/CHANGELOG.md (hub), website/CHANGELOG.md, scripts/CHANGELOG.md; end-of-task detail in REPORT.md.

• 2026-07-09 — fork-4 custody VALIDATED end-to-end live (supervised). The full custody cycle proven on the demo (agent v0.77.0 + controller v0.105.0): enable → controller pushes the repo password to the agent → real escrow ceremony (--selftest=escrow-create, K + restic-pw under a fresh R, uploaded to the hub, staged secret wiped, K byte-unchanged) → atomicity proof: a run skipped before confirm-escrow and proceeded after (2 apps, 2 snapshots) — the gate opens exactly on confirm. DR money shot: the restic password recovered from the production hub blob (host_escrow.identity_blob for demo-felhom-01, via identity-consume with R) is byte-identical to the live password (sha 24c44f49…) and decrypted the real box repo; the real inject-password→restore endpoints reproduced a 45 MB db-dump byte-perfect. Demo offbox left escrowed (un-paused); Viktor holds R out-of-band; no secret in any log; live repo_password unchanged. Record: documentation/audits/VALIDATION-fork4-custody-2026-07-09.md. NEXT: hub-provisioning + soft-quota spec (hub-verified auto-confirm replacing operator-trust; both storage models); per-customer onboarding runbook; the identity-consume OK-message fix (omits the recovered restic field).

• 2026-07-09 — offbox v0.104.0 VALIDATED live against the Storage Box (supervised). Closed the offsite arc: controller :0.104.0@908e4b9 ran the real /backup/offbox/* endpoints against a fresh box sub-account (u629193-sub1, RepoPath /home/felhom-demo-repo). The capstone proof — immich (undeployed, recovery unit only on the USB drive, the exact case v0.103.0 mis-resolved to systemDataPath) → discovery found the USB unit → ok, 1 snapshot; 45 MB db-dump restore sha256-matched. No-silent-success confirmed on the live box: 0/N → error + backup_failed operator alert; partial → ok + Hungarian LastWarning on /backups. Persists across restart. Forks: (1) direct topology + (2) sub-account-per-customer (RepoPath /home/<name>, ssh-copy-id -p 23 -s -f — console SSH-key field still broken, F3) confirmed; (3) PBS/WG tier deferred; (4) custody surface RECORDED, escrow design OPEN — the per-customer restic repo password + SSH key are 0600 in the controller data dir, NOT independently recoverable (ride DR only via the PBS whole-CT) → the load-bearing follow-up before real reliance. KEPT as a standing demo-offsite (€4.06/mo). Record: documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md.

• 2026-07-09 — TASK GL-7 SHIPPED: customer-page passphrase hardening + install-command generator (hub v0.36.0, 844fbfa7; deploy 02c748eb; LIVE, ArgoCD Synced/Healthy) — go-live G7/G9/G10/G11. Two coupled, security-first changes to the operator customer page (customer_unified.html + configs.go). (1) Passphrase hardening: the per-customer retrieval passphrase was cleartext twice (visible #retrieval-pw + baked into the Option-3 debug curl X-Retrieval-Password: header); now masked-by-default with reveal (toggleSecret) + copy (copySecret), value in data-secret (existing model), and the Option-3 command carries a <YOUR-RETRIEVAL-PASSWORD> placeholder — no secret in any copyable command. (Zero-secret-in-DOM fetch = noted future follow-up, NOT scoped.) (2) Install-command generator: the three hard-coded <code> blocks became a client-side builder (vanilla JS, no framework/CDN/network) emitting ONLY real host-install v1.12.0 flags, download-then- run, byo-requires-caps enforced client-side, CustomerID prefilled from the server, JS-off static fallback; the seven dangerous/operator-only flags (--force/--rotate-recovery/--enable-oob/ --remove-golden/--uninstall/--adopt-pool/--rescope-acl) are never offered as controls. Tests TestTemplates_PassphraseHardened + TestTemplates_InstallGenerator (green; passphrase test red-proofs). Tester agreement written (documentation/pilot/PETI-tester-agreement.md): BYO trust boundary, honest limitations (no offsite backup, F2 removable-drive, F7 pool-reassert, F6 :53-free), exit rights, Peti box facts (80c/128 GB → caps 12/32768, sajatfelhom.hu re-point), byo install command, onboarding sequence. GO-LIVE-PACKAGE: GL-7 ✔; G9/G10/G11 → done. One open pilot question (owned by the agreement §6): Peti's local backup TARGET — does his box have a second disk/pool? → sets --acl-storages at onboarding. Deploy gotcha: first build was stale (local commits not pushed before build.sh pulls origin) — pushed then rebuilt. NEXT: Peti day-0 onboarding (operator-run).

• 2026-07-08 — TASK GL-8 SHIPPED: BYO coexistence hardening (host-install v1.12.0, 18a556a5) — the GL-6 findings F3/F6/F1. F3: the populated-host leaf guard fires only when a felhom_guests guest exists (not pct list) → a fresh byo install on a host running only the OWNER's guests needs no --allow-new-leaf; the pin protection for real Felhom guests is unchanged. F6: byo preflight REFUSES a foreign process on :53 with remediation and mutates NOTHING (Viktor's ruling: refuse+ instruct only — the dnsmasq analog of the break-glass no-touch rule; appliance untouched). F1: uninstall purges the config .bak* siblings (a GL-6 residue held a live hub api_key). F6 live-proven on felhom-pve (byo preflight refuses at :53 — the exact GL-6 scenario, read-only die). Harness 36/36 (+GL8-F3/F6/F1 static, GL8-F1b behavioural, GL8-F6 live) + red-proofs RP-F3/F6/F1; GL-2/GL-4 regression green; shellcheck clean. Shipped BEFORE GL-7 so Peti's command needs no workaround. F2/F7 remain open (non-blocking). NEXT: GL-7.

• 2026-07-08 — TASK GL-5 SHIPPED: DR bring-up bind overrides (agent v0.75.0, LIVE + scratch-DR validated) — go-live G8; docs-only here (GO-LIVE-PACKAGE G8/G6/G1 flips + decision log; the code + full report live in felhom-agent b3446213). Guest-loss DR restore WORKS now: the two structural binds (mp8/mp9) are overridden with throwaways at restore and swapped for the REAL binds pre-start (root pct via the new Engine HostRunner seam), displaced unusedN deleted. TWO live-discovered PVE rules beyond the spike: explicit-params restore needs an explicit rootfs AND silently DROPS unlisted mountpoints — the full param set now derives from the archive's embedded config (ExtractArchiveConfig, 200 under the scoped token). Live proof: scratch 9310 from a real 9201 archive — mp0 200G + mp1 50G restored WITH content (7m23s), real binds, zero residue, clean teardown; the same op failed outright on v0.74.0. OPERATOR: publish agent 0.75.0 + bump the manifest to it (vouching 0.74.0 ships broken DR). Follow-ups: restore-test boot-verifies without mp0/mp1 (same PVE rule, reverse direction — candidate reuse of drRestoreOverrides); DR-selftest KeepMAC duplicates a live source's MAC briefly; no host-loss DR runbook exists yet (GL-6/S5 writes it).

• 2026-07-08 — TASK GL-4 SHIPPED: uninstaller gap-closure + operator-key day-0 fold (host-install v1.11.0) — go-live G6 + the G1 key-pin follow-up; awaiting GL-6 for the real (non-dry) teardown + armed-pin install. Uninstall now removes the self-update artifacts (4b4: guarded wrapper, .prev/.new.* slots, rollback unit, limits drop-in — derived from configs/felhom-selfupdate-guarded), unmounts enrolled drives under /mnt/felhom-drives (plain umount ONLY, busy = warn+guidance, root-bind guarded), and ends with a KEPT-vs-WIPED statement (drives/PBS/hub record/escrow/vaulted recovery credential live on; guest-only mode prints the vmid's bind-store drives). Key-pin: OPERATOR_KEY_* constants (EMPTY until the operator ceremony)

  • --operator-pubkey-file (validated at argv, comment=key_id required) → authz.signers written at step 6 per the agent SignerKey schema; no-keys-resolved KEEPS preserved signers (never un-pin a manually-pinned box); verify reports armed/dormant (dormant = WARN). Harness 28/28 on felhom-pve incl. the NEW GL4 H-U full-uninstall DRY transcript vs live 9201; red-proofs RP-1..3 green. GO-LIVE-PACKAGE.md was ABSENT AGAIN (spec said the operator attaches it; not present in the repo or workspace) — G6/G1 status recorded here, doc still pending its operator commit. Key CEREMONY (real keypairs + felhom-pve pin) = operator step.

• 2026-07-07 — RUNBOOK GL-1 EXECUTED: agent 0.74.0 + golden 0.103.0 PUBLISHED — go-live G1 (partial): the published-artifact chain is current. Agent = the LIVE felhom-pve bytes (sha 1ec3f588…76af05, provenance preserved, publish-agent.sh round-trip verified); golden baked clean-room in the drill VM (build-golden.sh v2.0.0 @ ced60ddf, controller 0.103.0 baked, all integrity gates green, sha 8481e8a1…6026e, 583MB) and published HTTP 201. Both artifacts fetch-verified ANONYMOUSLY (Gate 3c finding: generic packages are world-readable → G3 git.token read-only rotation CANNOT break fetches; but the INSTALLER dies on an empty token — script-side requirement, not Gitea's). Drill VM restored to virgin; evidence ~/drill/bake-0.103.0.log on 180 + documentation/pilot/RUNBOOK-GL1-publish-2026-07-07.md. OPERATOR NEXT: bump the hub Day-0 manifest to agent 0.74.0 / golden 0.103.0 (the four values are in the execution record) — until then fresh installs land 0.63.0/0.98.3. Also recorded: operator-key pin follow-up (fold into GL-4); GO-LIVE-PACKAGE.md still not in the repo (G1 status recorded here instead).

• 2026-07-07 — TASK GL-2 SHIPPED: --mode appliance|byo install profile (host-install v1.10.0 + harness) — go-live items G2/G4/G5 implemented, awaiting the supervised GL-6 drill for live provision/uninstall + C7 verify-drift. DECISIONS: the flag is MANDATORY (no default; the old --mode provision value retired with a specific die, dr stub kept); break-glass gated at its CALL SITE (byo never touches root@pam); byo requires --cores+--memory, refuses --enable-oob/--rotate-recovery/non-9.x-PVE/missing --acl-storages; disclosure + typed-hostname ack; the config write's lan_resolver default flips to OFF in byo (the spec assumed default-off — it was default-ON; appliance unchanged) + post-write asserts refuse lan_resolver/wg_tunnel/oob=true; pool+ACL verify asserts in BOTH modes (R2 lesson); --preflight-only (no state, PASS/FAIL verdict, proven live on felhom-pve). NEW scripts/hostinstall-mode-harness.sh (static C1–C4 + invariants everywhere; PVE tier: C5 + A/B dry transcripts) — 16/16 PASS on felhom-pve incl. C5 live; red-proofs RP-1..3 run→fail→revert. FINDING: demo-felhom's controller.yaml has EMPTY git.username/git.token → any step-5 (re)install for that customer dies until the operator sets real Gitea read creds (relates to the standing scope-down+rotate follow-up). documentation/pilot/GO-LIVE-PACKAGE.md is NOT in the repo and no operator copy was provided — the G2/G4/G5 status flip is recorded here instead; add the doc when the operator supplies it. Next: GL-6 supervised drill (appliance + byo end-to-end, uninstall/re-provision, C7, the ack prompt interactively).

• 2026-07-05 — TASK H1 SHIPPED: OOB operator access (hub v0.35.0 + installer + endpoint; agent v0.72.0) — merged E1+H1. Hub: operator OOB peer (store/wg_operator.go, PUT/GET /admin/wg/operator-peer global key) + oob_peer_ip/oob_operator_ssh_key in the desired-state wireguard block + monitor/host_oob (oob_degraded/recovered). Installer: --enable-oob/install_oob (static felhom-sshd + belt). Doc 06 §4.5/§4.6 amended (forwarding per-pair allow-listed, box↔box drop explicit, oob health stanza); endpoint runbook §9. Live-validated: operator→box SSH as felhom-op with scoped sudo; OF-1 /32 self-heal survival; coexistence (:22 PID unchanged); belt drops LAN + box↔box; PBS unaffected. Operator key = hub report_api_key.

• 2026-07-05 — TASK G1 SHIPPED: management-plane break-glass (hub v0.34.1 + installer; agent v0.71.0) — prereq for felhom-sshd (H1). Hub: store.host_recovery vault (per-host root@pam console password; PUT /hosts/{id}/recovery-credential self-scoped + GET /admin/hosts/{id}/ recovery-credential global-only, secret never logged) + host_mgmtplane monitor raising mgmt_plane_healed. Installer: step_break_glass (generate+set+vault root@pam, --rotate-recovery)

  • install_mgmt_watchdog (the 3 host artifacts, RuntimeDirectory-guarded). Live-validated: agent-independent /run/sshd auto-heal in 30.0s; warning fired end-to-end; break-glass vault→retrieve→PVE-ticket(200). felhom-pve root@pam is now the vaulted value.

• 2026-07-04 — S3 SHIPPED: agent WG tunnel (felhom-agent v0.64.0; docs-only here) — the doc-06 §3.3 handshake is now END-TO-END LIVE on demo: agent keygen → POST /hosts/{id}/wg → desired-state wireguard block → agent-managed wg-quick@wg-felhom → PBS page over 10.77.0.1:8007 in 3 s. wg_tunnel.enabled defaults FALSE agent-side (rollouts to other boxes are no-ops until the production endpoint exists). Revocation semantics completed (§3.5): removed peer → agent disables + keeps its marker + NEVER re-registers; operator re-adds via the pubkey the heartbeat report now carries. Escrow: IdentityBundle.WGPrivateKey auto-injected on new escrow-create runs (pre-S3 blobs can't be retrofitted — honest fallback = fresh-key re-register, /32 kept). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix, deferred non-blocking); S4 = PBS over the tunnel; hub-side S6 = tunnel-health alerting on the new report stanza. Detail: felhom-agent CHANGELOG v0.64.0 + REPORT.

• 2026-07-04 — S2 SHIPPED: box-facing WG registration + wireguard desired-state block + /offsite UI (hub v0.33.0) — doc-06 row S2. DECISIONS: the WG assignment is hub-owned state, merged into served desired-state at READ time — stored desired_json stays a pure operator blob; admin PUT with a top-level wireguard key → 400 (no copy-paste clobber); re-key-in-place keeps the /32 (stable tunnel addressing across rotation/DR); one bound peer per host (partial unique index); generation bumps ONLY on real change (register/re-key/adopt/bound-delete — idempotent paths move nothing). POST /hosts/{id}/wg self-scoped per-host key; NEW golden desired-state-wireguard.golden.json = the S3 cross-repo contract (byte-identical agent copy rule). peersync v1.0.1 (strip out of process substitution — the S1 exit-swallow finding; sandbox red-proofed) redeployed to the dev endpoint. Live-proven on the REAL demo-felhom-01 record via the real per-host key: gen 2→3, served block = golden shape, endpoint wg show carried the peer, v0.63.0 agent tolerated the unknown key (fetch clean, guests untouched), then removed → gen 4, all baselines restored. /offsite page unit-proven (live view is operator-password-gated — eyeball after next login). Next: S3 (agent keygen + registration + wg-felhom service + escrow join + CGNAT smoke — risky/supervised).

• 2026-07-04 — S1 SHIPPED: offsite endpoint live + hub peer-sync (hub v0.32.1) — the doc-06 roadmap's first slice, done-criterion met live. Peer-sync design point RESOLVED: hub pushes over SSH to a forced-command felhom-peersync script (pull rejected: weakens immediate revocation; HTTPS API rejected: new binary + third public port). Endpoint = the dev felhom-hetzner box, provisioned by the NEW runbook documentation/runbooks/offsite-endpoint.md (WG on 443/udp, subnet 10.77.0.0/24, PBS datastore felhom-offsite at /srv/pbs-felhom, nftables drop-all + wg0-only 8007, ep0.felhom.eu grey-cloud). Hub: wg_endpoints/wg_peers + /32 allocator + /admin/wg/* (global-key, pubkey-in-body) + internal/wgsync (FixedHostKey pin

  • pinned-type HostKeyAlgorithms — live-caught bug: multi-hostkey sshd presented ECDSA vs the ed25519 pin; fixed v0.32.1 + regression test) + full-list declarative reconciler (5-min drift repair). Live: add/remove→wg show, malformed→state-untouched, reboot→persist+converge, client tunnel via ep0:443→PBS login page, public 8007 closed. INCIDENT handled: first WG server key leaked via wg show dump into the session log → rotated immediately (warning now in the runbook). OPERATOR follow-up: fix the AAAA record (currently …7aa1::, must be …7aa1::1). Next: S2 (hub schema host-join + wireguard block on WireDesiredState + box-facing registration + UI).

• 2026-07-03 — OFFSITE-CONNECTIVITY DESIGN-OF-RECORD WRITTEN (architecture Part 06) — documentation/architecture/06-offsite-connectivity.md: records the six settled decisions (plain WG / host-side agent-managed wg-felhom as the agent-managed-unit PILOT / one shared hub-driven endpoint VM running WG+PBS, NO agent on it / hub = source of truth, wireguard block rides WireDesiredState on the existing DesiredGeneration channel / one datastore + per-customer namespaces / PBS ON the VM, relay-through-DooPlex REJECTED as non-scaling), the Day-0 join handshake, robustness set (WG-roaming-not-DynDNS, DNS re-resolve watchdog, MTU 1420, per-/32 isolation, tunnel-health via storage-target model), trust conformance (outbound only; operator-inbound + game-ingress out of scope), the honest ledger (CGNAT still unmeasured — mobile-hotspot smoke closure; peer-sync push-vs-pull = slice-1 design point), and the S1–S6 slice roadmap (MVP = S1 endpoint + S2 hub schema/desired-state + S3 agent WG service, then S4 PBS-over-tunnel; S5 merges with DR-completeness). day0-install backlog line now cites spike + design doc. Docs-only; slices are separate future tasks.

• 2026-07-03 — OFFSITE-BACKUP TRANSPORT DECIDED (connectivity spike, GO at the lowest rung) — plain WireGuard, host-side, box-dials-out to a small public endpoint; no Headscale/DERP needed for backup (separable fleet layer, adoptable later). Live-proven on demo-felhom → throwaway Hetzner: 11.4-min idle hold, PBS API through the tunnel with the exact-cert pin intact (positive + negative), real 2 GiB worst-case backup at 4.26 MiB/s (= the home uplink; tunnel overhead ~5%), UDP 51820 and 443 both pass. P0 honesty: the operator's line is plain-NAT (fixed public IP, One Hungary), NOT CGNAT, and has zero IPv6 — CGNAT confirmation deferred to Peti's VM 110 (second ISP). P7 surprise: kernel WG works inside the unprivileged guest (no TUN needed) — host placement stands on architecture (backups originate host-side; cloudflared precedent), not infeasibility. Evidence + architecture recommendation (endpoint VM shapes, key custody via IdentityBundle escrow, per-peer /32 isolation): documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md. This unblocks the DR-completeness work: next task = production connectivity spec (endpoint VM, agent-managed WG service, hub peer registry, escrow join). Throwaway Hetzner box parked powered-off (secrets shredded); demo-felhom verified residue-free.

• 2026-07-03 — 4th skill: felhom-app-catalog (catalog authoring workflow; points at app-catalog REUSE.md §1–2, no duplicated tables; auto-discovered by install_skills.py, 4/4 found fresh-session). Proven by finalizing SparkyFitness live on demo (image-inspected healthchecks, probe-container naming rule discovered + REUSE row added, UI-applied, controller probe healthy). App-count drift recorded: website says "45+", per-category spans sum 49, real catalog 53 — website fix is a separate task.

• 2026-07-03 — GOLDEN 0.98.3 LIVE + VOUCHED; guide D.1b RETIRED (drill B1+B5 FIXED) — build-golden.sh v2.0.0 (mandatory controller tag + baked bootstrap .path unit); golden baked with controller 0.98.3, clean-room validated (drill VM) through all four scenarios incl. the published-artifact --force-gitea-golden path, published + operator-vouched (Day-0 manifest now: agent 0.63.0 + golden 0.98.3). Fresh installs land CURRENT and self-manage — D.1b is a one-line check; the prior "vouch agent 0.63.0" operator follow-up is DONE. Evidence: documentation/audits/DRILL-golden-098-2026-07-03.md. NEW operator follow-ups: (1) SECURITY — the customer-config git.token is a Gitea admin token with package-WRITE (proven during the bake); scope down + rotate. (2) optionally raise the global controller floor to 0.98.3.

• 2026-07-03 — drill findings B2+B3 FIXED (agent v0.63.0, live on felhom-pve + published) — token reload-on-miss (no more fresh-install 401/manual restart) + guesthook snippets-dir mkdir. Guide D.1b restart step narrowed to "agent < v0.63.0"; drill ledger updated. OPERATOR: vouch agent 0.63.0 in the Day-0 manifest UI (until then fresh installs fetch 0.62.0).

• 2026-07-03 — GO-LIVE PACKAGE SHIPPED (host-install v1.9.1 + day0-install runbook + clean-room drill) — documentation/runbooks/day0-install.md is the operator Day-0 guide, every command executed verbatim on a throwaway nested PVE 9.2.2 (build server): install → verify → app deploy → uninstall → zero-residue diff → reinstall. v1.9.1 = residue-free uninstall (agent config w/ hub api_key, shared-parent unit/script/mount, mkfs wrapper, hook snippet, dnsmasq snippets) + post-provision guest reboot (without it the controller never deploys on slower hardware) + header sync. Evidence + go/no-go for Peti (GO, 3 install-day items): documentation/audits/DRILL-day0-cleanroom-2026-07-03.md. Key OQ-2 finding: golden bakes a pre-floor controller 0.85.1 → fresh installs need the one-time D.1b update (agent restart → update button → 0.98.3); follow-ups recorded: rebuild golden ≥0.86.0 + re-vouch (operator), agent mkdir -p for the guesthook snippet dir, fresh-install local-API 401 (pre-existing), hub has no host-delete path. Drill VM parked virgin-snapshotted on 180 (~/drill/).

• 2026-07-03 — A1 CLOSED (host-install v1.9.0 + agent v0.62.0) — Pool.Audit added to FelhomAgentGuest; the agent's stale-lock reaper now intersects its scan with GET /pools/felhom membership (fail-safe skip on read failure). Spike: documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md. Deploy order on any box: rescope ACL first, then the agent. STILL PENDING (operator): rotate + de-git the 3 committed secrets in manifests/felhom.secret.yaml (spike appendix; REUSE §2 errata added).

• 2026-07-03 — hub v0.31.0 LIVE (critical-severity ingest fix + UI badges; ArgoCD Synced/Healthy; live probe: critical event stored + operator email delivered).

• 2026-07-03 — Felhom skills exist: skills/{felhom-build-deploy,felhom-ui-design,felhom-testing}, installed via scripts/install_skills.py — junction mode (repo edits live immediately; no re-install needed after editing). Fresh-session discovery verified.

• 2026-07-03 — CLAUDE.md refreshed (all four repos): no version-pinned state in CLAUDE.md — current state lives here / CHANGELOG / REUSE.md.

• 2026-07-03 — REUSE.md exists at the repo root (hub helpers / website conventions / gate scripts / manifest patterns, code-verified); maintenance rule active: update it in the same commit that changes a shared helper. scripts/reuse_refs_check.py machine-checks its citations across all four felhom repos.