felhom.eu/documentation/pilot/GO-LIVE-PACKAGE.md

GO-LIVE PACKAGE — first external tester (Peti)

Tracking doc for the pilot go-live: gating items, statuses, decisions, task sequence. Lives at felhom.eu/documentation/pilot/GO-LIVE-PACKAGE.md. Update per task completion. Created 2026-07-07 (project-Claude, source-verified). Operator: Viktor.

Baselines at creation (verified live): agent ced60dd/v0.74.0 · controller 59eb3bea/v0.103.0 · catalog 2ebe082a · felhom.eu 1a1e42ad/hub v0.35.0 · host-install v1.9.1 · golden 0.98.3. Updated 2026-07-07 (GL-2 shipped): felhom.eu 02d63ed0 · host-install v1.10.0 + scripts/hostinstall-mode-harness.sh. Updated 2026-07-08 (GL-4 + GL-5 shipped): felhom.eu f7cc6a72 · host-install v1.11.0 (uninstall gap-closure + key-pin plumbing) · felhom-agent v0.75.0 (DR bind overrides, live on felhom-pve). Updated 2026-07-08 (GL-5b shipped): felhom-agent v0.76.0 live + PUBLISHED (9828c5f7…f50b) — restore-test full-fidelity + mount parity (new G12); the manifest bump target is now 0.76.0.

---

1. Pilot definition & locked decisions (2026-07-07)

#	Fork	Decision
D1	Pool model	One shared felhom pool. No multi-tenant boxes yet; the pool separates Felhom guests from the host owner's own workloads. Effort goes to preflight pool-membership assertion + local-API per-guest token scoping (verified structural, see G10).
D2	Pilot topology	BYO host. Peti installs Felhom onto his OWN Proxmox server, already running his own LXCs. This reframes containment: not customer-vs-customer, but Felhom-as-a-guest on a host the operator does not own. See §2.
D3	Backup posture	Local-only for the pilot. wg_tunnel.enabled and oob.enabled stay FALSE. Offsite (Tier-1/Tier-2 split + production WG endpoint) is a fast-follow, and the limitation is stated honestly in the tester agreement.
D4	Uninstall drill	Full provision → uninstall → re-provision drill on the demo box BEFORE Peti's day-0. Peti's box is never the first live test of anything destructive.
D5	DR bind-override	Fix now (small, spike-validated known-constant override per SPIKE-dr-bindmount-source-2026-07-07.md), not deferred to the intermediary-mount re-architecture.
D6	Packaging	This one tracking doc + a sequenced set of CC tasks (GL-1…GL-7).

---

2. The trust-model reframe (BYO host) — load-bearing

Everything so far was designed for the appliance model: Felhom-owned hardware, Viktor is the root-holding operator, the customer is a UI user. Peti's pilot is the BYO-host model: Peti is root on his own PVE box; Felhom is a guest system there (one LXC + a non-root agent + narrow sudoers + a pool-scoped token). Consequences, each source-verified:

• Break-glass inverts. step_break_glass (host-install.sh:1186, step 4b/8, NOT flag-gated) sets root@pam via chpasswd and vaults it to the hub. Right for appliances; on a BYO host it silently changes the owner's root password and hands the operator root on hardware he doesn't own — a consent, trust, and data-sovereignty violation. Must be profile-gated (G2).
• Host-wide artifacts need coexistence review, not just installation: dnsmasq snippets under /etc/dnsmasq.d/felhom-*.conf + systemctl restart dnsmasq (and the installer installs the dnsmasq package) can collide with the owner's existing DNS; the mgmt watchdog + tmpfiles heal /run/sshd on his sshd (additive, likely benign — verify); felhom-sshd/OOB stays off (D3).
• Already good citizens (verified): VMID selection collision-safe (default 9201, auto-next-free over the union of pct+qm ids, explicit --vmid dies unless --force — host-install.sh:223–241); uninstall targets only guests carrying the /etc/felhom-bootstrap bind mount and refuses non-Felhom guests (:337, :472); stale-lock reaper is pool-intersected (agent v0.62.0); local-API requests are authorized strictly against the caller-token's own guest (localapi/server.go:23, :53–55, :153).
• Local-API exposure: binds the vmbr0 IP:8443 by default (host-install.sh:1546–1548) — on a BYO host that's reachable from the owner's LAN and his other guests. Defense = TLS + per-guest bearer token. Acceptable for the pilot; documented in G10.

---

3. Gating items (must be green before Peti's day-0)

Status legend: ✔ DONE · ◐ PARTIAL · ○ OPEN · ⚠ BLOCKER

ID	Item	Status	Evidence / gap
G1	Publish + vouch chain current & self-update alive. ARTIFACTS PUBLISHED (RUNBOOK GL-1, felhom.eu a63cc715): agent 0.74.0 sha 1ec3f588…76af05 (the exact live-proven felhom-pve bytes, hash independently re-verified anonymously) and golden 0.103.0 sha 8481e8a1…6026e (clean-room bake on the virgin drill VM, controller 0.103.0 baked, both split volumes in the vzdump). Key-pin day-0 PLUMBING SHIPPED in GL-4 (v1.11.0: OPERATOR_KEY_* constants empty-by-default + --operator-pubkey-file → authz.signers; preserved-signers never clobbered; verify reports armed/dormant). Remaining: (a) operator manifest bump — target agent 0.76.0 (published 2026-07-08, sha 9828c5f7…f50b, the exact live felhom-pve bytes; 0.74.0's published artifact has broken guest-loss DR, 0.75.0 was superseded UNPUBLISHED); until the bump fresh installs still land 0.63.0/0.98.3; (b) the key CEREMONY (real keypairs + felhom-pve pin) = operator.	◐	RUNBOOK-GL1-publish-2026-07-07.md; host-install v1.11.0
G2	BYO-host install profile. SHIPPED as --mode appliance|byo (mandatory, no default; the flag was repurposed from the retired provision|dr — dr stub seam preserved). byo: break-glass gated OFF at its single call site (:2040–2044 — root@pam never touched, nothing vaulted), caps mandatory, --enable-oob/--rotate-recovery refused, host-mutation disclosure + typed-hostname ack, post-write config asserts (lan_resolver/wg_tunnel/oob must be off — asserts run BEFORE _state_mark agent_config, so --resume re-asserts). Red-proofs RP-1..3 run→fail→revert. Appliance flow unchanged (regression cases green).	◐	host-install v1.10.0 @ 02d63ed0; verified at file:line 2026-07-07. Remaining: GL-6 live drill (C7 verify-assert firing, interactive ack)
G3	git.token scope-down + rotate — UNBLOCKED, and now coupled to a visibility ruling. Gate 3c (GL-1): both generic packages are anonymously world-readable, so rotating the customer token to read-only cannot break fresh-install fetches; the empty-creds die on demo-felhom is the installer's resolve_git_creds precondition (:326–332), not a Gitea requirement. NEW FINDING (project-Claude probe, 2026-07-08): the ENTIRE Gitea instance is anonymously readable — all nine repos private=False, including homelab-manifests (k3s manifests — audit for raw Secret objects) and personal repos (work, revfulop-calendar). Only the geo-fence (HU/DE/US) limits reach. Coupling warning: if the reaction is to lock the instance ("require sign-in"), the anonymous day-0 package fetch dies and read-scoped tokens become MANDATORY — retest the fetch path with a token before any visibility change.	○ ⚠	GL-1 record §Gate 3c; anonymous repos/search probe 2026-07-08
G4	Preflight guards (refuse-to-provision gate). SHIPPED in v1.10.0: byo dies on non-9.x PVE (unless --force) and on missing --acl-storages entries (named); :53 DNS info line + existing-guest count; --preflight-only runs the full mode-aware checks with a PASS/FAIL verdict and zero state marks (live-proven on felhom-pve, incl. the FAIL verdict on a bogus storage). Verify step now asserts pool membership + all scoped-ACL grants (user+token) in BOTH modes — R2's lesson made structural.	◐	host-install v1.10.0 @ 02d63ed0 (verify asserts read at source). Remaining: GL-6 (asserts firing on real drift)
G5	LXC caps mandatory on BYO hosts. SHIPPED: byo refuses to run without explicit --cores AND --memory (:947, fires before the passphrase prompt); existing cap-sanity warns retained. Appliance unchanged (optional, golden default). Cap VALUES for Peti's guest still to be chosen (§6).	◐	host-install v1.10.0 :947. Remaining: pick values + GL-6
G6	Uninstaller gap-closure (customer offboarding). SHIPPED in host-install v1.11.0 (GL-4, f7cc6a72): 4b4 self-update-artifact removal (wrapper, .prev/.new.* slots, rollback unit, limits drop-in — derived from the guarded script itself), per-drive unmounts under /mnt/felhom-drives (plain umount ONLY, busy = warn+guidance, root-bind guarded), the KEPT-vs-WIPED statement in BOTH modes (drives/PBS/hub record/escrow/vaulted recovery credential named as living on), guest-only drive note from the bind store. Harness: disclosure↔uninstall parity check (GL4-D) + full-uninstall DRY transcript vs live 9201; red-proofs RP-1..3.	◐	host-install v1.11.0 @ f7cc6a72. Remaining: GL-6 (real teardown incl. a busy drive)
G7	Demo-box drill: provision → uninstall → re-provision (runbook, supervised). Proves G2/G4/G5/G6 end-to-end before they touch Peti's hardware.	○	D4
G8	DR bring-up bind-override fix (agent). SHIPPED as v0.75.0 (GL-5, b3446213, LIVE on felhom-pve) + scratch-DR live-validated end-to-end (9310 from a real 9201 archive: mp0 200G + mp1 50G restored with content, real mp8/mp9 binds, zero unusedN, clean auto-teardown; the same op failed outright on v0.74.0). TWO live-discovered PVE rules beyond the spike: explicit-params restore requires an explicit rootfs AND silently drops unlisted mountpoints — the full param set now derives from the archive's embedded config (ExtractArchiveConfig, 200 under the scoped token; bind LAYOUT stays the known constants).	◐	agent v0.75.0 @ b3446213; REPORT.md. Remaining: the full customer-data DR drill (GL-6/S5). (The publish follow-up landed as 0.76.0 — GL-5b/G12.)
G9	Auth-on onboarding. Auth + CSRF proven in campaign 2; remaining work is procedural: Peti's onboarding sets the dashboard password via the REAL settings pipeline (anti-F9) and verifies login. Runbook step in GL-7.	◐	controller v0.103.0 (F-C2-1 live-validated)
G10	Local-API posture documented + spot-verified. Token→vmid binding is structural (server.go:23, :53–55, :153 — verified this session); write it into the security notes, including the vmbr0-reachable surface and its TLS+token defense. One CC spot-check: a token minted for guest A is refused for a guest-B-scoped op (test exists? verify; add if hollow).	◐	localapi/server.go as cited
G12	Restore-test full-fidelity verification (GL-5b, agent v0.76.0). The restore-test had GL-5 finding #2's mirror image: its live-source-config override path tripped PVE's drop-unlisted-mountpoints rule, so it boot-verified scratch guests WITHOUT their storage mpN. Now: params derive from the ARCHIVE's embedded config (drRestoreOverrides, same as DR) + a mount-parity assert (restored mpN set vs the archive's; missing/mispathed/undersized/extra = FAIL naming the delta) so the rule can never regress into a green light. MountParity+MountInventory ride the hub wire record (additive). Honest cost, measured: 3m4s on the local tier (the extraction adds ~2min over data-less; cheaper than the DR-derived ~7m estimate; PBS tier will run longer). Live-proven on felhom-pve: parity ok, inventory mp0 200G + mp1 50G + 2 throwaways; a rotated-out archive volid refuses up front.	◐	agent v0.76.0 (published 9828c5f7…f50b); agent REPORT.md. Remaining: parity-on-real-drift (GL-6 family) + PBS-tier runtime
G11	Pilot backup statement. Assert local backups green on Peti's box at onboarding; the tester agreement states plainly: no offsite copy yet — a dead/stolen box or dead drive without a second local target loses data. Honesty is the sovereignty pitch.	○	D3; agent CONTEXT.md v0.66/0.67 (retarget reverted; Tier-1/2 split pending)

Non-gating / fast-follow (tracked, not blocking): Impl-3 shared-box operator format gate (existing wipe-binding + mkfs-guarded gates cover the dangerous core); deviceRole/roleForMountPath unification refactor; Tier-1/Tier-2 offsite split + production WG endpoint; CGNAT true-sim retest; BUNDLE cert/key/token migration item; hub-floor auto-update.

---

4. Task sequence

Task	Type	Repo(s)	Covers	Depends on
GL-1 Publish/vouch/key-pin chain	◐ artifacts published 2026-07-08 (a63cc715); manifest bump + key pin = operator	felhom-agent, felhom.eu (hub manifest), operator UI	G1	—
GL-2 BYO-host profile + preflight + mandatory caps	✔ DONE 2026-07-07 (02d63ed0, validated)	felhom.eu/scripts	G2, G4, G5	—
GL-3 git.token rotate + scope-down	Operator (small)	Gitea admin + config template	G3	—
GL-4 Uninstaller gap-closure + key-pin fold	✔ DONE 2026-07-08 (host-install v1.11.0, f7cc6a72)	felhom.eu/scripts	G6 + G1-plumbing	GL-2
GL-5 DR bind-override	✔ DONE 2026-07-08 (agent v0.75.0, b3446213, live-validated)	felhom-agent	G8	—
GL-6 Demo-box drill	RUNBOOK, supervised	—	G7	GL-1, GL-2, GL-4
GL-7 Peti day-0 runbook + onboarding	RUNBOOK, supervised	felhom.eu/documentation/pilot	G9, G10, G11	GL-6 green

Recommended order: GL-1 and GL-3 immediately (operator-heavy, unblock everything), GL-2 next (the biggest CC task), GL-5 in parallel (independent repo), then GL-4 → GL-6 → GL-7.

---

5. Decision log

• 2026-07-07 — D1–D6 locked by Viktor (see §1). BYO-host reframe adopted; break-glass step identified as a BYO blocker (G2); G1 stale-vouch chain identified as item #1; G3 security O1 promoted into the gating list. Catalog healthcheck sweep confirmed CLOSED (live, 2ebe082a).

• 2026-07-07 — GL-2 SHIPPED (host-install v1.10.0, felhom.eu 02d63ed0), CC report validated against source at file:line. Two spec corrections adopted, both verified at baseline: (a) --mode already existed as the retired provision|dr — repurposed to appliance|byo with a guiding die for provision and the dr stub seam preserved; (b) the installer's config write defaulted lan_resolver.enable: true (baseline :1592) — the spec's "default-off" claim was agent-side only; byo now defaults it OFF and the post-write assert still catches a --preserve-from carrying true (appliance keeps historical default-on). The byo skip line intentionally avoids the literal "4b/8" (the spec's Scenario B forbade the string). Harness runs red-proofs on scratch copies; every invocation uses a throwaway FELHOM_INSTALL_STATE_DIR (new env override, spec-permitted).

• 2026-07-08 — GL-1 EXECUTED (felhom.eu a63cc715, felhom-agent 4c408467), validated: agent 0.74.0 published from the exact running felhom-pve bytes (sha independently re-verified via anonymous fetch, byte-exact); golden 0.103.0 baked clean-room on the virgin drill VM and published (612MB, rootfs+mp0+mp1 all in the vzdump); drill environment restored to virgin. Deviations sound: debian-13 template pveam-downloaded in-VM; bake detached via systemd-run. Gate 3c: packages anonymously readable → G3 rotation unblocked. Follow-on probe found the whole Gitea instance anonymously readable (all repos public) — operator ruling required (see G3 + operator actions).

• 2026-07-08 — GL-4 SHIPPED (host-install v1.11.0, felhom.eu f7cc6a72): G6 uninstall gap-closure (self-update artifacts, per-drive plain-umount with busy-warn, KEPT-vs-WIPED statement both modes, guest-only bind-store drive note) + the G1 key-pin day-0 PLUMBING (OPERATOR_KEY_* constants empty-by-default + --operator-pubkey-file → authz.signers; no-keys-resolved KEEPS preserved signers — a pinned box is never silently un-pinned). Harness 28/28 on felhom-pve incl. a full-uninstall DRY transcript vs live 9201; red-proofs RP-1..3. Statuses: G6 → implemented-awaiting-GL-6; G1 key-pin plumbing → done, ceremony pending.

• 2026-07-08 — GL-5 SHIPPED (agent v0.75.0, b3446213, live on felhom-pve): G8 DR bind overrides + 4d real-bind swap, scratch-DR live-validated (see G8). The spike's known-constants verdict held for the bind LAYOUT but PVE's all-or-nothing explicit-params restore forced two live corrections (explicit rootfs; full storage-mpN pass-through from the archive's embedded config — without it the DR guest boots WITHOUT its data volumes, a data-loss restore that looks green). NOTE: no host-loss DR runbook exists yet to simplify (the GL-5 spec assumed one) — the DR procedure doc is GL-6/S5 material. Follow-ups recorded: publish agent 0.75.0 (the published 0.74.0 lacks working DR); restore-test has the same dropped-mountpoint exposure in reverse (boot-verifies without mp0/mp1 — candidate: reuse drRestoreOverrides); DR selftest KeepMAC=true duplicates a live source's MAC briefly.

• 2026-07-08 — GL-5b SHIPPED (agent v0.76.0, b1697874, live + PUBLISHED 9828c5f7…f50b): new G12 — the restore-test now verifies FULL fidelity (archive-derived params via drRestoreOverrides + a mount-parity assert; the old live-source-config path deleted with its tests). Live run: parity ok, inventory mp0 200G + mp1 50G + 2 throwaways, 3m4s local tier (measured — cheaper than the DR-derived ~7m estimate); a rotated-out archive volid refuses up front (good failure mode). 0.75.0 superseded unpublished; the ONE manifest bump goes straight to 0.76.0. Scenario-B red-proof run→fail→revert; bringup.go zero line changes (DR re-run trigger not fired).

6. Open questions & operator actions

Operator actions (Viktor):

• Hub manifest bump (Configs → Day-0 artifacts) — enter verbatim (UPDATED for GL-5b): agent 0.76.0 / 9828c5f75e719fb2e1fc3523f9c322b595a963ec1dcdf37ca42a96bedab7f50b; golden 0.103.0 / 8481e8a14e2aa0abe3831cc40e5aea4c32f2017e7561c15dca8a674df6a6026e. Until then fresh installs still land 0.63.0/0.98.3. Do NOT vouch 0.74.0 (broken guest-loss DR); 0.75.0 was superseded unpublished.
• Gitea visibility ruling (new, see G3): decide deliberately — (a) keep felhom repos public as an open-source posture, but audit + privatize homelab-manifests and the personal repos; or (b) lock the instance, which REQUIRES first proving the day-0 package fetch with a read-scoped token (the anonymous path dies). Recommendation: (a) — it preserves the working day-0 path; do the homelab-manifests secrets audit either way.
• G3 token work: mint the scoped read-only token, rotate out the package-WRITE one, fill the demo customer's empty git.username/git.token in the hub-served config.
• Operator-key pin ceremony on felhom-pve (+ the small day-0 follow-up — candidate GL-4 fold).
• Commit THIS doc to felhom.eu/documentation/pilot/GO-LIVE-PACKAGE.md (still absent after two tasks — paste it to CC with the next task so wrap-ups stop falling back to CONTEXT.md).

Open questions:

• G5: CPU/RAM cap values for Peti's guest (needed by GL-6/GL-7, not before).
• G11/GL-7: what is the local backup TARGET on Peti's box, given felhom-pbs is unreachable from his LAN — second local drive via per-app cross-drive backup, or a local PBS/vzdump storage?