From fb8e5111cf5888bbd5ec39591c40e7e1bb2e750a Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 8 Jul 2026 19:40:57 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20GL-8=20wrap=20=E2=80=94=20runbook=20F3/?= =?UTF-8?q?F6=20notes,=20findings=20FIXED,=20REPORT,=20decision=20log?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit day0-install: F3 (no --allow-new-leaf on a populated BYO host) + F6 (:53 must be free, refusal + remediation). GO-LIVE-PACKAGE + DRILL-GL6 ledger flip F1/F3/F6 -> FIXED v1.12.0; GL-7 unblocked; F2/F7 stay open. CONTEXT + REPORT. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CONTEXT.md | 11 +++ REPORT.md | 93 +++++++++++---------- documentation/pilot/DRILL-GL6-2026-07-08.md | 6 +- documentation/pilot/GO-LIVE-PACKAGE.md | 23 +++-- documentation/runbooks/day0-install.md | 14 +++- 5 files changed, 95 insertions(+), 52 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index f4134d2..725d68b 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,17 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-08 — TASK GL-8 SHIPPED: BYO coexistence hardening (host-install v1.12.0, `18a556a5`)** — + the GL-6 findings F3/F6/F1. **F3:** the populated-host leaf guard fires only when a `felhom_guests` + guest exists (not `pct list`) → a fresh byo install on a host running only the OWNER's guests needs + no `--allow-new-leaf`; the pin protection for real Felhom guests is unchanged. **F6:** byo preflight + REFUSES a foreign process on `:53` with remediation and mutates NOTHING (Viktor's ruling: refuse+ + instruct only — the dnsmasq analog of the break-glass no-touch rule; appliance untouched). **F1:** + uninstall purges the config `.bak*` siblings (a GL-6 residue held a live hub api_key). F6 + live-proven on felhom-pve (byo preflight refuses at :53 — the exact GL-6 scenario, read-only die). + Harness 36/36 (+GL8-F3/F6/F1 static, GL8-F1b behavioural, GL8-F6 live) + red-proofs RP-F3/F6/F1; + GL-2/GL-4 regression green; shellcheck clean. Shipped BEFORE GL-7 so Peti's command needs no + workaround. **F2/F7 remain open (non-blocking). NEXT: GL-7.** - **2026-07-08 — TASK GL-5 SHIPPED: DR bring-up bind overrides (agent v0.75.0, LIVE + scratch-DR validated)** — go-live G8; docs-only here (GO-LIVE-PACKAGE G8/G6/G1 flips + decision log; the code + full report live in felhom-agent `b3446213`). Guest-loss DR restore WORKS now: the two diff --git a/REPORT.md b/REPORT.md index 99a8eca..1127b86 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,57 +2,66 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## RUNBOOK GL-6 — supervised go-live drill (demo box, two full cycles) — 2026-07-08 +## TASK GL-8 — BYO coexistence hardening (host-install v1.12.0) — 2026-07-08 -Full per-phase record: `documentation/pilot/DRILL-GL6-2026-07-08.md`. Supervised, Viktor gating -every phase; destructive (guest 9201 destroyed + rebuilt twice on felhom-pve). Go-live item **G7** -and the live-validation landing zone for GL-1…GL-5b. Ended with the demo healthy on normal life. +**Baseline confirmed:** felhom.eu `1dc86523`, `SCRIPT_VERSION="1.11.3"` → **`1.12.0`** (`18a556a5`). +Three independent fixes from the GL-6 drill findings, one file, one theme (BYO = Felhom is a guest +on a host it doesn't own). felhom.eu only; agent untouched. §13 STOP honored — no live install/ +uninstall; the F6 refuse path is a read-only preflight die (safe to exercise), proven on felhom-pve. -**Baselines:** felhom.eu `bd97bc34` → **`87ba30ae`** (host-install `v1.11.1` → **`v1.11.3`** — two -fixes shipped mid-drill). felhom-agent `7f073936` (v0.76.0). Manifest confirmed vouching agent -0.76.0 + golden 0.103.0. +### The three fixes -### Result: DRILL COMPLETE — all gates GO +- **F3 — leaf guard narrowed.** The populated-host leaf-regeneration guard tested `pct list` (ANY + guest) and so over-fired on a BYO host's own non-Felhom guests, forcing `--allow-new-leaf` on + every fresh byo install. Now it tests `[[ -n "$(felhom_guests)" ]]` (the `/etc/felhom-bootstrap` + detector) — it fires only when a real Felhom guest exists (whose pinned fingerprint a leaf regen + would orphan — the 2026-06-28 incident). `--preserve-state-from` / `--allow-new-leaf` unchanged. +- **F6 — foreign `:53` refuse (byo), Viktor's §8 option 1.** In byo the agent's lan_resolver stays + off, so any `:53` bind is the owner's. GL-2's informational `:53` line is promoted to an actionable + gate that **`die`s with remediation and mutates nothing** — Felhom does not stop/mask/kill a DNS + service on a host it doesn't own (the break-glass analog). No stop/mask path, no consent-ack leg + (nothing to consent to). `ss` failure/absence degrades to a warn. Appliance mode untouched. +- **F1 — uninstall config purge.** The teardown removed `agent.json` but left `${agent_cfg}.bak*` + siblings — a GL-6 residue held a **live hub api_key**. Now it removes the config AND every `.bak*` + sibling, then the dir; the KEPT/WIPED statement + summary say "config (+ its .bak backups)". Paths + logged, contents never. GL-4 Scenario-D parity still green (the GL4-D token set unchanged — the + config removal was already covered). -- **Key-pin chain (GL-1's whole point) — legs 1–4 all green:** box==repo==password-manager - fingerprints; signers ARMED from first boot (byo + appliance); the **pinned operational key - authorized a real signed `agent_update` round-trip** 0.76.0→0.76.1→0.76.0 (verify vs pinned - `felhom-op-1` + nonce, A/B swap, dwell, commit, crash-guard silent), throwaway 0.76.1 deleted. -- **Two full destroy/rebuild cycles + two clean uninstalls.** Uninstall: busy-drive prop handled - (warn+guidance, never `-l/-f`, statement=retry), KEPT/WIPED statement verbatim, host audit clean, - **drive data intact on all three checks** (witness hash matched). byo (Phase 2) + appliance - (Phase 6) installs both green; appliance's 4b vaulted the root@pam recovery credential to the hub. -- **C7 live-validated:** an induced pool-membership drop made `step_verify` `log_error` + - WITH-WARNINGS (read-only); repaired via `--adopt-pool` → green. -- **Drive re-enrollment via the real UI** (Chrome bridge) — additive, pre-drill data reappeared; - restore-test `mount_parity: ok` on both PBS (5m35s) and local tiers. +### Tests (harness `hostinstall-mode-harness.sh`, extended) -### Two fixes shipped mid-drill +Static (anywhere): **GL8-F3** (guard uses `felhom_guests`, not `pct list`), **GL8-F6** (byo `:53` +gate refuses+instructs; no `systemctl stop/mask` / `kill` of any `:53`/dnsmasq/resolv service +anywhere), **GL8-F1** (`${agent_cfg}.bak*` removal present) + **GL8-F1b** (behavioural: the exact +glob-removal clears `agent.json` + three `.bak*` + the emptied dir). PVE tier on felhom-pve: +**GL8-F6 live** — the byo preflight **actually refused at `:53`** (felhom-pve's leftover dnsmasq is +bound — the exact GL-6 F6 scenario) with the remediation + `PRE-FLIGHT FAIL (exit 1)`; and the +free-`:53` arm (ss shimmed empty) passes the gate. **36/36 on felhom-pve, 0 failed**; shellcheck +clean; GL-2 Scenario-A + GL-4 Scenario-D regression green. -1. **host-install v1.11.2 (Gate-0 ruling, G3):** empty customer git creds → **anonymous artifact - fetch** (the Felhom artifacts are world-readable by operator ruling; the hub-vouched sha256 stays - the integrity root). All 12 installer fetch targets 200-anon-verified first. Harness `+GL6-ANON`. -2. **host-install v1.11.3 (finding F4, HIGH — Viktor HOLD):** `--resume` fully skipped completed - producer steps (`should_skip X || step_X`), but token/enroll/grows produce in-memory outputs - later steps consume → a resumed install wrote a config missing `hub.host_id`/`proxmox.token` - (daemon crash-loop) and passed `-rootfs-grow ""` (flag parse error). Fix: those three re-run every - pass (idempotent); golden's volid re-derived from local. **Live-proven** (crafted dry-resume + - the subsequent appliance install ran single-shot). Harness `+GL6-F4`; 27/27; shellcheck clean. +**Red-proofs (mutate scratch copy → invariant fails → discard):** RP-F3 (revert to `pct list`) → +GL8-F3 FAILS; RP-F6 (drop the `:53` die) → GL8-F6 FAILS; RP-F1 (narrow to `agent.json` only) → +GL8-F1 FAILS. All confirmed. -### Findings filed (small installer follow-up, none block Peti) +### Judgment calls -F1 config `.bak` residue on uninstall · F2 per-drive `mnt-*.mount` residue · F3 byo fresh install on -a populated host needs `--allow-new-leaf` (guard counts any guest) · F6 leftover broken dnsmasq -breaks LAN-client DNS on a byo host · F7 pool re-assert is bring-up-only (no continuous heal; -`--adopt-pool` repairs). (F5 was a self-corrected misdiagnosis — the public DNS wildcard works.) +- F3 behavioural runtime is impractical to isolate (the guard is inline in step 5, gated on the real + state dir's leaf; felhom-pve has BOTH a leaf and a Felhom guest) — validated grep-level + red-proof; + the true behavioural proof (fresh byo install on a populated non-Felhom host proceeds without + `--allow-new-leaf`) folds into GL-7's Peti install. +- F6 uses the existing `ss -H -ltnup` (UDP+TCP) invocation — catches a UDP-only resolver (dnsmasq) + too, not just TCP. +- F1 is surgical (`${agent_cfg}` + `${agent_cfg}.bak*`, then `rmdir`), not `rm -rf` the dir — respects + a non-default `-config` path that might share a directory. -### Deferred to GL-7 (with Peti) +### NOT live-validated (folds into GL-7 / a demo re-run) -Dashboard password (operator-set via hub — G9) + geo-restriction; **G10** (public dashboard is open -until the operator sets a password) is a GL-7 prerequisite. Full demo app redeploy + LAN-DNS -alignment (re-provision changed the guest DHCP IP) are Viktor's return-to-normal. +The real populated-host byo install proceeding without `--allow-new-leaf` (F3); the real foreign-`:53` +refusal on Peti's box (F6 — proven read-only on felhom-pve here); the real uninstall purging a `.bak` +(F1 — behaviourally unit-tested + grep-verified). All fold into GL-7's Peti day-0 as the live proof. -### Honest leftovers (unchanged) +### Findings status + what's next -S5 customer-data DR drill; G12 parity-on-real-drift + PBS-tier runtime; CGNAT retest; Impl-3; -OOB spike. **Next: GL-7** (Peti's day-0 runbook + tester agreement). +F1/F3/F6 → **FIXED (v1.12.0)** in the GL-6 ledger + GO-LIVE-PACKAGE. **Still open (non-blocking):** +F2 (per-drive `mnt-*.mount` units survive uninstall), F7 (pool re-assert bring-up-only, no continuous +heal — `--adopt-pool` repairs). **GL-7 is unblocked** — Peti's day-0 command needs no `--allow-new-leaf` +workaround. **Next: GL-7** (Peti day-0 runbook + tester agreement). diff --git a/documentation/pilot/DRILL-GL6-2026-07-08.md b/documentation/pilot/DRILL-GL6-2026-07-08.md index 44aea63..7a1bb2c 100644 --- a/documentation/pilot/DRILL-GL6-2026-07-08.md +++ b/documentation/pilot/DRILL-GL6-2026-07-08.md @@ -361,11 +361,11 @@ plain-umount busy-drive handling, self-update-artifact removal, **data intact × | ID | Sev | What | Status | |----|-----|------|--------| | **F4** | HIGH | `--resume` skipped producer steps → config missing hub.host_id/proxmox.token (daemon crash-loop) + provision `-rootfs-grow ""` | **FIXED v1.11.3, live-proven** (dry + the single-shot appliance install) | -| F1 | MED | `/etc/felhom-agent/` + `agent.json.bak-*` (secret-bearing) survive uninstall when prior `.bak` configs exist | open — purge config dir/.bak in uninstall 4b | +| F1 | MED | `/etc/felhom-agent/` + `agent.json.bak-*` (secret-bearing) survive uninstall | **FIXED v1.12.0 (GL-8)** — uninstall purges `${agent_cfg}.bak*` | | F2 | MED | per-drive `mnt-*.mount` units + `/mnt/` primary mounts survive uninstall ("physically removable" overclaims) | open — uninstall stop+remove the agent's `mnt-*` units | -| F3 | MED | byo FRESH install on a host with the OWNER's guests trips the populated-host leaf guard (`pct list | grep -q .`), needs `--allow-new-leaf` | GL-7 command + candidate guard-narrow to `felhom_guests` | +| F3 | MED | byo FRESH install on a host with the OWNER's guests trips the populated-host leaf guard (`pct list`), needs `--allow-new-leaf` | **FIXED v1.12.0 (GL-8)** — guard narrowed to `felhom_guests` | | F5 | — | initial "public dashboard DNS absent" — **MISDIAGNOSIS**: the `*.demo-felhom.eu` wildcard works (HTTP 200 via DoH); real cause was F6 | superseded by F6 | -| F6 | MED | leftover **broken dnsmasq** (active on `:53`, queries time out) on a byo host (`lan_resolver` off) breaks LAN-client DNS | open — byo install/uninstall stop+mask leftover dnsmasq | +| F6 | MED | leftover **broken dnsmasq** (active on `:53`) on a byo host breaks LAN-client DNS | **FIXED v1.12.0 (GL-8)** — byo preflight REFUSES a foreign `:53` + instructs (no mutation; Viktor ruling); live-proven | | F7 | MED | R2 pool re-assertion is **bring-up-only** (`bringup.go:498`); no continuous membership reconcile → out-of-band drift doesn't auto-heal (repair via `--adopt-pool`) | open — candidate periodic membership reconcile | | — | LOW | re-provision changes the guest DHCP IP (.105→.139) → manual LAN DNS entries go stale | GL-7 note (reserve lease / use lan_resolver) | | — | LOW | legacy `felhom-agent.v*.bak` binaries survive the uninstall `.bak-*` glob | trivial (demo-box history only) | diff --git a/documentation/pilot/GO-LIVE-PACKAGE.md b/documentation/pilot/GO-LIVE-PACKAGE.md index 206fe18..4f8ca67 100644 --- a/documentation/pilot/GO-LIVE-PACKAGE.md +++ b/documentation/pilot/GO-LIVE-PACKAGE.md @@ -9,6 +9,7 @@ catalog `2ebe082a` · felhom.eu `1a1e42ad`/hub v0.35.0 · host-install v1.9.1 · **Updated 2026-07-07 (GL-2 shipped):** felhom.eu `02d63ed0` · host-install **v1.10.0** + `scripts/hostinstall-mode-harness.sh`. **Updated 2026-07-08 (GL-4 + GL-5 shipped):** felhom.eu `f7cc6a72` · host-install **v1.11.0** (uninstall gap-closure + key-pin plumbing) · felhom-agent **v0.75.0** (DR bind overrides, live on felhom-pve). **Updated 2026-07-08 (GL-5b shipped):** felhom-agent **v0.76.0** live + PUBLISHED (`9828c5f7…f50b`) — restore-test full-fidelity + mount parity (new G12); the manifest bump target is now 0.76.0. +**Updated 2026-07-08 (GL-8):** felhom.eu `18a556a5` · host-install **v1.12.0** — BYO coexistence: F3 (leaf guard narrowed to Felhom guests → no `--allow-new-leaf` on a populated BYO host), F6 (byo refuses a foreign `:53`, never mutates), F1 (uninstall purges config `.bak*`). F6 live-proven on felhom-pve; harness 36/36. **GL-7 unblocked** (Peti's command needs no workaround). **Updated 2026-07-08 (GL-6 DRILL COMPLETE):** felhom.eu `87ba30ae` · host-install **v1.11.3** (F4 resume fix shipped mid-drill; v1.11.2 anon-fetch ruling). Two full destroy/rebuild cycles on felhom-pve; manifest confirmed vouching agent 0.76.0 + golden 0.103.0. G2/G4/G5/G6/G7 LIVE-VALIDATED; G1 DONE; key-pin legs 1–4 proven. Record: `DRILL-GL6-2026-07-08.md`. --- @@ -89,6 +90,7 @@ BUNDLE cert/key/token migration item; hub-floor auto-update. | **GL-4** Uninstaller gap-closure + key-pin fold | ✔ **DONE 2026-07-08** (host-install v1.11.0, `f7cc6a72`) | felhom.eu/scripts | G6 + G1-plumbing | GL-2 | | **GL-5** DR bind-override | ✔ **DONE 2026-07-08** (agent v0.75.0, `b3446213`, live-validated) | felhom-agent | G8 | — | | **GL-6** Demo-box drill | ✔ **DONE 2026-07-08** (`DRILL-GL6-2026-07-08.md`; F4 fix v1.11.3 mid-drill) | felhom.eu/scripts (v1.11.3) | G7 | GL-1, GL-2, GL-4 | +| **GL-8** BYO coexistence hardening | ✔ **DONE 2026-07-08** (v1.12.0, `18a556a5`; F3/F6/F1) | felhom.eu/scripts | F3/F6/F1 | GL-6 | | **GL-7** Peti day-0 runbook + onboarding | **NEXT** (GL-6 green) | felhom.eu/documentation/pilot | G9, G10, G11 + findings F1/F2/F3/F6/F7 | GL-6 green | Recommended order: **GL-1 and GL-3 immediately** (operator-heavy, unblock everything), GL-2 next @@ -159,16 +161,27 @@ Recommended order: **GL-1 and GL-3 immediately** (operator-heavy, unblock everyt installer follow-up (none block Peti). G2/G4/G5/G6/G7 → LIVE-VALIDATED; G1 → DONE; G9 → deferred to GL-7; G10 (open dashboard until operator password) → GL-7 prerequisite. +- 2026-07-08 — **GL-8 SHIPPED** (host-install **v1.12.0**, felhom.eu `18a556a5`): BYO coexistence + hardening from the GL-6 findings — **F3** leaf guard narrowed to `felhom_guests` (Peti's fresh + install on his populated box needs no `--allow-new-leaf` — the reason GL-8 shipped BEFORE GL-7), + **F6** byo preflight REFUSES a foreign `:53` with a remediation message (**Viktor's ruling: refuse + + instruct, NO stop/mask — Felhom does not mutate a DNS service on a host it doesn't own**; the + break-glass analog), **F1** uninstall purges the config `.bak*` siblings (a GL-6 residue held a + live hub api_key). F6 **live-proven** on felhom-pve (byo preflight refuses at :53 — the exact + GL-6 scenario). Harness 36/36 + red-proofs RP-F3/F6/F1; GL-2/GL-4 regression green. F2/F7 remain + open (non-blocking). **GL-7 unblocked.** + ## 6. Open questions & operator actions **Operator actions (Viktor):** - **Hub manifest bump — DONE** (confirmed at GL-6 Phase 0: the manifest vouches agent `0.76.0` + golden `0.103.0`). No action. -- **Small installer follow-up (GL-6 findings, none block Peti)** — a v1.11.4 pass: - F1 (purge `/etc/felhom-agent` + `agent.json.bak-*` on uninstall), F2 (stop+remove the agent's - per-drive `mnt-*.mount` units on uninstall), F3 (narrow the populated-host leaf guard to - `felhom_guests`, or document `--allow-new-leaf` in the byo command), F6 (stop+mask the leftover - broken dnsmasq on a byo host), F7 (optional periodic pool-membership reconcile). +- **Installer findings — F1/F3/F6 FIXED in host-install v1.12.0 (GL-8, `18a556a5`).** F3: leaf guard + narrowed to `felhom_guests` (fresh byo install on a populated host needs no `--allow-new-leaf`). + F6: byo preflight REFUSES a foreign `:53` with remediation (Viktor's ruling — no stop/mask; live- + proven on felhom-pve). F1: uninstall purges config `.bak*` siblings. **Still open (none block + Peti):** F2 (per-drive `mnt-*.mount` units survive uninstall), F7 (pool re-assert is bring-up-only + — no continuous heal; `--adopt-pool` repairs). A later small pass, or fold into a future task. - **Gitea visibility ruling** (new, see G3): decide deliberately — (a) keep felhom repos public as an open-source posture, but audit + privatize `homelab-manifests` and the personal repos; or (b) lock the instance, which REQUIRES first proving the day-0 package fetch with a diff --git a/documentation/runbooks/day0-install.md b/documentation/runbooks/day0-install.md index 236203e..551cea0 100644 --- a/documentation/runbooks/day0-install.md +++ b/documentation/runbooks/day0-install.md @@ -240,8 +240,18 @@ What byo does differently (everything else matches C.4's eight steps): - `--cores` AND `--memory` are **required** — size them WITH the owner (the guest must never starve the owner's guests). `--acl-storages` must name storages that actually exist on the box, or the preflight dies naming the missing ones. -- Preflight additionally prints what already listens on the host's `:53` (informational — Felhom - never configures DNS on a byo box) and the existing-guest count. +- **The host's `:53` must be free** (v1.12.0/F6): Felhom publishes the guest by name on the LAN, so a + resolver already bound to `:53` blocks it. byo preflight **REFUSES** (does not touch the owner's + service — it is a refusal, not a change) with: *"a resolver is already bound to :53 … stop or + reconfigure it, or point your LAN DNS at the guest's address, then re-run."* Clear `:53` (or point + the owner's DNS at the guest) and re-run. (Appliance mode is different — there Felhom's own + lan_resolver owns `:53`.) +- **A fresh install on a populated BYO host Just Works** (v1.12.0/F3): the leaf-regeneration guard + now fires only when a **Felhom** guest is already present — the owner's own non-Felhom guests no + longer trip it, so **`--allow-new-leaf` is NOT needed** on a first install. (You still need + `--preserve-state-from` / `--allow-new-leaf` on a reinstall over an existing *Felhom* guest, to + protect its pinned fingerprint — unchanged.) +- Preflight also prints the existing-guest count. - At the end of preflight the script prints the **complete host-mutation list** (pveum roles/user/token/ACL/pool, agent user/binary/unit/sudoers/config/state, the watchdog + guarded wrappers, the guest + golden) and requires typing the host's **short hostname** to proceed —