admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: GL-8 wrap — runbook F3/F6 notes, findings FIXED, REPORT, decision log

Browse Source 
day0-install: F3 (no --allow-new-leaf on a populated BYO host) + F6 (:53 must
be free, refusal + remediation). GO-LIVE-PACKAGE + DRILL-GL6 ledger flip
F1/F3/F6 -> FIXED v1.12.0; GL-7 unblocked; F2/F7 stay open. CONTEXT + REPORT.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
authentik Default Admin
2026-07-08 19:40:57 +02:00
parent 18a556a514
commit fb8e5111cf
5 changed files with 95 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTEXT.md
+11
View File
@@ -3,6 +3,17 @@
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
- **2026-07-08 — TASK GL-8 SHIPPED: BYO coexistence hardening (host-install v1.12.0, `18a556a5`)** —
the GL-6 findings F3/F6/F1. **F3:** the populated-host leaf guard fires only when a `felhom_guests`
guest exists (not `pct list`) → a fresh byo install on a host running only the OWNER's guests needs
no `--allow-new-leaf`; the pin protection for real Felhom guests is unchanged. **F6:** byo preflight
REFUSES a foreign process on `:53` with remediation and mutates NOTHING (Viktor's ruling: refuse+
instruct only — the dnsmasq analog of the break-glass no-touch rule; appliance untouched). **F1:**
uninstall purges the config `.bak*` siblings (a GL-6 residue held a live hub api_key). F6
live-proven on felhom-pve (byo preflight refuses at :53 — the exact GL-6 scenario, read-only die).
Harness 36/36 (+GL8-F3/F6/F1 static, GL8-F1b behavioural, GL8-F6 live) + red-proofs RP-F3/F6/F1;
GL-2/GL-4 regression green; shellcheck clean. Shipped BEFORE GL-7 so Peti's command needs no
workaround. **F2/F7 remain open (non-blocking). NEXT: GL-7.**
- **2026-07-08 — TASK GL-5 SHIPPED: DR bring-up bind overrides (agent v0.75.0, LIVE + scratch-DR
validated)** — go-live G8; docs-only here (GO-LIVE-PACKAGE G8/G6/G1 flips + decision log; the
code + full report live in felhom-agent `b3446213`). Guest-loss DR restore WORKS now: the two
REPORT.md
+51 -42
View File
@@ -2,57 +2,66 @@
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
## RUNBOOK GL-6 — supervised go-live drill (demo box, two full cycles) — 2026-07-08
## TASK GL-8 — BYO coexistence hardening (host-install v1.12.0) — 2026-07-08
Full per-phase record: `documentation/pilot/DRILL-GL6-2026-07-08.md`. Supervised, Viktor gating
every phase; destructive (guest 9201 destroyed + rebuilt twice on felhom-pve). Go-live item **G7**
and the live-validation landing zone for GL-1…GL-5b. Ended with the demo healthy on normal life.
**Baseline confirmed:** felhom.eu `1dc86523`, `SCRIPT_VERSION="1.11.3"` → **`1.12.0`** (`18a556a5`).
Three independent fixes from the GL-6 drill findings, one file, one theme (BYO = Felhom is a guest
on a host it doesn't own). felhom.eu only; agent untouched. §13 STOP honored — no live install/
uninstall; the F6 refuse path is a read-only preflight die (safe to exercise), proven on felhom-pve.
**Baselines:** felhom.eu `bd97bc34` → **`87ba30ae`** (host-install `v1.11.1` → **`v1.11.3`** — two
fixes shipped mid-drill). felhom-agent `7f073936` (v0.76.0). Manifest confirmed vouching agent
0.76.0 + golden 0.103.0.
### The three fixes
### Result: DRILL COMPLETE — all gates GO
- **F3 — leaf guard narrowed.** The populated-host leaf-regeneration guard tested `pct list` (ANY
guest) and so over-fired on a BYO host's own non-Felhom guests, forcing `--allow-new-leaf` on
every fresh byo install. Now it tests `[[ -n "$(felhom_guests)" ]]` (the `/etc/felhom-bootstrap`
detector) — it fires only when a real Felhom guest exists (whose pinned fingerprint a leaf regen
would orphan — the 2026-06-28 incident). `--preserve-state-from` / `--allow-new-leaf` unchanged.
- **F6 — foreign `:53` refuse (byo), Viktor's §8 option 1.** In byo the agent's lan_resolver stays
off, so any `:53` bind is the owner's. GL-2's informational `:53` line is promoted to an actionable
gate that **`die`s with remediation and mutates nothing** — Felhom does not stop/mask/kill a DNS
service on a host it doesn't own (the break-glass analog). No stop/mask path, no consent-ack leg
(nothing to consent to). `ss` failure/absence degrades to a warn. Appliance mode untouched.
- **F1 — uninstall config purge.** The teardown removed `agent.json` but left `${agent_cfg}.bak*`
siblings — a GL-6 residue held a **live hub api_key**. Now it removes the config AND every `.bak*`
sibling, then the dir; the KEPT/WIPED statement + summary say "config (+ its .bak backups)". Paths
logged, contents never. GL-4 Scenario-D parity still green (the GL4-D token set unchanged — the
config removal was already covered).
- **Key-pin chain (GL-1's whole point) — legs 1–4 all green:** box==repo==password-manager
fingerprints; signers ARMED from first boot (byo + appliance); the **pinned operational key
authorized a real signed `agent_update` round-trip** 0.76.0→0.76.1→0.76.0 (verify vs pinned
`felhom-op-1` + nonce, A/B swap, dwell, commit, crash-guard silent), throwaway 0.76.1 deleted.
- **Two full destroy/rebuild cycles + two clean uninstalls.** Uninstall: busy-drive prop handled
(warn+guidance, never `-l/-f`, statement=retry), KEPT/WIPED statement verbatim, host audit clean,
**drive data intact on all three checks** (witness hash matched). byo (Phase 2) + appliance
(Phase 6) installs both green; appliance's 4b vaulted the root@pam recovery credential to the hub.
- **C7 live-validated:** an induced pool-membership drop made `step_verify` `log_error` +
WITH-WARNINGS (read-only); repaired via `--adopt-pool` → green.
- **Drive re-enrollment via the real UI** (Chrome bridge) — additive, pre-drill data reappeared;
restore-test `mount_parity: ok` on both PBS (5m35s) and local tiers.
### Tests (harness `hostinstall-mode-harness.sh`, extended)
### Two fixes shipped mid-drill
Static (anywhere): **GL8-F3** (guard uses `felhom_guests`, not `pct list`), **GL8-F6** (byo `:53`
gate refuses+instructs; no `systemctl stop/mask` / `kill` of any `:53`/dnsmasq/resolv service
anywhere), **GL8-F1** (`${agent_cfg}.bak*` removal present) + **GL8-F1b** (behavioural: the exact
glob-removal clears `agent.json` + three `.bak*` + the emptied dir). PVE tier on felhom-pve:
**GL8-F6 live** — the byo preflight **actually refused at `:53`** (felhom-pve's leftover dnsmasq is
bound — the exact GL-6 F6 scenario) with the remediation + `PRE-FLIGHT FAIL (exit 1)`; and the
free-`:53` arm (ss shimmed empty) passes the gate. **36/36 on felhom-pve, 0 failed**; shellcheck
clean; GL-2 Scenario-A + GL-4 Scenario-D regression green.
1. **host-install v1.11.2 (Gate-0 ruling, G3):** empty customer git creds → **anonymous artifact
fetch** (the Felhom artifacts are world-readable by operator ruling; the hub-vouched sha256 stays
the integrity root). All 12 installer fetch targets 200-anon-verified first. Harness `+GL6-ANON`.
2. **host-install v1.11.3 (finding F4, HIGH — Viktor HOLD):** `--resume` fully skipped completed
producer steps (`should_skip X || step_X`), but token/enroll/grows produce in-memory outputs
later steps consume → a resumed install wrote a config missing `hub.host_id`/`proxmox.token`
(daemon crash-loop) and passed `-rootfs-grow ""` (flag parse error). Fix: those three re-run every
pass (idempotent); golden's volid re-derived from local. **Live-proven** (crafted dry-resume +
the subsequent appliance install ran single-shot). Harness `+GL6-F4`; 27/27; shellcheck clean.
**Red-proofs (mutate scratch copy → invariant fails → discard):** RP-F3 (revert to `pct list`) →
GL8-F3 FAILS; RP-F6 (drop the `:53` die) → GL8-F6 FAILS; RP-F1 (narrow to `agent.json` only) →
GL8-F1 FAILS. All confirmed.
### Findings filed (small installer follow-up, none block Peti)
### Judgment calls
F1 config `.bak` residue on uninstall · F2 per-drive `mnt-*.mount` residue · F3 byo fresh install on
a populated host needs `--allow-new-leaf` (guard counts any guest) · F6 leftover broken dnsmasq
breaks LAN-client DNS on a byo host · F7 pool re-assert is bring-up-only (no continuous heal;
`--adopt-pool` repairs). (F5 was a self-corrected misdiagnosis — the public DNS wildcard works.)
- F3 behavioural runtime is impractical to isolate (the guard is inline in step 5, gated on the real
state dir's leaf; felhom-pve has BOTH a leaf and a Felhom guest) — validated grep-level + red-proof;
the true behavioural proof (fresh byo install on a populated non-Felhom host proceeds without
`--allow-new-leaf`) folds into GL-7's Peti install.
- F6 uses the existing `ss -H -ltnup` (UDP+TCP) invocation — catches a UDP-only resolver (dnsmasq)
too, not just TCP.
- F1 is surgical (`${agent_cfg}` + `${agent_cfg}.bak*`, then `rmdir`), not `rm -rf` the dir — respects
a non-default `-config` path that might share a directory.
### Deferred to GL-7 (with Peti)
### NOT live-validated (folds into GL-7 / a demo re-run)
Dashboard password (operator-set via hub — G9) + geo-restriction; **G10** (public dashboard is open
until the operator sets a password) is a GL-7 prerequisite. Full demo app redeploy + LAN-DNS
alignment (re-provision changed the guest DHCP IP) are Viktor's return-to-normal.
The real populated-host byo install proceeding without `--allow-new-leaf` (F3); the real foreign-`:53`
refusal on Peti's box (F6 — proven read-only on felhom-pve here); the real uninstall purging a `.bak`
(F1 — behaviourally unit-tested + grep-verified). All fold into GL-7's Peti day-0 as the live proof.
### Honest leftovers (unchanged)
### Findings status + what's next
S5 customer-data DR drill; G12 parity-on-real-drift + PBS-tier runtime; CGNAT retest; Impl-3;
OOB spike. **Next: GL-7** (Peti's day-0 runbook + tester agreement).
F1/F3/F6 → **FIXED (v1.12.0)** in the GL-6 ledger + GO-LIVE-PACKAGE. **Still open (non-blocking):**
F2 (per-drive `mnt-*.mount` units survive uninstall), F7 (pool re-assert bring-up-only, no continuous
heal — `--adopt-pool` repairs). **GL-7 is unblocked** — Peti's day-0 command needs no `--allow-new-leaf`
workaround. **Next: GL-7** (Peti day-0 runbook + tester agreement).
documentation/pilot/DRILL-GL6-2026-07-08.md
+3 -3
View File
@@ -361,11 +361,11 @@ plain-umount busy-drive handling, self-update-artifact removal, **data intact ×
| ID | Sev | What | Status |
|----|-----|------|--------|
| **F4** | HIGH | `--resume` skipped producer steps → config missing hub.host_id/proxmox.token (daemon crash-loop) + provision `-rootfs-grow ""` | **FIXED v1.11.3, live-proven** (dry + the single-shot appliance install) |
| F1 | MED | `/etc/felhom-agent/` + `agent.json.bak-*` (secret-bearing) survive uninstall when prior `.bak` configs exist | open — purge config dir/.bak in uninstall 4b |
| F1 | MED | `/etc/felhom-agent/` + `agent.json.bak-*` (secret-bearing) survive uninstall | **FIXED v1.12.0 (GL-8)** — uninstall purges `${agent_cfg}.bak*` |
| F2 | MED | per-drive `mnt-*.mount` units + `/mnt/<name>` primary mounts survive uninstall ("physically removable" overclaims) | open — uninstall stop+remove the agent's `mnt-*` units |
| F3 | MED | byo FRESH install on a host with the OWNER's guests trips the populated-host leaf guard (`pct list | grep -q .`), needs `--allow-new-leaf` | GL-7 command + candidate guard-narrow to `felhom_guests` |
| F3 | MED | byo FRESH install on a host with the OWNER's guests trips the populated-host leaf guard (`pct list`), needs `--allow-new-leaf` | **FIXED v1.12.0 (GL-8)** — guard narrowed to `felhom_guests` |
| F5 | — | initial "public dashboard DNS absent" — **MISDIAGNOSIS**: the `*.demo-felhom.eu` wildcard works (HTTP 200 via DoH); real cause was F6 | superseded by F6 |
| F6 | MED | leftover **broken dnsmasq** (active on `:53`, queries time out) on a byo host (`lan_resolver` off) breaks LAN-client DNS | open — byo install/uninstall stop+mask leftover dnsmasq |
| F6 | MED | leftover **broken dnsmasq** (active on `:53`) on a byo host breaks LAN-client DNS | **FIXED v1.12.0 (GL-8)** — byo preflight REFUSES a foreign `:53` + instructs (no mutation; Viktor ruling); live-proven |
| F7 | MED | R2 pool re-assertion is **bring-up-only** (`bringup.go:498`); no continuous membership reconcile → out-of-band drift doesn't auto-heal (repair via `--adopt-pool`) | open — candidate periodic membership reconcile |
| — | LOW | re-provision changes the guest DHCP IP (.105→.139) → manual LAN DNS entries go stale | GL-7 note (reserve lease / use lan_resolver) |
| — | LOW | legacy `felhom-agent.v*.bak` binaries survive the uninstall `.bak-*` glob | trivial (demo-box history only) |
documentation/pilot/GO-LIVE-PACKAGE.md
+18 -5
View File
@@ -9,6 +9,7 @@ catalog `2ebe082a` · felhom.eu `1a1e42ad`/hub v0.35.0 · host-install v1.9.1 ·
**Updated 2026-07-07 (GL-2 shipped):** felhom.eu `02d63ed0` · host-install **v1.10.0** + `scripts/hostinstall-mode-harness.sh`.
**Updated 2026-07-08 (GL-4 + GL-5 shipped):** felhom.eu `f7cc6a72` · host-install **v1.11.0** (uninstall gap-closure + key-pin plumbing) · felhom-agent **v0.75.0** (DR bind overrides, live on felhom-pve).
**Updated 2026-07-08 (GL-5b shipped):** felhom-agent **v0.76.0** live + PUBLISHED (`9828c5f7…f50b`) — restore-test full-fidelity + mount parity (new G12); the manifest bump target is now 0.76.0.
**Updated 2026-07-08 (GL-8):** felhom.eu `18a556a5` · host-install **v1.12.0** — BYO coexistence: F3 (leaf guard narrowed to Felhom guests → no `--allow-new-leaf` on a populated BYO host), F6 (byo refuses a foreign `:53`, never mutates), F1 (uninstall purges config `.bak*`). F6 live-proven on felhom-pve; harness 36/36. **GL-7 unblocked** (Peti's command needs no workaround).
**Updated 2026-07-08 (GL-6 DRILL COMPLETE):** felhom.eu `87ba30ae` · host-install **v1.11.3** (F4 resume fix shipped mid-drill; v1.11.2 anon-fetch ruling). Two full destroy/rebuild cycles on felhom-pve; manifest confirmed vouching agent 0.76.0 + golden 0.103.0. G2/G4/G5/G6/G7 LIVE-VALIDATED; G1 DONE; key-pin legs 1–4 proven. Record: `DRILL-GL6-2026-07-08.md`.
---
@@ -89,6 +90,7 @@ BUNDLE cert/key/token migration item; hub-floor auto-update.
| **GL-4** Uninstaller gap-closure + key-pin fold | ✔ **DONE 2026-07-08** (host-install v1.11.0, `f7cc6a72`) | felhom.eu/scripts | G6 + G1-plumbing | GL-2 |
| **GL-5** DR bind-override | ✔ **DONE 2026-07-08** (agent v0.75.0, `b3446213`, live-validated) | felhom-agent | G8 | — |
| **GL-6** Demo-box drill | ✔ **DONE 2026-07-08** (`DRILL-GL6-2026-07-08.md`; F4 fix v1.11.3 mid-drill) | felhom.eu/scripts (v1.11.3) | G7 | GL-1, GL-2, GL-4 |
| **GL-8** BYO coexistence hardening | ✔ **DONE 2026-07-08** (v1.12.0, `18a556a5`; F3/F6/F1) | felhom.eu/scripts | F3/F6/F1 | GL-6 |
| **GL-7** Peti day-0 runbook + onboarding | **NEXT** (GL-6 green) | felhom.eu/documentation/pilot | G9, G10, G11 + findings F1/F2/F3/F6/F7 | GL-6 green |
Recommended order: **GL-1 and GL-3 immediately** (operator-heavy, unblock everything), GL-2 next
@@ -159,16 +161,27 @@ Recommended order: **GL-1 and GL-3 immediately** (operator-heavy, unblock everyt
installer follow-up (none block Peti). G2/G4/G5/G6/G7 → LIVE-VALIDATED; G1 → DONE; G9 → deferred
to GL-7; G10 (open dashboard until operator password) → GL-7 prerequisite.
- 2026-07-08 — **GL-8 SHIPPED** (host-install **v1.12.0**, felhom.eu `18a556a5`): BYO coexistence
hardening from the GL-6 findings — **F3** leaf guard narrowed to `felhom_guests` (Peti's fresh
install on his populated box needs no `--allow-new-leaf` — the reason GL-8 shipped BEFORE GL-7),
**F6** byo preflight REFUSES a foreign `:53` with a remediation message (**Viktor's ruling: refuse
+ instruct, NO stop/mask — Felhom does not mutate a DNS service on a host it doesn't own**; the
break-glass analog), **F1** uninstall purges the config `.bak*` siblings (a GL-6 residue held a
live hub api_key). F6 **live-proven** on felhom-pve (byo preflight refuses at :53 — the exact
GL-6 scenario). Harness 36/36 + red-proofs RP-F3/F6/F1; GL-2/GL-4 regression green. F2/F7 remain
open (non-blocking). **GL-7 unblocked.**
## 6. Open questions & operator actions
**Operator actions (Viktor):**
- **Hub manifest bump — DONE** (confirmed at GL-6 Phase 0: the manifest vouches agent `0.76.0` +
golden `0.103.0`). No action.
- **Small installer follow-up (GL-6 findings, none block Peti)** — a v1.11.4 pass:
F1 (purge `/etc/felhom-agent` + `agent.json.bak-*` on uninstall), F2 (stop+remove the agent's
per-drive `mnt-*.mount` units on uninstall), F3 (narrow the populated-host leaf guard to
`felhom_guests`, or document `--allow-new-leaf` in the byo command), F6 (stop+mask the leftover
broken dnsmasq on a byo host), F7 (optional periodic pool-membership reconcile).
- **Installer findings — F1/F3/F6 FIXED in host-install v1.12.0 (GL-8, `18a556a5`).** F3: leaf guard
narrowed to `felhom_guests` (fresh byo install on a populated host needs no `--allow-new-leaf`).
F6: byo preflight REFUSES a foreign `:53` with remediation (Viktor's ruling — no stop/mask; live-
proven on felhom-pve). F1: uninstall purges config `.bak*` siblings. **Still open (none block
Peti):** F2 (per-drive `mnt-*.mount` units survive uninstall), F7 (pool re-assert is bring-up-only
— no continuous heal; `--adopt-pool` repairs). A later small pass, or fold into a future task.
- **Gitea visibility ruling** (new, see G3): decide deliberately — (a) keep felhom repos public
as an open-source posture, but audit + privatize `homelab-manifests` and the personal repos;
or (b) lock the instance, which REQUIRES first proving the day-0 package fetch with a
documentation/runbooks/day0-install.md
+12 -2
View File
@@ -240,8 +240,18 @@ What byo does differently (everything else matches C.4's eight steps):
- `--cores` AND `--memory` are **required** — size them WITH the owner (the guest must never starve
the owner's guests). `--acl-storages` must name storages that actually exist on the box, or the
preflight dies naming the missing ones.
- Preflight additionally prints what already listens on the host's `:53` (informational — Felhom
never configures DNS on a byo box) and the existing-guest count.
- **The host's `:53` must be free** (v1.12.0/F6): Felhom publishes the guest by name on the LAN, so a
resolver already bound to `:53` blocks it. byo preflight **REFUSES** (does not touch the owner's
service — it is a refusal, not a change) with: *"a resolver is already bound to :53 … stop or
reconfigure it, or point your LAN DNS at the guest's address, then re-run."* Clear `:53` (or point
the owner's DNS at the guest) and re-run. (Appliance mode is different — there Felhom's own
lan_resolver owns `:53`.)
- **A fresh install on a populated BYO host Just Works** (v1.12.0/F3): the leaf-regeneration guard
now fires only when a **Felhom** guest is already present — the owner's own non-Felhom guests no
longer trip it, so **`--allow-new-leaf` is NOT needed** on a first install. (You still need
`--preserve-state-from` / `--allow-new-leaf` on a reinstall over an existing *Felhom* guest, to
protect its pinned fingerprint — unchanged.)
- Preflight also prints the existing-guest count.
- At the end of preflight the script prints the **complete host-mutation list** (pveum
roles/user/token/ACL/pool, agent user/binary/unit/sudoers/config/state, the watchdog + guarded
wrappers, the guest + golden) and requires typing the host's **short hostname** to proceed —