felhom.eu/documentation/pilot/DRILL-GL6-2026-07-08.md

DRILL GL-6 — supervised go-live drill (demo box, two full cycles) — 2026-07-08

Go-live item G7 + the live-validation landing zone for GL-1…GL-5b deferrals. Destructive- supervised: guest 9201 destroyed + rebuilt twice; Viktor present, gating every phase. This record is filled per phase as evidence lands; timings + deviations verbatim. No secrets.

Inputs verified at start: felhom.eu bd97bc34 (host-install v1.11.1, operator pubkeys pinned) · felhom-agent 7f073936 (v0.76.0) · hub Day-0 manifest vouches agent 0.76.0/9828c5f7…f50b + golden 0.103.0/8481e8a1…6026e (fetched via the box-facing API).

Phase 0 — preflight + key validation (nothing destroyed)

Step	Result
0a baselines	PASS — repo heads + SCRIPT_VERSION="1.11.1" + manifest all as stated above
0b key validation leg 1	PASS — all four fingerprints EQUAL (below)
0c key validation leg 2	[Viktor] — pending at Gate 0
0d abort-path backup + restore-test	PASS (local + PBS backups fresh; PBS restore-test parity ok, 5m35s)
0e busy-drive prop	STAGED — gl6-busyprop unit cwd-pinned into /mnt/felhom-drives/felhom-usb (4h)
0f mechanism re-read	DONE — Phase-3 delivery pinned (below)

0b — the four fingerprints, side by side

repo op  (felhom-op-1):  SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw
box  op  (felhom-op-1):  SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw
repo rec (felhom-rec-1): SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k
box  rec (felhom-rec-1): SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k

Box source: felhom-pve agent config (systemctl cat → -config path) authz.signers[]; repo source: the v1.11.1 OPERATOR_KEY_*_LINE constants. Both match the runbook's stated values.

0d — abort-path backups (the session's safety net)

• Local (agent pipeline): --selftest=backup -vmid 9201 → snapshot-mode, crash-consistent, local:backup/vzdump-lxc-9201-2026_07_08-11_43_42.tar.zst (6.82 GB, 5m52s). Expected uncovered_volumes: /etc/felhom-bootstrap + /mnt/felhom-drives (bind mounts — regenerated / re-enrolled, per doctrine).
• PBS (offsite tier, PVE-native vzdump per the S4-proven path): felhom-pbs:backup/ct/9201/2026-07-08T09:51:00Z — 3m04s (dedup) — NOTE: the live config deliberately pins local_backup_target: local; the PBS leg was driven with vzdump 9201 --storage felhom-pbs --mode snapshot (tunnel healthy, storage active).
• Restore-test vs the fresh PBS archive: PASS — mount_parity: ok, inventory mp0=/var/lib/docker (200G), mp1=/mnt/sys_drive (50G), mp8/mp9 throwaways; scratch 990000 restored+booted+verified+torn-down in 5m35s (first G12 PBS-tier runtime datum). Observation (cosmetic): the record's source_tier says "local" — the tier label derives from the CONFIG backup target, not the explicit -archive's storage; display-only.

0e — busy prop + data-intact witness

Prop: transient unit gl6-busyprop, cwd inside /mnt/felhom-drives/felhom-usb. Witness file (for the Phase-1/5 drive-data-intact checks): userdata/actualbudget/canary_hdd_actualbudget.txt on /dev/sdd1 (UUID da9e7089-cf8e-4617-adcb-a377743fae00), sha256 prefix 7b00bc3550b10e3952e0e0965f5a2628.

0f — Phase-3 signed-update mechanism (confirmed at source, NOT improvised)

1. [Viktor] sign: felhom-opsign -op agent_update -host <host_id> -key-id felhom-op-1 -agent-version <v> -sha256 <hex> -key <operational private key> [-ttl 30m] → signed-op blob.
2. Deliver: POST /api/v1/admin/hosts/{host_id}/jobs {job_id, blob_b64} — hub GLOBAL key required (Viktor), or the v0.70.0-precedent direct signed_jobs sqlite insert on the hub pod (CC via kubectl on 180). The agent's heartbeat sees has_signed_ops, fetches, verifies against the PINNED operational key + durable nonce store, downloads the binary from Gitea, re-verifies the SIGNED sha (the only integrity root), then felhom-selfupdate-guarded apply A/B-swaps with crash-rollback armed; dwell → commit.

Gate 0 — [Viktor]

• Input 1 (byo caps): --cores 4 --memory 8192
• Input 2 (Phase-6 repopulation): fresh + redeploy apps
• 0c fingerprint confirmation vs password manager: CONFIRMED — match
• Verdict: GO — with one pre-Phase-1 STOP + ruling (below)

Gate-0 deviation — the demo customer's empty git credentials (Viktor ruling → v1.11.2)

Pre-destroy check found demo-felhom's git.username/git.token still EMPTY (the GL-2 finding); the v1.11.1 installer would die at step 5/8. Viktor ruled: the artifacts are world-readable by design — anonymous fetch is the supported path. All 12 installer fetch targets (2 packages + 10 raw configs) validated HTTP 200 anonymously, then host-install v1.11.2 shipped mid-drill (felhom.eu 0640aa06): empty creds → loud WARN + anonymous fetch (conditional curl auth; the hub-vouched sha256 stays the integrity root; configured creds still used when present). Harness 26/26 (new GL6-ANON case; GL4-C2 assertion updated for the pinned constants).

Phase 1 — uninstall #1 (full mode, busy drive staged) — PASS with 2 findings

Real --uninstall --vmid 9201 (typed confirm, v1.11.2 fetched from the served URL):

• Busy-drive behavior (the staged prop on felhom-usb): WARN + "Eject the drive via the dashboard, or stop the apps using it and re-run" — NO -l/-f, teardown CONTINUED; the other two drives unmounted ("data stays on the drive"); the root bind correctly left mounted with its own guard warn. The KEPT-vs-WIPED statement listed felhom-usb as "still mounted (busy — stop the apps and retry)" — i.e. the code takes the statement-lists-it-as-retry path; after releasing the prop a plain manual umount completed the job (no re-run needed).
• KEPT-vs-WIPED statement: printed verbatim as designed (guest+volumes / agent / self-update artifacts / watchdog+OOB / pveum+pool / state = WIPED; drives+data / PBS backups / hub record / escrow = KEPT). The recovery-credential KEPT line correctly keyed off the state file — which on THIS box predates the 4b state marker, so the line was absent although a G1-vaulted credential exists (observation: state-keyed conditionals can under-report on pre-v1.10.0 installs).
• 1b audit: agent binary+baks, selfupdate wrapper+slots, rollback unit+drop-in, watchdog set, mkfs wrapper, snippets, dnsmasq drop-ins, sudoers, state dirs, pveum user/roles/ACL/pool — ALL GONE. PVE printed a benign "ignore invalid acl role 'FelhomAgentStore'" notice during ACL removal (ordering noise, recorded).
• Drive data intact: witness file re-hashed on the still-mounted primary path — 7b00bc3550b10e3952e0e0965f5a2628 == the Phase-0 value.

Findings

• GL6-F1 (MED, security-adjacent): /etc/felhom-agent/ SURVIVES the uninstall holding agent.json.bak-* files (incl. the operator's same-day ceremony backup) — old configs with secret material (pve token now dead; the per-host hub api_key is still live because the hub record survives). The uninstall removes agent.json but not .bak-* siblings. Fix candidate: purge "$agent_cfg"* / the dir in 4b (v1.11.3). Files left in place pending Viktor's call (one is his own backup).
• GL6-F2 (MED): the agent-created per-drive systemd mount units (mnt-felhom-usb.mount + flash + teszt_enroll) and their ACTIVE primary mounts at /mnt/<name> survive — the statement's "physically removable now" overclaims (the binds are gone but the partition is still mounted). Fix candidate: uninstall stops+removes the agent's mnt-* units for enrolled namespaces (data untouched). Left as-is for Phase 2 (the re-enroll behavior against pre-existing mounts is itself drill-relevant).

Gate 1 — [Viktor]: GO (delete the .bak secrets now)

GL6-F1 residue resolved at the gate: shred -u /etc/felhom-agent/agent.json.bak-* + rmdir (incl. Viktor's ceremony backup, confirmed disposable). F1/F2 kept as v1.11.3 fix candidates.

Phase 2 — byo-mode day-0 (Peti's exact path, on real hardware) — PASS (happy path), 3 findings

Command (Viktor typed the demo-felhom hostname ack — interactive consent leg): ./felhom-host-install.sh --customer-id demo-felhom --mode byo --vmid 9201 --cores 4 --memory 8192 --force-gitea-golden --acl-storages "local local-lvm" --passphrase-file <0600> (+ --allow-new-leaf --resume after F3/F4, see below).

End state — all green (verify step, verbatim highlights):

• disclosure printed; Viktor's typed-hostname ack accepted (byo install acknowledged for host demo-felhom);
• NO 4b / chpasswd / recovery-credential anywhere (grep of the full transcript = 0);
• anonymous artifact fetch worked (v1.11.2): agent v0.76.0 sha 9828c5f7… verified, golden 0.103.0 sha 8481e8a1… verified, both against the hub manifest;
• installed agent --version == 0.76.0; daemon active;
• authz signers: 2 (operator-signed self-update armed) from first boot — key validation leg 3 ✓ (config carries operational:felhom-op-1 + recovery:felhom-rec-1);
• pool member + all 4 scoped-ACL grants (user+token) present; caps on the guest: cores 4 / memory 8192;
• byo config asserts pass (lan_resolver/wg_tunnel/oob all off);
• controller Up (healthy) 0.103.0 + traefik + cloudflared up; dashboard HTTP 200 (Host-header probe); hub config-retrieve OK.

Findings — the byo install's resume path is fragile (happy single-shot path is clean)

• GL6-F3 (MED, GL-7-relevant): on a BYO host carrying the OWNER's (non-Felhom) guests, a FRESH install trips the populated-host leaf guard (pct list | grep -q . counts ANY guest) and dies demanding --preserve-state-from/--allow-new-leaf — even though no Felhom controller pin exists to protect on a fresh install. Peti's box has his own guests → his day-0 command MUST include --allow-new-leaf (GL-7). Better fix candidate: narrow the guard to felhom_guests so a fresh byo install on a populated non-Felhom host doesn't trip.
• GL6-F4 (HIGH, resume-correctness): --resume fully skips completed steps via should_skip X || step_X, but later steps consume the skipped steps' IN-MEMORY outputs, which are then empty → broken results: (a) enroll skipped → HOST_ID/HOST_API_KEY empty → step_agent_config writes a config missing hub.host_id (+ empty proxmox.token when token is skipped) → daemon crash-loops (hub.host_id is required); the mint-once token secret is unrecoverable, so recovery = re-run token+enroll. (b) grows skipped → ROOTFS_GROW/… empty → provision passes -rootfs-grow "" → flag parse error. Note step_token even has an internal resume-guard ([[ -n "$PVE_TOKEN" ]]) that the || step_token dispatch DEFEATS — strong signal the dispatch is the bug (steps whose outputs feed later steps must re-run, or re-derive from the written config, on resume). Drill recovery: dropped the affected completed marks from state.json to force re-run — the install then completed cleanly. The single-shot happy path (no interruption) is unaffected, which is Peti's path.
• GL6-F2 (from Phase 1, MED): the agent's per-drive mnt-*.mount units + primary /mnt/<name> mounts survived uninstall #1; on this reinstall the parent bind /mnt/felhom-drives is back but the drives are NOT yet bound under it — re-enrollment is the Phase-2b UI leg.

F4 fixed mid-drill (Viktor: HOLD → fix now) — host-install v1.11.3 (98bf5a43)

Root cause confirmed: should_skip X || step_X fully skips a completed step on --resume, but token/enroll/grows are PRODUCERS whose in-memory outputs later steps consume; step_token even had an internal resume-guard the || step_token dispatch DEFEATED. Fix: token/enroll/grows run every pass (all idempotent — reuse-or-rotate / mint-once-reuse POST / pure recompute); golden's GOLDEN_VOLID re-derived from the local archive in the resume block. Harness +GL6-F4 invariant, 27/27, shellcheck clean.

Live proof (v1.11.3, crafted --resume --dry-run on felhom-pve, throwaway FELHOM_INSTALL_STATE_DIR, zero mutations): marks = the exact pre-crash set [preflight,token,grows,enroll] (agent_config NOT done). Transcript: steps 2/8 token, 3/8 grows, 4/8 enroll all RE-RUN; the agent_config write shows host_id=<dry-run-host-id> (non-empty) and provision shows -rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42 — both the exact values that were empty (→ crash-loop / flag parse error) pre-fix. Real 9201 + its state.json untouched (verified: daemon active, controller Up healthy). GL6-F4 CLOSED + live-validated. (GL6-F3 stays a GL-7 command requirement --allow-new-leaf; guard-narrowing to felhom_guests recorded as a candidate. GL6-F1/F2 remain v1.11.x candidates.)

Phase 2b — bring-to-life: BLOCKED on public DNS (GL6-F5), browser bridge confirmed working

• Browser bridge: WORKS — created a tab group, navigated to the dashboard; the failure below is the SITE, not the bridge.
• GL6-F5 (onboarding, Viktor/Cloudflare): after the byo reinstall, felhom.demo-felhom.eu does NOT resolve publicly — curl from the box (outbound internet) returns Could not resolve host. The cloudflared tunnel is UP (QUIC + HTTP/2 to argotunnel, precheck all pass, same tunnel token), and the dashboard is healthy IN-GUEST (HTTP 200, "Beállítás" setup page) — so the controller + tunnel are fine; the missing piece is the public DNS CNAME felhom.demo-felhom.eu → <tunnel>.cfargotunnel.com. The controller does not auto-(re)create the dashboard's own DNS record on reinstall. This blocks the 2b real-pipeline rehearsal (password set + drive re-enroll need the dashboard reachable in a browser). NOT an install-correctness defect — the install is green; it is a Cloudflare-zone / onboarding gap that also matters for Peti (GL-7 / G9).

Phase 2b — bring-to-life (browser, CC-driven) — drive re-enroll + data-visible PASS

DNS unblock: Viktor added a Pi-hole split-horizon A record felhom.demo-felhom.eu → 192.168.0.105 (LAN path to the guest); dashboard then loaded in-browser. (The public Cloudflare path is ALSO correct — forced-DoH resolution returned HTTP 200 with the real page; only the client-side resolver via the box's broken dnsmasq (F6) was failing.)

• Dashboard live: "Demo Ügyfél", 3 running / 56 total apps, mem 1.1/8 GB, backup "Még nem futott".
• Drive re-enrollment (CC-driven UI): Storage → Meghajtók showed all 3 drives detected-but- UNREGISTERED (the agent's registration state was wiped in uninstall #1; the drives + data survived per F2, tagged "Adatot tartalmaz"). Clicked Regisztrálás on felhom-usb (uuid da9e7089…, /dev/sdd1) — it moved to Adattárolók → Aktív, ext4 /dev/sdd1[/felhom-data] HD710 PRO, additive (no format). Data-visible PASS: the witness file is now readable in the guest at /mnt/felhom-drives/felhom-usb/userdata/actualbudget/canary_hdd_actualbudget.txt, hash 7b00bc35… == the Phase-0 value. Additive doctrine proven end-to-end through the real UI.
• GL6-F6 (MED, byo coexistence): the box runs a leftover broken dnsmasq (active, bound to 0.0.0.0:53, but queries time out — empty /etc/dnsmasq.d/, no upstream) on a byo host where lan_resolver is OFF. It breaks DNS for LAN clients that use the box as a resolver (it broke the drill's browser resolution). Fix pile with F1/F2: byo install/uninstall should stop+mask the leftover dnsmasq. Immediate drill unblock was the Pi-hole entry.
• G9/G10 — password is OPERATOR-set, currently UNSET → dashboard OPEN. Settings → Biztonság: "A jelszavas védelem nincs beállítva. Kérd az üzemeltetőt a beállításhoz." The dashboard password comes through the real config pipeline (hub customer-config → controller.yaml bcrypt — the anti-F9 path), NOT a customer-set field here. Until the operator sets it the public-reachable dashboard is UNAUTHENTICATED (the known G10 exposure, now confirmed live). Geo-restriction also OFF. Operator leg pending: Viktor sets the dashboard password via the hub, then CC verifies login is enforced.

Gate 2 — [Viktor]: GO — Peti's byo path live-proven end to end

Green: anonymous fetch, sha-verified 0.76.0 + 0.103.0, signers ARMED from first boot, caps applied, no root@pam touched, controller healthy, drive re-enrolled additively with pre-drill data intact. F4 found + fixed (v1.11.3) + live-proven. Deferred to GL-7 with Peti: the dashboard password (operator-set via hub — G9) + geo-restriction; G10 (open dashboard until the operator sets a password) is now a GL-7 onboarding prerequisite. Findings F1/F2/F3/F6 → post-drill installer pile.

Phase 3 — armed key end-to-end (the only leg that needed real keys)

PASS — the armed-key chain proven both directions with the REAL operational key.

• 3a: drill agent 0.76.1 published (sha ff18972f…aed16c), version-only bump, throwaway.
• Delivery mechanism (deviation from the [CC] tag, recorded): the hub pod is a minimal image (no sqlite3/python — only the felhom-hub binary) and the admin-enqueue API is GLOBAL-key gated, so CC cannot inject the signed job. felhom-opsign has a built-in -upload+-hub-key, so the sign + enqueue is ONE operator command (Viktor's secrets stay in it). The -hub-key is the hub's report_api_key (live ConfigMap == committed manifests/hub.yaml:77; the operator/global key, NOT the web-UI login) — a point of confusion worth a GL-7 note.
• 3b/3c FORWARD (0.76.0→0.76.1): felhom-opsign -op agent_update -host demo-felhom-01 -key-id felhom-op-1 -agent-version 0.76.1 -sha256 ff18972f… -key <op-priv> -upload https://hub.felhom.eu -hub-key <report_api_key>. Agent (restart-forced poll) → audit gate decision class=agent_update disposition=destructive allowed=true reason=signed key_id=felhom-op-1 nonce=4176e8e0… → download (anon Gitea) → sha re-verify → felhom-selfupdate-guarded apply A/B flip → new version running 0.76.1 prev=0.76.0 dwell=1m0s → update committed version=0.76.1. .prev=0.76.0 retained; crash-guard silent.
• RETURN (0.76.1→0.76.0): same with the 0.76.0 published sha 9828c5f7… + a FRESH nonce → committed back to 0.76.0 (the manifest-vouched version; sha confirmed). Double round-trip = sign→verify→ nonce→swap proven twice with the real key.
• 3d: 0.76.1 package DELETED (HTTP 204 → 404 anon). Box ends at 0.76.0, daemon active.

Key validation leg 4 (the whole point of GL-1's key-pin): the pinned operational key actually authorizes a real signed op end-to-end. Combined with legs 1–3 (box==repo==manager fingerprints, armed-from-first-boot), the operator-signed self-update path is fully live-validated.

Gate 3 — [Viktor]: GO — armed-key chain proven both directions.

Phase 4 — drift-heal, live (the R2 fix earning its keep) — PASS, 1 finding

• 4a induce drift: pveum pool modify felhom --vms 9201 --delete 1 → pool members [9201] → []; the agent selftest immediately reflected it (pool read: pool "felhom", 0 member(s)) — the exact silent drift R2 addresses.
• GL6-F7 (MED, new): the R2 PoolAddVMID re-assertion is bring-up-only (bringup.go:498, after restore-over-existing) — there is NO continuous pool-membership reconcile (the stale-lock reaper READS membership, stalelock.go:183, but never repairs). Confirmed live: 2 min after the removal, no re-assert, pool stayed []. So out-of-band membership drift does NOT auto-heal; it needs a bring-up or --adopt-pool. (R2 itself DID fire correctly at this drill's provision — pool membership re-asserted vmid=9201 — so the fix works for its designed path.) Candidate: a periodic membership reconcile; acceptable for the pilot (verify catches it, --adopt-pool fixes).
• 4b C7 — the verify assert firing on real drift (read-only): installer --resume re-ran step_verify → [ERROR] pool: guest 9201 NOT in pool felhom (the scoped token cannot manage it — adopt with --adopt-pool) → Day-0 provision completed WITH WARNINGS. The ACL grants + authz signers: 2 stayed green (only membership was perturbed). C7 (GL-2's structural assert catching a real silent drop) is now LIVE-VALIDATED — closes the GL-2 "not testable pre-drill" gap.
• Repair: --adopt-pool --vmid 9201 re-added it (pool [9201]), re-verify → SUCCESS. Demo healthy. (Also incidentally re-proved the v1.11.3 F4 fix: the --resume re-ran token/enroll/grows cleanly and reached verify — no crash-loop, no flag error.)

Gate 4 — [Viktor]: GO — drift-heal proven (C7 live, F7 recorded, repaired).

Phase 5 — uninstall #2 (the byo box's clean exit) — PASS

Real --uninstall --vmid 9201 (typed confirm), no busy prop this time.

• felhom-usb unmounted cleanly ("data stays on the drive"); statement reads "the drives are physically removable now" (no busy exception this run) — the KEPT-vs-WIPED statement printed as designed; UNINSTALL complete.
• Host audit CLEAN: no felhom-agent unit/user/binary/selfupdate-wrapper/A-B-slots/rollback-unit/ limits-drop-in/watchdog/mkfs-wrapper/hook-snippet/dnsmasq-drop-ins; pveum user 0, roles empty, felhom pool gone; /var/lib/felhom-agent, /var/lib/felhom-install, /etc/felhom-agent all gone. (F1 did NOT recur — this install's config had no .bak siblings.)
• Drive DATA intact: witness 7b00bc35… == Phase-0 (the data-sovereignty exit, proven a 2nd time).
• Recurring known findings: F2 (per-drive mnt-*.mount units survive), F6 (leftover dnsmasq still active). Trivial demo-box note: 6 ancient felhom-agent.v0.1x.0.bak binaries (Jun 10–11, legacy naming) survive the .bak-* glob — historical cruft, not on a real customer box; a fix could widen the glob to .v*.bak.

Gate 5 — [Viktor]: GO — clean byo exit, data intact 2nd time.

6b — return to normal.

• Drive re-enrolled (CC-driven UI): felhom-usb → Adattárolók Aktív, ext4 /dev/sdd1 [/felhom-data]; witness 7b00bc35… intact in-guest (additive, 3rd data-intact proof). (First click collided with the in-progress snapshot backup and no-op'd; retry after the backup registered it — minor UI race worth a note. Only felhom-usb re-enrolled; flash/teszt left for Viktor's demo curation.)
• Manual backup + restore-test GREEN: selftest=backup OK (crash-consistent=true) → restore-test mount_parity: ok (29s — small: fresh install, no apps/drive-data yet).
• Controller + traefik + cloudflared + filebrowser all Up healthy; dashboard HTTP 200; hub reporting. Observation: during the snapshot vzdump the dashboard briefly flashed "Protected container not running: traefik/cloudflared/filebrowser" — a transient false-positive of the container monitor during the backup freeze (cosmetic; cleared when the backup finished).
• Viktor's ongoing return-to-normal (not drill blockers): redeploy the demo apps (his Phase-0 choice: fresh redeploy); set the dashboard password (deferred to GL-7); update/align LAN DNS (the re-provision changed the guest IP .105→.139 — a manual Pi-hole entry goes stale; rely on the appliance lan_resolver or a reserved DHCP lease — GL-7 note).

Phase 6 — appliance-mode reinstall (the demo returns to normal life)

6a — appliance install: SUCCESS, single-shot. --mode appliance … --allow-new-leaf.

• 4b/8 break-glass RAN (appliance): root@pam password set + vaulted to the hub + break-glass layers 1+2 installed — the recovery credential is in the hub (operator-retrievable). (SSH key access unaffected.)
• verify green: pool member, all 4 scoped-ACL grants (user+token), authz signers: 2, controller Up healthy 0.103.0, Day-0 provision SUCCESS.
• Bonus: this ran in ONE shot (vs the byo install's 4 resumes on v1.11.2) — the v1.11.3 F4 fix live-proven on a real end-to-end install, not just the dry proof.

(Phases 3–6 appended as they run.)

Gate 6 — [Viktor]: GO — DRILL COMPLETE.

Demo healthy on normal life. Two full destroy/rebuild cycles proven (byo #2, appliance #6), both uninstalls clean, drive data intact ALL THREE checks, armed-key chain proven, C7 live-validated.

Summary — what GL-6 validated

Key-pin chain (GL-1's whole point) — legs 1–4 all green: box==repo==password-manager fingerprints (op 7YqN…, rec /ixg…); signers ARMED from first boot on both byo + appliance installs; the pinned operational key AUTHORIZED a real signed agent_update op end-to-end (round-trip 0.76.0→0.76.1→0.76.0 with the real key, verify vs pinned signer + nonce, A/B swap, crash-guard silent).

Go-live gates now LIVE-VALIDATED on real hardware: G2 (byo profile — disclosure+ack, no root@pam, config asserts), G4 (preflight guards + verify pool/ACL asserts incl. C7 firing on real drift), G5 (mandatory byo caps applied to the guest), G6 (uninstall gap-closure — KEPT/WIPED statement, plain-umount busy-drive handling, self-update-artifact removal, data intact ×3), G7 (this drill). G1 DONE (published chain consumed end-to-end via anonymous fetch — the v1.11.2 Gate-0 ruling —

• armed keys proven). G12 re-exercised (restore-test mount_parity: ok on PBS 5m35s + local tiers). G9 rehearsed (dashboard password is operator-set-via-hub; location confirmed) → deferred to GL-7; G10 (open dashboard until the operator sets a password) → GL-7 prerequisite.

Findings ledger (post-drill installer pile unless noted)

ID	Sev	What	Status
F4	HIGH	--resume skipped producer steps → config missing hub.host_id/proxmox.token (daemon crash-loop) + provision -rootfs-grow ""	FIXED v1.11.3, live-proven (dry + the single-shot appliance install)
F1	MED	/etc/felhom-agent/ + agent.json.bak-* (secret-bearing) survive uninstall	FIXED v1.12.0 (GL-8) — uninstall purges ${agent_cfg}.bak*
F2	MED	per-drive mnt-*.mount units + /mnt/<name> primary mounts survive uninstall ("physically removable" overclaims)	open — uninstall stop+remove the agent's mnt-* units
F3	MED	byo FRESH install on a host with the OWNER's guests trips the populated-host leaf guard (pct list), needs --allow-new-leaf	FIXED v1.12.0 (GL-8) — guard narrowed to felhom_guests
F5	—	initial "public dashboard DNS absent" — MISDIAGNOSIS: the *.demo-felhom.eu wildcard works (HTTP 200 via DoH); real cause was F6	superseded by F6
F6	MED	leftover broken dnsmasq (active on :53) on a byo host breaks LAN-client DNS	FIXED v1.12.0 (GL-8) — byo preflight REFUSES a foreign :53 + instructs (no mutation; Viktor ruling); live-proven
F7	MED	R2 pool re-assertion is bring-up-only (bringup.go:498); no continuous membership reconcile → out-of-band drift doesn't auto-heal (repair via --adopt-pool)	open — candidate periodic membership reconcile
—	LOW	re-provision changes the guest DHCP IP (.105→.139) → manual LAN DNS entries go stale	GL-7 note (reserve lease / use lan_resolver)
—	LOW	legacy felhom-agent.v*.bak binaries survive the uninstall .bak-* glob	trivial (demo-box history only)
—	LOW	snapshot-mode backup briefly flashes "Protected container not running" (monitor false-positive during freeze)	cosmetic

Honest leftovers (unchanged, still open): S5 customer-data DR drill; G12 parity-on-real-drift + PBS-tier runtime characterization; CGNAT true-sim retest; Impl-3 shared-box format gate; OOB spike.

Drill deviations (recorded)

1. v1.11.2 shipped mid-drill (Gate-0 ruling): empty customer git creds → anonymous artifact fetch (the artifacts are world-readable by ruling; sha256 stays the integrity root). All 12 fetch targets 200-anon-validated first.
2. F4 fixed mid-drill (v1.11.3, Viktor HOLD): the resume-correctness bug, live-proven fixed.
3. Phase-3 delivery is operator-driven, not [CC]: the hub pod has no direct-insert tool and the admin-enqueue API is global-key-gated, so Viktor signs+enqueues in one felhom-opsign -upload command (-hub-key = the hub report_api_key, not the web login — a GL-7 clarity note).
4. G9 password + full app redeploy deferred to GL-7 (operator/customer curation).