felhom.eu/documentation/pilot/DRILL-GL6-2026-07-08.md

# DRILL GL-6 — supervised go-live drill (demo box, two full cycles) — 2026-07-08

> Go-live item **G7** + the live-validation landing zone for GL-1…GL-5b deferrals. Destructive-
> supervised: guest 9201 destroyed + rebuilt twice; Viktor present, gating every phase. This record
> is filled per phase as evidence lands; timings + deviations verbatim. No secrets.

**Inputs verified at start:** felhom.eu `bd97bc34` (host-install **v1.11.1**, operator pubkeys
pinned) · felhom-agent `7f073936` (v0.76.0) · hub Day-0 manifest vouches agent
`0.76.0`/`9828c5f7…f50b` + golden `0.103.0`/`8481e8a1…6026e` (fetched via the box-facing API).

## Phase 0 — preflight + key validation (nothing destroyed)

| Step | Result |
|---|---|
| 0a baselines | **PASS** — repo heads + `SCRIPT_VERSION="1.11.1"` + manifest all as stated above |
| 0b key validation leg 1 | **PASS** — all four fingerprints EQUAL (below) |
| 0c key validation leg 2 | [Viktor] — pending at Gate 0 |
| 0d abort-path backup + restore-test | **PASS** (local + PBS backups fresh; PBS restore-test parity ok, 5m35s) |
| 0e busy-drive prop | **STAGED** — `gl6-busyprop` unit cwd-pinned into `/mnt/felhom-drives/felhom-usb` (4h) |
| 0f mechanism re-read | **DONE** — Phase-3 delivery pinned (below) |

### 0b — the four fingerprints, side by side

```
repo op  (felhom-op-1):  SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw
box  op  (felhom-op-1):  SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw
repo rec (felhom-rec-1): SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k
box  rec (felhom-rec-1): SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k
```

Box source: felhom-pve agent config (`systemctl cat` → `-config` path) `authz.signers[]`;
repo source: the v1.11.1 `OPERATOR_KEY_*_LINE` constants. Both match the runbook's stated values.

### 0d — abort-path backups (the session's safety net)

- **Local (agent pipeline):** `--selftest=backup -vmid 9201` → snapshot-mode, crash-consistent,
  `local:backup/vzdump-lxc-9201-2026_07_08-11_43_42.tar.zst` (6.82 GB, 5m52s). Expected
  `uncovered_volumes`: `/etc/felhom-bootstrap` + `/mnt/felhom-drives` (bind mounts — regenerated /
  re-enrolled, per doctrine).
- **PBS (offsite tier, PVE-native vzdump per the S4-proven path):** `felhom-pbs:backup/ct/9201/2026-07-08T09:51:00Z` — 3m04s (dedup)
  — NOTE: the live config deliberately pins `local_backup_target: local`; the PBS leg was driven
  with `vzdump 9201 --storage felhom-pbs --mode snapshot` (tunnel healthy, storage active).
- **Restore-test vs the fresh PBS archive:** **PASS** — `mount_parity: ok`, inventory
  `mp0=/var/lib/docker (200G)`, `mp1=/mnt/sys_drive (50G)`, mp8/mp9 throwaways; scratch 990000
  restored+booted+verified+torn-down in **5m35s** (first G12 PBS-tier runtime datum).
  Observation (cosmetic): the record's `source_tier` says "local" — the tier label derives from the
  CONFIG backup target, not the explicit `-archive`'s storage; display-only.

### 0e — busy prop + data-intact witness

Prop: transient unit `gl6-busyprop`, cwd inside `/mnt/felhom-drives/felhom-usb`.
Witness file (for the Phase-1/5 drive-data-intact checks):
`userdata/actualbudget/canary_hdd_actualbudget.txt` on `/dev/sdd1`
(UUID `da9e7089-cf8e-4617-adcb-a377743fae00`), sha256 prefix `7b00bc3550b10e3952e0e0965f5a2628`.

### 0f — Phase-3 signed-update mechanism (confirmed at source, NOT improvised)

1. [Viktor] sign: `felhom-opsign -op agent_update -host <host_id> -key-id felhom-op-1
   -agent-version <v> -sha256 <hex> -key <operational private key> [-ttl 30m]` → signed-op blob.
2. Deliver: `POST /api/v1/admin/hosts/{host_id}/jobs` `{job_id, blob_b64}` — **hub GLOBAL key
   required** (Viktor), or the v0.70.0-precedent direct `signed_jobs` sqlite insert on the hub pod
   (CC via kubectl on 180). The agent's heartbeat sees `has_signed_ops`, fetches, verifies against
   the PINNED operational key + durable nonce store, downloads the binary from Gitea, re-verifies
   the SIGNED sha (the only integrity root), then `felhom-selfupdate-guarded apply` A/B-swaps with
   crash-rollback armed; dwell → `commit`.

## Gate 0 — [Viktor]

- Input 1 (byo caps): **--cores 4 --memory 8192**
- Input 2 (Phase-6 repopulation): **fresh + redeploy apps**
- 0c fingerprint confirmation vs password manager: **CONFIRMED — match**
- Verdict: **GO** — with one pre-Phase-1 STOP + ruling (below)

### Gate-0 deviation — the demo customer's empty git credentials (Viktor ruling → v1.11.2)

Pre-destroy check found demo-felhom's `git.username`/`git.token` still EMPTY (the GL-2 finding);
the v1.11.1 installer would die at step 5/8. **Viktor ruled: the artifacts are world-readable by
design — anonymous fetch is the supported path.** All 12 installer fetch targets (2 packages + 10
raw configs) validated HTTP 200 anonymously, then **host-install v1.11.2** shipped mid-drill
(felhom.eu `0640aa06`): empty creds → loud WARN + anonymous fetch (conditional curl auth; the
hub-vouched sha256 stays the integrity root; configured creds still used when present). Harness
26/26 (new GL6-ANON case; GL4-C2 assertion updated for the pinned constants).

## Phase 1 — uninstall #1 (full mode, busy drive staged) — PASS with 2 findings

Real `--uninstall --vmid 9201` (typed confirm, v1.11.2 fetched from the served URL):

- **Busy-drive behavior (the staged prop on felhom-usb):** WARN + "Eject the drive via the
  dashboard, or stop the apps using it and re-run" — NO `-l`/`-f`, teardown CONTINUED; the other
  two drives unmounted ("data stays on the drive"); the root bind correctly left mounted with its
  own guard warn. The KEPT-vs-WIPED statement listed felhom-usb as
  "still mounted (busy — stop the apps and retry)" — i.e. the code takes the
  statement-lists-it-as-retry path; after releasing the prop a plain manual `umount` completed the
  job (no re-run needed).
- **KEPT-vs-WIPED statement:** printed verbatim as designed (guest+volumes / agent / self-update
  artifacts / watchdog+OOB / pveum+pool / state = WIPED; drives+data / PBS backups / hub record /
  escrow = KEPT). The recovery-credential KEPT line correctly keyed off the state file — which on
  THIS box predates the 4b state marker, so the line was absent although a G1-vaulted credential
  exists (observation: state-keyed conditionals can under-report on pre-v1.10.0 installs).
- **1b audit:** agent binary+baks, selfupdate wrapper+slots, rollback unit+drop-in, watchdog set,
  mkfs wrapper, snippets, dnsmasq drop-ins, sudoers, state dirs, pveum user/roles/ACL/pool — ALL
  GONE. PVE printed a benign "ignore invalid acl role 'FelhomAgentStore'" notice during ACL
  removal (ordering noise, recorded).
- **Drive data intact:** witness file re-hashed on the still-mounted primary path —
  `7b00bc3550b10e3952e0e0965f5a2628` == the Phase-0 value.

### Findings

- **GL6-F1 (MED, security-adjacent):** `/etc/felhom-agent/` SURVIVES the uninstall holding
  `agent.json.bak-*` files (incl. the operator's same-day ceremony backup) — old configs with
  secret material (pve token now dead; the per-host hub api_key is still live because the hub
  record survives). The uninstall removes `agent.json` but not `.bak-*` siblings. Fix candidate:
  purge `"$agent_cfg"*` / the dir in 4b (v1.11.3). Files left in place pending Viktor's call
  (one is his own backup).
- **GL6-F2 (MED):** the agent-created per-drive systemd mount units
  (`mnt-felhom-usb.mount` + flash + teszt_enroll) and their ACTIVE primary mounts at
  `/mnt/<name>` survive — the statement's "physically removable now" overclaims (the binds are
  gone but the partition is still mounted). Fix candidate: uninstall stops+removes the agent's
  `mnt-*` units for enrolled namespaces (data untouched). Left as-is for Phase 2 (the re-enroll
  behavior against pre-existing mounts is itself drill-relevant).

## Gate 1 — [Viktor]: **GO** (delete the .bak secrets now)

GL6-F1 residue resolved at the gate: `shred -u /etc/felhom-agent/agent.json.bak-*` + `rmdir`
(incl. Viktor's ceremony backup, confirmed disposable). F1/F2 kept as v1.11.3 fix candidates.

## Phase 2 — byo-mode day-0 (Peti's exact path, on real hardware) — PASS (happy path), 3 findings

Command (Viktor typed the `demo-felhom` hostname ack — interactive consent leg):
`./felhom-host-install.sh --customer-id demo-felhom --mode byo --vmid 9201 --cores 4 --memory 8192
--force-gitea-golden --acl-storages "local local-lvm" --passphrase-file <0600>` (+ `--allow-new-leaf
--resume` after F3/F4, see below).

**End state — all green (verify step, verbatim highlights):**
- disclosure printed; **Viktor's typed-hostname ack accepted** (`byo install acknowledged for host demo-felhom`);
- **NO 4b / chpasswd / recovery-credential** anywhere (grep of the full transcript = 0);
- **anonymous artifact fetch worked** (v1.11.2): agent v0.76.0 sha `9828c5f7…` verified, golden
  0.103.0 sha `8481e8a1…` verified, both against the hub manifest;
- installed agent `--version` == **0.76.0**; daemon **active**;
- **`authz signers: 2 (operator-signed self-update armed)`** from first boot — **key validation leg 3 ✓** (config carries operational:felhom-op-1 + recovery:felhom-rec-1);
- pool member + all 4 scoped-ACL grants (user+token) present; **caps on the guest: cores 4 / memory 8192**;
- byo config asserts pass (lan_resolver/wg_tunnel/oob all off);
- **controller Up (healthy) 0.103.0** + traefik + cloudflared up; dashboard `HTTP 200` (Host-header probe); hub config-retrieve OK.

### Findings — the byo install's resume path is fragile (happy single-shot path is clean)

- **GL6-F3 (MED, GL-7-relevant):** on a BYO host carrying the OWNER's (non-Felhom) guests, a FRESH
  install trips the populated-host leaf guard (`pct list | grep -q .` counts ANY guest) and dies
  demanding `--preserve-state-from`/`--allow-new-leaf` — even though no Felhom controller pin exists
  to protect on a fresh install. **Peti's box has his own guests → his day-0 command MUST include
  `--allow-new-leaf`** (GL-7). Better fix candidate: narrow the guard to `felhom_guests` so a fresh
  byo install on a populated non-Felhom host doesn't trip.
- **GL6-F4 (HIGH, resume-correctness):** `--resume` fully skips completed steps via
  `should_skip X || step_X`, but later steps consume the skipped steps' IN-MEMORY outputs, which are
  then empty → broken results: **(a)** `enroll` skipped → `HOST_ID`/`HOST_API_KEY` empty →
  `step_agent_config` writes a config missing `hub.host_id` (+ empty `proxmox.token` when `token` is
  skipped) → **daemon crash-loops** (`hub.host_id is required`); the mint-once token secret is
  unrecoverable, so recovery = re-run `token`+`enroll`. **(b)** `grows` skipped →
  `ROOTFS_GROW`/… empty → provision passes `-rootfs-grow ""` → **flag parse error**. Note
  `step_token` even has an internal resume-guard (`[[ -n "$PVE_TOKEN" ]]`) that the `|| step_token`
  dispatch DEFEATS — strong signal the dispatch is the bug (steps whose outputs feed later steps must
  re-run, or re-derive from the written config, on resume). **Drill recovery:** dropped the affected
  completed marks from `state.json` to force re-run — the install then completed cleanly. The
  single-shot happy path (no interruption) is unaffected, which is Peti's path.
- **GL6-F2 (from Phase 1, MED):** the agent's per-drive `mnt-*.mount` units + primary `/mnt/<name>`
  mounts survived uninstall #1; on this reinstall the parent bind `/mnt/felhom-drives` is back but the
  drives are NOT yet bound under it — re-enrollment is the Phase-2b UI leg.

### F4 fixed mid-drill (Viktor: HOLD → fix now) — host-install v1.11.3 (`98bf5a43`)

Root cause confirmed: `should_skip X || step_X` fully skips a completed step on `--resume`, but
token/enroll/grows are PRODUCERS whose in-memory outputs later steps consume; `step_token` even had
an internal resume-guard the `|| step_token` dispatch DEFEATED. Fix: token/enroll/grows run every
pass (all idempotent — reuse-or-rotate / mint-once-reuse POST / pure recompute); golden's
`GOLDEN_VOLID` re-derived from the local archive in the resume block. Harness +GL6-F4 invariant,
27/27, shellcheck clean.

**Live proof (v1.11.3, crafted `--resume --dry-run` on felhom-pve, throwaway `FELHOM_INSTALL_STATE_DIR`,
zero mutations):** marks = the exact pre-crash set `[preflight,token,grows,enroll]` (agent_config
NOT done). Transcript: steps **2/8 token, 3/8 grows, 4/8 enroll all RE-RUN**; the agent_config write
shows **`host_id=<dry-run-host-id>` (non-empty)** and provision shows **`-rootfs-grow 0 -datavol-grow
184 -sysdata-grow 42`** — both the exact values that were empty (→ crash-loop / flag parse error)
pre-fix. Real 9201 + its state.json untouched (verified: daemon active, controller Up healthy).
**GL6-F4 CLOSED + live-validated.** (GL6-F3 stays a GL-7 command requirement `--allow-new-leaf`;
guard-narrowing to `felhom_guests` recorded as a candidate. GL6-F1/F2 remain v1.11.x candidates.)

### Phase 2b — bring-to-life: BLOCKED on public DNS (GL6-F5), browser bridge confirmed working

- **Browser bridge: WORKS** — created a tab group, navigated to the dashboard; the failure below is
  the SITE, not the bridge.
- **GL6-F5 (onboarding, Viktor/Cloudflare):** after the byo reinstall, `felhom.demo-felhom.eu` does
  NOT resolve publicly — `curl` from the box (outbound internet) returns `Could not resolve host`.
  The cloudflared tunnel is UP (QUIC + HTTP/2 to argotunnel, precheck all pass, same tunnel token),
  and the dashboard is healthy IN-GUEST (`HTTP 200`, "Beállítás" setup page) — so the controller +
  tunnel are fine; the missing piece is the public DNS CNAME `felhom.demo-felhom.eu → <tunnel>.cfargotunnel.com`.
  The controller does not auto-(re)create the dashboard's own DNS record on reinstall. **This blocks
  the 2b real-pipeline rehearsal** (password set + drive re-enroll need the dashboard reachable in a
  browser). NOT an install-correctness defect — the install is green; it is a Cloudflare-zone /
  onboarding gap that also matters for Peti (GL-7 / G9).

### Phase 2b — bring-to-life (browser, CC-driven) — drive re-enroll + data-visible PASS

DNS unblock: Viktor added a Pi-hole split-horizon A record `felhom.demo-felhom.eu → 192.168.0.105`
(LAN path to the guest); dashboard then loaded in-browser. (The public Cloudflare path is ALSO
correct — forced-DoH resolution returned HTTP 200 with the real page; only the client-side resolver
via the box's broken dnsmasq (F6) was failing.)

- **Dashboard live:** "Demo Ügyfél", 3 running / 56 total apps, mem 1.1/8 GB, backup "Még nem futott".
- **Drive re-enrollment (CC-driven UI):** Storage → Meghajtók showed all 3 drives detected-but-
  UNREGISTERED (the agent's registration state was wiped in uninstall #1; the drives + data survived
  per F2, tagged **"Adatot tartalmaz"**). Clicked **Regisztrálás** on `felhom-usb`
  (uuid `da9e7089…`, /dev/sdd1) — it moved to **Adattárolók → Aktív**, ext4 /dev/sdd1[/felhom-data]
  HD710 PRO, additive (no format). **Data-visible PASS:** the witness file is now readable in the
  guest at `/mnt/felhom-drives/felhom-usb/userdata/actualbudget/canary_hdd_actualbudget.txt`,
  hash `7b00bc35…` == the Phase-0 value. Additive doctrine proven end-to-end through the real UI.
- **GL6-F6 (MED, byo coexistence):** the box runs a leftover **broken dnsmasq** (active, bound to
  `0.0.0.0:53`, but queries time out — empty `/etc/dnsmasq.d/`, no upstream) on a byo host where
  `lan_resolver` is OFF. It breaks DNS for LAN clients that use the box as a resolver (it broke the
  drill's browser resolution). Fix pile with F1/F2: byo install/uninstall should stop+mask the
  leftover dnsmasq. Immediate drill unblock was the Pi-hole entry.
- **G9/G10 — password is OPERATOR-set, currently UNSET → dashboard OPEN.** Settings → Biztonság:
  *"A jelszavas védelem nincs beállítva. Kérd az üzemeltetőt a beállításhoz."* The dashboard password
  comes through the real config pipeline (hub customer-config → controller.yaml bcrypt — the anti-F9
  path), NOT a customer-set field here. Until the operator sets it the public-reachable dashboard is
  UNAUTHENTICATED (the known G10 exposure, now confirmed live). Geo-restriction also OFF. **Operator
  leg pending:** Viktor sets the dashboard password via the hub, then CC verifies login is enforced.

## Gate 2 — [Viktor]: **GO** — Peti's byo path live-proven end to end

Green: anonymous fetch, sha-verified 0.76.0 + 0.103.0, signers ARMED from first boot, caps applied,
no root@pam touched, controller healthy, drive re-enrolled additively with pre-drill data intact.
F4 found + fixed (v1.11.3) + live-proven. **Deferred to GL-7 with Peti:** the dashboard password
(operator-set via hub — G9) + geo-restriction; **G10 (open dashboard until the operator sets a
password) is now a GL-7 onboarding prerequisite.** Findings F1/F2/F3/F6 → post-drill installer pile.

## Phase 3 — armed key end-to-end (the only leg that needed real keys)

**PASS — the armed-key chain proven both directions with the REAL operational key.**

- 3a: drill agent **0.76.1** published (sha `ff18972f…aed16c`), version-only bump, throwaway.
- **Delivery mechanism (deviation from the [CC] tag, recorded):** the hub pod is a minimal image
  (no sqlite3/python — only the felhom-hub binary) and the admin-enqueue API is GLOBAL-key gated, so
  CC cannot inject the signed job. `felhom-opsign` has a built-in `-upload`+`-hub-key`, so the sign +
  enqueue is ONE operator command (Viktor's secrets stay in it). The `-hub-key` is the hub's
  **`report_api_key`** (live ConfigMap == committed `manifests/hub.yaml:77`; the operator/global key,
  NOT the web-UI login) — a point of confusion worth a GL-7 note.
- 3b/3c FORWARD (0.76.0→0.76.1): `felhom-opsign -op agent_update -host demo-felhom-01 -key-id
  felhom-op-1 -agent-version 0.76.1 -sha256 ff18972f… -key <op-priv> -upload https://hub.felhom.eu
  -hub-key <report_api_key>`. Agent (restart-forced poll) → audit `gate decision class=agent_update
  disposition=destructive allowed=true reason=signed key_id=felhom-op-1 nonce=4176e8e0…` → download
  (anon Gitea) → sha re-verify → `felhom-selfupdate-guarded apply` A/B flip → `new version running
  0.76.1 prev=0.76.0 dwell=1m0s` → `update committed version=0.76.1`. `.prev`=0.76.0 retained;
  crash-guard silent.
- RETURN (0.76.1→0.76.0): same with the 0.76.0 published sha `9828c5f7…` + a FRESH nonce → committed
  back to **0.76.0** (the manifest-vouched version; sha confirmed). Double round-trip = sign→verify→
  nonce→swap proven twice with the real key.
- 3d: 0.76.1 package DELETED (HTTP 204 → 404 anon). Box ends at 0.76.0, daemon active.

**Key validation leg 4 (the whole point of GL-1's key-pin): the pinned operational key actually
authorizes a real signed op end-to-end.** Combined with legs 1–3 (box==repo==manager fingerprints,
armed-from-first-boot), the operator-signed self-update path is fully live-validated.

## Gate 3 — [Viktor]: **GO** — armed-key chain proven both directions.

## Phase 4 — drift-heal, live (the R2 fix earning its keep) — PASS, 1 finding

- **4a induce drift:** `pveum pool modify felhom --vms 9201 --delete 1` → pool members `[9201]` → `[]`;
  the agent selftest immediately reflected it (`pool read: pool "felhom", 0 member(s)`) — the exact
  silent drift R2 addresses.
- **GL6-F7 (MED, new):** the R2 `PoolAddVMID` re-assertion is **bring-up-only** (`bringup.go:498`,
  after restore-over-existing) — there is NO continuous pool-membership reconcile (the stale-lock
  reaper READS membership, `stalelock.go:183`, but never repairs). Confirmed live: 2 min after the
  removal, no re-assert, pool stayed `[]`. So out-of-band membership drift does NOT auto-heal; it
  needs a bring-up or `--adopt-pool`. (R2 itself DID fire correctly at this drill's provision —
  `pool membership re-asserted vmid=9201` — so the fix works for its designed path.) Candidate:
  a periodic membership reconcile; acceptable for the pilot (verify catches it, --adopt-pool fixes).
- **4b C7 — the verify assert firing on real drift (read-only):** installer `--resume` re-ran
  `step_verify` → **`[ERROR] pool: guest 9201 NOT in pool felhom (the scoped token cannot manage it —
  adopt with --adopt-pool)`** → `Day-0 provision completed WITH WARNINGS`. The ACL grants + `authz
  signers: 2` stayed green (only membership was perturbed). C7 (GL-2's structural assert catching a
  real silent drop) is now LIVE-VALIDATED — closes the GL-2 "not testable pre-drill" gap.
- **Repair:** `--adopt-pool --vmid 9201` re-added it (pool `[9201]`), re-verify → **SUCCESS**. Demo
  healthy. (Also incidentally re-proved the v1.11.3 F4 fix: the `--resume` re-ran token/enroll/grows
  cleanly and reached verify — no crash-loop, no flag error.)

## Gate 4 — [Viktor]: **GO** — drift-heal proven (C7 live, F7 recorded, repaired).

## Phase 5 — uninstall #2 (the byo box's clean exit) — PASS

Real `--uninstall --vmid 9201` (typed confirm), no busy prop this time.
- `felhom-usb` unmounted cleanly ("data stays on the drive"); statement reads **"the drives are
  physically removable now"** (no busy exception this run) — the KEPT-vs-WIPED statement printed as
  designed; UNINSTALL complete.
- **Host audit CLEAN:** no felhom-agent unit/user/binary/selfupdate-wrapper/A-B-slots/rollback-unit/
  limits-drop-in/watchdog/mkfs-wrapper/hook-snippet/dnsmasq-drop-ins; pveum user 0, roles empty,
  felhom pool gone; `/var/lib/felhom-agent`, `/var/lib/felhom-install`, `/etc/felhom-agent` all gone.
  (F1 did NOT recur — this install's config had no `.bak` siblings.)
- **Drive DATA intact:** witness `7b00bc35…` == Phase-0 (the data-sovereignty exit, proven a 2nd time).
- Recurring known findings: **F2** (per-drive `mnt-*.mount` units survive), **F6** (leftover dnsmasq
  still active). Trivial demo-box note: 6 ancient `felhom-agent.v0.1x.0.bak` binaries (Jun 10–11,
  legacy naming) survive the `.bak-*` glob — historical cruft, not on a real customer box; a fix
  could widen the glob to `.v*.bak`.

## Gate 5 — [Viktor]: **GO** — clean byo exit, data intact 2nd time.

**6b — return to normal.**
- **Drive re-enrolled (CC-driven UI):** `felhom-usb` → Adattárolók **Aktív**, ext4 /dev/sdd1
  [/felhom-data]; **witness `7b00bc35…` intact** in-guest (additive, 3rd data-intact proof).
  (First click collided with the in-progress snapshot backup and no-op'd; retry after the backup
  registered it — minor UI race worth a note. Only felhom-usb re-enrolled; flash/teszt left for
  Viktor's demo curation.)
- **Manual backup + restore-test GREEN:** `selftest=backup OK (crash-consistent=true)` →
  restore-test **`mount_parity: ok`** (29s — small: fresh install, no apps/drive-data yet).
- Controller + traefik + cloudflared + filebrowser all Up healthy; dashboard HTTP 200; hub reporting.
  **Observation:** during the snapshot vzdump the dashboard briefly flashed "Protected container not
  running: traefik/cloudflared/filebrowser" — a transient false-positive of the container monitor
  during the backup freeze (cosmetic; cleared when the backup finished).
- **Viktor's ongoing return-to-normal (not drill blockers):** redeploy the demo apps (his Phase-0
  choice: fresh redeploy); set the dashboard password (deferred to GL-7); update/align LAN DNS
  (the re-provision changed the guest IP .105→.139 — a manual Pi-hole entry goes stale; rely on the
  appliance lan_resolver or a reserved DHCP lease — GL-7 note).


## Phase 6 — appliance-mode reinstall (the demo returns to normal life)

**6a — appliance install: SUCCESS, single-shot.** `--mode appliance … --allow-new-leaf`.
- **4b/8 break-glass RAN** (appliance): `root@pam password set + vaulted to the hub` + break-glass
  layers 1+2 installed — the recovery credential is in the hub (operator-retrievable). (SSH key
  access unaffected.)
- verify green: pool member, all 4 scoped-ACL grants (user+token), **`authz signers: 2`**, controller
  Up healthy 0.103.0, Day-0 provision SUCCESS.
- **Bonus: this ran in ONE shot** (vs the byo install's 4 resumes on v1.11.2) — the **v1.11.3 F4 fix
  live-proven on a real end-to-end install**, not just the dry proof.


*(Phases 3–6 appended as they run.)*

## Gate 6 — [Viktor]: **GO — DRILL COMPLETE.**

Demo healthy on normal life. Two full destroy/rebuild cycles proven (byo #2, appliance #6), both
uninstalls clean, drive data intact ALL THREE checks, armed-key chain proven, C7 live-validated.

## Summary — what GL-6 validated

**Key-pin chain (GL-1's whole point) — legs 1–4 all green:** box==repo==password-manager fingerprints
(op `7YqN…`, rec `/ixg…`); signers ARMED from first boot on both byo + appliance installs; the pinned
operational key AUTHORIZED a real signed `agent_update` op end-to-end (round-trip 0.76.0→0.76.1→0.76.0
with the real key, verify vs pinned signer + nonce, A/B swap, crash-guard silent).

**Go-live gates now LIVE-VALIDATED on real hardware:** G2 (byo profile — disclosure+ack, no root@pam,
config asserts), G4 (preflight guards + verify pool/ACL asserts incl. **C7 firing on real drift**),
G5 (mandatory byo caps applied to the guest), G6 (uninstall gap-closure — KEPT/WIPED statement,
plain-umount busy-drive handling, self-update-artifact removal, **data intact ×3**), G7 (this drill).
**G1 DONE** (published chain consumed end-to-end via anonymous fetch — the v1.11.2 Gate-0 ruling —
+ armed keys proven). **G12** re-exercised (restore-test `mount_parity: ok` on PBS 5m35s + local tiers).
**G9** rehearsed (dashboard password is operator-set-via-hub; location confirmed) → **deferred to GL-7**;
**G10** (open dashboard until the operator sets a password) → GL-7 prerequisite.

## Findings ledger (post-drill installer pile unless noted)

| ID | Sev | What | Status |
|----|-----|------|--------|
| **F4** | HIGH | `--resume` skipped producer steps → config missing hub.host_id/proxmox.token (daemon crash-loop) + provision `-rootfs-grow ""` | **FIXED v1.11.3, live-proven** (dry + the single-shot appliance install) |
| F1 | MED | `/etc/felhom-agent/` + `agent.json.bak-*` (secret-bearing) survive uninstall | **FIXED v1.12.0 (GL-8)** — uninstall purges `${agent_cfg}.bak*` |
| F2 | MED | per-drive `mnt-*.mount` units + `/mnt/<name>` primary mounts survive uninstall ("physically removable" overclaims) | open — uninstall stop+remove the agent's `mnt-*` units |
| F3 | MED | byo FRESH install on a host with the OWNER's guests trips the populated-host leaf guard (`pct list`), needs `--allow-new-leaf` | **FIXED v1.12.0 (GL-8)** — guard narrowed to `felhom_guests` |
| F5 | — | initial "public dashboard DNS absent" — **MISDIAGNOSIS**: the `*.demo-felhom.eu` wildcard works (HTTP 200 via DoH); real cause was F6 | superseded by F6 |
| F6 | MED | leftover **broken dnsmasq** (active on `:53`) on a byo host breaks LAN-client DNS | **FIXED v1.12.0 (GL-8)** — byo preflight REFUSES a foreign `:53` + instructs (no mutation; Viktor ruling); live-proven |
| F7 | MED | R2 pool re-assertion is **bring-up-only** (`bringup.go:498`); no continuous membership reconcile → out-of-band drift doesn't auto-heal (repair via `--adopt-pool`) | open — candidate periodic membership reconcile |
| — | LOW | re-provision changes the guest DHCP IP (.105→.139) → manual LAN DNS entries go stale | GL-7 note (reserve lease / use lan_resolver) |
| — | LOW | legacy `felhom-agent.v*.bak` binaries survive the uninstall `.bak-*` glob | trivial (demo-box history only) |
| — | LOW | snapshot-mode backup briefly flashes "Protected container not running" (monitor false-positive during freeze) | cosmetic |

**Honest leftovers (unchanged, still open):** S5 customer-data DR drill; G12 parity-on-real-drift +
PBS-tier runtime characterization; CGNAT true-sim retest; Impl-3 shared-box format gate; OOB spike.

## Drill deviations (recorded)

1. **v1.11.2 shipped mid-drill (Gate-0 ruling):** empty customer git creds → anonymous artifact fetch
   (the artifacts are world-readable by ruling; sha256 stays the integrity root). All 12 fetch targets
   200-anon-validated first.
2. **F4 fixed mid-drill (v1.11.3, Viktor HOLD):** the resume-correctness bug, live-proven fixed.
3. **Phase-3 delivery is operator-driven, not [CC]:** the hub pod has no direct-insert tool and the
   admin-enqueue API is global-key-gated, so Viktor signs+enqueues in one `felhom-opsign -upload` command
   (`-hub-key` = the hub `report_api_key`, not the web login — a GL-7 clarity note).
4. **G9 password + full app redeploy deferred to GL-7** (operator/customer curation).