felhom.eu/documentation/pilot/DRILL-GL6-2026-07-08.md

# DRILL GL-6 — supervised go-live drill (demo box, two full cycles) — 2026-07-08

> Go-live item **G7** + the live-validation landing zone for GL-1…GL-5b deferrals. Destructive-
> supervised: guest 9201 destroyed + rebuilt twice; Viktor present, gating every phase. This record
> is filled per phase as evidence lands; timings + deviations verbatim. No secrets.

**Inputs verified at start:** felhom.eu `bd97bc34` (host-install **v1.11.1**, operator pubkeys
pinned) · felhom-agent `7f073936` (v0.76.0) · hub Day-0 manifest vouches agent
`0.76.0`/`9828c5f7…f50b` + golden `0.103.0`/`8481e8a1…6026e` (fetched via the box-facing API).

## Phase 0 — preflight + key validation (nothing destroyed)

| Step | Result |
|---|---|
| 0a baselines | **PASS** — repo heads + `SCRIPT_VERSION="1.11.1"` + manifest all as stated above |
| 0b key validation leg 1 | **PASS** — all four fingerprints EQUAL (below) |
| 0c key validation leg 2 | [Viktor] — pending at Gate 0 |
| 0d abort-path backup + restore-test | **PASS** (local + PBS backups fresh; PBS restore-test parity ok, 5m35s) |
| 0e busy-drive prop | **STAGED** — `gl6-busyprop` unit cwd-pinned into `/mnt/felhom-drives/felhom-usb` (4h) |
| 0f mechanism re-read | **DONE** — Phase-3 delivery pinned (below) |

### 0b — the four fingerprints, side by side

```
repo op  (felhom-op-1):  SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw
box  op  (felhom-op-1):  SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw
repo rec (felhom-rec-1): SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k
box  rec (felhom-rec-1): SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k
```

Box source: felhom-pve agent config (`systemctl cat` → `-config` path) `authz.signers[]`;
repo source: the v1.11.1 `OPERATOR_KEY_*_LINE` constants. Both match the runbook's stated values.

### 0d — abort-path backups (the session's safety net)

- **Local (agent pipeline):** `--selftest=backup -vmid 9201` → snapshot-mode, crash-consistent,
  `local:backup/vzdump-lxc-9201-2026_07_08-11_43_42.tar.zst` (6.82 GB, 5m52s). Expected
  `uncovered_volumes`: `/etc/felhom-bootstrap` + `/mnt/felhom-drives` (bind mounts — regenerated /
  re-enrolled, per doctrine).
- **PBS (offsite tier, PVE-native vzdump per the S4-proven path):** `felhom-pbs:backup/ct/9201/2026-07-08T09:51:00Z` — 3m04s (dedup)
  — NOTE: the live config deliberately pins `local_backup_target: local`; the PBS leg was driven
  with `vzdump 9201 --storage felhom-pbs --mode snapshot` (tunnel healthy, storage active).
- **Restore-test vs the fresh PBS archive:** **PASS** — `mount_parity: ok`, inventory
  `mp0=/var/lib/docker (200G)`, `mp1=/mnt/sys_drive (50G)`, mp8/mp9 throwaways; scratch 990000
  restored+booted+verified+torn-down in **5m35s** (first G12 PBS-tier runtime datum).
  Observation (cosmetic): the record's `source_tier` says "local" — the tier label derives from the
  CONFIG backup target, not the explicit `-archive`'s storage; display-only.

### 0e — busy prop + data-intact witness

Prop: transient unit `gl6-busyprop`, cwd inside `/mnt/felhom-drives/felhom-usb`.
Witness file (for the Phase-1/5 drive-data-intact checks):
`userdata/actualbudget/canary_hdd_actualbudget.txt` on `/dev/sdd1`
(UUID `da9e7089-cf8e-4617-adcb-a377743fae00`), sha256 prefix `7b00bc3550b10e3952e0e0965f5a2628`.

### 0f — Phase-3 signed-update mechanism (confirmed at source, NOT improvised)

1. [Viktor] sign: `felhom-opsign -op agent_update -host <host_id> -key-id felhom-op-1
   -agent-version <v> -sha256 <hex> -key <operational private key> [-ttl 30m]` → signed-op blob.
2. Deliver: `POST /api/v1/admin/hosts/{host_id}/jobs` `{job_id, blob_b64}` — **hub GLOBAL key
   required** (Viktor), or the v0.70.0-precedent direct `signed_jobs` sqlite insert on the hub pod
   (CC via kubectl on 180). The agent's heartbeat sees `has_signed_ops`, fetches, verifies against
   the PINNED operational key + durable nonce store, downloads the binary from Gitea, re-verifies
   the SIGNED sha (the only integrity root), then `felhom-selfupdate-guarded apply` A/B-swaps with
   crash-rollback armed; dwell → `commit`.

## Gate 0 — [Viktor]

- Input 1 (byo caps): **--cores 4 --memory 8192**
- Input 2 (Phase-6 repopulation): **fresh + redeploy apps**
- 0c fingerprint confirmation vs password manager: **CONFIRMED — match**
- Verdict: **GO** — with one pre-Phase-1 STOP + ruling (below)

### Gate-0 deviation — the demo customer's empty git credentials (Viktor ruling → v1.11.2)

Pre-destroy check found demo-felhom's `git.username`/`git.token` still EMPTY (the GL-2 finding);
the v1.11.1 installer would die at step 5/8. **Viktor ruled: the artifacts are world-readable by
design — anonymous fetch is the supported path.** All 12 installer fetch targets (2 packages + 10
raw configs) validated HTTP 200 anonymously, then **host-install v1.11.2** shipped mid-drill
(felhom.eu `0640aa06`): empty creds → loud WARN + anonymous fetch (conditional curl auth; the
hub-vouched sha256 stays the integrity root; configured creds still used when present). Harness
26/26 (new GL6-ANON case; GL4-C2 assertion updated for the pinned constants).

## Phase 1 — uninstall #1 (full mode, busy drive staged) — PASS with 2 findings

Real `--uninstall --vmid 9201` (typed confirm, v1.11.2 fetched from the served URL):

- **Busy-drive behavior (the staged prop on felhom-usb):** WARN + "Eject the drive via the
  dashboard, or stop the apps using it and re-run" — NO `-l`/`-f`, teardown CONTINUED; the other
  two drives unmounted ("data stays on the drive"); the root bind correctly left mounted with its
  own guard warn. The KEPT-vs-WIPED statement listed felhom-usb as
  "still mounted (busy — stop the apps and retry)" — i.e. the code takes the
  statement-lists-it-as-retry path; after releasing the prop a plain manual `umount` completed the
  job (no re-run needed).
- **KEPT-vs-WIPED statement:** printed verbatim as designed (guest+volumes / agent / self-update
  artifacts / watchdog+OOB / pveum+pool / state = WIPED; drives+data / PBS backups / hub record /
  escrow = KEPT). The recovery-credential KEPT line correctly keyed off the state file — which on
  THIS box predates the 4b state marker, so the line was absent although a G1-vaulted credential
  exists (observation: state-keyed conditionals can under-report on pre-v1.10.0 installs).
- **1b audit:** agent binary+baks, selfupdate wrapper+slots, rollback unit+drop-in, watchdog set,
  mkfs wrapper, snippets, dnsmasq drop-ins, sudoers, state dirs, pveum user/roles/ACL/pool — ALL
  GONE. PVE printed a benign "ignore invalid acl role 'FelhomAgentStore'" notice during ACL
  removal (ordering noise, recorded).
- **Drive data intact:** witness file re-hashed on the still-mounted primary path —
  `7b00bc3550b10e3952e0e0965f5a2628` == the Phase-0 value.

### Findings

- **GL6-F1 (MED, security-adjacent):** `/etc/felhom-agent/` SURVIVES the uninstall holding
  `agent.json.bak-*` files (incl. the operator's same-day ceremony backup) — old configs with
  secret material (pve token now dead; the per-host hub api_key is still live because the hub
  record survives). The uninstall removes `agent.json` but not `.bak-*` siblings. Fix candidate:
  purge `"$agent_cfg"*` / the dir in 4b (v1.11.3). Files left in place pending Viktor's call
  (one is his own backup).
- **GL6-F2 (MED):** the agent-created per-drive systemd mount units
  (`mnt-felhom-usb.mount` + flash + teszt_enroll) and their ACTIVE primary mounts at
  `/mnt/<name>` survive — the statement's "physically removable now" overclaims (the binds are
  gone but the partition is still mounted). Fix candidate: uninstall stops+removes the agent's
  `mnt-*` units for enrolled namespaces (data untouched). Left as-is for Phase 2 (the re-enroll
  behavior against pre-existing mounts is itself drill-relevant).

## Gate 1 — [Viktor]: **GO** (delete the .bak secrets now)

GL6-F1 residue resolved at the gate: `shred -u /etc/felhom-agent/agent.json.bak-*` + `rmdir`
(incl. Viktor's ceremony backup, confirmed disposable). F1/F2 kept as v1.11.3 fix candidates.

## Phase 2 — byo-mode day-0 (Peti's exact path, on real hardware) — PASS (happy path), 3 findings

Command (Viktor typed the `demo-felhom` hostname ack — interactive consent leg):
`./felhom-host-install.sh --customer-id demo-felhom --mode byo --vmid 9201 --cores 4 --memory 8192
--force-gitea-golden --acl-storages "local local-lvm" --passphrase-file <0600>` (+ `--allow-new-leaf
--resume` after F3/F4, see below).

**End state — all green (verify step, verbatim highlights):**
- disclosure printed; **Viktor's typed-hostname ack accepted** (`byo install acknowledged for host demo-felhom`);
- **NO 4b / chpasswd / recovery-credential** anywhere (grep of the full transcript = 0);
- **anonymous artifact fetch worked** (v1.11.2): agent v0.76.0 sha `9828c5f7…` verified, golden
  0.103.0 sha `8481e8a1…` verified, both against the hub manifest;
- installed agent `--version` == **0.76.0**; daemon **active**;
- **`authz signers: 2 (operator-signed self-update armed)`** from first boot — **key validation leg 3 ✓** (config carries operational:felhom-op-1 + recovery:felhom-rec-1);
- pool member + all 4 scoped-ACL grants (user+token) present; **caps on the guest: cores 4 / memory 8192**;
- byo config asserts pass (lan_resolver/wg_tunnel/oob all off);
- **controller Up (healthy) 0.103.0** + traefik + cloudflared up; dashboard `HTTP 200` (Host-header probe); hub config-retrieve OK.

### Findings — the byo install's resume path is fragile (happy single-shot path is clean)

- **GL6-F3 (MED, GL-7-relevant):** on a BYO host carrying the OWNER's (non-Felhom) guests, a FRESH
  install trips the populated-host leaf guard (`pct list | grep -q .` counts ANY guest) and dies
  demanding `--preserve-state-from`/`--allow-new-leaf` — even though no Felhom controller pin exists
  to protect on a fresh install. **Peti's box has his own guests → his day-0 command MUST include
  `--allow-new-leaf`** (GL-7). Better fix candidate: narrow the guard to `felhom_guests` so a fresh
  byo install on a populated non-Felhom host doesn't trip.
- **GL6-F4 (HIGH, resume-correctness):** `--resume` fully skips completed steps via
  `should_skip X || step_X`, but later steps consume the skipped steps' IN-MEMORY outputs, which are
  then empty → broken results: **(a)** `enroll` skipped → `HOST_ID`/`HOST_API_KEY` empty →
  `step_agent_config` writes a config missing `hub.host_id` (+ empty `proxmox.token` when `token` is
  skipped) → **daemon crash-loops** (`hub.host_id is required`); the mint-once token secret is
  unrecoverable, so recovery = re-run `token`+`enroll`. **(b)** `grows` skipped →
  `ROOTFS_GROW`/… empty → provision passes `-rootfs-grow ""` → **flag parse error**. Note
  `step_token` even has an internal resume-guard (`[[ -n "$PVE_TOKEN" ]]`) that the `|| step_token`
  dispatch DEFEATS — strong signal the dispatch is the bug (steps whose outputs feed later steps must
  re-run, or re-derive from the written config, on resume). **Drill recovery:** dropped the affected
  completed marks from `state.json` to force re-run — the install then completed cleanly. The
  single-shot happy path (no interruption) is unaffected, which is Peti's path.
- **GL6-F2 (from Phase 1, MED):** the agent's per-drive `mnt-*.mount` units + primary `/mnt/<name>`
  mounts survived uninstall #1; on this reinstall the parent bind `/mnt/felhom-drives` is back but the
  drives are NOT yet bound under it — re-enrollment is the Phase-2b UI leg.

### F4 fixed mid-drill (Viktor: HOLD → fix now) — host-install v1.11.3 (`98bf5a43`)

Root cause confirmed: `should_skip X || step_X` fully skips a completed step on `--resume`, but
token/enroll/grows are PRODUCERS whose in-memory outputs later steps consume; `step_token` even had
an internal resume-guard the `|| step_token` dispatch DEFEATED. Fix: token/enroll/grows run every
pass (all idempotent — reuse-or-rotate / mint-once-reuse POST / pure recompute); golden's
`GOLDEN_VOLID` re-derived from the local archive in the resume block. Harness +GL6-F4 invariant,
27/27, shellcheck clean.

**Live proof (v1.11.3, crafted `--resume --dry-run` on felhom-pve, throwaway `FELHOM_INSTALL_STATE_DIR`,
zero mutations):** marks = the exact pre-crash set `[preflight,token,grows,enroll]` (agent_config
NOT done). Transcript: steps **2/8 token, 3/8 grows, 4/8 enroll all RE-RUN**; the agent_config write
shows **`host_id=<dry-run-host-id>` (non-empty)** and provision shows **`-rootfs-grow 0 -datavol-grow
184 -sysdata-grow 42`** — both the exact values that were empty (→ crash-loop / flag parse error)
pre-fix. Real 9201 + its state.json untouched (verified: daemon active, controller Up healthy).
**GL6-F4 CLOSED + live-validated.** (GL6-F3 stays a GL-7 command requirement `--allow-new-leaf`;
guard-narrowing to `felhom_guests` recorded as a candidate. GL6-F1/F2 remain v1.11.x candidates.)

## Gate 2 — [Viktor]: <pending — F4 fixed + proven; interactive 2b bring-to-life next (CC via browser)>

*(Phases 3–6 appended as they run.)*