189 lines
16 KiB
Markdown
189 lines
16 KiB
Markdown
# DRILL — Day-0 on the Demo-VM (nested PVE on felhom-pve), 2026-07-12
|
||
|
||
Executed per `RUNBOOK` (Day-0 drill on the Demo-VM). Actors: Viktor (console install, dry-run
|
||
go/no-go, fork decisions, the ceremony/R-moment, dialog clicks) + CC (everything scriptable over
|
||
SSH + the hub/dashboard browser tracks). **Outcome: the full arc ran to completion — Day-0
|
||
appliance install, first live escrow ceremony + AUTO-CONFIRM FIRST FIRING, offsite backup +
|
||
restore round-trip — at the cost of one mid-drill fork (PBS DR tier attached) forced by the
|
||
drill's headline findings (F-4, F-6).**
|
||
|
||
## 1. Baselines recorded (§1 of the runbook, verified LIVE before Phase 1)
|
||
|
||
| Item | Recorded |
|
||
|---|---|
|
||
| Day-0 artifact manifest | Agent **0.85.0** sha256 `31babb2c4f8fa5a1f961428519a0dfbb3a19b6e536fba01c2664afca339da93d`; Golden **0.120.0** sha256 `f7d7d02c76b49891719df9cf68624c9ae193e438e2a3a63b948a2a9d12887596`; MinAgent **0.81.0** — matches the 0.85/0.120 publish train, no bump needed |
|
||
| Global controller floor | Effective **v0.120.0**, source DB (hub_settings); env fallback also v0.120.0 |
|
||
| Customer `demo-vm-felhom` | exists, domain `enkisfelhom.hu`, CF tunnel + API tokens saved (API token perms incl. Zone WAF:Edit); git credentials EMPTY (correct per G3; hub edit-form already labels Git Sync "Opcionális" — only day0-install.md A.2 was stale, fixed in this commit); DEBUG MÓD was ON |
|
||
| Wildcard DNS | `*.enkisfelhom.hu` → CF proxied edge (104.21.3.175 / 172.67.130.252 + AAAA), verified via 1.1.1.1 on a real and a random subdomain |
|
||
| felhom-pve capacity | local-lvm thin pool 348.82g @ 7.47%; RAM available 12.9 GiB; nested virt Y |
|
||
| Installer | felhom-host-install.sh **v1.14.0** per `-h` (but see F-1) |
|
||
| Golden-baked controller | 0.120.0 (= floor; see 4.3 note) |
|
||
|
||
## 2. Per-phase gates
|
||
|
||
| Phase | Gate | Result | Evidence (abridged) |
|
||
|---|---|---|---|
|
||
| 0 — build VM | P0 | **PASS** | qm 300 `drill-day0` (8G/4c/host/250G thin); PVE **9.2.2** ISO 9.2-1; key SSH proven; snapshot `pre-day0-clean` 15:12:41. ISO detached + boot→scsi0 BEFORE the snapshot (clean config) |
|
||
| 1 — hub verify | — | **PASS** | §1 table above; nothing created/changed in the hub |
|
||
| 2 — box prereqs | P2 | **PASS** | single node; nested local-lvm **149.88 GiB** free (≥120 ✓ — note: a 250G VM disk yields ~150G nested pool after the installer's root/swap split); vmid 9201 absent; hub 302 / gitea 200 / felhom.eu 200 |
|
||
| 3 — install | P3 | **PASS** | dry-run reviewed by Viktor → GO → real run 15:25–15:28: **"Day-0 provision SUCCESS — vmid=9201 host_id=demo-vm-felhom-2482b0"**; both artifacts sha-verified vs the hub manifest; both expected ANONYMOUS-fetch warns; 4b break-glass vaulted; passphrase file 0600, shredded after (never in CC's transcript — Viktor wrote it himself) |
|
||
| 4 — post-install | D.1–D.4 | **PASS** (1 finding) | see §3 |
|
||
| 5 — hub tracks | G9/G10 | **PREMISE COLLAPSED** — F-4/F-5; geo exercised via the shipped path | see §4 |
|
||
| 6 — escrow/auto-confirm/offsite | fork-4 | **PASS after fork** (PBS DR attached) — F-6/F-7/F-10/F-11 | see §5 |
|
||
|
||
## 3. Phase 4 detail (post-install verification)
|
||
|
||
- **D.1:** agent active as non-root `felhom-agent`; full read-only selftest ALL-OK incl. pool read
|
||
(pool `felhom`, member 9201); guest running, onboot=1, mounts mp0 docker 50G / mp1 sys 20G /
|
||
mp8 felhom-drives / rootfs 32G; in-guest containers controller 0.120.0 (healthy) + filebrowser +
|
||
cloudflared + traefik; dashboard 200 in-guest (traefik https + Host header; :80 → 301).
|
||
- **4.2 appliance gating (first live appliance box):** agent.json `deployment_mode="appliance"`;
|
||
journal `selfheal: node watchdog starting mode=appliance interval_s=60` + storage watchdog
|
||
armed; felhom-mgmt-watchdog.timer enabled+active. NOT byo-defaulted.
|
||
- **4.3 floor self-update (tester-recruitment gate):** guest landed **0.120.0 = floor at its FIRST
|
||
report, zero manual steps, elapsed ≈ 0**. LIMITATION: baked == floor on this train, so the
|
||
floor-driven *update* path was not stressed — re-prove on the next train where golden < floor.
|
||
- **4.4 hub:** customer PENDING→ok ~2 min after provision; host ONLINE agent 0.85.0; guest 9201
|
||
appears at the next agent report (900 s cadence — the empty Guests panel in between is timing,
|
||
not a bug); only-degraded capabilities = the 3 `pbsdr-*` ("binary not found") — expected no-PBS
|
||
shape (and see F-7).
|
||
- **4.5 customer-visible via the REAL edge** (method: `curl --resolve` on both CF anycast IPs —
|
||
the split-horizon-proof variant): felhom.enkisfelhom.hu → 200 "Vezérlőpult", **OPEN, no auth**
|
||
(G10 "before", ~15:30).
|
||
- **4.6 first-app smoke:** ActualBudget via the open dashboard → "Telepítés sikeres"; real-edge
|
||
200 + `<title>Actual</title>` on budget.enkisfelhom.hu. (The old POST-via-public-URL no-op
|
||
gotcha did NOT reproduce on 0.120.0 for normal forms.)
|
||
- Snapshot `post-install` 15:35:37 (live, no fs-freeze — no qemu-guest-agent in the drill VM).
|
||
|
||
## 4. Phase 5 detail — the G9/G10 premise vs shipped code
|
||
|
||
- **5.1 operator password-set: IMPOSSIBLE (F-4).** No hub UI/API sets per-customer
|
||
`web.password_hash`; the controller's security page in the open state says "Kérd az
|
||
üzemeltetőt"; `settingsPasswordHandler` requires a current-password match (no initial-set);
|
||
the hub-preseeded Day-0 setup path skips the only wizard form having a password field.
|
||
- **5.2 geo-restriction:** apply lives in the CUSTOMER dashboard (controller
|
||
`settings_security.html` + `api/geo.go`), not the hub (hub has only `handleGeoDisable`) — F-5.
|
||
Exercised via the shipped path: HU-only enabled → controller created WAF rule
|
||
**"[felhom-geo] Global"** (`(not ip.src.country in {"HU"})`, action block) on the zone, sync
|
||
reported 1 active rule; HU access still 200 through the real edge. Non-HU block not testable
|
||
from an HU vantage.
|
||
- **5.3 G10 statement:** BEFORE — dashboard OPEN (proven in-guest + real-edge, ~15:30). AFTER —
|
||
**STILL OPEN; closure impossible until F-4 ships.** The tester-agreement onboarding gate that
|
||
points at this line CANNOT currently be satisfied.
|
||
|
||
## 5. Phase 6 detail — offsite → ceremony → auto-confirm → tier proof
|
||
|
||
Pre-ceremony state (all verified): offbox pre-provisioned at customer-create
|
||
(u629488-sub3@…your-storagebox.de:23, /home/felhom-repo, pinned host fingerprint, 0/50 GB);
|
||
controller `EscrowState=pending` — the /backups banner "a mentés addig nem fut" (the F6
|
||
no-single-copy guard, LIVE); staged `escrow-stage/restic_repo_password` on the agent (0600,
|
||
staged at provision — fork-4 stage-FIRST ordering held); ActualBudget toggled for NAS.
|
||
|
||
**The ceremony blocked → the drill's second headline (F-6):** `--selftest=escrow-create --upload`
|
||
refuses without a PBS storage/key; **identity-only mode does not exist in any shipped agent**
|
||
(the ≥0.80.0 claim in RUNBOOK-escrow-ceremony.md was false — fixed in this commit). On a no-PBS
|
||
appliance box the offsite arc can NEVER complete. **Fork decision (Viktor): attach the PBS DR
|
||
tier (ep0)** — which converted the drill into a full rehearsal of Peti's pending sequence:
|
||
|
||
1. Ship `configs/felhom-pbs-apply` → /usr/local/sbin (the binary is missing from host-install —
|
||
F-7; the FELHOM_PBSDR sudoers alias DOES ship). Agent restart → zero capability-DEGRADED.
|
||
2. Hub PBS-DR enable failed correctly: "host has not reported a WG key yet — the tunnel peer must
|
||
exist before the PBS DR tier" — the dependency is hub-enforced (good), and the drill runbook's
|
||
"no WG" scope was inconsistent with any PBS path.
|
||
3. WG: `wg_tunnel.enabled=true` in agent.json (installer never sets it — decide the appliance
|
||
default together with the F-6 spec) → keygen → **hands-free hub registration** (10.77.0.3/32,
|
||
gen 1, no operator vouch) → conf applied → handshake + ping 10.77.0.1 (33 ms).
|
||
4. Hub PBS-DR enable → "Configuration updated" → descriptor gen 2; apply-bridge first 403'd
|
||
(`Datastore.Allocate` on /storage/felhom-pbs) — consequence of the drill's narrowed
|
||
`--acl-storages "local local-lvm"`; the installer DEFAULT includes felhom-pbs exactly for
|
||
this. Retrofit dual-grant → next tick: **token consumed (single-use) → entry + K created →
|
||
`escrow.pbs_storage_id` seeded → state=applied** (16:17). pvesm ACTIVE; felhom-pbs.{enc,pw}
|
||
present.
|
||
5. **Ceremony** (Viktor, R on paper, nothing in CC's transcript): attempt 1 failed — `age`
|
||
missing (F-10, `apt-get install -y age` → 1.2.1); attempt 2 SUCCESS ~16:35; the bundle
|
||
auto-captured `+wg_private_key +restic_repo_password`; ceremony wiped the staged secret; hub
|
||
host page flipped to **DR RECIPE: present / KEY ESCROW: present**.
|
||
6. **AUTO-CONFIRM FIRST LIVE FIRING:** hands off, "Letét megerősítése" untouched → 16:42:36
|
||
controller log: *"hub-verified: the escrow covers the current repo password (hash
|
||
99c16e8d84e7…) — EscrowState auto-confirmed escrowed; offsite runs enabled."*
|
||
**Elapsed ≈ 7.5 min, zero clicks.** /backups gate banner gone.
|
||
7. **Tier proof:** "NAS-mentés most" → repo initialized on the Storage Box → 1 snapshot, 14 s,
|
||
"✓ Rendben". Restore-to-verify (native confirm() blocked automation twice and Viktor missed
|
||
the popup — F-11; also live operator confusion: he first launched the FULL local tier-1
|
||
restore, which itself completed healthy in 8.7 s) → offbox verification restore →
|
||
*"restored actualbudget → …/offbox-restore/actualbudget"*, 48K payload (app.yaml, .felhom.yml,
|
||
docker-compose.yml, manifest.json) decrypted from the box — **the offsite round-trip is
|
||
proven**. Bonus: the full local tier-1 restore was ALSO proven the same afternoon.
|
||
|
||
## 6. Headline metrics
|
||
|
||
| Metric | Value |
|
||
|---|---|
|
||
| Wall-clock Phase 2 → 6 complete | ~15:15 → ~17:09 (**~1 h 55 m**, including the mid-drill fork, 3 fix-and-continue stops, and two supervised STOPs) |
|
||
| Install run itself (Phase 3) | ~3 min |
|
||
| Floor self-update (4.3) | at-floor at FIRST report, 0 manual steps (baked == floor; update path not stressed) |
|
||
| **Auto-confirm (6.4)** | **~7.5 min ceremony→escrowed, zero manual clicks (FIRST LIVE FIRING)** |
|
||
| Offsite backup / restore | 14 s backup (1 snapshot) / verification restore round-trip proven |
|
||
|
||
## 7. Findings
|
||
|
||
| # | Sev | Phase | Finding | Disposition |
|
||
|---|---|---|---|---|
|
||
| F-1 | LOW | 3 | Version-string mismatches: `-h` v1.14.0 vs run banner v1.13.0 vs hub Setup-tab copy "1.12.0" | fix strings (installer + hub template) |
|
||
| F-2 | COSMETIC | 3 | dry-run prints `curl -u <git>` on the anonymous-fetch branch | fix placeholder |
|
||
| F-3 | MEDIUM | 4 | Root-run provision leaves `/var/lib/felhom-agent/guests{,/9201}` root:root 0700 inside the agent-owned state dir → non-root agent lanresolver "permission denied". LIVE-FIXED (chown the two parent dirs; the guest-root-owned bootstrap subtree untouched) | agent/installer: create parents agent-owned at provision; **check demo/felhom-pve for the same latent state** |
|
||
| **F-4** | **HIGH** | 5 | **No operator-set dashboard password path exists anywhere** (hub has no UI/API for per-customer `password_hash`; controller open-state page defers to the operator; Day-0 preseeded path skips the wizard's password form) → **G10 unclosable; every fresh box's dashboard stays OPEN on the internet** | **RESOLVED 2026-07-12 — customer-claim arc (controller v0.122.0 + hub v0.50.0).** The customer OWNS the password, set via a one-time claim code the hub emails to the registered address; an unclaimed box serves only the claim page (no open window). The operator-set premise was correctly abandoned. Live-proven on the drill box (gate ON via the real edge). |
|
||
| F-5 | MEDIUM | 5 | Geo-restriction APPLY is customer-dashboard-side; hub only disables. Runbook premise stale; also: the open dashboard (F-4) exposes the geo toggle unauthenticated | **RESOLVED with F-4** — the geo toggle (and every other route) is now behind the claim gate on an unclaimed box. The apply-is-customer-side design is accepted (documented). |
|
||
| **F-6** | **HIGH** | 6 | **Identity-only escrow ceremony was never implemented** (`escrow-create` hard-requires PBS storage + key; the ≥0.80.0 runbook claim was false) → on a no-PBS box (the documented appliance standard!) the offsite escrow chain can never complete | agent feature task (identity-only mode); ceremony runbook corrected in this commit |
|
||
| F-7 | MEDIUM | 6 | host-install ships the FELHOM_PBSDR sudoers alias but NOT the `felhom-pbs-apply` binary → pbsdr capabilities born DEGRADED on every fresh box | installer: ship the wrapper (like mkfs/selfupdate wrappers) |
|
||
| F-8 | LOW/UX | 3 | Step 4b rotates root@pam + vaults silently — operator surprised by 401 at the PVE GUI (live: Viktor) | installer: print "root@pam rotated + vaulted — retrieve at hub → host page" |
|
||
| F-9 | NOTE | 6 | Installer never sets `wg_tunnel.enabled`; WG registration itself is hands-free once enabled | decide appliance default alongside the F-6 spec |
|
||
| F-10 | MEDIUM | 6 | `age` (ceremony identity-wrap dependency) not installed by host-install — fresh-box ceremony dies; demo host masked it (spike-era install) | installer: add `age` to package set; runbook prereq added in this commit |
|
||
| F-11 | LOW | 6 | Native `confirm()` on the offbox restore-verify form: blocks browser automation, easy to miss (operator missed it twice live; meanwhile launched the full tier-1 restore from the adjacent form — the two restore controls invite confusion) | convert to the design-system inline confirm pattern; consider renaming |
|
||
|
||
Observations (not F-numbered): hub "REGISTRY LATEST v0.120.0 — up to date" vs the dashboard's own
|
||
"Új controller verzió elérhető: 0.121.0" banner disagree on "latest"; the hub dashboard row for
|
||
Peti shows "OK · minutes ago" (controller-derived, via his proxmox2 migration) while his agent
|
||
HOST is DOWN 23h — the roll-up masks a dead host; retrofit-ACL note: adding the PBS tier to a box
|
||
installed with narrowed `--acl-storages` needs the /storage/<id> dual-grant (documented default
|
||
avoids it).
|
||
|
||
## 8. What this proved for Peti — and what it did not
|
||
|
||
**Proven live on a fresh box (his exact pending sequence):** wrapper+age prep → WG enable →
|
||
hands-free peer registration → hub PBS-DR enable (dependency guard works) → apply-bridge fresh
|
||
path (consume → K → grant → escrow seed) → ceremony (with K) → **auto-confirm** → gated offsite
|
||
run → restore round-trip. Prep list for his box: `felhom-pbs-apply` + `age` + `wg_tunnel.enabled`
|
||
+ (if his ACL set was narrowed) the /storage dual-grant.
|
||
|
||
**Deliberately NOT covered:** WG OOB operator peer (separate arc); S5 DR restore from total loss
|
||
(identity-consume — queued as its own drill); byo-mode re-run from `pre-day0-clean` (queued);
|
||
non-HU geo-block verification (needs a non-HU vantage).
|
||
|
||
## 9. Snapshot inventory (qm 300 on felhom-pve, at drill end)
|
||
|
||
| Snapshot | When | State |
|
||
|---|---|---|
|
||
| `pre-day0-clean` | 15:12 | PVE 9.2.2 + keys, nothing Felhom — the universal re-drill zero |
|
||
| `post-install` | 15:35 | Day-0 SUCCESS + ActualBudget + guests-dir chown |
|
||
| `post-drill` | 17:11 | full drill end state (WG + PBS-DR + escrowed + geo HU + dashboard OPEN per F-4) |
|
||
| `post-claim-arc` | 19:08 | customer-claim arc: ctrl 0.122.0, demo-vm-felhom CLAIMED (customer-set password), gate proven |
|
||
|
||
## 10. F-4/F-5 resolution — customer-claim arc (2026-07-12, same day)
|
||
|
||
Shipped **hub v0.50.0 + controller v0.122.0** (both LIVE): the dashboard password is customer-owned,
|
||
set via a one-time claim code the hub emails to the registered address. An unclaimed box serves only
|
||
the claim page — F-4's open-dashboard exposure and F-5's unauthenticated geo toggle are both closed.
|
||
Live-proven on this drill box (gate ON via the real edge; Viktor claimed it, code generation consumed,
|
||
box now password-gated) and on the live demo after the floor raise (0.121→0.122 → claim-gated). Two
|
||
follow-up observations for the backlog: (a) the hub issues a claim code to EVERY reporting customer on
|
||
first report (peti + demo also emailed) — benign for pre-0.122 boxes (no gate, ACK ignored), but a
|
||
real customer on an old controller gets an unusable "beállító kód" email; consider gating issuance on
|
||
reported controller version ≥ 0.122. (b) the drill's reset flow (Scenario C) was deferred by the
|
||
operator — worth one supervised pass before Peti onboarding.
|
||
|
||
Blast radius honored: guest demo 9201 on felhom-pve, Peti's hub entry, and the demo offbox were
|
||
never touched. Drill leftovers to tidy at teardown (deliberately kept for re-drills now): hub
|
||
customer `demo-vm-felhom` (its api_key + CF tokens surfaced in the operator UI during the drill —
|
||
delete/rotate at teardown), DEBUG MÓD on, geo HU rule on the drill zone, VM 300 running.
|