DRILL-day0-vm 2026-07-12: full report + runbook corrections (F-4/F-6 headline findings)
Audit doc for the Day-0 VM drill: appliance install, floor-at-first-report, escrow ceremony + auto-confirm FIRST LIVE FIRING (~7.5 min, zero clicks), offsite backup + restore round-trip, PBS-DR/WG fork (Peti-sequence rehearsal). Corrects day0-install.md A.2 (git creds optional since v1.11.2, anonymous fetch is the normal shape) and RUNBOOK-escrow-ceremony.md (identity-only mode does NOT exist — F-6; age prereq — F-10). REPORT.md overwritten; CONTEXT.md one-liner added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NptTCFtu7dz2Ru89qHRagN
This commit is contained in:
@@ -2,19 +2,14 @@
|
||||
|
||||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||||
|
||||
## CAMPAIGN-3 — unattended "no mercy" night run (NAS · deploy · backup · chaos · observability) — 2026-07-11/12
|
||||
## DRILL — Day-0 on the Demo-VM (nested PVE) + escrow/auto-confirm first firing — 2026-07-12
|
||||
|
||||
**Full report: [`documentation/audits/CAMPAIGN-3-2026-07-11.md`](documentation/audits/CAMPAIGN-3-2026-07-11.md).** Run 22:09 → 04:27 CEST against the demo box (controller 0.117.0 / agent 0.84.0), operator-unattended, real surfaces only, no code changes. Ledger (final incl. morning RCA/recovery): **31 PASS · 17 FAIL · 12 FINDING · 1 DISCREPANCY** (70+ scenario entries, evidence at `180:~/campaign3/`).
|
||||
**Full report: [`documentation/audits/DRILL-day0-vm-2026-07-12.md`](documentation/audits/DRILL-day0-vm-2026-07-12.md).** Supervised drill (Viktor + CC), ~15:15–17:09 CEST, on a throwaway nested PVE 9.2.2 VM (qm 300 on felhom-pve) against hub customer `demo-vm-felhom` / `enkisfelhom.hu`. No code changes; two runbook docs corrected (day0-install.md A.2 anonymous-fetch wording; RUNBOOK-escrow-ceremony.md's false "identity-only ≥0.80.0" claim + `age` prereq).
|
||||
|
||||
### Headlines
|
||||
|
||||
- **CRITICAL F12 — the overnight host loss, RCA closed (hardware exonerated):** the agent's network-storage **automount** template (`After=`+`Wants=network-online.target`, implicitly `Before=local-fs.target`) creates a boot **ordering cycle**; systemd breaks it by deleting an arbitrary job. Boot at 23:31 sacrificed `networking.service` → host up **7 hours with no network**; the 06:45 power-cycle boot hit the same cycle and sacrificed the **automount** instead (NAS dead, healed manually). **Every boot of a host with an enrolled share is a coin flip until the template drops the network-online ordering** (`_netdev` on the `.mount` suffices). 4e caught exactly what it was designed to catch.
|
||||
- **CRITICAL F10 + HIGH F11/F9 — the reboot/recovery plane around NAS automounts is broken:** `mount-start-limit-hit` is never re-armed by any heal path (once even blocked guest start → guest DOWN); guest reboot with an idle share leaves the autofs trigger unpropagated into the container — the post-start reassert **logs its own WARNING and then skips** the automount restart that provably heals ("skip-active" branch). Every such reboot = 4 NAS apps dead-at-boot. Reproduced on 3 of 3 guest reboots.
|
||||
- **HIGH F7 — backup dumps are written in place (no tmp+rename):** a mid-backup NAS cut left a 0-byte tar *replacing* the last good 247M dump; in that window restore = empty volume. Next run self-heals; run-level `success:false` is the only signal.
|
||||
- **The data plane held:** all 5 refusal categories ×2 correct + fast (2–5 s, retry=0), verify/rollback/single-flight/orphan flows clean, deploy-view truth holds, restore round-trips **byte-identical**, EIO same-second under outage, organic stub → badge + deploy-409 live-validated, hardlinks work on NFSv4.1.
|
||||
- **Fix-6 answered with numbers:** ring cap horizon = ~55 min idle but **~6.5 min under load**; every restart/reboot wipes both rings — persistence, not just size, is the gap.
|
||||
- **Policy discovery (docs):** tier-1 = volumes+config only (NAS media userdata excluded by design); NAS apps' tier-1 lands *on the NAS*, tier-2 is what gets it off; volume-only apps back up to sys_drive with blank drive label and **no tier-2 copy**.
|
||||
|
||||
### Box state / cleanup (final, 06:53)
|
||||
|
||||
DooPlex NAS restored **md5-identical to baseline** (exports + smb.conf; campaign user/share/dirs/creds removed); services never touched (exportfs-only rail held). Host recovered post-power-cycle; guest 9201 in **defined state**: 6 wave apps deployed + healthy with data, `privatebin` stop+removed via the real flow, verification backup `success:true`, nas-media `ok`, stub 0. Hub untouched throughout. ⚠ Next host reboot re-rolls the F12 dice until the agent template is fixed (interim: systemd drop-in on the automount units).
|
||||
- **The whole arc ran to completion on a fresh box:** appliance-mode Day-0 (agent 0.85.0 + golden 0.120.0, both sha-verified, "Day-0 provision SUCCESS"), guest at floor 0.120.0 at FIRST report with zero manual steps, ActualBudget deployed through the tunnel, escrow ceremony sealed (R with Viktor only), **AUTO-CONFIRM FIRST LIVE FIRING: pending → escrowed in ~7.5 min with zero clicks**, offsite backup (1 snapshot, 14 s) **and the verification-restore round-trip proven** (48K payload decrypted back from the Storage Box).
|
||||
- **F-4 (HIGH): G10 is unclosable** — no operator-set dashboard password path exists in shipped code (hub has no per-customer `password_hash` UI/API; controller's open-state page defers to the operator; Day-0 preseeded setup skips the wizard's password form). Every fresh box's dashboard stays OPEN on the internet. Blocks the tester onboarding gate.
|
||||
- **F-6 (HIGH): identity-only escrow ceremony was never implemented** — on a no-PBS box (the documented appliance standard) the offsite escrow chain can never complete. Mid-drill fork (Viktor): attached the **PBS DR tier (ep0)** instead, which live-rehearsed Peti's exact pending sequence — wrapper+`age` prep → `wg_tunnel.enabled` → hands-free WG peer registration → hub PBS-DR enable (WG-first dependency guard works) → apply-bridge fresh path (consume-once → K born → escrow seed) → ceremony → auto-confirm.
|
||||
- Fresh-box installer gaps found live: `felhom-pbs-apply` binary not shipped (F-7), `age` not installed (F-10), root-owned `guests/` parents break the non-root agent's lanresolver (F-3, live-fixed; **demo host likely has the same latent state**), silent root@pam rotation surprises the operator (F-8), version strings disagree ×3 (F-1).
|
||||
- Snapshots on qm 300 for re-drills: `pre-day0-clean` / `post-install` / `post-drill`. Demo 9201, Peti's hub entry, demo offbox untouched.
|
||||
|
||||
Reference in New Issue
Block a user