From a16896af865df18ad6f86c25c7562c39fb9c1f23 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 27 Jul 2026 11:03:02 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20R-88=20root=20cause=20established=20?= =?UTF-8?q?=E2=80=94=20no=20limiter,=20and=20the=20nil=20age=20bypasses=20?= =?UTF-8?q?the=20window=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Corrects two wrong severity readings with evidence from the box and the code. The PBS outage was ~15 min (07:00-07:18 UTC), caused by a global OOM at 06:58:12: proxmox-backup-proxy peaked at 3.2G on a 3.8G box and a concurrent 1.9G rsync tipped it over. Root SSH to that box works from DooPlex via the public IP, not from felhom-pve via the tunnel IP — the documented path I failed to try first. R-88: internal/quiesce has NO failure limiter, backoff or breaker; the loop stopped after three cycles only because PBS recovered. Verified additionally that scheduledRunAllowed (quiesce.go:476-478) returns true whenever lastAgeSecs is nil, so the same missing value that makes every poll due also bypasses the time-of-day gate — the cycles ran outside the [04:30,08:30) window. Fixing the due-verdict without fixing the nil-age bypass would leave the hole open. --- REPORT.md | 73 ++++++++++++++++++++------------ documentation/backlog/ROADMAP.md | 8 +++- 2 files changed, 54 insertions(+), 27 deletions(-) diff --git a/REPORT.md b/REPORT.md index 909764f..76efacf 100644 --- a/REPORT.md +++ b/REPORT.md @@ -237,38 +237,59 @@ An estimate extrapolated from a degraded measurement is not a measurement. Recorded after §1–§10 were written. Two facts, both still true at hand-off. -### 11.1 The offsite PBS service is DOWN — cause unknown, box reachable +### 11.1 The offsite PBS outage — ~15 minutes, OOM-driven, RESOLVED -`felhom-hetzner` (167.233.158.164) reports `status=running` via the Hetzner API; the `wg-felhom` -tunnel is healthy (handshake seconds old, ping 0% loss, ~40 ms); **SSH 22 answers but 8007 refuses**, -repeatedly over several minutes from felhom-pve. The box is up and `proxmox-backup-proxy` is not -listening. **No SSH key to that box exists from DooPlex or felhom-pve**, so diagnosis stopped there — -the box's own journal and `dmesg` are unread. +**Root cause established, on the box.** `felhom-hetzner` (167.233.158.164) did **not** reboot +(`up 18 days`). At **06:58:12 UTC a global OOM** fired: `proxmox-backup-proxy` had grown to a +**3.2 GB peak on a 3.8 GB box** (systemd's own accounting at the later restart: +`Consumed 42.395s CPU time, 3.2G memory peak`) and a concurrent root `rsync` at **1.9 GB RSS** tipped +it over. The kernel killed the rsync. PBS stopped serving from ~07:00 to **07:17:59 UTC**, then +recovered on its own — evidenced by the PVE API access log for the agent's ground-truth reads: +`500` at 09:02/09:07/09:12 CEST, `200` from 09:17:59 CEST onward, uninterrupted since. A **separate** +deliberate proxy restart at 07:52:52 UTC cleared unrelated `read fs info on "/srv/pbs-scratch" failed +- ENOENT` spam; it was not the recovery. -**A cause I cannot rule out: this session's own restore-test.** The unattended PBS restore-test read -**14.46 GB** off that datastore 06:44–06:58 UTC and completed `OK`; PBS was refusing five minutes -later. On a **cx23 (2 vCPU / 4 GB)** an OOM of the proxy under that read is a plausible mechanism. -Correlation only — **not established**, and it must not be written up as though it were. First checks -for whoever gets into the box: `journalctl -u proxmox-backup-proxy` and `dmesg | grep -i oom`. -If it IS the restore load, it bears directly on **R-86**: restore-testing a tier weekly means putting -that read on a small offsite box on a schedule. +**This session's restore-test is the most likely driver of the proxy's 3.2 GB peak** — it read +14.46 GB off that datastore 06:44–06:58 UTC, finishing 9 seconds before the OOM. Not provable from +what is on the box, but the timing and the memory figure both point at it, and the rsync was the +*victim* rather than the cause. **This bears directly on R-86:** restore-testing a tier weekly means +putting a multi-GB read on a 4 GB offsite box *on a schedule*. Either the box needs more RAM before +that lands, or the restore-test needs to not run concurrently with whatever else touches that +datastore. See also the existing note on rsync over a PBS chunk store. -### 11.2 An outage loop is RUNNING on demo-felhom, deliberately left running +**Access correction worth carrying:** root SSH to that box works **from DooPlex to the public IP** +(`root@167.233.158.164`), and **not** from felhom-pve to the tunnel IP `10.77.0.1`. I concluded "no +access exists" from the second failing and stopped diagnosing — the project memory recorded the +working path and I did not check it until later. The whole root cause above came from finally trying +the documented route. -The agent restart that applied the reverted 3.5-day cadence exposed **R-88** (filed `eb3f0b8`, -severity corrected `5aca709`): an unreachable target reads as *no backup exists*, so the offsite tier -is perpetually "due". The controller re-polls every ~5 min and runs the **full quiesce cycle** each -time — `quiescing 4 stack(s): [bookstack calibre-web docmost immich]` → `unquiescing (backup failed)` -— roughly **19 s of app downtime per cycle, unbounded**, until PBS answers. +### 11.2 R-88 — an unreachable target takes the customer's apps down on a loop -**Operator ruling 2026-07-27: leave it running.** It is a demo box, the impact is contained, it -self-heals the instant PBS returns, and leaving it keeps the fault visible rather than masked. The -alternatives (disable the tier; ship the R-88 fix) were declined in favour of not masking it. +The agent restart that applied the reverted 3.5-day cadence exposed **R-88**: an unreachable target +reads as *no backup exists*, so the offsite tier is perpetually "due". Three full quiesce cycles ran +(07:02:57, 07:07:58, 07:12:57 UTC), each stopping and restarting all four app stacks for a backup +that could not succeed — ~19 s of app downtime per cycle, ~50 s per full cycle. -**I recorded R-88 as bounded — "one spurious event per restart" — before measuring it.** It is -neither bounded nor event-only; it is a repeating availability fault. The roadmap entry carries the -correction. The error was the same shape as the ~2-hour estimate in §9: a severity asserted from the -mechanism I had reasoned about, before looking at what the mechanism actually did on the box. +**It stopped after three only because PBS recovered.** There is no limiter: `internal/quiesce` has no +failure counter, backoff, breaker or attempt budget, and the driver is a plain 5-minute ticker +(`quiesce.go:149`). Had the outage lasted, the loop would have continued indefinitely. + +**The amplifier found while verifying that:** the agent answers `AgeSecs: nil`, and +`scheduledRunAllowed` (`quiesce.go:466-480`) returns `true` whenever the age is nil — *"never +withhold the first one"*. So the same missing value that makes every poll due **also bypasses the +time-of-day gate**. The gate was `[04:30, 08:30)`; the cycles ran 09:02–09:12 Budapest, outside it. +A safety valve written for a genuine first-ever backup is being tripped by a failed storage read. + +**Operator ruling 2026-07-27: leave it running** (given while PBS was still down) — demo box, +contained impact, self-heals, and leaving it keeps the fault visible rather than masked. It has since +self-resolved; no action is outstanding on the box. + +**I got the severity wrong twice before measuring it.** First as "one spurious event per restart" +(it was a repeating loop), then as "three tries, so something limits it" (nothing does — the +condition ended). Both readings were asserted from the mechanism I had reasoned about rather than +from what the box and the code actually showed; the second was only caught by reading +`internal/quiesce` instead of inferring a breaker from three log lines. Same shape as the ~2-hour +estimate in §9. ### 11.3 Verified clean at hand-off diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 55475e2..6377d61 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -116,7 +116,13 @@ | R-85 | **The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD** | M | **Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE** | R-82 proved a PBS restore round-trip **by hand**. Three defects kept the unattended path from covering it, each verified at source: **(a)** `buildRestoreTestScheduler` took `cfg.Backup.BackupTarget()` and built ONE runner on it, so the offsite tier's archives were never candidates; **(b)** the `Spec` was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; **(c)** a failed restore-test was a `[WARN]` log line with **no event, no notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. **Selection ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported **so the achieved interval can be MEASURED before anyone decides whether a knob is needed** — measure, then configure, not the reverse. **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS, never fails); persisted `RestoreTestState` (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); `backup.InFlight`, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub `restore_test_failed` / `restore_test_stale`, kept **distinct** and anchored on R-81's verdict structure, operator-tier only. **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). | | R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — operator ruling 2026-07-27** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. | | R-87 | **The restic (app-data offsite) tier is NEVER restore-tested** | M | idea — surfaced 2026-07-27 while closing R-85 | **R-85 covers whole-guest vzdump tiers only** (`local`, `felhom-pbs`); the agent has no restic surface at all. restic is the CONTROLLER's app-data offsite backup to the Hetzner Storage Box, a separate mechanism — so the tier that is arguably most important to a customer is the one nothing verifies. It is the only tier that survives losing the box **and** carries their actual app data: the whole-guest snapshot deliberately excludes the bind-mounted data drives (`/mnt/felhom-drives`). Restore code exists and has been exercised BY HAND (the immich destroy-and-recover drill, PROVEN-LIVE), but nothing tests it unattended — **exactly the state PBS was in before R-85: it works when someone tries it, and nobody would know if it stopped.** Needs its own design: a restic restore-test is controller-side, has no scratch-guest analogue, and would verify into a scratch dir rather than a booted guest, so R-85's machinery does not transfer. | -| R-88 | **An UNREACHABLE backup target reads as "no backup exists" → the agent fires a doomed backup at it** | S | idea — observed live 2026-07-27 | **The R-81 class (absence-is-not-failure), reappearing one layer down in the AGENT's due-check.** R-84 replaced the in-memory `Store` with storage ground truth (`NewestArchiveTime`), and fails safe on a read error by *falling back to the in-memory record* — "never fake freshness, never suppress". That fallback is correct in isolation but wrong in the one composition where it actually runs: **a cold store plus an unreachable target.** After a restart the in-memory record is empty, so the fallback answers *no backup has ever been recorded* → tier due → the agent fires a backup at the very storage it just failed to read. **Observed:** `felhom-agent` restarted on demo-felhom at 07:02:57 UTC; `local-api: could not read the backup storage for the due-check — falling back to the in-memory record` (vmid=9201 target=felhom-pbs); 11 s later `vzdump 9201 → felhom-pbs` failed with `could not activate storage 'felhom-pbs' … Can't connect to 10.77.0.1:8007 (Connection refused)`. The offsite PBS service was down at the time — so the trigger was real, not synthetic. **NOT bounded — it is a repeating outage loop, and this correction matters.** The first reading ("fails in ~0 s, so the cost is one spurious event per restart") was wrong on both halves. It is not once: the controller re-polls `/backup/due` on its normal cadence, gets `due` again — the storage is still unreachable and the store is still cold — and retries **every ~5 minutes indefinitely**, for as long as the offsite target stays down. And the cost is not an event: each attempt runs the **full quiesce cycle**, so all four customer app stacks are STOPPED and RESTARTED for a backup that cannot possibly succeed. Measured on demo-felhom: `07:07:58 quiescing 4 stack(s): [bookstack calibre-web docmost immich]` → `07:08:17 unquiescing (backup failed)` = **~19 s of customer-visible app downtime per cycle, every 5 minutes, unbounded.** So the defect's real shape is *an unreachable offsite tier takes the customer's applications down on a loop* — a availability fault, not a logging nuisance. It self-resolves the moment the target answers again (the storage read succeeds, sees the recent archive, and the tier stops being due), which is why it can hide for a long time: it needs an offsite outage to appear at all. **Shape:** distinguish *storage unreachable* from *storage readable and empty*. Unreachable is UNKNOWN — defer the due-verdict rather than resolving it either way, exactly as R-81 made the hub do with a missing report. Only a target that is reachable AND has no archive is genuinely due. Note this does NOT weaken R-84's fail-safe intent: a tier whose storage is merely slow or briefly unreadable should still err toward backing up — it is specifically the **cold-store + unreachable** pair that must defer, because there the fallback has no information at all, only an empty default that looks like a fact. | +| R-88 | **An UNREACHABLE backup target reads as "no backup exists" → the agent fires a doomed backup at it** | S | idea — observed live 2026-07-27 | **The R-81 class (absence-is-not-failure), reappearing one layer down in the AGENT's due-check.** R-84 replaced the in-memory `Store` with storage ground truth (`NewestArchiveTime`), and fails safe on a read error by *falling back to the in-memory record* — "never fake freshness, never suppress". That fallback is correct in isolation but wrong in the one composition where it actually runs: **a cold store plus an unreachable target.** After a restart the in-memory record is empty, so the fallback answers *no backup has ever been recorded* → tier due → the agent fires a backup at the very storage it just failed to read. **Observed:** `felhom-agent` restarted on demo-felhom at 07:02:57 UTC; `local-api: could not read the backup storage for the due-check — falling back to the in-memory record` (vmid=9201 target=felhom-pbs); 11 s later `vzdump 9201 → felhom-pbs` failed with `could not activate storage 'felhom-pbs' … Can't connect to 10.77.0.1:8007 (Connection refused)`. The offsite PBS service was down at the time — so the trigger was real, not synthetic. **NOT bounded, and it also DISABLES the business-hours protection — an availability fault, not a logging nuisance.** The first reading ("fails in ~0 s, so the cost is one spurious event per restart") was wrong on both halves, and a second reading ("it stopped after three tries, so something limits it") was wrong too — the live run stopped after three only because the storage came back. **Verified in code: there is no limiter.** `internal/quiesce` holds no failure counter, no backoff, no breaker, no attempt budget; `runOnce` keeps no state between ticks, and the driver is a plain 5-minute `time.Ticker` (`quiesce.go:149`, default `quiesce.go:107-109`). Had the target stayed down, the loop would have continued indefinitely. + +Each attempt runs the **full quiesce cycle**, so every customer app stack is STOPPED and RESTARTED for a backup that cannot succeed. Measured on demo-felhom: `07:07:58 quiescing 4 stack(s): [bookstack calibre-web docmost immich]` → `07:08:17 unquiescing (backup failed)` → `07:08:45 failed` — **~19 s of app downtime per cycle (~50 s per full cycle), every 5 minutes.** + +**The amplifier, and the part worth designing against:** the agent answers `Due: true, Reason: "no successful backup recorded yet", AgeSecs: nil`, and that **nil age does double duty**. `scheduledRunAllowed` (`quiesce.go:466-480`) returns `true` whenever `lastAgeSecs == nil` — *"no recorded backup yet — never withhold the first one"* — so the same nil that makes every poll due **also bypasses the time-of-day gate** `[W+2h, W+6h)`. On the live box the gate was `[04:30, 08:30)` and the cycles ran at 09:02–09:12 Budapest, i.e. **outside the backup window entirely**. So the fault stops customer apps every 5 minutes *at any hour, including business hours* — the one protection specifically built to prevent that is switched off by the same missing value. A safety valve written for a genuine first-ever backup is being triggered by an unreachable storage read, which is not the same thing at all. + +Self-resolves the moment the target answers (the storage read succeeds, sees the archive, tier stops being due) — which is why it can hide indefinitely: it needs an offsite outage to appear at all. **Shape:** distinguish *storage unreachable* from *storage readable and empty*. Unreachable is UNKNOWN — defer the due-verdict rather than resolving it either way, exactly as R-81 made the hub do with a missing report. Only a target that is reachable AND has no archive is genuinely due. **Fix the window bypass in the same slice:** `AgeSecs == nil` must stop meaning "run now regardless of the hour". Either the agent distinguishes *never backed up* from *cannot tell* in what it reports, or `scheduledRunAllowed` gates on the former only — otherwise any future nil-age path re-opens the same hole. Note this does NOT weaken R-84's fail-safe intent: a tier whose storage is merely slow or briefly unreadable should still err toward backing up — it is specifically the **cold-store + unreachable** pair that must defer, because there the fallback has no information at all, only an empty default that looks like a fact. | | R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog | | R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | **SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result.** The universal secret-free ISO was booted on a completely different machine — an **HP t740 (Ryzen V1756B, AMI M42 firmware)** versus the N100 it was proven on — and the whole chain ran on **virgin hardware in one pass**: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host `demo-hp-bb76ea`. **Two things generalise from the second board specifically:** the **shim** loader booted with **Secure Boot ENABLED** (`mokutil --sb-state` → `SecureBoot enabled`), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's **1TB NVMe untouched and unenrolled** (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. **Not clean, and the failures are filed:** the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (**R-59**), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (**R-61**) | R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** |