From 57ba3c7c8c2b347e727295d19ddc7c233217a702 Mon Sep 17 00:00:00 2001 From: Claude Code Date: Mon, 27 Jul 2026 07:33:40 +0200 Subject: [PATCH] =?UTF-8?q?R-85=20Phase=204:=20docs=20=E2=80=94=20the=20UN?= =?UTF-8?q?ATTENDED=20path=20is=20documented=20as=20unproven?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - ROADMAP: R-85 row. Code SHIPPED; rotation NOT YET OBSERVED LIVE, stated as such rather than written as done. - Capability map: a new row for UNATTENDED restore-proof, IMPLEMENTED not PROVEN-LIVE, kept distinct from the R-82 row that a MANUAL selftest earned. That distinction is the same one the activation-vs-arrival split made. - 03-host-agent §8: the scheduler covers every tier, oldest-proven first; the spec is per-run; a restore-test joins the one-heavy-op gate. The safety properties that must not be re-derived are listed. - 07: restore-proof recorded as a per-tier property. Doc still NOT ratified. - 06: corrects S4.1's 'the offsite restore-test now runs unattended' — it silently stopped being true when local_backup_target was retargeted to 'local', the SECOND time in that doc that a correct mechanism was broken by its input changing underneath it. - CONTEXT + REUSE. Hub gate green (17 packages, rc=0). --- CONTEXT.md | 27 +++++++++++++++++++ REUSE.md | 1 + .../architecture/00-capability-map.md | 1 + documentation/architecture/03-host-agent.md | 27 +++++++++++++++++++ .../architecture/06-offsite-connectivity.md | 2 +- .../architecture/07-backup-architecture.md | 9 +++++++ documentation/backlog/ROADMAP.md | 1 + 7 files changed, 67 insertions(+), 1 deletion(-) diff --git a/CONTEXT.md b/CONTEXT.md index 91a0a98..3c6b561 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -21,6 +21,33 @@ reader meets the warning before the content, not after acting on it. Origin: versions stale (live v0.173.0), and cited as authoritative throughout the R-80 diagnostic. Ratifying or retiring it is → **R-83**. +- **2026-07-27 — R-85: the DR tier is restore-tested UNATTENDED, and its failure is HEARD.** agent + v0.104.0 + hub v0.77.0. R-82 proved a PBS restore round-trip **by hand**; three defects kept the + unattended path from covering it. (a) `buildRestoreTestScheduler` built ONE runner on + `cfg.Backup.BackupTarget()`, so the offsite tier's archives were never candidates — the DR tier + could not be restore-tested unattended AT ALL. (b) the `Spec` was an immediately-invoked function + evaluated once at daemon start, so nothing tier-varying was expressible and any config change + stayed stale until restart. (c) a failed restore-test was a `[WARN]` line with **no event, no + notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR + signal this system produces was inaudible. **(c) was the one that mattered:** rotating tiers + without it would only mean two tiers can fail silently instead of one. + **Ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, no cadence + knob; the per-tier last-proven timestamp is reported so the achieved interval can be **measured** + before anyone decides a knob is needed. + **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS); + persisted `RestoreTestState` (survives restart; credit only on SUCCESS); `backup.InFlight`, the + host-wide one-heavy-op gate shared with the backup path (a LINK concern — an offsite restore pulls + multi-GB over the tunnel a backup pushes one); hub `restore_test_failed` / `restore_test_stale`, + kept DISTINCT and anchored on R-81, operator-tier only. + **Also fixed a time bomb I planted in Slice C:** a deadline test hard-coded the incident timestamp + `2026-07-18T18:31:06Z` while comparing against the REAL clock — harmless under one 26h threshold, + but once the offsite tier got an 8-day limit it passed all day and began failing at exactly 18:31 + UTC, 8 days later. A test that passes at commit time and fails hours later is worse than one that + fails immediately. + **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive + UNATTENDED cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). + Spec: `documentation/backlog/SPEC-r85-phase4-5-2026-07-26.md`. + - **2026-07-26 — R-82 SHIPPED: the backup target split (local daily + offsite weekly).** agent v0.97→v0.102.0, controller v0.174→v0.175.0, hub v0.76.0, host-install 1.20.0. `BackupTarget()` returned ONE string and `BackupCadence()` ONE 24h window, so "local daily AND PBS weekly" was not diff --git a/REUSE.md b/REUSE.md index 5c0653a..7bc3ec5 100644 --- a/REUSE.md +++ b/REUSE.md @@ -102,6 +102,7 @@ | `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. | | `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). | | `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. | +| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. | | `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. | | `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. | | `cfg.DRTier` + offsite coupling | hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) | bool | Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) | One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field `dr_tier` (formBool helper). | diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index ac681da..af1f417 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -38,6 +38,7 @@ | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test | | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). | | **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** | +| **Restore-proof is UNATTENDED — the scheduler covers EVERY tier and a failure is heard** | agent v0.104.0, hub v0.77.0 | **IMPLEMENTED** (not PROVEN-LIVE) | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why NOT PROVEN-LIVE:** rotation has not been observed selecting both tiers across consecutive UNATTENDED cadences. At a 24h cadence that is a multi-day observation window, and it had not elapsed when this shipped. A single passing run proves the code path, not the rotation. → **R-85** | | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below | | **Customer DELETE cascade** (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **UNIT-PROVEN; live leg PENDING** | `hub/internal/web/customer_delete_test.go` — leg ORDER observed from inside leg 2 (hosts already gone, customer row still present, custody still retained); 9 fail-closed gate cases each asserting zero mutations + zero external calls + no journal row; resume-after-external-failure converges; `purgeEscrow` custody semantics; preview leaks no secret. **5 red-proofs** (ack gate, stale-preview gate, ONLINE-host gate, leg order inverted, `purgeEscrow=true`) | Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **v0.70.0 (2026-07-21):** added the **residue** leg — `GetCustomers()` is REPORT-derived, so before it a fully deleted customer stayed on the Customers list and its report stream kept the staleness/offsite checkers alerting (live: `demo-vm-felhom` deleted 07-18, still emailing `offsite_stale` on 07-21). The leg also purges the credential-bearing `appliance_registrations` + `selfbind_tokens`. **Ghost customers (config row already gone) are now deletable** — 404 means "nothing here", not "no config row"; the Hetzner/descriptor legs record `skipped_no_config`. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run | | Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 | diff --git a/documentation/architecture/03-host-agent.md b/documentation/architecture/03-host-agent.md index c13bf51..4d5db77 100644 --- a/documentation/architecture/03-host-agent.md +++ b/documentation/architecture/03-host-agent.md @@ -228,6 +228,33 @@ Tiers double as backup *and* restore-source priority (fastest surviving source f per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a backup) → **local second storage** (vzdump to dir/NFS/CIFS) → **PBS offsite** (the DR substrate). +> **Self-restore-test: the scheduler covers EVERY tier, oldest-proven first (R-85, agent v0.104.0).** +> It used to build ONE runner on `cfg.Backup.BackupTarget()`, so **the offsite tier's archives were +> never candidates** — the DR tier could not be restore-tested unattended at all, which is how +> demo-hp's tier sat `applied` with zero snapshots for five days unnoticed. Three things changed: +> +> - **The spec is built PER RUN** (`backup.SpecBuilder`), not frozen at daemon start. It previously +> came from an immediately-invoked function, so `storageTier()`/`restoreTaskTimeout()` were +> evaluated once and reused forever — nothing tier-varying was expressible, and any storage-type or +> config change stayed stale until the daemon restarted. `SourceTier` comes from **the archive** +> (`restoreTierForArchive`), never the configured target: config-derived is what classified a PBS +> archive as `local` and killed a 14.46 GB WAN restore at the 10-minute local bound. +> - **Rotation is oldest-first** (operator ruling 2026-07-26), persisted at +> `/restore-test-state.json` so it survives a restart — an in-memory map would reset +> every tier to "never proven" on each deploy and could starve one indefinitely. **Credit only on +> SUCCESS**, or a permanently failing tier would look freshly proven and stop being retried. A tier +> with no archive **no-ops and the next tier is tried** — a brand-new offsite tier legitimately has +> nothing to restore, and that is not an error. +> - **A restore-test joins the host-wide one-heavy-operation gate** (`backup.InFlight`), shared with +> the local-API backup path. Not a lock concern — the scratch VMID never touches the live guest's +> vzdump lock — but a **LINK** one: an offsite restore PULLS multi-GB over the same tunnel an +> offsite backup PUSHES one, and at the measured ~33 MB/min upstream running both drives each +> toward its timeout, which is how a healthy tier ends up recorded as failed. Callers **defer**; +> they never cancel what is already running. +> +> Unchanged and load-bearing: `onboot=0` on the scratch at restore time, **every NIC link-down before +> boot**, journal-before-mutate, guaranteed teardown, and the per-tier restore-task timeout. + - **Quiescing (controller-driven for app-consistency) — implemented (slice 8B):** an LXC has no fsfreeze (`proxmox-platform.md` §4.2), so app-consistency is the controller's job: it learns a backup is due (`GET /backup/due`, §6) → **quiesces** (stops its app stacks) → `POST /backup` → diff --git a/documentation/architecture/06-offsite-connectivity.md b/documentation/architecture/06-offsite-connectivity.md index 3783479..0aaa2db 100644 --- a/documentation/architecture/06-offsite-connectivity.md +++ b/documentation/architecture/06-offsite-connectivity.md @@ -321,7 +321,7 @@ vantage (Peti VM 110) remains the thorough confirmation but no longer gates anyt | **S1 — endpoint provisioning + peer-sync surface** — **SHIPPED 2026-07-04** (dev endpoint; hub v0.32.1). Peer-sync design point RESOLVED: **hub pushes over SSH** (`x/crypto/ssh`, `FixedHostKey` pin + pinned-type `HostKeyAlgorithms`) to a forced-command reconcile script; runbook `documentation/runbooks/offsite-endpoint.md`. Done-criterion met live: hub add/remove → `wg show` on the endpoint; tunnel `ep0.felhom.eu:443` → PBS through the wg0-only 8007 rule. | ~~provision the production VM…~~ (see runbook; production endpoint = a later re-run) | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing ✓ | | **S2 — hub schema + desired-state** — **SHIPPED 2026-07-04** (hub v0.33.0). `POST /hosts/{id}/wg` (self-scoped) + merge-at-read `wireguard` block (§3.2 shape = the golden) + one-per-host index + re-key-in-place + `/offsite` UI page. Live-proven against the real felhom-pve record incl. old-agent (v0.63.0) tolerance. | ~~endpoint coords + peer registry…~~ | a registered box's desired-state carries its assignment ✓; hub UI shows the peer registry ✓ | | **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) | -| **S4 — PBS over the tunnel — SHIPPED 2026-07-04** (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep `DatastoreBackup` token, dual-grant user+token on `/datastore//`, cross-tenant 403); box `felhom-offsite` PVE storage `Server=10.77.0.1`, box-born encryption key, token-secret WARN fix, `local_backup_target` retarget; agent **v0.66.0** v4-pin + re-resolve watchdog + FELHOM_WG Critical, **v0.67.0** namespace-aware PBS client. | ~~point the box's PBS storage at `pbs_tunnel_ip:8007`~~ | real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: ~~restore-test scheduler WAN deadline + scratch `VM.Allocate`~~ **RESOLVED S4.1 (agent v0.68.0)** — tier-aware restore-task timeout (pbs→120m, local→10m); the offsite restore-test now runs unattended on the agent-token path (`pass:true`, teardown clean). The `VM.Allocate` item was a **phantom** — the earlier 403 was the 10m-timeout consequence (guest not yet in `/pool/felhom`); no ACL change. ~~Remaining: Tier-1-local-vs-Tier-2-offsite target split~~ **SHIPPED 2026-07-26 (R-82)** — agent v0.97.0 gave `BackupConfig` per-target tiers (`backup_targets[]`, each with its OWN cadence + retention), controller v0.174.0 reconciles them into ONE quiesce window, hub v0.76.0 judges each tier against its own threshold, host-install 1.20.0 defaults a fresh box to local-daily + offsite-weekly. Live: local 24h + `felhom-pbs` 168h on both demo boxes. +| **S4 — PBS over the tunnel — SHIPPED 2026-07-04** (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep `DatastoreBackup` token, dual-grant user+token on `/datastore//`, cross-tenant 403); box `felhom-offsite` PVE storage `Server=10.77.0.1`, box-born encryption key, token-secret WARN fix, `local_backup_target` retarget; agent **v0.66.0** v4-pin + re-resolve watchdog + FELHOM_WG Critical, **v0.67.0** namespace-aware PBS client. | ~~point the box's PBS storage at `pbs_tunnel_ip:8007`~~ | real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: ~~restore-test scheduler WAN deadline + scratch `VM.Allocate`~~ **RESOLVED S4.1 (agent v0.68.0)** — tier-aware restore-task timeout (pbs→120m, local→10m); the offsite restore-test now runs unattended on the agent-token path (`pass:true`, teardown clean). **⚠️ That claim SILENTLY STOPPED BEING TRUE and was restored 2026-07-27 (R-85, agent v0.104.0) — for the second time in this doc, by the same mechanism: a correct fix whose INPUT changed underneath it.** S4.1 was verified while `local_backup_target` **was** `felhom-pbs`, so "the scheduler's one tier" and "the offsite tier" were the same thing. The R-82 target split retargeted it to `"local"`, and from then on `buildRestoreTestScheduler` built its only runner on the LOCAL tier: the offsite tier's archives were never candidates and it was **never restore-tested again**, unattended or otherwise, while this note still said it was. R-85 rotates across every configured tier (oldest-proven first) and makes a failure audible. The lesson is the same one the timeout taught: **a mechanism keyed on "the configured target" stops holding the moment there is more than one target.** The `VM.Allocate` item was a **phantom** — the earlier 403 was the 10m-timeout consequence (guest not yet in `/pool/felhom`); no ACL change. ~~Remaining: Tier-1-local-vs-Tier-2-offsite target split~~ **SHIPPED 2026-07-26 (R-82)** — agent v0.97.0 gave `BackupConfig` per-target tiers (`backup_targets[]`, each with its OWN cadence + retention), controller v0.174.0 reconciles them into ONE quiesce window, hub v0.76.0 judges each tier against its own threshold, host-install 1.20.0 defaults a fresh box to local-daily + offsite-weekly. Live: local 24h + `felhom-pbs` 168h on both demo boxes. > ⚠️ **S4.1's tier-aware timeout silently REGRESSED and was restored 2026-07-26 (agent v0.100.0) — the mechanism was never removed, its INPUT changed.** S4.1 derived the tier with `storageTier(BackupTarget())`, which was correct while `local_backup_target` **was** `felhom-pbs`. The target split above retargeted it to `"local"`, so a `felhom-pbs:` archive was thereafter classified **local** and got the 10-minute bound again — reproducing the S4.1 symptom exactly (wait expires mid-restore → teardown fires at a not-yet-pool-associated guest → the same `403 missing privilege VM.Allocate`, and the same **phantom** reading of it). v0.100.0 derives the tier from **the archive's own storage** (`restoreTierForArchive`), which is input-independent. The lesson is not "add a timeout" — it was already there — but that a fix keyed on *"the configured target"* stops holding the moment more than one target exists. | | **S5 — DR re-establishment — SAFE HALVES SHIPPED 2026-07-04 (agent v0.69.0); destructive drill operator-gated.** Part 1: `wgtunnel.InstallRecoveredKey` (escrow-recovered WG key → same identity/pubkey/`/32`, no keygen; create-only; `--selftest=identity-consume -install-wg-key`). Part 2: `internal/dr` consumes the host_loss `restore_directive` into an inspectable RestorePlan (per-guest restore coords+sizing, per-drive durable_id→mount) via the AddConsumer seam — derive-and-surface, execute nothing. Re-attach wrong-disk safety = `ResolveStorageDevice` (match resolves, mismatch REFUSED — unit-proven). | ~~consume `WGPrivateKey` in restore mode~~ | the tunnel re-establishes from the recovered key (same `/32`) ✓ (unit); the host_loss directive yields the correct PLAN ✓ (unit). **Operator-gated:** the destructive in-place 9201 restore (destroy→restore-from-offsite→re-attach-by-durable_id) is operator-present + STOP-gated — the operator runs the R-consume steps (CC never does). Pre-flight green (9201 offsite-restorable, S4.1). | diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index fdf4030..437e46b 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -82,6 +82,15 @@ the Task 1 lesson). Resolution to absolute paths happens at capture time against | **Manual `.fab`** | v0.130.0 full-root capture | locked-in (not deselectable) | checkbox, **pre-selected** | opt-in, behind the two-number size warning + FileBrowser pointer | download / chosen drive | tar, **exclusion-scoped** root (SQ5 verdict; manifest v1 unchanged) | existing import (old controllers import new bundles correctly) | | **PBS** whole-guest (R-82) | rootfs + `/var/lib/docker` + `/mnt/sys_drive`; bind mounts (`/mnt/felhom-drives`, `/etc/felhom-bootstrap`) out of reach | *unchanged* | | | **`felhom-pbs` → datastore `felhom-offsite` on ep0 (Hetzner), per-customer namespace, reached over `wg-felhom`** — NOT "PBS on DooPlex" (that was the 2026-07 spike store) | vzdump, **WEEKLY** (`cadence_seconds: 604800`), retention **keep_last=2** (two weeks, operator ruling 2026-07-26) | whole-guest restore | +**Restore-proof per tier (R-85, 2026-07-27).** A tier is not proven by having archives; it is proven +by one of them restoring into a bootable, mount-complete guest. Both tiers are now restore-tested +**unattended**, rotating oldest-proven-first at the restore-test cadence, and each tier's last +successful proof is reported. The hub raises two DISTINCT operator signals: `restore_test_failed` +(a run did not pass — broken now) and `restore_test_stale` (not proven within ~7 days — *unverified*, +which is not the same claim). Before this the scheduler could only ever see the primary tier, so the +PBS row above was scheduled-but-never-verified. See `03-host-agent.md` §8. + + Row-level decisions folded in: 1. **Classified apps' tier-2 appdata leg becomes fully class-driven** — per-bind paths, not the diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index efebe8f..36e0e03 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -113,6 +113,7 @@ | R-82 | **The backup target split — local daily + offsite weekly** | M | **SHIPPED (agent v0.102.0 + controller v0.175.0 + hub v0.76.0 + host-install 1.20.0, 2026-07-26)** | Phase 0: `audits/SPIKE-r82-phase0-2026-07-26.md` (weekly CONFIRMED; the `pvesm status` 0/0/0 resolved as a namespace-scoped-token reporting artifact; capacity measured). `backup_targets[]` gives each tier its OWN cadence and retention; the controller runs both due tiers in ONE quiesce window; the hub judges each against its own threshold (host 26h / offsite 8d); host-install 1.20.0 defaults a fresh box to local-daily + offsite-weekly, and an unprovisioned tier DEFERS rather than firing at a missing storage. Operator rulings 2026-07-26: 2-week offsite retention, first backup runs as long as needed, one backup at a time per guest, drill box dropped from the rollout. **Live:** demo-felhom + demo-hp both on local-24h + felhom-pbs-168h; demo-felhom's first real PBS backup landed (`TASK OK`, 41 min, 14.46 GB) and restored cleanly. **Four defects found by RUNNING it, not reviewing it:** 30-min wait bound vs a 41-min backup (v0.98.0), restore tier read from the configured target not the archive (v0.100.0 — a silent regression of the S4.1 fix), a leaked scratch keeping `onboot:1` (v0.101.0), and a tier firing at a not-yet-provisioned storage (v0.102.0). **REMAINING GATE:** the installer default is written but the fleet-wide flip waits on a full weekly cycle holding; and the offsite tier is never AUTOMATICALLY restore-tested (the scheduled test only selects the primary tier) — that gap, plus the hub inferring cadence from storage TYPE, are the two follow-ups. | | R-83 | **Ratify or retire `07-backup-architecture.md`** | S | **DISCHARGED (2026-07-26) — brought current, NOT ratified** | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ **STALE outside the PBS tier** verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names `felhom-pbs` → `felhom-offsite` on ep0 over `wg-felhom`, weekly, keep_last=2 — and §9.1 records the measured capacity. **Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's.** R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. | | R-84 | **The agent's cold backup `Store` no longer causes a redundant backup** | S | **SHIPPED (agent v0.103.0, 2026-07-26)** | Observed live: three redundant local backups on demo-felhom in one afternoon of deploys. The `Store` is in-memory ("lost on restart; the cadence re-populates"), so after every restart `/backup/due` said *no successful backup recorded yet* and the controller took another — on the OFFSITE tier a wasted multi-hour WAN upload after every agent deploy. **Resolved by asking the STORAGE rather than persisting the store** (`BackupRunner.NewestArchiveTime` + the optional `localapi.BackupArchiveLister`): ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists; no new on-disk state, no migration. It answers only *when did a backup last land* — the richer fields stay with real records, so the host-report never carries invented numbers. Fail-safes: read error → fall back to memory (never fake freshness, never suppress); genuinely empty → due; old archive → still due. **Proven live on both boxes** immediately after a restart with the store cold: demo-felhom `local age=5298s / felhom-pbs age=14373s`, demo-hp `local age=498s / felhom-pbs age=2346s`, all `due:false`. | +| R-85 | **The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD** | M | **Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE** | R-82 proved a PBS restore round-trip **by hand**. Three defects kept the unattended path from covering it, each verified at source: **(a)** `buildRestoreTestScheduler` took `cfg.Backup.BackupTarget()` and built ONE runner on it, so the offsite tier's archives were never candidates; **(b)** the `Spec` was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; **(c)** a failed restore-test was a `[WARN]` log line with **no event, no notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. **Selection ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported **so the achieved interval can be MEASURED before anyone decides whether a knob is needed** — measure, then configure, not the reverse. **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS, never fails); persisted `RestoreTestState` (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); `backup.InFlight`, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub `restore_test_failed` / `restore_test_stale`, kept **distinct** and anchored on R-81's verdict structure, operator-tier only. **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). | | R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog | | R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | **SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result.** The universal secret-free ISO was booted on a completely different machine — an **HP t740 (Ryzen V1756B, AMI M42 firmware)** versus the N100 it was proven on — and the whole chain ran on **virgin hardware in one pass**: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host `demo-hp-bb76ea`. **Two things generalise from the second board specifically:** the **shim** loader booted with **Secure Boot ENABLED** (`mokutil --sb-state` → `SecureBoot enabled`), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's **1TB NVMe untouched and unenrolled** (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. **Not clean, and the failures are filed:** the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (**R-59**), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (**R-61**) | R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** |