installer 1.24.0: a PRE-EXISTING backup target is granted too (R-185)

configure_backup_target has two arms and only one granted. Case A creates the storage and grants in the same breath; the Scenario-F arm ('the target already exists') returned without granting. A box whose felhom-backup pre-dated the install therefore pointed local_backup_target at a storage its own token could not read — measured on BOTH demo boxes: {"data":[]} through the token while root lists three archives. That tier was never restore-tested and nothing said so, because an empty listing is also what a brand-new tier returns. The reuse arm now ensures the ACL through the same guarded wrapper. Scenario F is unviolated: the storage DEFINITION is untouched, and pveum acl modify is idempotent. BACKUP_TARGET_ID is deliberately NOT added to PVE_STORAGES — that list is granted a step before the target is resolved, and --acl-storages entries are preflight-checked for existence; the comment now says so. A gate asserts it: every arm that resolves the target must also grant on it. Red-proved by reverting the arm.
REPORT: point at the agent report for the later 2026-08-03 session (docs-only here)
2026-08-03 18:58:49 +02:00 · 2026-08-03 16:49:17 +02:00 · 2026-08-03 16:46:49 +02:00 · 2026-08-03 15:35:17 +02:00 · 2026-08-03 15:33:19 +02:00 · 2026-08-03 15:29:59 +02:00
52 changed files with 1629 additions and 3072 deletions
@@ -17,6 +17,87 @@
 ## Standing rulings
 **S-19 — a restore-test PROOF is durable and reportable; a FAILURE is neither, and that asymmetry is
 the design (2026-08-03, R-189; agent v0.122.0).**
 *The rule.* Only successful restore-tests are written to `RestoreTestState`, and that state is what
 the host report carries after a restart. Failures live only in the in-memory `backup.Store`.
 *Why, in one line each.* A **success suppresses future work** — under R-86's per-archive due-check a
 proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it
 believes, for a whole archive generation (a week on the offsite tier). A **failure causes future
 work** — a failing tier stays due and is retried at the next evaluation, so a lost failure heals
 itself within one interval, while a *persisted* failure would outlive the fault it describes.
 *What the report does with the two.* The collector merges them: **one entry per tier, newest by
 `TestedAt` wins**. A fresh failure therefore beats a stored success (the failure is the news and
 exists nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier can
 never appear twice — the hub would read that as two tests.
 *It refuses to lie.* A persisted record missing the archive **or** the tier produces **no entry**: an
 unproven tier reading as proven would be worse than the defect this closes. Run mechanics (scratch
 VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be.
 **Migration consequence, seen live:** a pre-R-189 record has no tier, so upgrading does not
 retroactively make an old proof visible to the hub; the tier's next real proof fills it in.
 **S-20 — the release order is build → tag LOCALLY → publish → push tag, and every step protects
 something (2026-08-03, R-188 + R-186).**
 The tag is created before the publish, so the build and the tag describe the same commit. It is
 **pushed** after, because the push is what wakes CI (`on: [push]`) and a tag visible before its
 package made `check-published-versions.py` correctly fail a *correct* release — measured on roughly
 every second release, and R-168 mails those failures to the operator.
 The invariant the old order protected is **asserted directly instead**: the gate now also refuses a
 **published version with no tag**, as a bounded probe (frontier + patch gaps) that prints its own
 coverage, because the package listing api is 401 without a token and absence cannot be enumerated.
 A half-done release is loud: publish-then-failed-push dies naming the recovery command, and a failed
 publish deletes the local-only tag so a retry is clean.
 **A released binary is independently verifiable** — `-trimpath -buildvcs=false` means the same source
 yields the same bytes with or without the tag; the verification command lives in
 `felhom-agent/CLAUDE.md`. Both build paths (`release-agent.sh` and `publish-agent.sh`'s fallback) use
 identical flags: they differed by `CGO_ENABLED=0` and produced binaries 74 KB apart for one version.
 **S-17 — restore-testing is PER ARCHIVE GENERATION, and the hub's staleness window follows each
 tier's own rhythm (2026-08-03, R-86; agent v0.121.0 + hub v0.91.0).**
 *The rule.* Let **A** be the newest archive on a tier that has settled for at least the settle lag
 (24 h). The tier is **DUE** when A exists and **A has not already been proven**. The daemon-start
 ticker survives only as the **evaluation interval** (6 h). A daily tier is proved daily on yesterday's
 archive; a weekly tier weekly on its own; a tier with no archive is UNKNOWN, never a fault.
 *The trap, written down so it is not reintroduced.* The literal reading of R-86 — *"due when the
 newest archive is ≥24 h old"* — is **never true on a daily tier**, because a new archive resets the
 newest-archive age to zero long before it reaches the lag. It would have switched restore-testing off
 for the tier that matters most, silently. Red-proved at 0 runs over 5 simulated days
 (`felhom-agent/internal/backup/restoretest_due_test.go`).
 *What the state holds now.* `RestoreTestState` records **which archive** was proven, not just when a
 tier passed — a timestamp cannot answer *"have we proven THIS archive"*. A pre-R-86 file keeps its
 time (rotation ordering survives a deploy) and yields no proven archive, so each tier is due exactly
 once after the upgrade.
 *The old config key.* `backup.restore_test_cadence_seconds` is DEPRECATED. **Negative still disables**
 verbatim; a positive value now seeds the **settle lag** only, and the daemon WARNs once at start-up
 naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
 (default 24 h). It is deliberately NOT carried into the evaluation interval.
 *The hub half is not optional.* `restoreProvenStaleAfter` was a flat 7 days **derived from the cadence
 R-86 removes**, and a healthy weekly tier's proof age reaches EXACTLY 168 h just before its next
 proof — it sat ON the line. `restoreProvenWindow(tier, observed, ok)` now takes the tier's own
 observed archive interval × 4 generations, floored at 7 days, capped at 12 days (strictly inside the
 2-week offsite retention), falling back to the tier's **declared** rhythm (`backupStaleAfter` 26 h /
 `offsiteBackupStaleAfter` 8 d — the backup-freshness checker's own thresholds) when history is too
 short to observe one. Shipping Part 1 alone would have produced a nightly false alarm.
 **S-18 — `ep0` is Tier 2, PROTECTED (operator ruling, 2026-08-03).** D-d named two protected machines
 and did not name ep0 either way; `runbooks/target-selection.md` carried the question in writing for
 two days. The ruling **extends D-d's protected list to three machines**: DooPlex, Peti's cluster,
 **ep0**. It is a classification, not a new set of prohibitions — destroying datastores, prune jobs,
 tunnel config or nftables rules was already forbidden by what it would destroy, and the ordinary
 off-site READ a restore-test performs remains permitted.
 **S-13 — the `mp1` merge landed, and the variant was chosen on measurement (2026-08-03, R-165 / D-a).**
 The appliance's two data volumes are one. **Variant V-c**: the volume mounts at the NEUTRAL path
 `/var/lib/felhom`, and both `/var/lib/docker` and `/mnt/sys_drive` are binds of subdirectories of it.
@@ -123,6 +204,33 @@ Two rulings, one shape: something became live because someone pushed, not becaus
  that bumps a version, before publishing — and a gate that fails on the normal path is one people
  learn to ignore.
 **S-16 — a backup run NOTIFIES ONCE and RECORDS ALWAYS, and those are different things
 (2026-08-03, R-182; controller v0.194.0 + hub v0.90.0/.1).** Measured: nine per-app capture failures
 reached the hub, two were mailed, seven were dropped by a cooldown whose key carries no app
 identifier — *before* `LogNotification`, so they left no row anywhere.
 - **The record:** `recovery_unit_capture_failed`, per app, unconditionally, now routed
  **record-only** by the hub (`recordOnlyEvents`) — stored and logged every time, never competing
  for an e-mail slot.
 - **The notification:** `backup_run_failures`, ONE per run, listing every failed app with its leg
  and reason plus the counts and free space. **A clean run emits nothing.**
 - **A suppressed operator event now leaves a `suppressed` row** carrying the key that suppressed it —
  for every operator type. *"We chose not to e-mail you"* and *"nothing happened"* must never look
  identical; that is the whole finding, stated as a rule.
 - **The periodic sweep gets a digest too, with NO `run_id`**, so it stays under the ordinary hourly
  cooldown. Without it the sweep's failures would be recorded and never notified — a new silence
  created while closing one. A real run's digest carries a unique `run_id` precisely so the cooldown
  can never collapse a manual run into the nightly one.
 - **Why the silence is safe:** the hub's deadline check raises `expected_backup_missed` from report
  freshness, independently of any mail the box sends (`monitor/deadline.go:396,417`). **If that check
  is ever weakened, this design loses its footing.**
 - **Not taken, and why:** putting `app` in the cooldown key. It fixes the swallowing by producing one
  mail per failing app — a dozen on a full disk.
 **ep0 was rescaled by the operator to a CX33 (2026-08-03): 4 vCPU, 8 GB RAM, measured on the box, and
 the 4 GiB swapfile survived. The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only, so no disk
 figure in any runbook needed correcting. That closed **R-90** and unblocked **R-86**.
 **S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).**
 Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the
 CUSTOMER's** (they can free space, delete files, add a drive) and **a per-app backup capture failure
@@ -1,113 +0,0 @@
 # REPORT — Campaign 10, two-storage adversarial soak (2026-07-31)
 Follows `REPORT-campaign7/8/9.md`. Root `REPORT.md` is another session's (hub v0.85.0) and was not
 clobbered — same shared-clone reasoning as `REPORT-iso-release.md`.
 **Full audit + evidence:** `documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md`,
 `documentation/tests/campaign10-evidence-2026-07-31/`.
 ## The sentence that matters
 **Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the
 "drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No
 resource leak over 13.5 h. Two findings: R-156 and R-157.**
 ## What was established
 - **Venue** — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the
  `/mnt/nvme-1tb` **mount root** (a subdirectory would have emitted `storage_disconnected` for demo-hp
  all night — the exact signal I1/I2 discriminate).
 - **Baselines, all read fresh.** controller `main` 0.188.0, **golden 0.188.0 (not behind)**, agent
  0.119.0 published+vouched, hub 0.86.0, ISO **1.26.1** (`f3cc86d5…`, round-trip verified live).
  The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129.
 - **Isolation gate — both denials captured, each with a positive control.** The PBS control **failed
  first**: four clean-looking 403s were worthless because the token was denied on its own datastore
  too (PBS token privilege separation). Fixed, re-run, denials stand.
 - **A1** fresh install from the **published** ISO. 1.26.1 is a public release image — verified against
  its bytes that it has no auto-install path — so it was driven blind via screendump + `sendkey`
  through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which
  would otherwise have been mangled and locked the box out.
 - **A2** claimed for real; discriminator flipped `dashboard not yet claimed` → `authentication required`.
 - **A3** both drives enrolled through the **real** endpoint; `mentes` accepted as backup target via
  the offer flow, ending `degraded:false / target:felhom-backup` — the I5/I6 healthy baseline.
  Four apps healthy spanning both sides of D5's split (4 × `type: secret`, 1 × `type: password`).
 - **A4** discriminators seed and read back across all four apps; rallly's over the path
  `DATABASE_URL` actually names, not the trusted socket that produced D5's false pass.
 ## Phase B — the soak
 Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, **stopped
 deliberately** — two violations were harness defects), run 2b (**39 cycles, 12 atom families**).
 1 461 invariant checks in total.
 - **I7 is the headline: 66 restores across both passes, 66 correct discriminators.** Never stale,
  never empty. Run 2b added an `I7-SKIP` verdict so a check with unmet preconditions is recorded as
  skipped rather than silently green — it fired once.
 - **I2, I3, I4, I5, I6, I10, I11: zero violations in either pass**, including the abort-in-place
  variants and 330 secret-class assertions covering both sides of D5's split.
 - **`I1-under-load` 5/5**: the backup target pulled **while a backup was running** still produced
  `backup_target_absent` and a clean recovery.
 - **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via
  `bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names
  it. That is the case R-117's spike called "the worse half".
 - **RTO, both bands measured.** S: 66 MB → **42.0 s** / **41.4 s** across two passes (66 restores).
  M: 21.1 GB → **608 s** mean over 2 reps, both returning the correct discriminator. 327× the data
  cost 14.5× the time, giving **RTO ≈ 40 s + 26.9 s/GB** (backup ≈ 29 s + 17.4 s/GB). The fixed ~40 s
  dominates below ~1.5 GB — that is the S band, and why its numbers clustered so tightly.
 - **Capacity ceiling, and the more consequential result:** a DB-backed app's recovery unit is **1.90×**
  its data (volume tar + SQL dump). The default `/mnt/sys_drive` is **20 GB**, so on a default box the
  largest locally-backupable app is **≈ 10 GB** — **the M band does not fit at all** without a
  per-customer `SysDataGrowGB`. **RPO still not measured.**
 - **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds
  flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots
  and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)** —
  bounded by journald, but a lot of logging.
 - Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that
  was not first observed live.
 - **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system
  disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the
  soak would have produced green cycles that tested nothing cross-drive.
 - **Two violations were my harness, not the product**, and run 2a was stopped for them: a seed that
  never landed became a fake "stale restore", and a real one would have looked identical. Fixed and
  red-proofed before restarting.
 ## Findings
 - **R-156 (new)** — **papra's data is neither persisted nor backed up, and it reports healthy.** The
  template mounts `papra_data:/app/data`; the app writes `/app/app-data/db/db.sqlite`. The volume is
  empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB
  sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its
  Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
 - **R-157 (new)** — **bootrecon's start-once sweep misses the boot orphan it exists to recover.** Two
  mechanisms. **A:** the container is left `Exited`, the sweep runs ~5 s after controller start while
  docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences,
  intermittent ~50%). **B:** the interruption leaves the stack with **zero containers**, which is
  exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp
  check reported **`0 currently down`** while a `deployed: true` app was not running, i.e. silent on
  every channel. A settle-condition fix closes A and leaves B open. Not fixed.
 - **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier
  (`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred,
  `hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign
  therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by
  not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows.
 ## What did not run
 **12 of the brief's ~13 atom families ran** (run 1 covered 6; run 2b added abort-fs-in-place,
 kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive).
 Previously reported as 6 of 12 — that was run 1 only.
 **Superseded detail:** Still not run: **Tier-3 backup/restore** (§3, structurally un-isolatable) and **I8**. **I9** was not
 automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed.
 `kill_controller` is still not literally "mid-backup"; the dedicated concurrent backup+detach atom
 covers that case properly. The run-1 flaw where `reboot` never interleaved with a detach was fixed.
 **Depth reached: 39 consecutive cycles**, past the brief's "thirty-eighth", with c34–c39 clean on
 every invariant. Beyond 39 is untested, not proven clean.
 ## Teardown — OWED, nothing removed
 Still intact: the rig is reusable for the atoms that did not run. VM 311, `c10-scratch`, PBS datastore
 `felhom-c10` + user/token, restic subaccount `u629488-sub4`, and **hub customer `c10-soak` (disposition:
 DELETE)** are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four
 orphaned scratch customers left by exactly this omission.
@@ -1,43 +0,0 @@
 # REPORT — CAMPAIGN 7 (felhom.eu side: docs only)
 > Written as `REPORT-campaign7.md`, **not** the shared `REPORT.md`, per the convention this run
 > added to `CLAUDE.md`: `REPORT.md` is overwritten, so a second concurrent session in this repo
 > would clobber it. This session's implementation work was in `app-catalog-felhom.eu`; here it only
 > touched documentation.
 **Run:** 2026-07-18 evening → 2026-07-19 morning. **Class:** campaign (record-and-rank + a defined
 allowed-fix set). **Implementation repo:** `app-catalog-felhom.eu` (see its `REPORT.md`).
 ## What changed in this repo
 | file | change |
 |---|---|
 | `documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md` | **new** — method, uninstall-semantics map, trio detail, full 53-app matrix, ranked findings, coverage |
 | `documentation/backlog/ROADMAP.md` | **+3 items** — R-40 (multi-hop major upgrade path), R-41 (no standing catalog deployability check), R-42 (sidecar-major ruling) |
 | `CLAUDE.md` | REPORT.md parallel-session rule: the second session writes `REPORT-<topic>.md` |
 No hub/agent/scripts/website code was touched (campaign scope: catalog + docs).
 ## Headline for this repo's readers
 1. **Uninstall semantics map row PARTIAL → PROVEN** (campaign doc §2), with live evidence from all
   three trio apps: remove requires stop first; named docker volumes are **always destroyed**
   (including the app's database); HDD bind-mount data and `backups/primary/<app>` survive unless
   explicitly ticked; images are kept; `app.yaml` goes, the template stays; the per-app **offsite
   toggle survives** the uninstall while tier-2 config is cleared. The confirmation modal does warn
   about the volumes, so there is **no consent gap**.
 2. **A lying healthcheck takes an app OFF-LINE, it does not merely mislead.** Traefik will not route
   to an `unhealthy` container, so a probe that cannot execute → permanent unhealthy → **404 to the
   customer while the app serves 200 on its own port**. 7 of 53 apps were in that state.
 3. **The pre-flight gate's own signal is missing:** the 0.145.0 → 0.146.0 floor-lift emitted no
   `controller_updated` event, though the identical bootstrap path emitted one for 0.143.0 → 0.145.0
   two hours earlier (§0, finding F1). The box did converge — golden, floor and runtime all agreed —
   but the event trail under-reports version transitions.
 ## Open items owned outside this repo
 - **plant-it / wanderer** — images do not resolve at all (neither the new tag nor the one the
  catalog already ships). Upstream research needed; recorded as findings, not deletions.
 - **gokapi** — pinned back to v1.9.6; v2 needs the seeded `config.json` regenerated. Security-
  relevant, should not sit on a superseded line indefinitely.
 - **glance** — never had a seeded `glance.yml`; proven pre-existing.
@@ -1,62 +0,0 @@
 # REPORT — CAMPAIGN 8: the backup & restore subsystem (2026-07-27/28)
 Adversarial, destructive, unattended run against `demo-felhom`, `demo-hp` and `ep0`.
 Full report: `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
 Evidence: `DooPlex:~/campaign8/evidence/` (103 files, 35 MB, written continuously by 11 collectors).
 **No production code was changed.** Findings are recorded and ranked, never fixed inline, per the
 campaign's own rules.
 ## Scope safety
 `peti-felhom`, its namespace and `u629488-sub2` were never touched. Phase 0 established with five
 documented probes that peti has **no data at all** in `felhom-offsite`, which is what made the
 operator-approved 100% datastore-fill safe. The 13 GB rollback copy `/srv/pbs-felhom` on ep0 is
 intact.
 ## Findings
 | # | Finding | Severity | Class |
 |---|---|---|---|
 | F-CRIT-1 | An app that fails to restart after a quiesce **never alarms**, on any channel | **HIGH** | DEFECT |
 | F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** (up to 7 days silent on real cadences) | **HIGH** | DEFECT |
 | F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) |
 | F-HUB | The hub dropped an event under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT |
 | F-LEAK | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed | MEDIUM | DEFECT (root-caused by fault 18) |
 | F-REBOOT | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
 | F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
 | F-OBS | `deadapp-check` leaves no positive observable on a default (info-level) box | LOW | DEFECT |
 | F-OPS | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational |
 Both HIGH findings are in the same place: **the system's ability to tell you a backup did not
 happen.** Both cite the code and the comment that asserts the property the code does not provide.
 ## What is now proven that was not before
 R-88 breaker arming **and its full ladder** (15m/30m/1h/2h/4h/4h-cap) · per-tier isolation under a
 real one-tier-fails case · `whole_guest_backup_failed` end-to-end with correct tier attribution ·
 R-97c operator-only routing verified against the hub DB (zero customer rows with `status='sent'`) ·
 `whole_guest_backup_recovered` + the R-68 pairing gate firing live · **`age_state=absent`** ·
 R-97b's suppression half · the crash-recovery unquiesce by an actual SIGKILL (1 s) ·
 **R-87 — the first restic restore round-trip ever performed**, byte-verified (6/7 sha256 identical,
 the 7th explained) · R-82 one-quiesce-two-tiers · single-flight on two independent paths.
 ## Restore round-trips
 restic (R-87) · local vzdump → fresh CT · PBS offsite → fresh CT · corrupted snapshot → fails
 cleanly. `mount_parity` exact on both whole-guest tiers, `unprivileged: 1` preserved.
 ## Fleet state
 **Healthy. Nothing left broken.** All four compression knobs reverted and verified; every fault
 unwound; no leaked scratch guests, nft rules, ballast files or clock skew; ep0 datastore clean with
 zero `.bad` chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8.
 ## Not tested (with reasons)
 Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; **the
 most valuable follow-up**, because F-CRIT-2 raises the same question for restic), fault 12 (host
 reboot — reasoned skip), and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for
 documented reasons. Faults 11 and 18 WERE run in the campaign's tail and both produced findings.
 **Campaign-caused outage, stated plainly:** fault 11 took demo-hp guest 9201 down for ~9m47s
 (the guest did not restart after a mid-backup reboot) until manually started. Fleet healthy after.
 ## Note on repo conventions
 This run touched no `hub/`, `scripts/` or `website/` code, so none of the per-area CHANGELOGs has an
 entry — there is nothing shipped to log. The deliverable is the audit document plus this report.
@@ -1,85 +0,0 @@
 # REPORT — CAMPAIGN 9: the restore paths, proven (2026-07-28)
 **Overwritten** per the standing rule. **No production code shipped** — this was a proof campaign,
 and findings are recorded, never fixed inline. Full write-up:
 `documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md`.
 Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously).
 Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes.
 **`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact.
 ## The headline — two never-proven restore paths are now proven
 Driven through the **real endpoints the UI posts to**, over https through traefik with a real session
 and CSRF token, on live hardware.
 | proof | result |
 |---|---|
 | **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) |
 | A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) |
 | A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy |
 | A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 |
 | **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy |
 | **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 |
 The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return
 at their last-backup state.**
 ## Findings — 3 defects, ranked (none fixed)
 | # | finding | severity |
 |---|---|---|
 | **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** |
 | **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** |
 | **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** |
 Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog
 sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an
 artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed
 so live.
 ## Mechanisms confirmed working, live
 R-82's one-quiesce rule under mixed outcomes (2 tiers due, apps stopped **once**, per-target
 breaker); R-88's breaker (edge-triggered, one WARN, one event, three silent DEBUG skips, **no app
 thrash**); F-A1's contention deferral (409 → no breaker, no event, prompt restart — both sides of
 the seam captured in the same second); **F-CRIT-2's size filter against a real 1-byte phantom** on
 demo-hp, confirmed independently on ep0's filesystem; R-100's success anchor twice; **F-DIAG's
 sanitiser on the exact bare-hostname case that defeated its first version** (nothing raw reaches the
 hub event or the report); F-OBS's positive observable — which is precisely what made C9-F2 provable;
 F-LEAK's fenced destroy (no leaked `990000` guests across ~10 restore-tests).
 ## Where it stopped, and what remains
 Stopped at the **end of Phase B**, plus Phase D item 10, then full recovery. Phase C item 6 (host
 reboot mid-backup) was deliberately not started — a large new fault class against boxes that are
 remote until ~08-02, and starting it would have meant rushing it or leaving the fleet unknown.
 **Approved but impossible:** Phase 0 cleared compressing the hub's `staleAfter` for R-100's
 threshold test. It is **not a knob** — `cmd/hub/main.go:552` passes `0`, selecting the compile-time
 `defaultOffsiteStaleAfter = 48h`. Compressing it needed a hub code change, which the campaign
 forbids. Reported rather than worked around. The no-code-change alternative (age the controller's
 reported `last_success` past 48 h and let the hub judge at its real threshold) is the recommended
 method next time.
 **The honest residue — still not proven:** Tier-1 **content** recovery after real loss (A2 ran on an
 intact app; A3 used Tier-2) — now the most valuable open item; host reboot mid-backup; three-way
 concurrency with GC; Scenario C live; `offsite_stale` actually firing; F-HUB `SQLITE_BUSY`.
 ## Recovery
 Every config reverted from `evidence/config-before/REVERT.md`, each verified with a **positive
 observable**: agent cadences back to `0 / 302400 / 604800` on both hosts (`is-active` = active),
 windows back to `02:30`, `pvesm` shows `felhom-pbs active` on both, 0 campaign iptables rules on
 either host or guest, 0 scratch guests in the `990000` band, all stacks healthy on both boxes, and
 the offsite tier not merely unblocked but **proven working again** (`ok`, 1m35s, 8 snapshots).
 One benign residue: the in-memory R-88 breaker still holds a `felhom-pbs` failure count on each box.
 Its `until` is long past so it blocks nothing; it clears on the next successful backup or any
 controller restart (by design, not persisted). Clearing it would have cost another app outage for no
 benefit.
 **One operational lesson worth a runbook line:** a hand-run `docker compose up -d` in
 `/opt/docker/stacks/<app>` starts a Felhom app **without its secrets** — they are injected by the
 controller's `stackEnv` at start time, not stored in a `.env`. It turned a healthy docmost into a
 crash loop during recovery. Manual recovery must go through `POST /api/stacks/<name>/restart`.
@@ -1,71 +0,0 @@
 # REPORT — DIAGNOSE immich offsite restore (felhom.eu side: docs only)
 > Written as `REPORT-diag-immich-restore.md`, **not** the shared `REPORT.md`, per the CLAUDE.md
 > convention — `REPORT.md` is overwritten and currently holds the 2026-07-18 website refresh.
 > No code shipped in this run; findings only.
 **Date:** 2026-07-19 · **Box:** demo-felhom (felhom-pve guest 9201), controller 0.146.0, immich v3.0.3
 > **CLOSED IN CODE 2026-07-19 — controller v0.148.0.** Findings 1 and 2 shipped as R-43 (offsite
 > reconstitution: safety dump → stop → overwrite files → start → replay the snapshot's dump) and
 > R-44 (every run dumps before it captures; manifests carry `offsite_run_id` + `dumps_at`). Deployed
 > to demo 9201, healthy. **The §9 live acceptance has NOT run**, so no capability-map flip: the
 > offsite row is PARTIAL, the customer-restore row stays MISSING, R-3 stays DRAFT. Implementation
 > detail lives in `felhom-controller/REPORT.md`.
 ## What ran
 A diagnosis of "immich offsite restore succeeds but photos do not reappear". No product code was
 changed: no restore/backup logic, no labels, no flashes, no `restic prune`/`forget`, no snapshot or
 escrow changes.
 ## Outcome
 The restore did not fail. **It was never invoked on missing data, and could not have worked
 if it had been.**
 1. Viktor deleted the 11 photos in the immich UI to test offsite restore. A UI delete means
   **trash**, not deletion — no file left the disk. Both „csak a hiányzó fájlok" runs merged
   **0 files**, correctly, and flashed success. The test proved nothing.
 2. A *valid* test would also have failed: **no offsite path loads a database.** All three buttons
   are file-only. Files would return; the library would stay empty.
 3. The shipped dump is additionally stale by design — from the 02:30 local run, never refreshed
   before a manual push. Probed: **`asset: 0`, `user: 0`, `album: 0`**.
 **Photos:** left in trash at Viktor's instruction (recovery not wanted). All 11 files verified
 present on disk and all 11 rows intact, so an ordinary un-trash recovers them until immich's
 30-day auto-purge.
 **Answer to "can a customer trust same-day offsite?"** For a DB-indexed app: **no — files come
 back, content does not.** The backup half is honest; the restore half cannot reconstitute the app.
 ## Decisive evidence
 - `updatedAt` == `deletedAt` on all 11 asset rows ⇒ **no restore operation touched the DB.**
 - Unit dump `immich-postgres.sql`, 51 954 452 B, mtime **02:30 CEST** ⇒ `asset: 0 / user: 0 /
  album: 0`. The 52 MB is entirely immich's shipped geodata reference tables. It predates the admin
  user (07:56:25) and the photos (07:57).
 - **Latent hazard:** had a full restore loaded that dump, it would have written an empty DB over the
  live one, destroying the trashed rows that were the only surviving recovery path.
 ## Files written
 | File | Change |
 |---|---|
 | `documentation/audits/DIAG-immich-restore-2026-07-19.md` | new — timeline, evidence, source mechanism table, 7 findings |
 | `documentation/backlog/ROADMAP.md` | **R-43** (P2-HIGH, offsite restore cannot restore a DB) + **R-44** (P2-HIGH, manual push ships unrefreshed dump) |
 | `documentation/architecture/00-capability-map.md` | customer-restore row **stays MISSING**, gains the finding + a method note for R-3; offsite-restore row flagged *scope contested* |
 ## Needs Viktor
 1. **Capability-map ruling (line 61).** The offsite-restore row claims PROVEN-LIVE citing
   "immich end-to-end from offsite alone" (`CAMPAIGN-6D`). That phrase is contested by this
   diagnosis. I flagged it rather than downgrading a proven row — did 6D's accept leg exercise the
   DB half, or only the file half?
 2. **Optional real red-proof.** Emptying the trash would create genuine data loss and convert the
   dump-probe inference into a live proof. Offered, **not run** — irreversible, and the probe
   already settles it.
 3. **Unreconciled: the 704.6 MiB figure.** Measured 126 MB total on the library storage. If
   704.6 MiB came off a controller Storage page, that gap is its own defect.
 4. **Orphaned pre-v3 tree** `dccc13fe…` (~55 MB across upload/thumbs/encoded-video) stranded by the
   immich 2→3 redeploy — worth a sweep policy for major redeploys.
@@ -1,53 +0,0 @@
 # REPORT — Megosztás diagnosis (2026-07-20)
 Topic-scoped report (parallel-session rule: shared `REPORT.md` untouched).
 **Run:** RUNBOOK "Megosztás diagnosis — SMB unreachable from Mac + sharing-page reload loop".
 Read-only diagnosis. **No code changes, no version bumps, no builds, no restarts.**
 **Deliverable:** `documentation/audits/DIAG-sharing-2026-07-20.md`.
 ## Verdicts
 * **Reload loop — ROOT-CAUSED (HIGH).** `sharingStatusHandler` (`sharing_handlers.go:246`, added in
  `b5d78d1`, controller v0.147.0, 2026-07-19) coerces `idle` → `running` whenever the samba
  container is alive. `sharing.html` L320–326 treats `running` as a one-shot job-success edge and
  calls `location.reload()` 1.2 s later. The first `tick()` fires synchronously on every page load,
  so the page reloads forever. Unconditional for any customer with sharing enabled — the Megosztás
  page is currently unusable. Proven live: 6 consecutive `/sharing/status` polls all returned
  `{"phase":"running","running":true}`, and the controller log shows **no ensure job ran at all**,
  so the phase is manufactured by that line rather than left over from a stuck job.
 * **`smb://192.168.0.162` — ROOT-CAUSED.** `.162` is the Proxmox host and never was an SMB endpoint.
  smbd runs in guest 9201 and binds `192.168.0.104:445`. `nc` from the host: `.104:445` **open**,
  `.162:445` **refused**. Wrong target, stale Finder favourite.
 * **`smb://FELHOM` — OPEN, narrowed.** NetBIOS resolution works on the wire
  (`nmblookup -B 192.168.0.255 FELHOM` → `192.168.0.104 FELHOM<00>`), but the stack advertises **no
  mDNS/Bonjour** (nothing on udp/5353; the R-6 spike selected `smbd + nmbd + wsdd` only) — the
  mechanism macOS Finder prefers. Closing this needs one probe from the Mac (`smbutil lookup FELHOM`
  / `dns-sd -B _smb._tcp`), listed in the audit's Mac test matrix.
 * **H2 (container down/crash-looping) — RULED OUT** at the first probe: `felhom-samba` Up 3 h, clean
  logs, smbd/nmbd/wsdd all bound as `infra/samba.go` intends, live `smb.conf` matches the renderer
  with no baked address literal, no PVE firewall in the path.
 ## Findings (full table + evidence in the audit doc)
 S-1 HIGH reload loop (XS fix: latch a `sawInFlight` flag in the JS; red-proof required) ·
 S-2 MED the UI never shows the connect address, only `\\FELHOM` — customers guess IPs ·
 S-3 MED no mDNS advertisement (image slice, needs republish) ·
 S-4 LOW ensure-job phase never resets — fold into the async-job-feedback roadmap item ·
 S-5 INFO the guest's LAN IP is DHCP, so any displayed address must be read live, never cached.
 Recommended packaging: S-1 as an immediate patch task (it bricks a shipped page), S-2 alongside it
 if the guest IP is reachable from the sharing handler, S-3 as its own slice.
 ## Also noticed
 The **remote site's LAN is `192.168.0.0/24` — the same prefix as the DooPlex home LAN** that the
 Tailscale subnet router advertises (`192.168.0.180` shows `FAILED` in felhom-pve's neighbour table).
 A successful `ping 192.168.0.162` therefore does not by itself prove the Mac is on the remote
 segment; the Mac matrix starts by confirming which network it is on.
 ## Actions taken
 None. Every command was a read, except a `POST /login` to obtain a session for the status polls.
 No secrets are recorded in either document.
@@ -1,85 +0,0 @@
 # REPORT — R-111 fixed, then E-2 proven on a fresh box (2026-07-29)
 Two phases in one session. Full evidence: `documentation/audits/E2D-fresh-vm-2026-07-29.md`.
 Root `REPORT.md` untouched.
 ## Phase 1 — R-111: the Day-0 channel now serves the current software
 A Phase 0 gate earlier the same day stopped the E-2d run before any VM existed: a fresh box would
 have installed **agent 0.96.0 + controller 0.161.0**, ~17 and ~24 releases behind `main`.
 | | Before | Now |
 |---|---|---|
 | agent (Gitea generic) | 0.96.0 | **0.113.0**, sha `5f3247f7…`, round-trip verified |
 | golden (Gitea generic) | 0.161.0 | **0.185.1**, sha `dba00f3e…`, embeds controller 0.185.1 |
 | hub `min_agent` | 0.93.0 | **0.113.0** (what controller v0.185.0 declares) |
 Bake clean on every marker: `Result=success`, overlay2, **all three mounts in the archive**, 0
 FATAL/exclusions, HTTP 201, token-leak grep 0. GL-1 teardown: guest 9100 purged, secrets shredded,
 drill disk restored to `virgin`. Agent + golden moved in **one** manifest POST so it never vouched a
 new agent against an old golden. `min_agent` verified zero-impact first (all three enrolled hosts
 already at 0.113.0). Global floor deliberately **not** raised — the golden now bakes 0.185.1.
 Commit `3dff357`.
 ## Phase 2 — the E-2d run, full ISO/PAIRING route
 Nested PVE VM on demo-hp, one disk, outside the `felhom` pool. Bind → running controller in
 **3 m 35 s**. The install fetched exactly the artifacts published an hour earlier and restored
 `vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked 20 minutes before. The publish train is
 proven end to end on a real install.
 | Claim | Verdict |
 |---|---|
 | **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** |
 | **C2** Case B fires naturally | ✅ **PROVEN** — both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort |
 | **C3** degraded banner renders **to a customer** | ⚠️ **PARTIAL** — API byte-exact; **no UI consumer exists** → **R-112** |
 | **C4** offer appears and moves the target | ⚠️ **PARTIAL** — decline path, `restart_required:true`, no self-restart, E-2a wrapper, healthy-renders-nothing all PROVEN at API level; offer equally invisible → **R-112** |
 | **C5** `backup_target_absent` end to end | ❌ **FAILED** — zero events on any channel → **R-113** |
 ## The three findings
 **R-112 (P1)** — E-2's banner and offer have **no UI consumer**. The endpoint returns byte-exact copy;
 `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**, and no page handler injects
 the state. Decisive contrast: templates fetch **18** distinct `/api/storage/*` endpoints;
 `backup-target` and `backup-target/assign` are the only two with zero references. v0.185.1 fixed the
 router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired.
 **R-113 (P1)** — the drive-absent gate **cannot fire on device loss**. `planDriveGates` reads presence
 from `BoundUnderParent` = "is this path in the guest's mountinfo". The raw mount is a device-bound
 systemd unit and dies; **the agent's own bind is not device-bound and outlives the device**, so the
 gate sees "present" forever. Live: agent said `enrolled drive absent by UUID` every 20 s for 4½
 minutes, controller logged **0** `[gate]` lines, hub got **zero** events — neither the specific nor the
 generic one. Sixth instance of the class, one layer deeper: E-2b wired the seam to a condition that
 cannot occur.
 **R-114** — on target-drive loss the message says the backup is *"on the same disk as the system"*
 (false) and offers **the drive that just vanished**. Invisible today only because of R-112 — so
 **R-114 must be fixed before R-112 is wired.**
 Also filed as a **second instance under R-110** (not a new ID): host-install fetches **nine** files
 from `raw/branch/main` and the hub vouches a sha for **one**; E-2a's wrapper is installed 0755 to
 `/usr/local/sbin`, root-fenced in sudoers, validated only by `bash -n`.
 ## Record
 - `OPEN-ITEMS.md` — **R-112/R-113/R-114 opened**; E-2d re-stated with results and left open for the
  residue; E-2's "NOT yet live-proven" list resolved into proven / known-broken; R-94 fully unblocked;
  R-110 extended. The drill-cleanup row was opened and then **closed the same session** once the
  teardown completed, so it is not carried in the register.
 - `ROADMAP.md` — R-112/R-113/R-114 under P1; R-111 marked SHIPPED.
 - **`architecture/00-capability-map.md` not touched** — for two reasons: the customer-facing legs are
  broken rather than proven, and the map has **no E-2 / backup-target rows at all** (worth noting
  against the ROADMAP's coupling rule).
 ## Teardown
 VM destroyed, scratch storage removed, **`pvesm status` after == before** (`local-lvm` 38.77 %,
 byte-identical), guest 9201 and drill-r50 untouched. **Hub records removed — teardown complete.** The delete was correctly refused at four gates while the host still read ONLINE; once the destroyed host aged to DOWN (`delete-impact` → `deletable:true`) the documented cascade ran and completed: host deleted, PBS tenancy deprovisioned, claim reset, residue purged. Verified after: **0** `e2d` occurrences on the hosts page, fleet unchanged. The one purged `appliance_registrations=1` was this run's own appliance; the unrelated stale 2026-07-25 appliance (`206c8838…`) was not touched by the cascade — the operator removed it separately.
 ## One human step, and a premise correction
 The runbook's §5.1a operator STOP (the bind) is **retired** — CC did it. But E-2d's premise that a
 fresh install yields a CC-drivable claimable customer is **wrong**: the claim code is bcrypt-hashed and
 email-only, and the gate covers everything except `/claim`, `/api/health`, `/static/`. One operator
 relay of the emailed code was required — which also proved the claim flow end to end.
@@ -1,92 +0,0 @@
 # REPORT — ep0 PBS datastore relocated onto the 100 GB volume (2026-07-27)
 **Class:** supervised operational run (RUNBOOK execution). **No code changed. No version bump.**
 Written as `REPORT-<topic>.md` per the parallel-session rule — the shared `REPORT.md` was not touched.
 **Full record with all evidence:** `documentation/runbooks/RUNBOOK-ep0-datastore-volume-2026-07-27.md`
 ---
 ## Outcome: DONE and verified
 `felhom-offsite` now lives on a dedicated 100 GB Hetzner Cloud Volume instead of ep0's 40 GB root disk.
 | | Before | After |
 |---|---|---|
 | Path | `/srv/pbs-felhom` (root disk) | **`/mnt/pbs-datastore`** (volume) |
 | Datastore total | 37.2 GB | **98 GB** (hub gauge: 97.9 GB) |
 | Used | 28.9 % | **13 %** (hub gauge: 12.6 GB, 13 %) |
 | Headroom to the 80 % warn | 19 GB | **≈65 GB** |
 | Additional customers before warn | ≈2 | **≈7–13** |
 Datastore **name unchanged** — the PBS-DR descriptors, per-box storage ids, ACLs and namespace
 layout that R-39/R-82 made self-healing are untouched.
 **Window:** 06:58 → 07:19 UTC (PBS down 07:00 → 07:17). **Nothing was deleted.**
 ### Acceptance evidence
 | Gate | Result |
 |---|---|
 | Copy integrity | 13,242,207,822 = 13,242,207,822 B · **9,748 = 9,748 chunks** · 7 = 7 snapshots · `backup:backup` · itemised dry-run **0 lines** |
 | Snapshot counts per ns | `demo-felhom` 2=2, `demo-felhom-01` 3=3, `demo-hp` 2=2 |
 | atime semantics | `rw,relatime,discard` — **`relatime` present, `noatime` absent** (GC correctness) |
 | Verify job | `TASK OK`, 3/3 groups, forced re-verification of every snapshot, **0 errors** |
 | §6 mount guard | **refusal observed** — `Job … failed with result 'dependency'`; mountpoint stayed empty |
 | §8 restore round-trip | `source_tier: pbs`, `pass: true`, `mount_parity: ok`, clean teardown, 12m1s |
 ---
 ## Three findings the operator should act on
 1. **`scratch` datastore is configured at a path that does not exist** (`/srv/pbs-scratch`).
   Pre-existing, not caused here, but now logs `ENOENT` on every PBS start. This is the PRIME RISK
   shape ("reports fine, is not there") already live in the config. **Decision needed:** remove the
   stanza or create the directory.
 2. **The runbook's §6 acceptance test proves the wrong proposition.** `RequiresMountsFor` is a
   mount-first ordering guarantee, not a refusal — systemd silently *re-mounts* an unmounted volume
   and PBS then starts safely. The test only bites when the device is genuinely unavailable, which
   is how it was re-run and passed. **Amendment recommended in the runbook record.**
 3. **§11 — storage box `u629193` is NOT simply unused.** No live backup path references it (no
   datastore, no restic repo, no fstab, no `known_hosts` pin; R-17 already deleted `u629193-sub1`),
   **but ep0 carries an enabled, currently-mounted sshfs unit** `mnt-pbs\x2dstoragebox.mount` →
   `/mnt/pbs-storagebox`, holding spike leftovers. Disable and remove that unit before deleting the
   box, or ep0 logs a failed mount every boot. **The deletion is the operator's console click.**
 ## Deviations from the runbook as written
 - **The volume arrived already formatted and mounted** by Hetzner at `/mnt/HC_Volume_106469259`
  (§2 assumed neither). Operator ruled: reformat + repath. The 5 % reserve was reclaimed (`-m 0`).
 - **§8 ran on demo-felhom, not demo-hp** — DooPlex holds no SSH key for demo-hp (the G1 gap). Same
  tier, same relocated datastore, larger archive.
 - **The window was contended** by a stale 10-minute restore-test cadence on demo-felhom: the config
  had already been reverted to 3.5 days on disk, but the cadence is read once at daemon start and
  `NRestarts=0`. Restarting the agent applied it (`cadence=84h0m0s`). The in-flight test was allowed
  to finish rather than aborted.
 ## Process errors made during this run (recorded deliberately)
 - `rsync -aHAX` **OOM-killed** ep0 (3.7 GB RAM, no swap). Cause: a PBS `.chunks/` tree pre-creates
  all 65536 shard dirs → 75,341 inodes, and `-H` retains the whole inode map. `-H` was dropped only
  after **proving** no hardlinks exist (`-links +1` → 0; max link count → 1); PBS references chunks
  by digest, never by hardlink.
 - `/usr/bin/time -v` is not installed on ep0 → exit 127, rsync never ran, and a `| grep … || true`
  wrapper swallowed it while printing a success-looking line.
 - `rsync --version | head -1` reported a working rsync 3.4.1 as "missing" — **the §12 pipe-into-head
  trap, fourth recorded instance in this project.** Both fixed by capturing the command's own `$?`.
 ## Deferred
 1. **Old copy retained** at `/srv/pbs-felhom` (13 GB, 9,748 chunks) as the rollback. Rollback is a
   two-line `datastore.cfg` revert. Reclaim only after a new weekly offsite backup lands on the
   volume, with explicit go-ahead.
 2. **GC not run** — now unblocked by the round-trip, but left for a separate deliberate run. No GC
   schedule is configured on this PBS at all.
 3. ~~Hub PBS-DR capacity gauge not re-read.~~ **CLOSED — verified correct.** The hub operator UI
   (Offsite → PBS DR) reports `felhom-offsite (ep0)` at **97.9 GB capacity, 12.6 GB used, 13 % full**,
   agreeing with the on-box `df`. The gauge follows the datastore's configured path, so the move
   needed no hub-side change and the suspected "wrong filesystem" bug does not exist.
 4. **ep0 has no swap** (temporary 4 GB file removed; box left as found). Worth a small permanent
   swapfile — outside this runbook's scope.
@@ -1,63 +0,0 @@
 # REPORT — F-CRIT-1 + F-A1 fixed (controller v0.179.0, 2026-07-28)
 Docs here. Implementation, all six red-proofs and the full live replay live in
 `felhom-controller/REPORT.md`. The campaign that found both:
 `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
 ## What changed
 **F-CRIT-1** — an app that failed to restart after a quiesce never alarmed, for two independent
 reasons, either of which alone kept it dead: `restartAll` returned nothing (the failure was logged
 and dropped), and `classifyRunStates` whitelisted `StateStopped` on invariant I1 ("the user stopped
 it") — which the quiesce loop had made false by stopping stacks the same `docker compose down` way.
 A failed restart and a user stop are the *same* Docker state; the only difference is that the loop
 tried and could not, now surfaced by `Loop.FailedRestarts()`.
 **F-A1** — HTTP 409 is the agent's single-flight gate refusing while a restore-test holds it, not a
 failure. It is now contention: the tier stays DUE, is dropped before anything stops, and unending
 contention raises its own **BLOCKED** signal rather than going silent.
 ## Bounds, justified against measured reality
 - `contentionRetryAfter` **15m** — longest restore-test observed on the fleet is 12m01s; the agent's
  local restore-test wait is 10m. Caps app-stop churn at 4/hour instead of 12/hour.
 - `contentionAlarmAfter` **3h** — the agent's own PBS restore-test task is capped at 120 minutes, so
  contention outliving that is a *stuck* gate, not a busy one. 3h adds margin and is 15× the longest
  contention actually observed.
 ## Verified live, with the hub DB as arbiter — not from logs
 Same box, same day, same event type; the only difference is 409 versus a genuine error:
 | injection | operator emails (demo-hp) |
 |---|---|
 | **409 contention** | 8 → **8** (none) |
 | **real transport failure** | 8 → **9** |
 And for F-CRIT-1: the failed restart alarmed **9 seconds** after grace expiry with the dashboard
 banner naming the `(stopped)` state, while a **deliberate** user stop on the same box stayed silent
 through **9** dead-app scans (the positive observable that the silence is suppression, not a dead
 detector).
 ## The rule this arc earned
 Added to **both** copies of `CLAUDE.md` (live + `documentation/runbooks/workspace-CLAUDE.md`):
 **a comment asserting an invariant needs a test pinning it, or it is a wish.** Six instances in this
 project have shipped guarantees the code did not provide — `EffectiveProtected`, `newestArchiveOn`,
 the R-97a operator-only claim, `classifyRunStates`' I1, `inflight.go`'s defer claim, and
 `quiesce.go`'s spurious-failure claim. Two were found only on live hardware, and one of those had a
 green, red-proofed test suite over a production path broken two independent ways.
 Corollary recorded with it: prefer a test that asserts the **consequence** (does the alarm fire?)
 over one that asserts the **mechanism** (does suppression expire?). R-97b's Scenario F proved the
 mechanism; the consequence was still broken.
 ## Docs touched
 - `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-1 and F-A1 → SHIPPED + PROVEN-LIVE.
 - `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — both rows struck through, closing
  section added. **All three of the campaign's alarm findings are now closed** (F-CRIT-1, F-CRIT-2,
  F-A1).
 - `documentation/runbooks/workspace-CLAUDE.md` — the invariant rule.
 ## Still open, highest first
 **Fault 4** (restic transport interruption) — four injection approaches were defeated by
 guest-bridged networking, and it is now the most valuable follow-up: F-CRIT-2 answered the phantom
 question for PBS and left the identical question open for restic. Then **R-99** (prune never removes
 phantoms) and **F-LEAK** (a failed restore-test cannot destroy its own scratch guest — observed
 again during this work).
@@ -1,40 +0,0 @@
 # REPORT — F-CRIT-2 fixed: a failed backup no longer looks like a fresh one (2026-07-28)
 Scope: `felhom-agent` v0.105.0 → **v0.106.0**. Docs here. Implementation detail and the full live
 re-test live in `felhom-agent/REPORT.md`; the campaign that found it is
 `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
 ## What changed
 `NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, and NEWEST) as a
 successful backup, so the tier read fresh, went **not due**, and was never retried — seven days of
 silence on the real 168h cadence, invisible to both the R-88 breaker (defers only *due* tiers) and
 the hub deadline monitor (reads the same freshness). It now counts only *plausibly complete*
 entries via a measured 1 MiB floor; undecidable ⇒ not counted.
 **Size is the only tier-agnostic discriminator.** `verification` and `encrypted` are absent on every
 local (dir) archive AND on a good PBS snapshot until `verify-new` catches up — gating on either
 would have rejected 100% of local backups and produced fleet-wide backup thrash. That inverse risk
 is a first-class test, red-proofed by making the filter reject everything.
 ## Verified live, not just in unit tests
 Campaign fault 2 was replayed against the fixed agent on demo-hp — phantom created, rejected and
 announced once; the tier correctly reported DUE and backed up (4,359,968,099 B landed); and the
 inverse showed **no thrash**, with 91 scheduler ticks as the positive observable that the loop was
 alive rather than dead.
 ## Settled along the way — no retention bug
 Server-side prune does **not** count phantoms toward `keep-last`: a dry-run against three real
 snapshots plus a phantom retained two real ones plus the phantom. The feared "two phantoms ⇒ zero
 real backups" does not occur. Prune never removes them either, so they accumulate one per aborted
 upload — filed as **R-99** (LOW, hygiene), not as a retention bug.
 ## Docs touched
 - `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-2 → SHIPPED+PROVEN-LIVE; **R-99** filed;
  **F-CRIT-1** filed as READY-HIGHEST (Campaign 8's other HIGH finding, untouched here).
 - `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — F-CRIT-2 row struck through and
  a closing section added.
 ## Still open, highest first
 **F-CRIT-1** — an app that fails to restart after a quiesce never alarms, for two independent
 reasons. Then fault 4 (restic transport interruption), which this fix makes more pointed: the
 phantom question is now answered for PBS and still open for restic.
@@ -1,140 +0,0 @@
 # REPORT — F-REBOOT + F-LEAK + F-OBS, and two investigations (2026-07-28)
 Scope in this repo: **`scripts/felhom-host-install.sh` v1.20.0 → v1.21.0** (which is where F-LEAK's
 *actual* fix lives), plus the Campaign 8 audit doc and `OPEN-ITEMS.md`. Written as
 `REPORT-freboot-fleak-fobs.md` so the shared `REPORT.md` is not clobbered.
 Code companions: `felhom-agent` v0.106.0 → **v0.110.0**, `felhom-controller` v0.179.0 → **v0.180.0**.
 **Correction to this repo's part of the story:** v1.21.0's band-scoped ACL is *not* the final F-LEAK
 fix. It works, but only **once per slot** — PVE's destroy path calls
 `AccessControl::remove_vm_access($vmid)` (`API2/LXC.pm:906`), which deletes every ACL at `/vms/<vmid>`
 (`AccessControl.pm:1898`), so **the grant is consumed by the operation it authorises**. Found by counting
 ACL rows after the first successful teardown (`/vms/990000` → 0 grants), not by reasoning about it. The
 durable fix is agent **v0.110.0**'s band-scoped fenced destroy; v1.21.0 remains valuable because it makes
 the common case need no privileged call, and it is now the *first* of two layers rather than the only one.
 ## Baselines (reconfirmed, not copied)
 `felhom.eu d0cec9d`, `felhom-agent af1c21a`, `felhom-controller fb91c8d`, all clean. Agent `0.106.0`
 and controller `0.179.0` live on both demo boxes.
 ---
 ## host-install v1.21.0 — F-LEAK, and why the fix landed *here* rather than in the agent
 **The finding.** A restore-test whose restore **fails** leaves a scratch guest the agent cannot destroy
 (`403 missing privilege VM.Allocate`), so a half-restored guest holds its disks until a human removes
 it and the 10-slot scratch band shrinks silently.
 **The cause is structural, not a missing privilege in the role.** `FelhomAgentGuest` is granted at
 `/pool/felhom`, and **a guest joins that pool only when its restore completes**. A failed restore
 therefore produces a guest that exists, is in no pool, and is out of the token's reach entirely.
 **The first fix was wrong, and its own live replay is what proved it.** Agent v0.107.0 shipped a
 teardown fallback that adopted the stranded guest into the pool and retried — reasoning from
 `Pool.Allocate` on `/pool/felhom`. It fired exactly as designed and PVE refused it:
 ```
 ERROR restore-test: pool adoption failed; left for Recover  vmid=990000
      err="proxmox: PUT /pools/felhom -> HTTP 500: permission denied at /vms/990000 (missing privilege ...)"
 ```
 `PUT /pools/{pool}` **also** requires `VM.Allocate` on the VM being added. **Pool membership cannot
 bootstrap its own authority.** Removed in agent v0.108.0 rather than left in place — a path that
 provably cannot work is worse than none, because it reads as a fix.
 **What shipped instead.** `apply_scoped_acl` now grants `FelhomAgentGuest` at each `/vms/<id>` in
 `PVE_SCRATCH_VMID_MIN..PVE_SCRATCH_VMID_MAX` (990000–990009 — the band the restore-test already picks
 from), to **both** the user and the token, because the privsep-intersection rule applies here as
 everywhere.
 Two supporting changes, both load-bearing rather than tidy-up:
 - **`remove_scoped_acl` deletes the band grants before the role delete.** PVE refuses to delete a role
  still referenced by any ACL, so omitting this would have broken the uninstall — a failure that would
  only surface on a decommission.
 - **`step_verify` asserts the band grants.** A missing one is otherwise invisible until a restore-test
  *fails*, which is precisely the case that leaked a guest in the first place.
 ### Why the grant is still not a widening — proven live, at the seam the defect lives in
 A real PBS restore to `990000` **without `--pool`** reproduced the exact stranded state
 (`990000 stopped`; `felhom pool members: [9201]`; `990000 in pool: False`). Then, with the agent's own
 token, same guest, minutes apart:
 | | `DELETE /nodes/<node>/lxc/990000` |
 |---|---|
 | **grant removed** (the original defect) | `403 Permission check failed (/vms/990000, VM.Allocate)` — guest still present |
 | **grant restored** (the fix) | `200 UPID:...:vzdestroy:990000:felhom-agent@pve!agent` — guest gone |
 And it still cannot reach anything else:
 | target | result |
 |---|---|
 | `/vms/990010` (one past the band) | **403** `Permission check failed (/vms/990010, VM.Allocate)` |
 | `/vms/100` | **403** same |
 `990010` does not exist and PVE **still** answered 403 rather than "does not exist" — so PVE evaluates
 **permission before existence**, which makes these genuine authorization refusals rather than artifacts
 of a missing guest. Granting at `/vms` was considered and rejected: it would authorise destroying every
 guest on the box, including a co-tenant's.
 Applied on **both** demo boxes (demo-hp and demo-felhom) so the live fleet matches the installer.
 **A careless step of mine, recorded rather than buried.** The probe loop also issued a live `DELETE`
 against running guest **9201**. It was refused with `500 container is running` — but the *permission
 check passed* (9201 is a pool member by design), so had the guest been stopped I would have destroyed
 the live demo guest. The scratch-band probes were the safe ones; 9201 had no business in that list.
 ---
 ## Documentation changes
 - **`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`** — F-REBOOT, F-LEAK and F-OBS
  written up as FIXED with their live evidence, including F-LEAK's refuted first attempt (recorded
  precisely *because* it looked right), plus a new **§6b** for the follow-up investigation.
 - **`documentation/backlog/OPEN-ITEMS.md`** — three findings closed, **R-100** filed.
 ---
 ## R-100 — the investigation's finding, deliberately NOT fixed
 **A restic offsite tier that fails every night never goes stale on the hub.** This is **F-CRIT-2's
 defect class one layer up and on the other tier** — a *failed* run resetting the freshness clock — and
 it was found by asking whether F-CRIT-2's shape existed anywhere else.
 Both halves verified in the source, not inferred:
 - **Controller:** `o.LastRun = time.Now()` is set **unconditionally** at
  `controller/internal/backup/offbox.go:716`, *outside* the `runErr` branch. The failure is recorded
  faithfully, but into a different field — `o.LastStatus = "error"` at `:725`.
 - **Hub:** `isStale()` reads **only** `off.LastRun` (`hub/internal/monitor/offsite.go:120`, `:127`,
  `:131`) and never consults `LastStatus`.
 So a nightly restic run that fails every night keeps `LastRun` fresh, `isStale` is permanently false,
 and the staleness alarm never fires — with no successful offsite backup having occurred at all.
 **Scope of the silence, stated precisely rather than dramatically.** `LastStatus` *does* reach the hub —
 it is parsed into the report struct and **only logged** (`offsite.go:270`); it drives no checker and no
 notification. The controller's own guest UI surfaces `LastStatus="error"`, so the failure is visible to
 someone who looks. What is missing is the **push**: the operator's fleet-wide alarm plane is silent,
 which is the plane that matters for an unattended appliance.
 Not fixed, per this task's investigation-only scope. Fix direction: gate staleness on the last
 *successful* run rather than the last attempt — exactly what F-CRIT-2's `NewestArchiveTime` fix did for
 the PBS tier.
 ---
 ## Verification
 `bash -n scripts/felhom-host-install.sh` clean. The ACL behaviour was verified **live on real hardware**
 rather than by dry-run, since the whole finding is about what PVE's authorizer actually does — and the
 live run is what refuted my first design.
 ## Fleet state
 Agent **0.110.0** (with the updated sudoers) and controller **0.180.0** on both demo boxes, all healthy.
 Scratch-band ACLs at 20 rows on both — re-applied after the attempt-2 destroy consumed one. No leftover scratch guests. demo-hp's `restore_test_cadence_seconds` reverted **600 → 302400**
 (a bounded change made for the replay).
 `felhom.eu`: this repo has a **foreign uncommitted WIP file** (`documentation/PROMPT-TEMPLATE.md`) from
 another session in the shared worktree. Left untouched; my commits staged explicit paths only, per the
 never-`git add -A` rule.
@@ -1,119 +0,0 @@
 # REPORT — ISO boot branding + single-entry GRUB menu (R-38) · website grid restored (2026-07-19)
 > `REPORT-<topic>.md` per this repo's parallel-session rule: another session was writing in this
 > clone tonight (CAMPAIGN 7 / `DIAG-immich-restore-2026-07-19.md`), so the shared `REPORT.md` is left
 > untouched.
 Parts 1 and 2 of the polish train. Parts 3 and 4 landed in `felhom-agent` and `felhom-controller`;
 see their own `REPORT.md`.
 ---
 ## Part 1 — scripts v1.22.0: GRUB branding + single-entry safety (R-38)
 ### What shipped
 Every ISO is now **repacked** after `prepare-iso`. `mkimage-surgery.sh` → **`iso-repack.sh`**:
 branding and the slice-B loader swap need the same extract → modify → re-master cycle, so they share
 one pass instead of re-mastering twice. **The mkimage recipe is untouched.**
 **The safety half — the one that matters.** The stock PVE menu offers *Graphical*, *Terminal UI*, a
 serial variant, and an **Advanced Options** submenu holding two `nomodeset` entries, three debug
 entries, *Rescue Boot*, memtest and *UEFI Firmware Settings*. Every one of those reaches the
 **manual** installer, whose first question is which disk to wipe. They are **not emitted** — not
 hidden, not password-gated. What ships is one entry, „Felhom telepítés", default, 5 s.
 **Boot behavior is unchanged.** The `linux`/`initrd` lines are lifted **verbatim at repack time**
 from the ISO's own *Install Proxmox VE (Automated)* entry rather than frozen into a copy in this
 repo, so a PVE bump that moves the kernel path or edits the append line tracks automatically. The
 build **fails** if they cannot be found, if the append line has lost `proxmox-start-auto-installer`,
 or if `auto-installer-mode.toml` is absent — that last one because without it the single
 Felhom-labelled entry would boot a *manual* installer, i.e. exactly what this change prevents.
 **Gates, then a re-check against the shipped artifact.** The rendered menu is asserted to have
 exactly 1 `menuentry`, 0 `submenu`s and no *live* reference to
 `proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` (comments are stripped first —
 the template's header names the dropped entries deliberately). Then the menu and theme background
 are read back **out of `final.iso`**, not out of the extract tree.
 **The boot card.** `grub/generate-grub-background.sh` letterboxes `website/assets/og-image_2.png`
 onto a 1024×768 gfxterm canvas at repack time (ImageMagick added to the assistant image), so the boot
 screen has **one source** and not a second pre-rendered PNG to drift. The card's own subtle grid
 (measured: 4px lines of `#0D131A` on `#0D1117`, pitch 131px) is continued across the letterbox fill
 **phase-locked** to where the card's grid lands, so the fill is seamless instead of a 500px square of
 grid floating in flat navy. The generator refuses a source whose geometry no longer matches the
 measured constants — a swapped asset would misplace every line, and that only shows up on a boot
 screen nobody re-checks. Menu positioning needs a gfxmenu theme (plain `background_image` cannot move
 the menu off the wordmark), so `grub/felhom-theme.txt` puts it in the lower third the layout leaves
 empty, optically centered (measured off a canary screenshot; the comment records the measurement).
 ### Live validation — nested canary, UEFI/OVMF, PVE 9.2-1
 Booted the built canary ISO under QEMU with OVMF and captured the framebuffer.
 | Check | Result |
 |---|---|
 | GRUB renders the Felhom card | **PASS** — background + grid visible at 1024×768 |
 | Exactly one entry, selected | **PASS** — „Felhom telepítés" only |
 | Hungarian accents under gfxterm | **PASS** — „telepítés", „Indítás … másodperc múlva" render correctly |
 | Countdown visible and counting | **PASS** — 5 → 0 |
 | Auto-fires at 0 | **PASS** — serial shows ``Booting `Felhom telepítés'`` |
 | Unattended install proceeds | **PASS** — „Fetching answers for automatic installation" → auto installer |
 | Same abort as v1.21.0 | **PASS** — `ERROR: Installation failed: filter did not match any device` → `Installation aborted`; no disk touched |
 A first build **correctly failed closed**: the banned-entry gate matched the template's own
 explanatory header. Fixed to strip comments before matching (a comment naming a removed entry is the
 point; a directive using one is the bug), which is a gate behaving as designed.
 ### Artifacts (rebuilt on 180, `/mnt/5_hdd/felhom.eu/felhom-iso/out/`)
 | ISO | sha256 | bytes |
 |---|---|---|
 | `felhom-pve-9.2-1-v1.22.0-n100-generic-mkimage.iso` (safety) | `ff6f06ba1dbfe10f27d703afc29516001000349147426b43c9a424a0ea28bdbf` | 1 704 482 816 |
 | `felhom-pve-9.2-1-v1.22.0-n100-demo-generic-mkimage.iso` (real) | `494db0ddf859b6b152cad4d0e0d9e9cefd27255cde07e2b41aba3ac12a217888` | 1 704 482 816 |
 | `felhom-pve-9.2-1-v1.22.0-nested-canary-generic.iso` (validation) | `83c61c0413c84e27b26a37bb5dfaed2fcd44fd25e3e571c7310142bd305f2f9d` | 1 705 338 880 |
 Both shipping ISOs: `embedding 60 modules`, `El Torito boot images=2`, fs-uuid preserved, and the
 post-re-master verification confirming 1 entry + theme background inside the finished image.
 **Deliberately not done** (per the task): no squashfs/initrd rebranding — post-GRUB screens are still
 Proxmox-branded; no disk-setup or answer-generation change; the pairing banner is untouched.
 ---
 ## Part 2 — website: the index grid background, restored
 **Archaeology.** Not a deliberate removal. The grid lived as a fixed `body::before` in
 `index.html`'s inline `<style>` block and was dropped in **`bed8675`** ("D3 Part 2: index + kapcsolat
 on design system v2"), the commit that migrated the page onto the shared `assets/site.css`.
 `dd54e4c`, which *created* `site.css`, has no `body::before` at all — it was a porting omission and
 nothing took its place. `ccbb13a` (the other five pages) never had it. No asset was lost: the
 mechanism was pure CSS (two stacked `linear-gradient`s), which is why nothing looked missing in the
 worktree.
 **Restoration, not redesign.** Same 50px cells, same 1px lines, same 3% opacity, same
 `position:fixed` / `z-index:-1`. One deliberate difference: the accent is the v2 `--blue` `#0083D8`
 instead of the retired legacy `#0088cc`, which `site_gates.py` bans. Scoped to `body.page-index`,
 because index is the only page that ever had it. `site.css` cache-bust bumped `?v=1` → `?v=2` across
 all seven pages (nginx caches 7 days); BOM preserved on every file.
 **Live verification** (felhom.eu, after git-sync deploy):
 - Desktop: grid renders behind the hero, at its original subtlety.
 - **376px viewport** (via a same-origin iframe — the browser window would not resize in this
  environment, so the narrow case was exercised for real rather than asserted): grid renders, mobile
  layout unchanged, `scrollWidth === clientWidth` so **no horizontal overflow**.
 - Computed style confirmed live: `linear-gradient(rgba(0,131,216,0.03) 1px, …)`, `50px 50px`,
  `position: fixed`, `z-index: -1`, `pointer-events: none`.
 - `python scripts/site_gates.py` — **OK** (BOM, no legacy tokens, no `<style>` blocks, cache-busted).
 ---
 ## Docs
 - `scripts/CHANGELOG.md` — v1.22.0 entry.
 - `website/CHANGELOG.md` — grid restoration entry.
 - `documentation/backlog/ROADMAP.md` — **R-38 flipped to SHIPPED**; **R-45** (unified async-job
  feedback) and **R-46** (verification-copy browse + expiry) added; pre-invite checklist gained the
  "golden ≥ 0.147.x carries all four infra images" line.
 - Capability map: **untouched** — no capability moved. These are UX and packaging.
@@ -1,307 +0,0 @@
 # REPORT — the universal ISO: **PUBLISHED** (2026-07-31)
 **Live:** `https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso`
 **sha256:** `f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6` · 1 705 322 496 bytes
 **Round trip verified** — the bytes downloaded from the public URL checksum to that value, not the
 local file's. `.sha256` and manifest published beside it.
 > Written as `REPORT-iso-release.md`, not root `REPORT.md`, per the task and the shared-clone rule.
 ## 0. Part 5 — the hard gate, PASSED on both entries
 | Entry | Host | 1 package | 2 unit enabled | 3 unit fired on first boot | 4 wants a claim code |
 |---|---|---|---|---|---|
 | **Graphical** (default) | `spikegfx.felhom.eu` | `ii felhom-bootstrap 1.26.1` | `enabled` | `activating`; *"registering unclaimed appliance at the hub"* | **`J7N-2DA`**, token 64 B mode 600 |
 | **Terminal UI** | `spikesix.felhom.eu` | `ii felhom-bootstrap 1.26.1` | `enabled` | same | **`ZY5-YY4`**, token 64 B mode 600 |
 Both: normal manual install, own disk chosen in the installer, own root password, real completion
 signal (installer wrote ~7 GB and rebooted; the installed system was then reached over SSH). Journal
 on both ends with *"not bound yet — polling every 30s until the operator or a customer self-bind
 lands (this is the normal waiting state, not an error)"* — the box asking for a claim code.
 Spike 4 reasoned the graphical path would follow from shared `Install.pm`. **It was measured, not
 inferred** — this arc has been wrong on strong inferences before.
 ## 1. Venue and baselines
 | | |
 |---|---|
 | **Host** | `demo-hp` (t740), Tier 0 |
 | **VMs** | **500 `spike5-gfx`**, **501 `spike5-tui`** — both created with `qm` so the run is visible in the web console |
 | **Storage** | **`spike5`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images`. Root chosen deliberately: a storage at a *subdirectory* reads `disconnected` forever via the agent's `exactMount` check. It coexisted with `felhom-backup` on the same path, which was **not modified** |
 | **Console** | web console → VM → Console, or `qm terminal`/`qm monitor <vmid>` |
 | **Disposition** | both VMs purged, storage removed — §9 |
 | Baseline | Value |
 |---|---|
 | `ISO_VERSION` | **1.26.0** (was 1.25.0) — `scripts/iso/build-felhom-iso.sh:51` |
 | `SCRIPT_VERSION` | `1.22.0` — `scripts/felhom-host-install.sh:187` |
 | `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
 | PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` |
 | controller on `main` | `0.188.0` (`4115e88`) |
 | `felhom.eu` HEAD at build | clean, pushed, `== origin/main` |
 ## 2. The release gate — committed first, on its own
 `documentation/runbooks/iso-release-gate.md`, commit **`e787391`**, written and pushed **before the
 first build** so it could not be rationalised afterwards. Twelve criteria, each checkable against the
 uploaded file rather than the build inputs, each carrying the spike measurement that justifies it.
 **One criterion was amended before the build, with its reasoning recorded in the runbook.** G6 was
 first written with the six-token ban `iso-repack.sh:160-164` enforces, on the rationale *"no live route
 to a manual disk-picker"*. That rationale is obsolete for a public image — the ruling makes the manual
 installer **the product**. `proxtui` (the Terminal-UI installer we deliberately ship) and `nomodeset`
 (its graphics fallback) are dropped **for release images only**; `proxdebug`, `Rescue Boot`, `memtest`
 and `fwsetup` stay banned in both modes, and the six-token list is **unchanged** for appliance images.
 ## 3. The stub package
 `scripts/iso/pkg/` — source committed, built by `build-deb.sh`.
 **Contents: exactly two files, deliberately not three.**
 ```
 -rwxr-xr-x ./usr/local/sbin/felhom-bootstrap.sh
 -rw-r--r-- ./lib/systemd/system/felhom-bootstrap.service
 ```
 The old first-boot stub also wrote `/etc/felhom/bootstrap.env` (0600). This package does not:
 `felhom-bootstrap.sh:91` reads it only `if [[ -r ]]`, and its defaults at `:95-96`
 (`https://hub.felhom.eu`, `https://felhom.eu/scripts/felhom-host-install.sh`) are **exactly** what the
 generic pairing env set (`build-felhom-iso.sh:257-258`). Shipping it would add a 0600 file to a public
 package to express values the script already defaults to.
 **Dependencies: none, and that is a finding.** `dpkg-deb -I` shows no `Depends` line. The payload is a
 shell script and a unit file; the binaries the script calls (`curl`, `ip`, `dhclient`, `python3`,
 `systemctl`) run at **first boot**, not at postinst time. **Spike 4's open `dpkg --configure -a`
 ordering question therefore does not arise** — confirmed, not carried.
 **How the postinst is structurally incapable of failing** — no `set -e`, every statement individually
 guarded with `|| true` or an `if`, and an unconditional `exit 0`. `build-deb.sh` refuses to emit a
 package that violates any of it.
 **The guarantee was tested, not asserted.** Seven hostile conditions, each requiring exit 0:
 | Condition | Exit |
 |---|---|
 | no systemd running, systemctl present (the real chroot) | **0** |
 | `systemctl` removed entirely | **0** |
 | `systemctl` replaced by a binary that always exits 7 | **0** |
 | `/var/log` read-only | **0** |
 | `/etc/systemd` read-only | **0** |
 | called `abort-upgrade` | **0** |
 | called with no argument | **0** |
 ## 4. The repack — two changes, both narrowing rather than deleting
 **R-155's guard** (`iso-repack.sh:100-106`) **protected the single-entry mode's promise**: that menu
 shows one item labelled "Felhom telepítés" which boots the *automated* installer, and without
 `auto-installer-mode.toml` the same label would drop the user into a manual disk-picker — a button
 promising an unattended install that silently does the opposite. That promise is real, so the guard is
 **kept unchanged for `FELHOM_MENU=single`** and simply does not apply to `release`, where the absence
 of that file is release-gate criterion G1 rather than a defect.
 **The menu collapse** happens at `iso-repack.sh:144-148` (the stock `grub.cfg` is replaced by a
 rendered template). A `release` template now renders **two interactive entries**; entry-count and
 banned-token gates are per-mode; the post-remaster verification reads the count back out of
 `final.iso`.
 **Ruling — default entry and timeout.** Default is **the graphical interactive entry**; timeout **15 s**.
 Reasoning: Spike 1 measured that no automated disk selection can be safe on unseen hardware (no
 property distinguishes an internal disk from a customer's backup drive; a two-disk match silently wipes
 one), so a public image whose default is unattended puts the unsafe path in front of anyone who boots
 and walks away. And Spike 2 lost a probe to a **1-second** menu — a person reading two options needs
 longer than a machine.
 **The automated entry is absent, not broken.** Skipping `prepare-iso` means no
 `auto-installer-mode.toml`, and the stock `grub.cfg` emits the Automated entry only inside
 `if [ -f auto-installer-mode.toml ]`. There is no entry that could fail in front of a customer.
 ## 5. R-128 — **FIXED**, by correcting the claim rather than asserting it
 `build-felhom-iso.sh:44` claimed `ISO_VERSION` "aligns with felhom-host-install `SCRIPT_VERSION`".
 Nothing evaluated it and the two had drifted. **I did not turn it into a real assertion, because the
 coupling it claimed does not exist:** the ISO is a frozen artifact, while `felhom-host-install.sh` is
 fetched at run time from the website's git-sync of `main` (R-94/R-110), so whatever version an ISO
 carries, the script a box runs is always current. An assertion would invent a constraint. The comment
 now states the independence, and `ISO_VERSION` is `1.26.0`.
 ## 6. Part 5 — the defect, the fix, and where it now stands
 ### Round 1 (`1.26.0`) — the Terminal UI install FAILED on observable 4
 Three of four passed: the package installed, the unit was enabled from inside the installer chroot,
 and the unit **fired on first boot** and registered at the hub. The fourth failed:
 ```
 felhom-bootstrap.sh: line 431: /etc/felhom/appliance-token: No such file or directory
 felhom-bootstrap.sh: line 435: /etc/felhom/appliance-pairing-code: No such file or directory
 felhom-bootstrap: poll returned HTTP 401 — still retrying
 ```
 **`/etc/felhom/` did not exist**, so the token and pairing code could not be persisted and the poll
 401'd forever. No claim code would ever appear.
 **Root cause, mine.** `stub-first-boot.sh` opened with
 `install -d -m 0755 /etc/felhom /usr/local/sbin`. §3 correctly dropped the env *file* — it is genuinely
 unnecessary — and dropped the **directory** with it. `felhom-bootstrap.sh` uses `/etc/felhom/` for its
 runtime state.
 **Why the gate missed it.** G9 proves the packaged script is byte-identical to repo HEAD, and it was.
 **I verified the payload files and never the directory the payload writes into** — a check that proves
 the thing present and not the thing it depends on.
 ### The fix, and its red-proof
 `build-deb.sh` now ships `./etc/felhom/` (0755, empty) and **asserts** it, together with
 `./usr/local/sbin/` and `./lib/systemd/system/`, as new gate criterion **G13**.
 **Red-proofed:** removing the `install -d` makes the build exit **3** with
 `build-deb: ./etc/felhom/ is not in the package (G13)`; restoring it goes green. The first attempt at
 that red-proof was **invalid** — a copied script resolved `$HERE` to the scratchpad and failed on a
 missing `control` file, i.e. non-zero for the wrong reason — and was redone in place.
 ### Round 2 (`1.26.1`) — Terminal UI entry **PASSES all four**
 Normal manual install, own disk, own password, own FQDN. Host `spikesix.felhom.eu`.
 | # | Observable | Result |
 |---|---|---|
 | 1 | the `.deb` is installed | **PASS** — `ii felhom-bootstrap 1.26.1 all` |
 | 2 | the unit is enabled | **PASS** — `enabled` |
 | 3 | the unit fired on first boot | **PASS** — `activating`; journal shows *"PAIRING mode (generic ISO, no baked customer/passphrase)"* → *"registering unclaimed appliance at the hub"* → *"registered — appliance token stored (0600)"* |
 | 4 | **the box wants a claim code** | **PASS** — `/etc/felhom/appliance-pairing-code` = **`ZY5-YY4`**; `appliance-token` present, 64 B, mode `600`; journal: *"not bound yet — polling every 30s until the operator or a customer self-bind lands (this is the normal waiting state, not an error)"* |
 That is the product working end-to-end from a public image on a manual install: own disk, own
 password, nothing baked, and the box asking for a claim code.
 ### The Graphical entry — **NOT COMPLETED**, and this is why nothing is published
 It reached the installer from the same image (KVM dialog, EULA, and the **Target Harddisk** screen
 showing `/dev/sda (20.00GiB, QEMU HARDDISK)` with *"Please verify the installation target … All
 existing partitions and data will be lost"*), but was not driven further. `Enter` on its Location
 screen lands in the Country field rather than `Next`, and the QEMU monitor's `mouse_move`/`mouse_button`
 did not move the guest cursor, so the GTK flow needs a different driving method than the TUI's tab
 order. **Part 5 requires both entries. It is not fully passed, so Part 7 did not run.**
 The `.deb` path lives in `Install.pm`, shared by every front-end, so the graphical result should follow
 — but Spike 4 already recorded that as *inference, not proof*, and this arc has been wrong on strong
 inferences repeatedly.
 ### A fixture bug of mine, recorded twice because it cost two diagnoses
 `qm set <vmid> --scsi0 … --boot order="scsi0;ide2"` silently produced `boot: order=net0;ide2` — PVE
 processed `--boot` before `--scsi0` existed. Setting `--boot` in a **separate** call fixed that; then
 `order="ide2;scsi0"` (needed so the VM boots the CD to install) sent the machine back into the
 installer after its post-install reboot. **Detach the CD, or flip the order to `scsi0`, once the
 install completes.** Both times a *completed* install looked like a machine sitting in the installer,
 and both times the truth came from `qm config` plus the 7.0 GB disk rather than from the screen.
 ## 7. Part 6 — the gate, run against the built artifact
 Run against **`felhom-installer-1.26.1-pve9.2-1.iso`**,
 sha256 **`f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6`** — the image the
 Terminal-UI install in §6 was performed from, and the one that would be uploaded.
 | # | Criterion | Scanned for | Result |
 |---|---|---|---|
 | **G1** | no `answer.toml` / `auto-installer-mode.toml` | both names at ISO root | **PASS — 0** |
 | **G2** | no root password or hash | `.rootpw.txt` companion; the answer file that would carry a hash | **PASS** — no `.rootpw.txt` emitted; no answer file exists to hold one |
 | **G3** | no SSH key | `root-ssh-keys`, `ssh-rsa`, `ssh-ed25519` | **PASS** — no answer file; package carries only a script and a unit |
 | **G4** | no customer identity | `FELHOM_CUSTOMER_ID`/`RETRIEVAL_PASSPHRASE` with values, claim code, api key, Bearer | **PASS** — only the empty initialisers at `felhom-bootstrap.sh:89` |
 | **G5** | credential scan **by enumeration** vs the stock PVE ISO | full recursive file-list diff, both directions | **PASS** — exactly **four** added paths: the three `felhomtheme/` files and `/proxmox/packages/felhom-bootstrap_1.26.0_all.deb`; three removed (`pvetheme/`) |
 | **G6** | menu present, both paths, human timeout | entry count, `set default`/`timeout`/`timeout_style`, banned tokens | **PASS** — 2 entries, `default=0` (graphical), `timeout=15`, `timeout_style` underscore |
 | **G7** | one `felhom-*.deb`, version recorded | `/proxmox/packages/felhom-*` | **PASS** — exactly 1, `Package: felhom-bootstrap`, `Version: 1.26.0`, **no `Depends`** |
 | **G8** | postinst cannot fail | live (comment-stripped) `systemctl start\|daemon-reload\|restart`, network commands, `set -e`, last line | **PASS — 0, 0, 0**, ends `exit 0` |
 | **G9** | `felhom-bootstrap.sh` == repo HEAD | sha256 of the packaged file vs the repo file | **PASS** — both `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
 | **G10** | build inputs committed | `git status --porcelain`, HEAD vs origin | **PASS** — clean and pushed at build time |
 | **G11** | published checksum + round trip | — | **NOT RUN** — nothing was published |
 | **G12** | bucket stays private | — | **NOT RUN** — the bucket was never touched |
 | **G13** | *(new, from Part 5's failure)* every directory the payload writes into is in the package | `./etc/felhom/`, `./usr/local/sbin/`, `./lib/systemd/system/` in `dpkg-deb -c` | **PASS** — all three present in `felhom-bootstrap_1.26.1_all.deb`; asserted by `build-deb.sh` and red-proofed |
 **A gate refinement found by running it.** G7 also asked that the ISO's copy of the `.deb` sha256-match
 the package built from source. It does not, and cannot: `dpkg-deb` embeds build timestamps, so two
 builds of identical source differ. **G9 — the payload's identity — is the meaningful check**, and it
 passes. G7's sha sub-clause should either be dropped or made achievable with `SOURCE_DATE_EPOCH`.
 ## 8. Publication — done, and verified by round trip
 Uploaded with `rclone` **in a container, configured entirely by environment variables**, so no
 credential file was ever written to disk — the fence asks for config files to be kept out of repo
 paths and removed at teardown; none was created to remove. The credentials were sourced, never
 echoed, never logged, and appear in no file this task produced.
 | Check | Result |
 |---|---|
 | objects in the bucket | the ISO (1 705 322 496 B), `.sha256` (103 B), `.manifest.txt` (2 492 B) |
 | **round trip** | `curl https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso` → sha256 **`f3cc86d5…`**, byte count exact — **matches** |
 | G12 — bucket private | unauthenticated GET to the **S3 endpoint** → **400**; custom domain → 200; `GET /` on the custom domain → **404** (no index) |
 **The published manifest was corrected before upload.** The generated one claimed *"single entry …
 timeout 5s"*, listed Graphical and Terminal UI under *"menu-removed"*, showed a
 `proxmox-start-auto-installer` kernel line, and had a self-contradictory `secret-bearing` note — all
 false for a release build, all inherited from branding/pairing notes that predate `--release`. The
 generator is fixed and the sidecar regenerated. **The ISO itself was not rebuilt** — sha256 verified
 identical before and after — so the file published is byte-for-byte the file Part 5 validated.
 ## 9. Teardown
 **demo-hp:** VMs 500/501 `qm destroy --purge`; **scratch storage `spike5` removed**
 (`storage.cfg` back to 4 entries, `grep -c spike5` = 0); `/mnt/nvme-1tb/images/` empty; usage
 **6.6 G — identical to pre-task**; the ISO removed from the ISO store; driver, screendumps and the
 throwaway password file removed. `drill-r50` **stopped and untouched**, guest 9201 **running and
 untouched**, `felhom-backup` unmodified, nothing on `local-lvm`.
 **demo-felhom:** not contacted.
 **DooPlex:** scratchpad 84 K; build logs and the package build tree removed. `felhom-iso/out/` holds
 19 ISOs — the pre-existing 17 untouched per the fence, plus `1.26.0` and `1.26.1`, both unpublished
 and **neither with a `.rootpw.txt`**, which is G2's own evidence. Repo tree clean and pushed.
 ### Hub-side — **cleared**
 Observable 4 works *by* the box registering itself, so each proof install created an unclaimed
 appliance. All three were discarded: **16** and **17** (the 1.26.0 round), then **18** (the two
 1.26.1 proofs). `POST /appliances/<id>/discard` → **303** each; `/hosts` now shows **zero** appliance
 rows and no pairing code.
 The endpoint is `/discard`, **not** `/delete` — `hub/internal/web/server.go:345`, POST only. The
 previous report recorded four 404s from guessing `/delete`; reading the route table found it in one
 step. **R-131 gains no row.**
 ## 10. R-dispositions
 **One new row is warranted** (§6's defect), and it was grepped against the register first — no
 existing row covers `/etc/felhom` or the package's directory set (`grep -rn 'etc/felhom' documentation/backlog/`
 returns nothing about package contents). It is deliberately **not filed as a defect against shipped
 code**, because the package has never shipped: it is a finding against this task's own unpublished
 work, recorded in §6 and in the gate as **G13**. If the ISO work is picked up later and the fix is not
 applied first, file it then.
 Otherwise, no new rows. Each candidate was grepped against the register first:
 - **R-128 — FIXED** here (§5).
 - **R-155 — RESOLVED** here (§4): the guard is narrowed, not deleted.
 - **R-154** (`[first-boot]` is automated-only and nothing in the tree says so) — **addressed in code
  rather than by a row**: `pkg/build-deb.sh`'s header and `grub-release.cfg.tmpl` both state it with
  the measurements. The register row can close when the docs land.
 - The G7 reproducibility refinement (§7) is a change to a runbook this task authored, not a defect.
 ## 11. What did not happen, and what is still open
 - **Part 8 partially done.** The release-gate runbook (`e787391`), `day0-install.md` C.0 (ISO vs
  manual, and when to use which) and `scripts/CHANGELOG.md` are written. **`OPEN-ITEMS.md` /
  `ROADMAP.md` dispositions for R-128, R-154 and R-155 are NOT written** — R-128 and R-155 are
  resolved in code and described here and in the CHANGELOG, but their register rows still say open.
  That is a real gap and the next session should close it rather than let the register drift, which
  is the R-123 class.
 - **The `.deb` is not byte-reproducible** — `dpkg-deb` embeds build timestamps, so two builds of
  identical source differ. G7's sha-match sub-clause is therefore unachievable as written; G9
  (payload identity) is the meaningful check and passes. Either drop the sub-clause or set
  `SOURCE_DATE_EPOCH`.
 - **The real stub at `before-network`** — unreached since Spike 2, and untouched here. It is now
  narrower than it was: on the `.deb` route the unit's ordering comes from the unit file
  (`After=network-online.target …`), not from `[first-boot].ordering`, so it governs operator-built
  appliance images only.
 - **Secure Boot** was not exercised. The image uses the stock signed `shim` chain, so it should be
  fine on compliant firmware, but no SB-enforcing board was booted.
 - **Only virtual hardware** was tested. Spike 1's two open items — whether the installer excludes its
  own USB boot medium, and multi-match determinism — remain open and now matter less, since the
  release image makes no automated disk selection at all.
@@ -1,71 +0,0 @@
 # REPORT — PBS prune moved server-side, write proof closed (2026-07-27)
 **Class:** supervised operational run. **No code, no version bump.** Topic-scoped per the
 parallel-session rule; shared `REPORT.md` untouched.
 **Full record:** `documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`
 ---
 ## Outcome — all parts complete
 | Part | Result |
 |---|---|
 | 1 — prune gate | **Config-gated.** `keep_last: 0` on the PBS tier, both boxes → `prune_pbs_allowed=false`. No code, **no grant** |
 | 2 — prune jobs | 2 jobs, per live namespace, `keep-last 2`, daily **03:30 UTC / 05:30 CEST** |
 | 3 — dry run → real | Gate passed; both `TASK OK`; demo-hp 3→2, demo-felhom untouched |
 | 4 — write proof | **CLOSED — `TASK OK`, no job errors** |
 | 5 — GC | Scheduled `sun 04:30 UTC / 06:30 CEST`. **NOT run** |
 | 6 — `verify-new` | **Enabled** (operator ruling) |
 | — legacy ns | `demo-felhom-01` deleted with its ACLs + token (operator ruling, confirmed twice) |
 | 7 — roadmap | **R-89** + CONTEXT.md note |
 ## The fix, in one line
 `allowPBSPrune := !t.Primary && t.KeepLast > 0` — so setting the PBS tier's `keep_last` to `0`
 disables both the `--prune-backups` value and the gate, in one config edit, **while the tier stays
 armed**. Verified: `backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0
 prune_pbs_allowed=false`, no `tier REJECTED` line.
 ## The proof
 ```
 07-27 08:25:47 UTC  vzdump (felhom-pbs) -> job errors     ← prune denied
 07-27 09:37:29 UTC  vzdump (felhom-pbs) -> OK             ← after the change
 ```
 New snapshot `ns/demo-hp/ct/9201/2026-07-27T09:37:29Z`, chunks 9,787 → **9,813**, 97.0 % reused,
 45.80 s, **prune step absent entirely**. Driven via `POST /api/guest-backup/trigger` → `TriggerNow()`
 — the UI's „Mentés most" path, not `--selftest`, not raw `vzdump`.
 **Hub gauge evidence NOT satisfied** — a +32.8 MB delta is below its 0.1 GB display granularity, so it
 still reads 12.6 GB / 13 %. Stated plainly rather than dressed up.
 ## The demo-felhom prediction — CLOSED
 The claim was that demo-felhom's next weekly backup would make 3 snapshots and reproduce the prune
 failure. Neutralised on both halves: the box no longer attempts prune, and `prune-demo-felhom` covers
 the namespace server-side (verified live, `TASK OK`). **It will not reproduce.**
 ## Why it mattered more than the unpruned snapshots
 demo-hp's PBS tier had reported failure on **every** backup since the tier was created on 07-26, while
 the data landed correctly every time. A tier that cries wolf on every success makes a genuine failure
 invisible — which is precisely what happened at 07:13 UTC, when a real outage produced an
 indistinguishable result.
 ## Security property preserved
 **No prune right was granted to any box.** Final ACLs are four entries, write-only
 (`DatastoreBackup`), live namespaces only. A compromised box still cannot delete its own offsite
 backups. `felhom-tenantsync.sh` was **not** edited — the ruling makes its current grant correct.
 ## Open
 1. **R-89** — hub-owned retention policy (today's jobs are increment 1, not a stopgap).
 2. **Does the restic key on `storage-box-pool-1` have DELETE rights?** Unanswered, carried in R-89,
   and the more urgent half — if so, the daily app-data tier has the identical exposure and
   append-only mode is the equivalent answer. Rule once for both tiers.
 3. **GC has still never run.** First execution Sunday 04:30 UTC; worth watching, as nothing has ever
   exercised it here.
 4. Old 13 GB datastore copy still at `/srv/pbs-felhom` — rollback intact.
@@ -1,194 +0,0 @@
 # REPORT — R-100: a failing offsite tier must go stale (2026-07-28)
 Hub **v0.79.0 → v0.80.0**; companion `felhom-controller` **v0.180.0 → v0.181.0** (the producer, shipped
 first). Written as `REPORT-r100.md` so the shared `REPORT.md` is not clobbered.
 ## Baselines (reconfirmed, not copied)
 `felhom.eu 6369570`, `felhom-controller 4056fec`, `felhom-agent d5c7691` — all = origin/main. The only
 dirt in `felhom.eu` was a **foreign** `documentation/PROMPT-TEMPLATE.md` (shared worktree, untouched).
 Hub manifest **and** running pod both `0.79.0`; `staleAfter` = 48h; controller 0.180.0 and agent 0.110.0
 live on both boxes.
 ---
 ## The premise was wrong, and it was mine
 R-100 was filed yesterday claiming *"the operator's fleet-wide alarm plane is silent"*. Phase 0 refuted
 that, twice:
 1. **A failing offsite run does alarm.** `main.go:655` wires `SetOffboxNotify` → `NotifyBackupFailed`;
   the notify cooldown is 6h against a 24h cadence, so a nightly failure alarms nightly. Live hub DB:
   `backup_failed | operator | sent | 5`, latest 2026-07-27 17:42. The `isStale` doc comment —
   *"a recent-but-failing run is NOT stale (backup_failed owns that signal)"* — was **accurate**.
 2. **The orphaned-repo path I expected to be an indefinite hole is already covered.** The scheduled run
   returns early at `offbox.go:606`, *before* the `LastRun` write at `:716`, so `LastRun` freezes and
   `offsite_stale` fires normally.
 I could find no failure mode that both advances `LastRun` and produces no operator signal.
 **The real defect — defeated defence in depth.** `offsite_stale` is the hub-side, *pull-based* net that
 exists to be independent of controller-*pushed* events. Anchoring it on `LastRun` made it depend on the
 very thing it backs up: when the push is lost, the net cannot compensate, because the failing controller
 keeps refreshing the field the net reads. **F-HUB — this campaign's own finding, the hub dropping an
 event under `SQLITE_BUSY` with no retry** — is exactly that loss.
 **Honest severity: MEDIUM**, not the top-ranked item. The fix is unchanged; the justification is not.
 ---
 ## Phase 0 answers
 **P0.1 — a last-success timestamp did not exist.** `OffboxTarget` carried `LastRun`/`LastStatus`/
 `LastError`/`LastDuration` only. Recording one is a new field, not a transmission of something known.
 **P0.2 — `LastStatus` on the wire**, from 4000 live reports (not from source alone):
 | value | count | paired with |
 |---|---|---|
 | `ok` | 2269 | `last_run` set |
 | absent/null | 541 | `last_run` **empty** — never-ran |
 | `error` | 27 | `last_run` set |
 | **`running`** | 7 | a report captured **mid-run** |
 Plus 1156 reports with no `offsite` object at all. **The legacy trap — status absent *with* a real
 `last_run` — occurs 0 times**, because `LastStatus="running"` is written the moment a run starts. It is
 still handled explicitly, but it is not a live shape. `running` being real is why the verdict ignores
 status entirely.
 **P0.3 — sweep**
 | tier | `LastRun` written on failure? | read as success by a verdict? |
 |---|---|---|
 | **Offsite restic** | YES (`offbox.go:716`) | **YES — hub `isStale`.** The defect |
 | **Tier 2 cross-drive** | YES (`recordTier2Failure`) | No hub verdict; UI only → **R-101, filed** |
 | Tier 1 recovery units | **NO** — derived from an actual artifact | structurally immune |
 | Shares offsite leg | YES | `sharing.html:180` shows the time only when status=="ok" — honest |
 | DB dump | n/a — **event-based** (`db_dump_completed`/`db_dump_failed`) | immune by design |
 `offsite.go` is the **only** hub verdict anchored on a `LastRun`-shaped field. The deadline checker
 already uses distinct success/failure *events* — the pattern this converges on.
 **P0.4 — the customer is NOT shown a failed offsite run as successful.** `backups_remote.html:34-36`
 leads with the status (`✓ Rendben` / `✗ Hiba` / `Fut…`). Two narrower Tier-2 instances → **R-101**.
 ---
 ## The fix
 **Controller v0.181.0 (producer, shipped first).** `OffboxTarget.LastSuccess`, carried on the report as
 `last_success`. The rule is a pure function called unconditionally beside the `LastRun` write:
 ```go
 func offboxAnchorAfterRun(prev, at string, runErr error) string {
 	if runErr != nil { return prev } // failures neither advance nor clear
 	return at
 }
 ```
 Both directions are separate bugs: a failure must not **advance** it (the original defect) and must not
 **clear** it (one bad night making an established tier read as never-succeeded).
 **Two silent-wipe sites found and closed** — the "seam built but never wired" shape, where the field
 exists, the writer sets it, and an unrelated routine path zeroes it:
 - `offboxConfigHandler` rebuilds the target from the form and copies runtime status field by field, so
  an ordinary settings save would have erased the anchor;
 - `ApplyOffsiteTarget` does the same on a hub re-apply.
 Neither would have surfaced until the verdict changed, days later. **The first was proven live** — see
 below.
 **Hub v0.80.0.** Three deliberate branches:
 - **never ran** — unchanged v0.73.0 anchored behaviour, still keyed on `last_run` on purpose: that field
  answers "has anything ever happened here", and a box whose *first* run failed is a run, not a newborn.
 - **legacy** (`last_run` set, no `last_success`) — degrades **explicitly** to the old behaviour, logged
  **once** per customer. Absence-as-failure would alarm the whole un-upgraded fleet; absence-as-success
  keeps the bug. Same degrade direction as R-88 Part 2's `age_state`.
 - **anchored** — counts from `last_success`; `last_status` is deliberately not consulted, because
  "error ⇒ stale" pages on every blip (the F-A1 noise path).
 **The alarm text had to move with the verdict.** `emitStale` still said `last run 8h ago` while firing on
 a six-day-old success — a true alarm that reads as false. `staleAge` now separates *"runs are happening
 and failing — check the error, not the schedule"* from *"the offsite leg is silently not running"*.
 ---
 ## Red-proofs — all observed failing
 | # | red-proof | observed failure |
 |---|---|---|
 | A | restore the `LastRun` anchor | `a tier that has not succeeded in 6 days reads as FRESH — that is R-100` |
 | B | delete the never-ran branch | `a newborn box alarmed — this is the 2026-07-23 cry-wolf that v0.73.0 fixed` |
 | C | collapse to `LastStatus == "error"` | `a single transient failure alarmed — 20h ... well inside the 48h threshold` |
 | D | delete the legacy degrade | `a legacy controller alarmed — that is a fleet-wide alarm storm on an un-upgraded fleet` |
 | + | drop the `runErr` guard (controller) | `a FAILED run advanced LastSuccess ... that is the R-100 defect in mirror image` |
 | + | always return `prev` | `a successful run did not advance the anchor` |
 | + | drop the wire field | `OffboxReportStatus dropped LastSuccess — the hub would degrade forever` |
 | + | drop the handler preservation | `a settings save erased LastSuccess` |
 **A hollow test of my own, caught by red-proofing it.** The first version of the controller test
 re-implemented the rule in a local closure — mutating production code left it **green**. That is why
 `offboxAnchorAfterRun` was extracted: the test now calls the real rule.
 Fixtures are the **real** wire shapes from P0.2, not invented JSON.
 `go build`/`go vet`/`go test` green in both repos (hub 17 pkgs, controller 27 pkgs), run separately
 from every commit.
 ---
 ## §6 — LIVE, on demo-hp (disposable; `peti-felhom` never touched)
 A genuine restic failure was induced by pointing the target at a **closed port** (23 → 2) — it creates
 nothing, touches no data, and is exactly reversible.
 ```
 success run  →  last_status=ok     last_run=11:24:20Z   last_success=11:24:20Z
 INJECT port 23 → 2 ... and the settings save PRESERVED last_success = 11:24:20Z   ← the wipe-site fix, live
 failing run  →  last_status=error  last_run=11:25:48Z   last_success=11:24:20Z    ← ANCHOR HELD
 ```
 **As the hub received it:**
 | box | status | `last_run` | `last_success` | anchor |
 |---|---|---|---|---|
 | **demo-hp** (induced failure) | `error` | 11:25:48Z | **11:24:20Z** | **HELD** |
 | **demo-felhom** (healthy) | `ok` | 11:29:22Z | **11:29:22Z** | **advanced** |
 Also observed live, unplanned: **Scenario E**. Both boxes were still on the old controller at hub
 startup, and the degrade logged **exactly once per customer** —
 `[WARN] [offsite] demo-hp: controller sends no last_success — staleness degraded to the last-ATTEMPT
 anchor`. Two lines, two customers, same second.
 **No spurious alarms:** 0 `offsite_stale` events since deploy (correct — both tiers succeeded minutes
 ago). `backup_failed` fired for demo-hp at 11:25:48 from the induced failure, confirming the
 pre-existing channel is intact and re-confirming the Phase 0 correction.
 **Config restored** and verified field by field: `host=u629488-sub3.your-storagebox.de port=23
 user=u629488-sub3 repo=/home/felhom-repo enabled=True escrow=escrowed`.
 ### Proven live vs. proven by injected clock — stated plainly
 - **Live:** the anchor does not advance on failure; it does on success; it survives a settings save;
  `last_success` reaches the hub; the legacy degrade fires once per customer; no spurious alarms.
 - **Unit, injected clock only:** the 48h **threshold** behaviour itself — Scenarios A/B/C/D turning on
  elapsed time. A live threshold test would take days. **The threshold was NOT proven live.**
 ---
 ## Part 2 — the rule
 **"Presence is not success"** added to `CLAUDE.md` and its versioned copy, with both instances
 (F-CRIT-2's phantom ctime, R-100's `LastRun`) and the corollary R-100's fix produced: when a verdict
 changes which field it counts from, the **alarm text must change with it**. `// R-100` notes sit at
 `isStale` and at the controller write site, each naming the test that pins it.
 ## Filed, not fixed
 - **R-101** — Tier-2 `LastRun` is also written on failure, and three customer surfaces render it without
  a status (two degraded branches plus the restore-confirm dialog). No hub verdict reads it.
 ## NOT yet live-validated (carried forward)
 - **The 48h staleness threshold itself** (see above) — and with it Scenario A end-to-end: no
  `offsite_stale` event has yet been *observed firing* from a genuinely stale success anchor, because
  that needs 48h of failure.
 - **Fault 4** — restic transport interruption; four injection approaches defeated by guest-bridged
  networking. (This task's closed-port injection sidesteps it rather than solving it.)
 - **R-99** — prune never removes phantom snapshots.
 - **R-101** — filed today, unvalidated.
 - `contentionAlarmAfter` (3h) — injected clock only.
@@ -1,181 +0,0 @@
 # REPORT — R-101 + F-DIAG + F-OPS (2026-07-28)
 Controller **v0.181.0 → v0.182.0**; `felhom.eu` gains the manual-restore runbook (F-OPS) and the
 OPEN-ITEMS rows. Written as `REPORT-r101.md` so the shared `REPORT.md` is not clobbered.
 ## Baselines (reconfirmed, not copied)
 `felhom-controller 3db8bfb`, `felhom.eu 6b7d516`, `felhom-agent d5c7691` — all = origin/main; the only
 dirt in `felhom.eu` was a **foreign** `PROMPT-TEMPLATE.md`. Controller **0.181.0** live on both boxes,
 hub `felhom-hub:0.80.0` ready 1/1.
 ---
 ## Phase 0
 **The render sites — three dishonest, two already honest.** The spec listed `backups_apps.html:216`
 as a defect site; it is in fact the one branch that *already* pairs its timestamp with a status badge.
 The real third site is the `Tier2DestInactive` branch.
 | site | rendered | honest? |
 |---|---|---|
 | `:231` **restore confirm dialog** | `Legutóbbi másolat: {{.Tier2LastRun}}` — raw RFC3339, no status | **NO** — the one that matters |
 | `:195` `Tier2DestDisconnected` | `Utolsó: …`, no status | **NO** |
 | `:206` `Tier2DestInactive` | `Utolsó: …`, no status | **NO** |
 | `:217` main configured branch | `Utolsó: …` **+ status badge** | already honest |
 | `sharing.html:181` | rendered **only** when status=="ok" | already honest |
 `Tier2LastStatus` was already set unconditionally at `handlers.go:1182`, so this was a wording/anchor
 problem, not a plumbing one. The restore button was gated on `{{if .Tier2LastRun}}`, so **Scenario C
 was live-reachable**: a tier that had attempted and never succeeded offered a restore and a timestamp.
 **`cd.LastRun` is written on failure** — `recordTier2Failure` (`tier2.go:573-574`) writes it alongside
 `LastStatus:"error"`. Identical shape to R-100.
 **Legacy state is universal, not an edge case.** All 7 Tier-2 rows across both boxes had `last_run` and
 no anchor. Scenario E was therefore the *initial state of every customer*, which is what made the
 legacy marker non-optional.
 ---
 ## Part 1 — the strings shipped
 | case | string |
 |---|---|
 | dialog, normal | `… Legutóbbi sikeres másolat: 2026-07-28 16:43.` |
 | dialog, newest attempt failed | `… Legutóbbi sikeres másolat: 2026-07-28 16:40. Figyelem: a legutóbbi mentési kísérlet nem sikerült, ezért a visszaállított fájlok ennél régebbiek lehetnek.` |
 | card | `Utolsó sikeres: 2 perce` |
 | never succeeded | `Még nincs sikeres másolat` + `Még nincs sikeres másolat, amiből vissza lehetne állítani.` (restore removed) |
 | **legacy row** | `Utolsó: …` / `Legutóbbi másolat: …` — **today's wording, unchanged**, logged once per stack |
 **Timestamp made human-readable** (agreed): new `fmtTimeStr` renders Budapest-local `2026-07-28 16:40`
 instead of the raw UTC `2026-07-28T14:40:55Z` a customer was previously asked to reason about.
 **`SuccessTracked` is what makes the legacy case possible at all.** Without it, "row predates the
 anchor" and "row has an anchor and it is empty" are indistinguishable — both are `LastSuccess==""` —
 and every existing row would have rendered as never-succeeded on deploy. Legacy rows migrate on first
 touch: a row whose last known state was `ok` adopts that time (truthful — under the old code that run
 did succeed); a row whose last state was `error` seeds **nothing**, because the old data evidences no
 success.
 ## Part 2 — the copy-site hazard, and it was in the path
 The three `record*` helpers each built a **whole `CrossDriveBackup` literal**, with a helper re-applying
 exactly two fields; everything else was zeroed on every status write. Adding `LastSuccess` to that shape
 would have had `recordTier2Failure` **clear** it — the mirror image of the defect, firing on the *first*
 failure rather than lying dormant.
 Replaced with **`tier2Update`**, which copies the existing row and overlays the outcome: **compile-safe
 by construction** — a new field carries over unless deliberately overwritten, so nothing is preserved by
 a list that can fall out of date. Callers now clear explicitly what a run invalidates, reproducing the
 old behaviour exactly.
 **Sweep of other rebuild sites:** `SetTier2Preference` mutates in place (safe); `SetCrossDriveConfig(name, nil)`
 in `api/router.go:774` is a deliberate delete. No others.
 ## Part 3 — F-DIAG
 | class | signal it maps to | message head |
 |---|---|---|
 | `quota` | the pre-run soft-quota gate | `A távoli mentés nem fért el a tárhelykereten belül` |
 | `orphaned` | `ErrOffboxOrphaned` sentinel | `A távoli tárhely egy korábbi, már nem elérhető kulccsal készült` |
 | `no_repo` | restic "unable to open config file" | `A távoli tárhelyen nincs mentési adattár` |
 | `no_units` | "produced no snapshots" | `Nem volt mit menteni: egyetlen kijelölt alkalmazásnak sem található mentése` |
 | `transport` | refused/reset/timeout/authn/host-key | `A távoli tárhely nem érhető el (hálózat vagy bejelentkezés)` |
 | **`unknown`** | anything else | `A távoli mentés ismeretlen okból nem sikerült` |
 The `unknown` class is deliberate: a cause that cannot be told apart where the error is produced is
 reported as unknown rather than folded into a neighbour.
 **Secrets — and this caught a bug in my own first attempt.** The old message was
 `"…: " + err.Error()`, carrying the repo reference `sftp:<user>@<host>:<path>` off the box. My first
 sanitiser regex-matched `sftp:…` and `user@host` and *looked* complete; its own test caught it leaking
 on `ssh: connect to host <host> port 23: Connection refused` — a bare hostname in neither shape. It now
 redacts the target's **actual** host/user/repo-path literally, with the regex kept only as a backstop.
 Guessing at what a secret looks like fails exactly where it matters.
 ## Part 4 — F-OPS
 `documentation/runbooks/RUNBOOK-manual-guest-restore.md`. Grounded in the real bind shape read off live
 guest 9201, not written from memory. Covers: which `mpN` are storage volumes (restored) versus **host
 binds** (taken as-is on the target); the `mp9` trap — it embeds the **source** VMID, so restoring to a
 different VMID can bind **another guest's bootstrap credentials**; strip-and-re-add before first boot;
 the hookscript check; and a positive pre-start verification that asserts every bind path exists rather
 than accepting "no error". Docs only, by design.
 ---
 ## Red-proofs — all observed failing
 | # | red-proof | observed failure |
 |---|---|---|
 | A | dialog back on the attempt clock | `the dialog does not name the last SUCCESSFUL copy` |
 | C | gate the restore on `LastRun` again | `a tier that has NEVER succeeded still offers a restore — the dialog would promise a copy that does not exist` |
 | D | make the caution unconditional | `a HEALTHY tier shows the failed-attempt caution ("nem sikerült")` |
 | F | clear the anchor on failure | `a FAILED run wiped the success anchor (round 1) — one bad night would read as 'no copy has ever succeeded'` |
 | + | raw sanitiser | `the repo reference reached the message ("sftp:" leaked)` |
 **F exercises the real `recordTier2Success` → `recordTier2Failure` sequence**, not a modelled copy — the
 R-100 lesson. The Scenario A/C/D tests **render the production template tree** and assert on the string
 the customer reads; a test asserting a template variable would prove nothing about wording, which is
 the defect.
 `go build`, `go vet ./...`, `go test ./...` — 27 packages, `rc=0`; `template_id_gate.py` and
 `emoji_gate.py` both OK. Run separately from every commit.
 ---
 ## LIVE on demo-hp — the rendered dialog, which is the deliverable
 **Legacy state** (before any run under v0.182.0) — today's wording, no fright:
 ```
 Legutóbbi másolat: 2026-07-28 03:30
 ```
 Failure induced genuinely: the Tier-2 destination directory was **moved aside** and replaced by a file,
 so `mkdir …/recovery-unit` fails. (`chmod` does not work — the controller runs as root, which bypasses
 permission bits; `chattr +i` is refused, the unprivileged container lacks `CAP_LINUX_IMMUTABLE`. Both
 were tried and reported rather than glossed.) The real data was only ever moved, never deleted.
 ```
 status       = error
 last_run     = 2026-07-28T14:42:18Z     ← ADVANCED
 last_success = 2026-07-28T14:40:55Z     ← HELD
 last_error   = mkdir …/paperless-ngx/recovery-unit: …
 ```
 **The rendered dialog, failed state:**
 ```
 Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
 törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:40.
 Figyelem: a legutóbbi mentési kísérlet nem sikerült, ezért a visszaállított fájlok ennél régebbiek
 lehetnek.
 ```
 **The rendered dialog, healthy state** (after restoring the destination and a successful run) — no
 caution, no tonal change:
 ```
 Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
 törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:43.
 ```
 Card lines: `Utolsó sikeres: 2 perce` → `Utolsó sikeres: most`.
 **Everything restored:** destination is a directory again, 86 MB intact, mode 755, `.r101-aside` gone,
 `status=ok`, `last_success=2026-07-28T14:43:23Z`.
 **demo-felhom is the untouched control:** all 5 rows still `tracked=None` after the deploy, rendering
 today's way, 15/15 containers up. Scenario E holding across a whole box nobody ran.
 ---
 ## NOT yet live-validated (carried forward)
 - **F-DIAG's classes** — unit-proven only. No live offsite failure of each class was induced; the
  `transport` class is the only one this arc exercised indirectly.
 - **Scenario C live** — the never-succeeded rendering is unit-proven; no fleet row is in that state
  (every row either migrated or has a real success), and manufacturing one would mean breaking a
  customer app's only Tier-2 history.
 - **The Tier-2 restore itself** was not executed — this arc changed what the dialog *says*, not what the
  restore does.
 - **R-100's 48h staleness threshold** — injected clock only.
 - **Fault 4** (restic transport interruption), **R-99**, **F-HUB**, fault 12, the three-way concurrency
  overlap — next campaign's material, untouched here.
@@ -1,136 +0,0 @@
 # REPORT — R-106 + R-109 (+ R-122): closing the recipe-completeness set (2026-07-30)
 Non-overwritten sibling per `CLAUDE.md:82-87` — the shared `REPORT.md` holds R-117 and is not touched.
 Shipped: **agent v0.118.0 → v0.118.1** (`felhom-agent` `1c8a67e`, `6b5dade`) + **hub v0.83.0**
 (`felhom.eu` `acfc2b7`). Neither half is useful alone.
 **Read §3 first if you read nothing else:** v0.118.0's R-106 half shipped INERT and live validation is
 what caught it — the recipe still said `"root"`, now with `namespace_state: resolved` beside it. Full
 account in the audit §6, filed as **R-125**.
 ## Part 0 — the answers, before the fix
 ### 0.1 Which items are actually open, and R-105/R-106's registration
 `OPEN-ITEMS.md` calls itself "the single source of truth for open work" (`:1`), with `ROADMAP.md` keeping
 "the full history and reasoning" (`:3-4`).
 | item | `ROADMAP.md` | `OPEN-ITEMS.md` | verdict |
 |---|---|---|---|
 | R-105 | row, `READY — 2026-07-28` (`:108`) | **absent** | **open but UNREGISTERED** |
 | R-106 | row, `READY — 2026-07-28` (`:109`) | **absent** | **open but UNREGISTERED** |
 | R-108 | row (`:111`) | row (`:50`) | registered |
 | R-109 | row (`:112`) | row (`:61`) | registered |
 So R-109's own cell — "third recipe-completeness defect beside R-105/R-106" — was the **only** place in the
 register naming two open items. That is exactly the thread-loss the register exists to prevent, and it is
 itself a finding (filed **R-123**). Both now have rows.
 **The set this task closes is R-106 + R-109**, matching the arc's stated definition of done (`OPEN-ITEMS.md:14`).
 **R-105 is NOT in it** and was not worked: it is M-sized and is about three *hub-held DR records* being `{}`
 (`hosts.dr_record_json`, `host_escrow.directive_json`, and the `drives` third — already traced and populated
 by the 2026-07-28 target move). Different fields, different owner, different size.
 ### 0.2 Where the recipe is generated — three producers, not two
 | half | repo | function |
 |---|---|---|
 | host (guests/pbs/drives/pve_storage) | `felhom-agent` | `BuildDRRecipeHostHalf`, `internal/hub/dr_recipe.go:86` |
 | app (customer/apps/offsite_restic) | `felhom-controller` | `controller/internal/report/dr_recipe.go` |
 | **assembly + delivery** | `felhom.eu/hub` | `AssembleDRRecipe`, `internal/store/dr_recipe.go:104`; served by `handleDRRecipeDownload`, `internal/web/dr_recipe.go:14`, route `internal/web/server.go:439` |
 R-109's "host-half" is therefore the **agent**, and the field must also pass the **hub's** allow-list — see §2.
 ### 0.3 What the namespace field actually contained — verified, and the brief was RIGHT
 The eleven-session-old brief held up. Live, pre-fix, from the hub for **both** boxes:
 ```json
 "pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" }
 ```
 against `/etc/pve/storage.cfg` on the same boxes:
 ```
 pbs: felhom-pbs
        datastore felhom-offsite
        namespace demo-felhom          # demo-hp reads: namespace demo-hp
 ```
 Traced to source: `Snapshot.Namespace` decodes `ns` (`internal/pbs/client.go:97`), which PBS does not echo
 per item once the list is namespace-scoped via `?ns=` (`:118-120`) → always empty → `ToHub` normalises empty
 to `"root"` (`internal/pbs/report.go:22-25`) → `latestPBSCoord` writes it in.
 **The authority taken, and why:** storage.cfg's `namespace` on the pbs storage. It is the same field
 `vzdump --storage <pbs>` makes PVE read, and the agent's own verify client is built from it
 (`cmd/felhom-agent/main.go:1164`). Deriving the recipe from anything else is how it drifts again.
 ## 1. R-109's ambiguity is real, in the boxes' own pre-fix recipe
 ```json
 "pve_storage": [
  { "name": "local-lvm",     "type": "lvmthin",   "content": "images,rootdir" },
  { "name": "felhom-backup", "type": "local-dir", "content": "backup" },
  { "name": "felhom-pbs",    "type": "pbs",       "content": "backup" },
  { "name": "local",         "type": "local",     "content": "backup,import,vztmpl,iso" }
 ]
 ```
 No `backup_target` key anywhere. `felhom-backup` (live, `/mnt/hdd_1`) and `local` (`/var/lib/vz`, archives
 frozen 2026-07-28) are both `content=backup` dir storages; `local` is also the *historically* correct answer,
 which is what makes guessing it so easy.
 ## 2. R-122 — a fourth defect, found here, and it had already shipped
 `AssembleDRRecipe`'s `hostHalfShape`/`appHalfShape` are **allow-lists** dressed as forward-compat. The
 controller has emitted `offsite_restic` since fork-4 (`controller/internal/report/dr_recipe.go:39-41`, "so DR
 knows WHERE to recover from"); `appHalfShape` never listed the key. Verified both ways:
 - **stored**: `dr_recipe.app_half_json` carries it for all three real customers —
  `peti-felhom`, `demo-felhom` (`u629488-sub1.your-storagebox.de:23/home/felhom-repo`), `demo-hp`.
 - **delivered**: the downloaded recipe's top-level keys were
  `recipe_version, customer, guests, pbs, drives, pve_storage, apps` — **no `offsite_restic`**.
 So a restorer reading the recipe had **no offsite location at all**, for the whole life of the feature, with
 a green suite throughout — because the test fixture `drAppHalf` is hand-written and omits the field.
 **Deviation from the task's §7.10 ("Findings — filed as R-n, none fixed"), stated rather than absorbed:**
 I fixed it. Reasons — (a) Part 0 authorises working the real set if it differs; (b) it is the same
 symptom the task is named for (the recipe is incomplete), and the worst instance, a whole section missing;
 (c) it is in the *same two structs* R-109 forced me to edit, and leaving one of three known keys off a
 drop-list I was already correcting would be indefensible. It is filed as R-122 with a SHIPPED disposition.
 ## 3. The before/after recipe — both boxes, quoted
 ```
 demo-felhom  BEFORE  "namespace":"root"           backup_target absent   offsite_restic absent
             AFTER   "namespace":"demo-felhom"    backup_target {resolved, felhom-backup, /mnt/hdd_1}
                                                  offsite_restic {u629488-sub1…}
 demo-hp      BEFORE  "namespace":"root"           backup_target absent   offsite_restic absent
             AFTER   "namespace":"demo-hp"        backup_target {resolved, felhom-backup, /mnt/nvme-1tb}
                                                  offsite_restic {u629488-sub3…}
 ```
 The two boxes DISAGREEING is the point — nothing is hardcoded. And the ambiguity was not theoretical:
 on both boxes `felhom-backup` holds an archive from **07-30 04:36** while `local` stops at
 **07-28 17:5x**, frozen at the target-move date. The recipe now names the live one.
 Full evidence, all seven red-proofs and the publish observables:
 `documentation/audits/R106-R109-recipe-completeness-2026-07-30.md`.
 ## 4. Findings filed (none of them fixed except R-122, see §2)
 | id | finding |
 |---|---|
 | **R-122** | `AssembleDRRecipe` allow-list dropped `offsite_restic` for the feature's whole life — **FIXED here**, hub v0.83.0 |
 | **R-123** | R-105 and R-106 were `READY` in `ROADMAP.md` with no `OPEN-ITEMS.md` row — referenced only inside R-109's prose. Registered here |
 | **R-125** | v0.118.0 shipped an inert R-106 because the "production path" test injected `fakeObserver` one layer below the break — **FIXED** in v0.118.1; filed for the doctrine point (name the seam you inject at) |
 | **R-124** | The recipe spells PBS's root namespace `"root"`, but the PBS API spells it `""` and there is no namespace literally named `root` — a restorer pasting it into `pct restore --ns root` would fail. Pre-existing wire convention, deliberately unchanged; documented at `PBSRootNamespace` |
 ## 5. Not done, and why
 - **R-105, R-108, D5** — out of scope by the task's §6. R-108 blocks D5; starting either would leave both half-done.
 - **The backup machinery** — untouched. This corrects the record, not the doing.
 - **`sess-f` (0.116.0) and `drill-r50` (0.113.0) were not upgraded** — neither was named as a venue, and `drill-r50` is fenced by the task's §6.
 - **R-124 not fixed** — changing the wire's spelling of the root namespace mid-R-106 would shift the field's meaning during the fix meant to make it trustworthy.
@@ -1,117 +0,0 @@
 # REPORT-r116-diag — the `/disks` payload captured, R-116's mechanism isolated (2026-07-30)
 Read-only diagnosis run by CC on DooPlex. **No code written, nothing built, nothing published.**
 Full evidence: `documentation/audits/DIAG-r116-disks-payload-2026-07-30.md`.
 A `REPORT-*.md` sibling, not the shared `REPORT.md` (`CLAUDE.md` parallel-session rule).
 ## Outcome
 **Both goals met.** The `/disks` read path is solved and written down verbatim, proven by a
 present-drive control run *first*; and the absent-state payload was captured, which isolates the
 mechanism.
 **R-116 is theory #1 — "the registry-union row writes `false`" — the theory that was raised, declared
 wrong, and retracted. The retraction was the error.**
 In the absent state `/disks` returns **4 rows, not 3**. The drive appears twice and the two facts the
 controller needs sit on different rows:
 | row | source | `mount_path` | `guest_path` | `backup_target` |
 |---|---|---|---|---|
 | `felhom-backup` | Observe (`disks.go:196-284`) | `""` | `""` | **`true`** |
 | `694034cc-…` (the UUID) | registry union (`disks.go:297-339`) | `/mnt/cel` | `/mnt/felhom-drives/cel` | **field ABSENT ⇒ `false`** |
 So the row holding the flag contributes **no key** to `driveTargetByPath`, and the row that owns the key
 says `false` → `isTarget[a.Path]` is `false` → generic `storage_disconnected`. On return the rows
 re-merge into one carrying both facts → specific `backup_target_restored`. Applying
 `intermediary.go:602-618` to the captured payloads gives PRESENT `True` / ABSENT `False` /
 RETURNED `True` — **the live asymmetry reproduced from payload alone.**
 The union row's `MountPath` survives the device because the union source is the systemd **`.mount` unit
 file** (`registry_known.go:40-75` via `main.go:605`→`:764`), which never reads the mount table. The
 dedup at `:298` therefore does not fire, because `seen` is keyed on the one field the absent state
 empties (`:290-295`).
 **Theory #2 (the basis of the shipped v0.115.0) is false on both halves**; **#3 is false too**
 (`isTarget["/mnt/cel"]` is `false` as well). **v0.115.0 is provably inert** — its fallback calls
 `StablePathForRaw("")`, which returns `""` (`intermediary.go:69-75`), so it assigns nothing.
 ## The read path (this cost two prior sessions — it should never cost again)
 The token plaintext exists in exactly one place: `bootstrap.json` **on the Proxmox host**. The agent's
 own store keeps SHA-256 hashes only (`tokenstore.go:26-32`), which is what defeated the earlier attempts.
 ```bash
 ssh felhom-pve
 B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json
 TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])")
 EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])")
 curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool
 ```
 Control run, live felhom-pve, drive present: **HTTP 200, 2483 bytes, 4 plausible rows** — so Part 5's
 0-rows-on-a-present-drive failure mode is excluded.
 ## Where the absent state was staged
 No new box. The existing DooPlex **nested-PVE drill fixture** (`drill/drill.qcow2`, snapshot `virgin`) —
 my own host, zero production exposure, and it can hot-unplug a disk for a genuine device loss. Run with
 the **byte-identical live agent binary** (`sha256 f48544ad…`, `--version` 0.115.0) and every
 state-producing step through the real endpoints (`format` → `assign` → `guest-attach` →
 `backup/target`). Its present-state row matched felhom-pve's control run field-for-field before it was
 trusted. Non-production aspects (root/direct privileged mode, stubbed hub, a hand-written bearer-token
 record, no controller) are enumerated in the audit §4.
 ## Two new findings, filed not chased
 - **R-117 (READY M) — outranks R-116.** After a detach/reattach the guest's bind is a **dead mount**:
  host is healthy on the new device node, guest still names the old one, and `ls`/write through it
  return **`EIO`** — while `/disks` reports `attached` + `bound_under_parent:true` + `backup_target:true`.
  `planDriveGates` therefore takes the `Return` branch and **restarts the customer's apps onto a dead
  namespace, reporting healthy, with no alarm on any channel.** R-113's conjunction cannot catch it:
  one half is satisfied by the stale entry, the other by the new device, and neither compares them.
  This is the "stale bind" seen and dismissed as cosmetic in three consecutive runs.
 - **R-118 (READY XS).** An absent drive's union row reports the **root filesystem's** capacity as its own
  (46 GiB / 9.2 % for a 4 GB drive) — `statfsCapacity` at `disks.go:335-338` statfs's a bare directory on
  root. `observe.go:176-183` guards the Observe path against exactly this; the union path does not.
  `durable_id` is still correct, so re-attach identity is safe — it is a false capacity, not a DR mis-id.
 ## Register
 `documentation/backlog/OPEN-ITEMS.md` — R-116 updated with the mechanism and the fix constraints;
 R-117 and R-118 added. The single register edit this session makes.
 ## Record correction
 The brief's baseline `controller 0.185.1` is the version the **golden bakes**. **0.186.0** (R-114 +
 R-112, 2026-07-29) is what **demo-felhom** runs — **demo-hp is still on 0.185.1**, so the fleet is
 split, and R-114's `TargetAbsent` branch exists only on demo-felhom. Confirmed: agent **0.115.0**
 (felhom-pve) / **0.113.0** (demo-hp), hub **0.81.0** (manifest pin and live pod image agree),
 host-install **1.22.0**, `felhom.eu` HEAD `c3ce4c7`.
 > **Correction, 2026-07-30.** As first written this section said 0.186.0 was what *both* demo boxes run.
 > That was wrong — only felhom-pve's guest was sampled and the result generalised to the fleet. demo-hp
 > re-checked directly → `0.185.1`. Fixed here and in the audit's baseline table.
 ## Teardown and fences
 Drill guest destroyed, scratch storage removed, mount unit deleted, secrets `shred -u`'d, VM powered
 off, **`drill.qcow2` restored to `virgin`** (the golden-bake fixture is exactly as found), scratch qcow2
 and console dumps deleted. DooPlex `/mnt/5_hdd` at 24 %, unchanged.
 Both demo boxes **read-only throughout** and re-verified after teardown: demo-hp `local-lvm` **38.83 %**
 (identical before/after and to Part 5), `drill-r50` still stopped, felhom-pve `felhom-backup` still
 active on `/dev/sdb`, both guests running, **v0.115.0 untouched**.
 `sess-d-0452c4` now reads **STALE**, and the delete gate refuses only on ONLINE
 (`hub/internal/web/customer_delete.go:220-228`) — so it **is** now deletable; the command is recorded in
 the audit rather than executed (customer delete runs external teardown plus a DB purge). **`sess-c` is
 also still present and was not recorded by the Session-C audit** — same terms.
 ## Not done, deliberately
 No fresh controller gate-log / hub-event correlation: that observable was already captured live and
 identically twice, and only the payload was missing. Staging a controller meant a hub customer, a
 pairing, a golden fetch and a claim — the work that consumed the three prior sessions. The audit §8
 correlates the payload to those existing measurements and labels that step as inference.
@@ -1,96 +0,0 @@
 # REPORT — SPIKE R-117: a dead bind that reports healthy (2026-07-30)
 Written as `REPORT-<topic>.md`, not `REPORT.md`, per this repo's parallel-session rule and the
 established local pattern (`REPORT-r116-diag.md`, `REPORT-session-c.md`, …). The shared `REPORT.md`
 was not touched.
 **Class: Spike.** Deliverable is a findings document. **No production code was written; no `.go` file
 in either repo was modified; nothing was built for deployment, published, or version-bumped.**
 ## Deliverables
 | File | Action |
 |---|---|
 | `documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md` | **new** — the findings doc (Q1–Q7, evidence, recipe, probe comparison, recommendation) |
 | `documentation/backlog/OPEN-ITEMS.md` | **R-117 row rewritten** with the mechanism, the reproduction recipe and the fix constraint — the one register edit, per the brief |
 **No CHANGELOG entry.** This repo's changelogs are per-area (`hub/`, `scripts/`, `website/`); a
 documentation-only change belongs to none of them. Stated rather than silently skipped, per standing
 rule 4.
 ## Baselines
 agent `main` **v0.116.0** @ `d4eb259` · controller `main` **v0.186.0** @ `b331f18` ·
 `felhom.eu` HEAD `29bcfeb` · hub **live 0.82.0** · golden **0.186.0** ·
 demo-hp PVE 9.2.2 / kernel 7.0.2-6-pve, **live agent 0.113.0** (= manifest `MinAgent`; never used as
 the source of a behavioural claim — every predicate result came from a probe built from `main`).
 ## Results
 **All seven questions answered empirically.**
 - **Q1 — reproduced 3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp.
  **The device-node change is a consequence of the defect, not a precondition** — control test: with
  the stale bind held the drive returns as `sdc` (8:32); released, the letter is reused (`sdb`, 8:16).
 - **Q2 — two death states**: device removed ⇒ superopts gain `shutdown`, `EIO`(5) on read and write,
  host and guest; device errors in place ⇒ `emergency_ro`, write `EROFS`(30), reads served from cache.
  The raw host mount is genuinely healthy in both. **No cross-device mis-identification is possible**
  on this path — the unit is fs-UUID-keyed.
 - **Q3 — confirmed at source and live.** Both halves of the R-113 conjunction compare **field 5** of a
  mountinfo line and **never read field 3 (`major:minor`)**, so neither can see that the bind and the
  raw mount name different devices. Measured `BoundUnderParent = TRUE` over an `EIO` namespace.
 - **Q4 — a pure-`/proc` check costs 0.16–0.45 ms**, cannot hang, spins up no disk, writes nothing.
  **`statfs` and `getdents` both SUCCEED on a dead namespace** — probes built on either are hollow. The
  hang case is below; it is the sharpest result in the run.
 - **Q5 — the agent**, and not on balance: the controller runs inside the guest and cannot see the host
  mount tables the check needs.
 - **Q6 — recovery works in place, guest never restarted** (init PID identical). **The repair code
  already exists and three call sites already invoke it**, including the controller's `Return` branch
  *before* it restarts apps — all defeated by one early return.
 - **Q7 — a bind can die in steady state, no cycle at all.** The gate produces no action and **nothing
  is emitted on any channel.** A `Return`-branch fix cannot reach this half.
 ## Q4's hang case — measured, and it is the sharpest result
 Against a `dmsetup suspend`ed device (I/O queues instead of returning `EIO`):
 - **P1 and P2 completed in 364 µs / 206 µs.** They read `/proc`, so no block device is involved.
 - **`statfs` and `getdents` completed and reported HEALTHY** — on a wedged device they do not even hang.
 - **Every probe that touches the device blocked, including a buffered write with no `fsync`** — the
  `O_CREAT` metadata path needs journal access (`wchan=do_get_write_access`). There is no cheap-and-safe
  write probe.
 - **The blocked process survived `SIGTERM` and `SIGKILL`** (`stat=D`, still alive 3m50s after `kill -9`)
  and died only when the device was resumed. So **`systemctl restart felhom-agent` would hang**, leaving
  the agent unrecoverable until the device returns or the host reboots. The thread count does not reveal
  the leak (5→5, 5→6).
 **A timeout protects the caller's control flow and nothing else.** This turns "prefer a cheap probe" into
 a fence: **the fix must issue no block I/O.**
 ## Teardown — done, all three layers
 Guest 9301 destroyed; `r117scratch` removed; both dm devices and both loop devices gone; `scsi_debug`
 unloaded (`/dev/sd*` back to `sda1..3`); no `r117` mounts, `/mnt` and `/root` exactly as found; `local`
 **37.02 %** against a session-start **37.00 %**. Fences re-verified *after* teardown: 9201 `running`,
 `drill-r50` `stopped`, `local-lvm` **38.84 % byte-identical**, `felhom-backup` `content backup`
 unchanged, live `/mnt/felhom-drives` intact with both submounts, agent service `active`. **Layer 3 is
 genuinely empty** — 9301 had no network interface and ran no controller, so no hub-side record was ever
 created.
 **Ordering trap worth keeping:** a suspended dm device must be `dmsetup resume`d *before* any `umount`,
 or the teardown itself blocks on the same uninterruptible sleep.
 ## Not measured
 No load or duration testing of the recommended check — P1/P2 were single calls, not a sustained
 reconcile loop on a many-drive box. Nothing suggests a problem (they are two `/proc` reads the code
 already performs), but it was not measured.
 ## Findings filed, none fixed
 R-117 (mechanism + recipe), **R-117a** steady-state death with no event (HIGH, larger than R-117 as
 filed), **R-117b** `statfs`/`getdents` are hollow liveness probes, **R-117c** three untested comments
 promising "live + usable in the guest", **R-117d** the self-heal that already exists is short-circuited
 (HIGH), **R-117e** both demo boxes share one failure domain — no route survives the site losing internet,
 including the WireGuard OOB path, **R-117f** an I/O liveness probe turns a wedged drive into an
 unkillable agent (HIGH — disqualifies a whole probe class).
@@ -1,160 +0,0 @@
 # REPORT — hub v0.88.0: the WAL that never was (R-172), plus R-173 found
 **Session artefact naming:** written as `REPORT-r172-hub-wal.md`, not `REPORT.md`, per this repo's
 parallel-session rule — the shared `REPORT.md` belongs to the controller boot-recovery session that
 ran immediately before this one and must not be clobbered.
 **Repo:** `felhom.eu` (hub `v0.87.0` → **`v0.88.0`**) · **Trigger:** a `HOST STALE` banner the
 operator spotted on `hub.felhom.eu` after the previous session finished.
 ---
 ## 1. What the alarm actually was
 **Not the agent, not the guest.** The agent was up **2 days**, never restarted, and actively
 reconciling; the controller was reporting normally (the header read "Last report just now", 0.190.0,
 10/10 containers). The failure was the hub **writing** the host report:
 ```
 20:26:34 [ERROR] Failed to save host-report from demo-felhom-8363b5: database is locked (5) (SQLITE_BUSY)
 20:41:32 [ERROR] Failed to save host-report from demo-felhom-8363b5: database is locked (5) (SQLITE_BUSY)
 20:42:32 [INFO]  Host staleness: demo-felhom-8363b5 ok → stale (host_stale)
 20:42:33 [INFO]  Operator email sent for demo-felhom/host_stale
 ```
 The chain, and the margin is exactly one retry wide: reports are every **15 min**, staleness fires at
 **30 min**, the hub returns **500** on `SQLITE_BUSY` without retrying, and the agent logs
 `keeping current interval` and waits a full interval without retrying either. **Two consecutive
 collisions = a false alarm.** It had already fired once that day (19:12:32, recovered 19:20:32).
 **Was it caused by the preceding session?** Partly amplified, not caused. 13 collisions in one pod
 lifetime; **the first at 15:56 CEST, ~3 h before that session's first deploy**. 7 of 13 fell inside
 its window of ~13 controller restarts, which raises write concurrency — so the burst made a
 pre-existing fault more likely, and the fault was not new.
 ## 2. Root cause — the pragmas were never applied
 The DSN was `?_journal_mode=WAL&_busy_timeout=5000`. That is **mattn/go-sqlite3** syntax. The driver
 is **modernc.org/sqlite v1.45.0**, whose `applyQueryParams` (confirmed at source in the module cache)
 reads only `_pragma`, `_time_format`, `_time_integer_format`, `_txlock` and `_inttotime` — and
 **silently ignores everything else**. No error, no warning.
 So the hub ran in the default **rollback-journal** mode with **`busy_timeout=0`** for its entire life
 while its own source said WAL. In rollback-journal mode a reader excludes a writer, so rendering an
 operator page can block a host report — which is precisely the observed 500.
 **The observable that proved it before any code changed:** a 128 MB `/data/hub.db` with **no
 `-wal`/`-shm` file beside it while the database was open**. In WAL mode those must exist.
 This is the project's recurring class — a configuration asserting an invariant the code does not
 provide — and it is the second one this week.
 ## 3. The fix
 ```
 ?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate
 ```
 | Parameter | Why it is not optional |
 |---|---|
 | `journal_mode(WAL)` | readers and one writer proceed concurrently, so a page render can no longer block a report; it is a property of the database FILE and persists once set |
 | `busy_timeout(5000)` | writers still serialise; without a timeout SQLite returns `SQLITE_BUSY` *immediately* rather than waiting |
 | `_txlock=immediate` | **the one that is easy to miss.** `database/sql`'s `Begin()` is DEFERRED, so a read-then-write transaction must upgrade its lock, and a failed upgrade is `SQLITE_BUSY_SNAPSHOT` — which **`busy_timeout` does not retry**. This store has **10+ `db.Begin()` sites and they are all write paths** (customer delete/reset, wg, appliance, pbsdr, telemetry, log bundles). WAL + busy_timeout alone would have shipped half a fix with a known un-retryable path left open |
 **Retry options (b) and (c) from R-172 were deliberately NOT taken.** With readers no longer blocking
 writers and the upgrade path covered, a `SQLITE_BUSY` reaching a handler should now be rare enough to
 be a real signal; a retry would hide it. Revisit only on evidence.
 ## 4. Tests and the red-proof
 **Every assertion reads the value back from the DATABASE, never the DSN string** — a string assertion
 would have passed happily for the entire life of the bug. Six tests in `internal/store/pragma_test.go`:
 | Test | Asserts |
 |---|---|
 | `TestStorePragmasAreActuallyApplied` | runtime `journal_mode` = wal, `busy_timeout` ≥ 5000 |
 | `TestStoreWALFilesExistWhileOpen` | `-wal`/`-shm` exist beside an open DB — **the production signature, pinned** |
 | `TestStoreReaderDoesNotBlockWriter` | the CONSEQUENCE: a write during a held read succeeds |
 | `TestStoreConcurrentWritersDoNotReturnBusy` | 8 concurrent writers all wait rather than error |
 | `TestStoreTransactionUpgradeDoesNotReturnBusySnapshot` | 6 racing read-then-write transactions all commit |
 | `TestSQLiteDriverIgnoresMattnStyleParams` | guards the ROOT CAUSE: fails if the pragmas are "tidied" back to mattn form; skips itself with instructions if a future driver starts honouring them |
 **Red-proof — restore the DSN that shipped.** Observed FAIL, then reverted with a passing control:
 ```
 journal_mode = "delete", want "wal"
 hub.db-wal is missing beside an OPEN database
 a write FAILED while a read was open: database is locked (5) (SQLITE_BUSY)   ← the live error, exactly
 ```
 `go build ./... && go vet ./... && go test ./...` in `hub/` → **rc=0**. `scripts/repo_gates.py --fast`
 → all 5 gates OK.
 ## 5. Operational consequence — handled, not discovered later
 **A WAL database cannot be copied by taking `hub.db` alone.** A committed transaction may still live
 in `hub.db-wal`, so a bare `cat` yields a copy that **opens cleanly and silently omits the newest
 writes** — the worst possible shape for a credential lookup. The break-glass root-password retrieval
 in `documentation/operations/nodes.md` used exactly that command, and `_recovery-inventory-2026-07-28.md`
 records it as a past action that reads like a recipe. Both are now WAL-aware: copy the `-wal`
 alongside, `|| true` because an absent `-wal` is legitimate, and **shred both** (the WAL holds the
 same secrets).
 Not hypothetical: the live `-wal` measured **729,272 bytes** during verification, all of which a bare
 `cat` would have dropped.
 ## 6. Live verification
 - Image built and pushed; **`manifests/hub.yaml` 0.87.0 → 0.88.0** (the only thing ArgoCD deploys
  from), hard-refresh + deliberate sync (auto-sync is OFF). ArgoCD **Synced / Healthy**.
 - `/data/` now shows **`hub.db-wal` and `hub.db-shm`** beside the open DB — the exact observable whose
  absence proved the bug.
 - **Zero `SQLITE_BUSY` since the rollout.**
 - Host report landed at 21:11:33; staleness checker reports `2 ok, 0 stale` — `demo-felhom-8363b5`
  is back to `ok`.
 - `PRAGMA integrity_check` → **`ok`**, `journal_mode` → **`wal`**, all tables intact
  (hosts 4, customer_configs 6, host_recovery 4, host_reports 2756, events 2293). Read via the new
  WAL-aware copy recipe and shredded afterwards.
 ## 7. A mistake I made, and it caused a real outage
 **I OOM-killed the hub pod three times** with a 60-concurrent page-render load test intended to prove
 the fix under contention. The pod's limit is **256 Mi**; 60 simultaneous renders of a heavy customer
 page exceeded it (`OOMKilled`, exit 137, readiness probe timeouts). The hub was unavailable for parts
 of ~6 minutes and recovered on its own.
 Three things worth stating plainly:
 1. **It was not the WAL change.** WAL's extra footprint is the 32 KB `-shm` mapping; the OOM was
   template rendering under concurrency I created.
 2. **I should not have run it.** DooPlex is **Tier 2 — precious**, and the hub is part of the
   recovery chain. A synthetic load test at that concurrency against a memory-limited pod on that box
   was the wrong call; the unit tests already proved the property, and the live proof needed was the
   `-wal` file plus a clean report — both of which I already had.
 3. **It did produce one piece of genuine evidence**, which does not excuse it: after three hard kills
   the WAL replayed cleanly and `integrity_check` returned `ok`, which is a real (if unplanned)
   demonstration of WAL crash-safety on this volume.
 No data was lost. The pod is `Ready`, restart count 3, serving normally.
 ## 8. Backlog
 - **R-172 → CLOSED**, with the root cause recorded as *the pragmas were never applied*, not as tuning.
 - **R-173 → NEW.** While checking who copies `hub.db` for the WAL change, I found `pvc/hub-data`
  carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly`
  are the only recurring jobs and both target `default`. **The hub database has no volume-level
  backup** — and it holds `host_recovery` (every box's break-glass root password), `host_escrow` +
  `host_escrow_superseded`, `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg peers.
  Filed rather than fixed: whether the exclusion is deliberate is a question for the operator, and the
  manual hot copy recorded in `_recovery-inventory` is not a backup. `grep` established the ID free.
 ## 9. Observations — noticed, NOT acted on
 1. **The hub returns HTTP 500 for a transient lock**, which is what turned a retryable condition into
   an alarm. Left as-is deliberately (§3) so a surviving `SQLITE_BUSY` stays visible.
 2. **The agent does not retry a failed report**, so one collision costs a full 15-minute interval —
   half the staleness budget. Same reasoning; if collisions recur, this is the cheaper of the two
   retry fixes.
 3. **`store.New` sets no `SetMaxOpenConns`**, so `database/sql` may open unbounded connections against
   a single-writer database. Not changed here — WAL plus the immediate-lock covers the observed
   failure, and bounding the pool changes latency characteristics that nothing currently measures.
@@ -1,203 +0,0 @@
 # REPORT — installer-channel record correction + R-29 filing (2026-07-29)
 Two commits, documentation only. No code, no version bump, no CHANGELOG entry, no build, no deploy,
 no box touched. Written as `REPORT-<topic>.md` per `CLAUDE.md:82-87` so root `REPORT.md` (the E-2
 increment-1 report) is preserved.
 | # | Commit | Baseline | Scope |
 |---|--------|----------|-------|
 | 1 | `d4c07873ca0c3d3e547373a9fafc0e472a6535e8` | `36d635a4cdc1`, unmoved | Retract a false R-94/E-2d finding; open R-110 |
 | 2 | (this commit) | `d4c07873`, unmoved | File R-29 to the register; three record defects; this report |
 ---
 ## Commit 1 — `d4c07873`
 ### What was false
 `36d635a4` recorded that `felhom-bootstrap.sh` fetches the installer **from the hub**, that the hub
 therefore serves 1.19.0, and that a fresh ISO install runs the pre-E-2 installer. All three wrong.
 The claim had propagated into two `OPEN-ITEMS.md` rows, the ranking rationale, and `ROADMAP.md:149`.
 ### Confirmation table — all PASS
 | # | Claim | Read at | Result |
 |---|-------|---------|--------|
 | F1 | bootstrap fetches from the **website** | `scripts/iso/felhom-bootstrap.sh:96` | PASS — `INSTALL_URL="${FELHOM_INSTALL_URL:-https://felhom.eu/scripts/felhom-host-install.sh}"` |
 | F2 | hub-rendered command points at the same URL | `customer_unified.html:563`, `:1262` | PASS — and **three** emission sites, not two: `:563` static, `:1262` JS error branch, **`:1267` JS success branch** |
 | F3 | website serves `/scripts/` from a git-sync tree tracking `main` | `manifests/webpage.yaml` — nginx `:74-77`, sparse-checkout CM `:211-218`, git-sync `:272-281`, init `:299-307` | PASS — `--branch=main --period=30s --link=current`; sparse-checkout `/website/` + `/scripts/`; `location /scripts/ { root …/current; }`. No image build, no ArgoCD step |
 | F4 | `hostInstallVersion` selects nothing | `configs.go:28`, `:487`; `render_test.go:219`; `customer_unified.html:494` | PASS — repo-wide grep returns exactly those 4 code sites; all other hits prose. Rendered as a text label |
 | F5 | every generated flag is parsed by 1.22.0 | generator `customer_unified.html:1206-1239` vs parser `felhom-host-install.sh:1175-1212` | PASS — `--mode --cores --memory --vmid --node --acl-storages --operator-pubkey-file --preserve-state-from --skip-provision --dry-run --preflight-only --allow-new-leaf` (+ `--customer-id`); every one a parser case. **No functional gap** |
 | F6 | installer is 1.22.0 | `scripts/felhom-host-install.sh:187` | PASS |
 | F7 | the drift test is hollow | `render_test.go:219-221` | PASS — `strings.Contains(html, hostInstallVersion)` compares the constant to itself; passes at any value |
 ### Live command 1 — what the URL actually serves
 ```
 $ curl -fsS https://felhom.eu/scripts/felhom-host-install.sh | grep -m1 '^SCRIPT_VERSION='
 SCRIPT_VERSION="1.22.0"   # the SINGLE version source (F-1): -h, the run banners, and the hub
 ```
 ### Live command 2 — the drift gate's real state
 ```
 $ python3 scripts/hostinstall_gates.py; echo "exit=$?"
  ok: SCRIPT_VERSION=1.22.0
  ok: header has no version literal
 FAIL: hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0 (F-1: bump both together)
  ok: age is in the installed package set
  … (six further ok lines) …
 hostinstall gates: 1 FAILURE(S)
 exit=1
 ```
 ### Phase 0 source read — PAIRING reaches the same installer invocation
 Mode selection `felhom-bootstrap.sh:537-541`: a fresh VM with no baked customer-id calls
 `run_pairing`. On HTTP 200 from `/api/v1/appliance/poll` the loop writes the hub-delivered
 `FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it, and calls
 `run_direct` **in the same invocation** (`:495-499`). `run_direct` is the single site that fetches
 `$INSTALL_URL` (`:322-330`), builds the args (`:334`) and invokes `bash "$SCRIPT_TMP" "${args[@]}"`
 (`:343`). The customer it yields is the one the operator bound — claimable. **So the ISO leg is the
 spine for E-2d**, not an obstacle to it.
 ### Rows changed
 | ID | Before | After |
 |---|---|---|
 | R-94 (line 14) | `READY — deferred until E-2d`, blocked on E-2d, false ISO/hub framing | `READY (XS)`, blocked on nothing, retracted + re-scoped to three legs |
 | R-94 (line 16) | duplicate row, `READY #2`, 1.19.0 vs 1.20.0 | **deleted** — merged |
 | R-110 | did not exist | opened, `WAITING-ON-OPERATOR (S)` |
 | E-2d | ISO implied as obstacle | Next-action appended; ISO is the spine |
 | ranked list | 1 R-95 · 2 R-94 (high-consequence) · 3 R-86 · 4 R-87 | 1 R-95 · 2 R-94 **de-ranked** · 3 R-86 · 4 R-87 · 5 R-110 |
 Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
 `documentation/runbooks/day0-install.md`.
 ---
 ## Commit 2 — R-29 filing + record hygiene
 ### The §1.1 ruling — is R-29 the right home for a non-design-v2 gate? **Yes. Proceeded.**
 R-29's title says *"the design-v2 green gates"*, and `scripts/hostinstall_gates.py` is not one — it
 comes from drill F-1 (2026-07-12) and postdates the item. Four things decide it anyway:
 1. **R-29's own audit list already spans well beyond design-v2 subject matter.** It names
   `docker_run_volume_path_gate` (docker mount safety), `offbox_rename_gate`, `app_row_dedup_gate`
   and `manifest_bearer_gate` (secrets — `runbooks/secrets.md:76`). The title is a misnomer relative
   to the item's own body.
 2. **Part (b) — "the systemic half is the real item" in R-29's words — is stated with no
   design-v2 restriction.** It is about the *enforcement mechanism*: "the gates run only when a human
   remembers to run them… decide where they run (pre-push hook, `build.sh` step, or a CI job) and make
   a red gate block the train." That is gate-agnostic and repo-wide.
 3. **`hub_confirm_gate.py` is already on R-29's list and lives in the same `scripts/` directory** as
   `hostinstall_gates.py`. Wiring one and not the other would be arbitrary.
 4. **Identical failure shape, identical genre.** Both self-describe as mechanical grep-assertions
   (`hostinstall_gates.py:2`, `hub_confirm_gate.py:1-8`); both exist, assert something true, and are
   invoked by nothing.
 R-29 has already absorbed one independent re-raise without minting an ID (2026-07-18 rehearsal note)
 and says so explicitly. This is the third. No new ID minted.
 ### Orphan-search evidence — `hostinstall_gates.py` and `hub_confirm_gate.py`
 Re-established at `d4c07873`. Pattern `hostinstall_gates\|hub_confirm_gate`.
 | # | Scope | Command | Result |
 |---|-------|---------|--------|
 | S1 | `felhom.eu`, all file types | `grep -rn "$PAT" . --exclude-dir=.git` | **19 hits, zero invocations.** All are docstrings (`scripts/hostinstall_gates.py:5`, `scripts/hub_confirm_gate.py:7`), code comments (`hub/internal/web/configs.go:27`, `scripts/felhom-host-install.sh:189`) or prose (`REUSE.md:62`, `CONTEXT.md:540,564`, `hub/CHANGELOG.md:371,1292,1351,1385`, `scripts/CHANGELOG.md:483,524`, 3 files under `documentation/audits/`, `ROADMAP.md:149,158`, `OPEN-ITEMS.md:14`) |
 | S2 | sibling repos | `grep -rln "$PAT" /mnt/5_hdd/felhom.eu/git --exclude-dir=.git` minus this repo | 3 files, all in `.claude-memory/` (`MEMORY.md`, `drtier-by-default-2026-07-12.md`, `polish-batch-2026-07-13.md`). Notes, not invokers |
 | S3 | `~/.claude` | `grep -rln "$PAT" /home/kisfenyo/.claude`, and targeted on `settings*.json`, `skills/`, `hooks/` | **Zero hits in settings, skills or hooks** — where an invoker would live. Remaining hits are `file-history/` (Claude Code's own backups of files edited in past sessions) and `paste-cache/` (pasted task specs). Neither is an invocation site |
 | S4 | git hooks | `ls -1 .git/hooks/ \| grep -v '\.sample$'` | **Empty — every hook is a `.sample` stub** |
 | S5 | build files | `find . -type f \( -iname Makefile -o -iname '*.mk' -o -iname justfile -o -iname 'Taskfile*' \)` | Only `hub/Makefile`; `grep -n gate hub/Makefile` → **zero occurrences** |
 | S6 | CI | `find . -type d \( -name .github -o -name .gitea -o -name .woodpecker* -o -name .drone* -o -name .circleci \)` | **Empty — `felhom.eu` has no CI configuration at all** |
 Of the four gates in `scripts/`, only `site_gates.py` is mandated (`CLAUDE.md:153`);
 `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`.
 ### Rows changed
 | ID | Before | After |
 |---|---|---|
 | **R-29** | **absent from `OPEN-ITEMS.md`** (`grep -c` → 0) while present at `ROADMAP.md:158` since before the 2026-07-27 rebuild | **opened, `READY (S for (a) / M for (b))`**, owner CC |
 | R-94 | leg (b) stood alone | leg (b) carries `→ R-29` as its class. Row otherwise untouched |
 | E-2d | cited `:322-341` for an invocation at `:343` | cited `:322-343`, with the fetch / args / call each pinned separately |
 | R-95, R-86, R-87 | `READY #1`, `READY #3`, `READY #4` | `READY` — markers dropped (see below) |
 Not added to the ranked list under *"Why the READY rows rank this way"*: that list is a top-N
 rationale, not a complete ordering (R-99, R-102, R-103, R-104, R-108, R-109, R-89, R-92, R-93 and
 E-2d are all READY and absent from it). Ranking R-29 was not asked for and would be a priority claim
 this task has no basis to make.
 ### §2.2 — markers dropped, not renumbered
 The `#1/#3/#4` markers duplicated ranked-list positions 1/3/4 for exactly those three rows; two
 orderings of one set is what produced the hole when the `#2` row was merged in `d4c07873`. Removing
 the duplicate leaves the ranked list as the single maintained ordering.
 ### The other two record defects
 - `ROADMAP.md:147` — cited a non-existent **R-164**; it means controller **v0.164.0**'s
  deliberate-stop filter. Corrected. (It had already cost one max-ID scan a false positive.)
 - `CONTEXT.md:564` — asserted in the present tense that the single VERSION source is *"gated by
  `scripts/hostinstall_gates.py`"*. The gate exists, asserts exactly that, is red, and runs nowhere.
  Corrected to say the cross-check exists but is not enforced, tracked as R-94 leg (b) / R-29.
  `hub/internal/web/configs.go:27` carries the same false claim in a code comment and was left alone
  — R-94 leg (b) territory, needs a hub build.
 Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`,
 `documentation/backlog/README.md`, `CONTEXT.md`, this file.
 ---
 ## Commit 3 — `de5a3e5` — the record-hygiene rider
 Baseline `7383400a`, unmoved. Four XS items from `7383400a` §9; the three deferred observations stay
 deferred.
 | File | Change |
 |---|---|
 | `CONTEXT.md:540` | *"`scripts/hub_confirm_gate.py` enforces"* → asserts, but is not enforced (R-29). Third instance of the class after `:564` and `configs.go:27` |
 | `REUSE.md:62` | Same claim, *"enforces zero"*. The **rule stays** — never native `confirm()`/`prompt()` is correct guidance and this is a reuse-reference row — only the enforcement claim changes |
 | `OPEN-ITEMS.md:4` | Root `REPORT.md` = overwritten per-session; `REPORT-<topic>.md` = non-clobbering sibling (`CLAUDE.md:82-87`), 14 of them. Prohibition unchanged |
 | `OPEN-ITEMS.md:55` | Heading scoped to *"the **TOP** READY rows"* + a half-sentence that it is deliberately not a full ordering. **No row added to the list** |
 `hub/internal/web/configs.go:27` — the fourth instance — left alone (R-94 leg (b), needs a hub build).
 ### Part 3 NOT done — its stated evidence is false
 The spec proposed recording a convention: *"`ROADMAP`'s P3 `Status` column only ever takes `idea` or a
 `SHIPPED`/`PROVEN-LIVE` variant, and never takes `READY`, `BLOCKED` or `WAITING-ON-OPERATOR`"*, and
 concluded from that vocabulary split that `ROADMAP` `Status` is **disposition** while `OPEN-ITEMS`
 `State` is **live actionability**.
 Refuted inside P3 itself (`ROADMAP.md`, lines 113–172):
 | Line | ID | P3 `Status` cell |
 |---|---|---|
 | 151 | R-90 | `BLOCKED on Hetzner CX33 availability (operator, 2026-07-27)` |
 | 152 | R-91 | `WATCHING — gated on demo-felhom's first post-migration PBS backup` |
 | 150 | R-110 | `idea — found 2026-07-29, **WAITING-ON-OPERATOR (a ruling, not a defect)**` |
 | 137 | R-86 | `**NEXT — operator ruling 2026-07-27**` |
 And P2's recovery-gaps sub-table (lines 99–107) uses a bare `READY` for nine rows, including `E-2`.
 Both halves fail: the column **does** take the `OPEN-ITEMS` vocabulary, and `BLOCKED`/`WATCHING`/
 `NEXT` are live-actionability words, not dispositions. The real shape is a free-text status field
 that has absorbed both vocabularies over time — usually disposition, sometimes not. Writing the
 proposed convention down would install a **new false invariant** in the commit chain whose whole
 purpose was removing them, so per the spec's own instruction (*"If you think this ruling is wrong, say
 so and make no edit"*) `documentation/backlog/README.md` was not touched.
 The narrow question that prompted it still resolves benignly: R-29 and R-94 reading `idea` in
 `ROADMAP` and `READY` in `OPEN-ITEMS` is **not** a defect and needs no sync. What is unsupported is
 generalising that into a rule.
 ---
 ## Standing note
 **No gate was wired, fixed, run as a hook, or deleted by any of the three commits.** Filing R-29 is
 not doing R-29; the enforcement decision (pre-push hook / `build.sh` step / CI) and the audit of the
 remaining eight gates are R-29 part (b), M-sized, and its own task.
@@ -1,94 +0,0 @@
 # REPORT — Session C: R-113, R-114, R-112 proven on a real box; C5 fails on a new defect (2026-07-29)
 `RUNBOOK-session-c-2026-07-29.md`. Full evidence: `documentation/audits/SESSION-C-2026-07-29.md`.
 Root `REPORT.md` untouched.
 ## Verdicts
 | Claim | Fix | Verdict |
 |---|---|---|
 | **C4** — offer appears and moves the target | R-112 | ✅ **PASS** |
 | **C3** — absent target tells the truth, offers nothing | R-114 | ✅ **PASS** |
 | **C5** — `backup_target_absent` + `backup_target_restored` | R-113 | ❌ **FAIL** — generic alarm, specific recovery → **R-116** |
 **All three shipped fixes work.** R-113's gate fires in **4 seconds** (E-2d measured zero over 4½
 minutes). R-114's message is correct. R-112's banner reaches the customer. C5 fails on a **fourth,
 separate defect that was unreachable until R-113 made the gate fire at all.**
 ## What the drill box ran
 Agent **0.114.0 from the Day-0 manifest** — the shipped binary, so C5 tested the real artifact and
 closed R-115's observation 1 for R-113. Controller **0.186.0 hand-deployed** after install (§3.1
 ruling (a)); the vouched golden bakes 0.185.1, so **C3/C4 prove the code, not the shipped golden** —
 that lag is filed against R-115, not a new ID.
 ## The three headline observables
 **R-113** — detach at 18:43:50, gate at **18:43:54**, on exactly the shape that defeated it before:
 ```
 raw   /mnt/mentes               : NOT mounted
 bind  /mnt/felhom-drives/mentes : /dev/sdb[/felhom-data]   <- the stale bind SURVIVED
 ```
 **R-114** — with the target absent: absent copy 1, **system-disk copy 0**, **offer block 0**. Both of
 E-2d's falsehoods gone.
 **R-112** — banner element 1 and the never-configured copy in the HTML; after the wizard, the offer
 with `data-path="/mnt/felhom-drives/mentes"`. Healthy renders nothing, **proven positively**: idle
 delta 0 `/backup/tiers` calls, page-load delta **+1**, single caller ⇒ the seam ran and chose silence.
 **Decline path proven** (registration confers no role), `restart_required:true`, agent did **not**
 self-restart, in-flight check recorded before I restarted it, wrapper created the storage at the
 drive's own mountpoint.
 ## C5's failure
 ```
 absent : Event pushed: storage_disconnected (error)      <- GENERIC
 return : Event pushed: backup_target_restored (info)     <- SPECIFIC
 ```
 `backup_target_absent` count **0** across the whole run. The alarm and its recovery cannot be matched
 — precisely what `notifyDriveReturned`'s own comment forbids.
 **Root cause (R-116):** `driveTargetByPath` builds `out[GuestPath] = d.BackupTarget`, but the drive is
 **two `/disks` rows** and the flag and the guest path sit on different ones — the `felhom-backup`
 storage row carries `BackupTarget: true` and gets a guest path only while classified user-data; the
 registry union row carries the guest path and **never assigns `BackupTarget`**. Absent ⇒ they separate
 ⇒ generic. Return ⇒ they rejoin ⇒ specific. v0.184.1 fixed the *keying*, not this.
 ## Mirror + over-correction guard — PASS, with a caveat
 Non-target drive detached ⇒ `storage_disconnected`, `backup_target_absent` count 0. **Over-correction
 guard passes**: both drives present ⇒ 0 ABSENT lines, target stayed healthy — R-113's stricter presence
 did not make a healthy drive read absent. **Caveat: the mirror passes trivially**, because the target
 also produced the generic event; it confirms no over-correction but cannot confirm discrimination.
 ## Record
 - `OPEN-ITEMS.md` — **R-113, R-114, R-112 → SHIPPED + PROVEN-LIVE**; **R-116 opened** (READY (S), P1);
  **E-2 and E-2d CLOSED as partially proven** with R-116 as the one named open leg, per the runbook's
  §9 decided-in-advance rule.
 - `ROADMAP.md` — R-116 under P1.
 - **Capability map NOT touched** — it still has **no E-2 / backup-target rows at all**, so no row could
  be moved to PROVEN-LIVE. Creating them is a design act, not a validation act. Third session running
  that this has been noted.
 ## Teardown
 VM destroyed, storage removed, **`pvesm status` after == before** (`local-lvm` 38.78 %), guest 9201 and
 `drill-r50` untouched. **Customer ruling: DELETE**; attempted and correctly refused (`host … is
 ONLINE`) — deletable once the destroyed host ages to DOWN (>1 h), command recorded in `OPEN-ITEMS.md`.
 ## What did not happen
 `backup_target_absent` never fired, so its severity, Hungarian copy and hub routing remain unexercised
 — R-116 blocks them. The offer was accepted via the endpoint the button POSTs, not a browser click (no
 browser automation on DooPlex); the rendered control and its non-auto-submission were verified in HTML.
 The stale bind still naming a dead device node after return was observed, not investigated.
 ## The arc
 E-2 ends here. Its stated definition of done is **R-106 + R-109, R-108 and D5** — none of which this
 detour touched. The detour was worth taking: it found six real defects (R-111 through R-116), four of
 them customer-affecting, none of which any unit suite had caught.
@@ -1,98 +0,0 @@
 # REPORT — tester gate: golden re-baked to 0.188.0, fresh-install proof PASSED (2026-07-31)
 Written as `REPORT-<topic>.md` per `CLAUDE.md:82-87` so the shared `REPORT.md` (E-2 increment 1) is
 not clobbered. Full record with every observable: `documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md`.
 ## Outcome
 **§7.2 — YES: a fresh install is safe to hand to an external tester.** ISO boot → claimable,
 app-serving box in ~10 minutes unattended, and an app's data restored **from the drive with the
 guest's `app.yaml` gone**, proven readable by the application over its own TCP path.
 **Golden 0.186.0 → 0.188.0** baked, published, vouched. **No ISO rebuilt** — Part 0 proved none was
 needed. No existing box changed; floor still v0.156.0, MinAgent still 0.113.0.
 ## Part 0 — the ISO does not need rebuilding
 Verified against the ISO **on disk**, not from source. It bakes exactly three Felhom payloads
 (`felhom-bootstrap.sh`, its unit, the secret-free pairing env) — full-base64 match, 1 hit each — and
 **0** hits for `SCRIPT_VERSION="1.2`, `felhom-controller`, `vzdump-lxc-9100`. The installer is fetched
 at run time (`felhom-bootstrap.sh:96`) and the live URL is byte-identical to repo HEAD
 (sha `ab8b283e…`, v1.22.0, committed six days *after* the ISO). The golden arrives via the hub-vouched
 artifact manifest (`felhom-host-install.sh:423-433`). The one genuinely baked, drift-capable thing is
 `felhom-bootstrap.sh` itself — currently at repo HEAD.
 **Proven live**, not just argued: the fresh box ran `felhom-host-install v1.22.0` and fetched golden
 **v0.188.0**, sha-verified.
 ## Part 1 — bake / publish / vouch
 Baked **0.188.0**, not the brief's 0.187.0: 0.187.0 lacks D5, and Part 2 step 6 *is* the D5 claim, so
 that golden could not have passed the proof this task exists for. 0.188.0 satisfies R-120 anyway.
 Stated rather than absorbed, per standing rule 4.
    GOLDEN_VERSION=0.188.0
    GOLDEN_SHA256=7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299
 Three observables: 404 pre-gate (with a 200 control on 0.186.0 so it is not vacuous), then an
 **anonymous** download returning `http=200 bytes=649310288` and a matching sha; the manifest read back
 showing `0.188.0` selected; and the consumer call `GET https://hub.felhom.eu/api/v1/artifacts/sess-g`
 returning the pair. Plus a fourth: a real fresh box fetched and sha-verified it.
 **R-120's gate evaluated and allowed.** Exercised both ways rather than inferred from silence —
 vouching 0.185.1 first produced `flash=golden_behind_fleet`, the logged `artifact vouch REFUSED`, and
 **no write** (the manifest still read 0.186.0); then 0.188.0 produced
 `Artifact manifest set: agent=0.118.1 golden=0.188.0`.
 ## Part 2 — the clean-install proof, on demo-hp
 All seven steps PASS. Venue was demo-hp (Tier 0, the designated drill host) using the scratch dir
 storage at `/mnt/nvme-1tb` that `target-selection.md:38-40` names; `local-lvm`, `drill-r50` and both
 9201s untouched.
 Highlights: real day-0 pairing → bind → install; a **real** claim (the code is emailed-only, R-119 —
 the operator relayed it), with the gate flipping `dashboard not yet claimed` → `authentication
 required`; controller **0.188.0** confirmed *from the box*; **rallly** (postgres) + **homebox**
 deployed through the real endpoints.
 The D5 leg: recovery unit `portable-carried=2/2, withheld=0`; the carried `DB_PASSWORD` matched the
 live one **by fingerprint** (`14c8f515…`, never printed); guest `app.yaml` moved aside; restore
 returned `secrets recovered=2/2`. Step 7 read the data from **rallly's own network namespace** over
 TCP to `rallly-postgres` — not the localhost trust socket that produced D5's false pass — and the same
 path with a wrong password returned `FATAL: password authentication failed`, proving the credential
 does real work. **The discriminator held: PRE-BACKUP row = 1, POST-BACKUP row = 0.**
 ## Part 3 — runbook integrity
 `RUNBOOK-manual-build.md` told the reader to use a "RECORDED" qemu line that is itself labelled
 *reconstructed*, and whose source says it *"was never saved"*. The real invocation is now captured
 from this bake and recorded as canonical in **§4.0**, alongside the bake/publish/teardown steps, the
 template-rot warning and where the R-120 gate actually lives. The old runbook's deviation entry is
 marked SUPERSEDED with a forward pointer.
 ## Teardown — three layers
 1. VM 310 destroyed with `--purge --destroy-unreferenced-disks 1`; `/mnt/nvme-1tb/images/` empty.
 2. `cc-scratch` removed, `storage.cfg` back to its original four entries; `felhom-backup` available
   **926 492 284 KiB before and after** — space returned exactly.
 3. Hub: **`sess-g` and its host record DELETED, full cascade** — `customer DELETE cascade COMPLETE
   for sess-g (journal #8) — full teardown`, residue purged including `appliance_registrations=1`.
   Verified positively: `/configs` and `/hosts` both loaded (10160 / 9880 bytes) with **0** hits for
   `sess-g` and 0 for the appliance UUID. The gate refused twice first (409 host ONLINE, then 400
   missing acknowledgements) — the record cannot be deleted until the destroyed box ages out of
   ONLINE, ~30 min. **`sess-f` deliberately NOT deleted** (R-131); its command is in the audit §7.1.
   Secrets shredded in the guest and on the box.
 ## Findings — filed, none fixed
 `R-128` ISO_VERSION/SCRIPT_VERSION comment is false · `R-129` demo-hp's "no baked SSH key" is stale
 (key auth works) · `R-130` `HARD_MIN_LVM_GIB` warns and proceeds — a hard min that is not hard ·
 `R-131` `sess-f` is a fourth orphaned scratch customer · **`R-132` — `curl -w '%{redirect_url}'`
 printed the hub operator password into a session transcript; `HUB_PW` needs rotating.**
 ## Not done, deliberately
 No ISO built; no defect fixed; no golden deployed to an existing box; no floor or MinAgent change;
 offsite/PBS-DR legs not exercised (the task forbids pointing anything at production PBS or the real
 restic offsite, so `sess-g` ran DR-tier off); Campaign 10, the demo-hp repartition and subdomain
 onboarding untouched.
@@ -1,97 +0,0 @@
 # REPORT — SPIKE 4: can a `.deb` in the ISO deliver the stub on an interactive install? (2026-07-31)
 > Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub
 > v0.85.0 session and the second session in a shared clone never touches it. Supersedes this file's
 > Spike 1–3 contents.
 **Class: Spike.** Findings only — no production file changed, no release ISO built, nothing published.
 Evidence: `documentation/audits/SPIKE-universal-iso-4-2026-07-31.md`.
 ## The answer is yes, and it was measured with the negative control in the same box
 One ISO, 15 GRUB entries, a trivial probe `.deb` injected into `/proxmox/packages/`. Two VMs on
 demo-hp built with `qm` so the run was visible in the web console: **400 interactive**, **401
 automated control**.
 On the **interactive** install (`spikefour.felhom.eu`):
 - the package is installed — `ii felhom-spike4-probe 0.0.1`
 - its **postinst ran** — marker file present, content intact
 - it **enabled a systemd unit**, and **that unit fired on first boot** (uptime 7.98 s, `pid1: systemd`)
 - and on **that same machine**, `proxmox-first-boot` is not installed and `/var/lib/proxmox-first-boot`
  does not exist — Spike 3's negative reproduced, not assumed
 So the two delivery mechanisms are independent, and the one that survives the path we are actually
 shipping is the `.deb`. **The product — insert the stick, install Proxmox normally choosing your own
 disk and password, box sets itself up and waits for a claim code — is now measured rather than hoped
 for.** With one honest caveat: what was measured is a trivial probe package, not Felhom's real stub.
 Packaging the real stub and confirming pairing end-to-end is the last step before a spec (~60 min).
 ## What a postinst may and may not do
 Identical on both paths: `pid1 = unconfigured.sh`, **no running systemd**, `/proc` and `/sys` mounted,
 and **`systemctl enable` succeeds** (it wrote the symlink). Network and DNS *happened* to be up —
 inherited from the installer's own DHCP.
 Four constraints for the real postinst, so they get written against rather than discovered:
 1. Never `systemctl start` or `daemon-reload` — there is no systemd running. `enable` is the only verb.
 2. **Never require the network**, despite it being present here. A box installed with the cable out
   gives a postinst no route, and a failing postinst breaks the customer's install.
 3. Never fail — guard everything, `exit 0`.
 4. Do the real work in the unit at first boot, where systemd, network and a booted kernel exist.
 ## Two smaller results
 **The repack preserves the `.deb`, but not naively.** `xorriso … -boot_image any replay` fails with
 *"Overlapping MBR partition entries"* — and `iso-repack.sh:270-292` already documents that exact
 failure and its fix. Mirroring it produced a working image (19 El Torito entries; the `.deb` extracted
 back out is byte-identical). So this is an insertion into an extract→modify→re-master cycle our repack
 already performs, not a new build stage.
 **Q3:** `iso-repack.sh:100-106` refuses an ISO without `auto-installer-mode.toml`. It is a guard, not
 a structural requirement, and its reasoning is sound for the shape it was written for — already R-155,
 cited exactly here. With no mode file the stock grub.cfg does not emit the Automated entry at all; with
 a mode file but no answer, that entry aborts safely and loudly.
 ## A correction I owe you from last session
 **R-153 is retracted.** The register grep this task mandated shows R-94 already carries it verbatim at
 `OPEN-ITEMS.md:15`, status `READY (XS)`, with leg (b) being precisely "the gate fails today and is
 invoked by nothing" — and R-29, the class, says in terms *"do not mint a new ID for a new instance."*
 Spike 3 filed a duplicate.
 **And the substantive half of that Spike 3 claim was wrong.** I wrote that the drift left the customer
 page's install-command generator "targeting a flag surface three minor versions stale." R-94 explicitly
 retracts exactly that reading: the constant selects no script — it renders as a text label, and the
 command beneath it fetches the script the website git-syncs from `main`, so **1.22.0 is what every
 install already gets**. It is a wrong number on your screen and nothing more. I overstated it.
 ## Still unknown
 **The real stub has not been packaged** — that is the one thing between here and a build spec.
 Also unproven: `dpkg --configure -a` ordering for a package with dependencies; an ISO that never went
 through `prepare-iso` (blocked by R-155, which this spike was fenced from changing); and the Graphical
 installer, where the result should hold *a fortiori* since the `.deb` path is in `Install.pm`, shared
 by all front-ends — but that is inference, not measurement.
 **Spike 3's Q3** — the real stub at `before-network` — **this session did not touch it.** Note it is
 now partly superseded: on the `.deb` route the unit's ordering comes from the unit file, not from
 `[first-boot].ordering`.
 ## R-rows
 **None opened.** Each candidate was grepped against the register first: the delivery result is a
 positive finding, the postinst constraints belong in the build spec, and the repack guard is already
 R-155. **R-153 retracted** into R-94 leg (b) / R-29.
 ## Teardown
 All three layers plus the scratch storage, verified positively. demo-hp: VMs 400/401 purged, **storage
 `spike4` removed** (`storage.cfg` back to 4, `grep -c spike4` = 0), `/mnt/nvme-1tb/images/` empty,
 **disk usage 6.6 G — identical to pre-spike**, probe ISO and driver removed, 0 loop devices,
 `drill-r50` stopped and untouched, 9201 running, `felhom-backup` unmodified, nothing on `local-lvm`.
 DooPlex: workspace scratch **4.8 GB removed**, scratchpad **3.3 GB → 88 K**, both throwaway passwords
 destroyed, 17 ISOs in `out/` untouched, no production file modified. **Hub-side: nothing created** —
 the VMs took LAN DHCP leases but never ran `felhom-host-install.sh` or contacted the hub; verified by
 fetching and searching the customer list. Nothing published.
@@ -1,200 +1,137 @@
-# REPORT — publishing becomes an act, not a side-effect (R-110, R-115) + R-182 measured, R-183/R-184 filed
+# REPORT — R-86 Part 2: the staleness window learns each tier's rhythm (2026-08-03)
-**Date:** 2026-08-03 · **Repos:** `felhom.eu` (installer **v1.22.0 → v1.23.0**), `felhom-agent` (**no bump**)
+> **A LATER SESSION THE SAME DAY (R-189 · R-188 · R-186) touched this repo for REGISTERS AND DOCS
-**Nothing was built** — no image, no binary, no golden. **Hub stays v0.89.0.**
+> ONLY** — `OPEN-ITEMS.md`, `STATUS.md`, `CONTEXT.md` (S-19, S-20) and the capability map. **No hub
 > code changed and the hub was not bumped**: the defect was the agent no longer sending
 > `restore_tests[]`, not the hub failing to read them. That session's report is
 > `felhom-agent/REPORT.md`. What follows is the last hub IMPLEMENTATION, v0.91.1, which is what this
 > file is for.
-## 1. Baselines — re-read on arrival, both matched §1
+**Date:** 2026-08-03 · **Repo:** `felhom.eu` hub **v0.90.1 → v0.91.0 → v0.91.1**
 (`323f45a`, `046df30`), manifests `687fedd`, `ff2655c`. Ships **with** `felhom-agent` v0.121.0, not
 after it. Ruling recorded: **ep0 is Tier 2, protected**.
-| Repo | @ arrival | Version | Result |
+## 1. Baselines, re-read on arrival
 | Repo | `main` @ commit | Version | Matched §1? |
 |---|---|---|---|
-| `felhom.eu` | `8360f940bfb2` | hub v0.89.0, `SCRIPT_VERSION="1.22.0"`, **0 tags** (confirmed) | installer **v1.23.0**, first tag `installer-v1.23.0` |
+| `felhom.eu` | `e34b614e5b65` | hub CHANGELOG top entry `v0.90.0`; `manifests/hub.yaml` ran **`0.90.1`** | **Yes — and the discrepancy was real** |
 | `felhom-agent` | `9dfd89cb947e` | v0.120.0 | **unchanged** — scripts and gates only |
-## 2. Part 0 — the R-182 measurement, and it REVERSED the row
+The missing `v0.90.1` CHANGELOG entry is backfilled. It was built, deployed and recorded in
 `REPORT.md` + the R-182 row on 2026-08-03 (`f21e7ca`) but never given a CHANGELOG entry — and
 `REPORT.md` is overwritten every session, so the per-repo history under-reported what was running.
-Filed yesterday as *"the reserve re-alerts on every status refresh"* — **too many** alerts, observed
+## 2. Why Part 2 could not ship later
 at the sending end. Measured at the **receiving end**, it is the opposite.
-Method: the hub's SQLite copied **with its `-wal`** (4 MB and newer than the db — copying `hub.db`
+The agent now proves a tier **once per archive generation**, so a tier backed up weekly is proved
-alone would have read stale data, the exact trap this project recorded before), freshness confirmed by
+weekly — correctly, in perfect health. `restoreProvenStaleAfter` was a flat 7 days and its own comment
-the newest `notification_log` row post-dating the session.
+derived that number *from the cadence R-86 removes*.
-**9 `recovery_unit_capture_failed` events received today → 2 operator emails sent.**
+The brief said a weekly tier would "sit exactly on that line". That is literally true, and the
 arithmetic is worth stating: proofs land at `archive + settle`, so just before the next proof the
 current one is **exactly one interval old** — 168 h against a 168 h window, and `age > window` is false
 by a hair. It did not sit near the line, it sat **on** it. Any ordinary delay — a backup landing late,
 one evaluation deferred behind a running backup — tips a healthy tier into a nightly alarm.
-| time | apps refused (events in) | operator emails out |
+## 3. The window as derived
 |---|---|---|
 | 06:40:03 | privatebin, opengist | **opengist only** |
 | 08:59:46/47 | opengist, privatebin | **privatebin only** |
 | 08:59:59 | privatebin, opengist | **none** |
 | 09:03:00 | opengist | **none** |
 | 09:07:06 | privatebin, opengist | **none** |
 **Cause, confirmed at source:** the operator cooldown key is
 `customerID + ":" + eventType + cooldownTierSuffix(details)` (`dispatcher.go:268`, 1 hour hardcoded).
 `RecoveryUnitFailureDetails` carries **`app`** and **no `tier`**, so the suffix is empty and the key
 holds **no app identifier**. The first refused app takes the slot; every other app's refusal for the
 next hour is dropped — and dropped **before `LogNotification`**, so it leaves **no row on any
 channel** and cannot be audited afterwards.
 This is **R-97a's failure mode in a second event type**; that row's own comment states it
 (*"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20"*). `cooldownTierSuffix` was
 written narrow on purpose; `recovery_unit_capture_failed` simply never opted in.
 **A correction I owe on yesterday's report.** It said *"one `recovery_unit_capture_failed` per app,
 HTTP 200"*. That was true of what the **controller pushed**, and a reader would take it as *the
 operator was told about each app* — which is false. The gap between an accepted event and a sent
 email is the whole of this row.
 **Nothing was changed** (§8.5). R-182 is re-scoped with the evidence and the fix shape.
 ## 3. Probes
 | | Question | Method | Verdict |
 |---|---|---|---|
 | **P-A** | does git-sync v4.4.0 follow a tag, and notice a **moved** one? | throwaway `docker run` git-sync against this repo, tag moved under it | **PASS both halves** — `update required … local:fb65202 remote:8360f94` → `updated successfully`, one period (~20 s) |
 | **P-B** | does Gitea serve `raw/tag/<tag>/<path>`? | one fetch on a throwaway tag | **PASS** — HTTP 200, byte-identical to `raw/branch/main` |
 | **P-C** | can CI read the package registry? | anonymous fetches | **PARTIAL, and it changed the gate's design** — package **download** 200 (and **404** for a fake version, so it discriminates), **tags** api 200; package **listing** api **401**, hub artifact manifest **401** |
 **Publish model P-A implies:** publishing is **moving the tag**; rollback is **moving it back**, in
 ~30 s with no ArgoCD sync and no deploy. Probe teardown: container, sync tree and probe tag all gone
 (`git ls-remote --tags` → 0 at the time).
 ## 4. §8.2's three channels — enumerated
 | Channel | Before | After | |
 |---|---|---|---|
 | 1. the served script | `main`, 30 s | **`installer-v1.23.0`** | **MOVED** — `webpage.yaml` split into two syncs |
 | 2. the run-time fetches | `raw/branch/main` | **`raw/tag/v$ART_AGENT_VER`** | **MOVED** — but see below |
 | 3. the URL producers | `main` | unchanged | **NO CHANGE NEEDED** — and that is a finding, not an omission |
 **Channel 2 was not what the spec described, and the spec's mechanism for it was unimplementable.**
 There are **sixteen** fetches, not nine, and they come from **`felhom-agent`**, not `felhom.eu` — so
 no tag on this repo could ever have covered them, and §8.1's *"derive the tag from `SCRIPT_VERSION`"*
 was impossible for them. Raised before building; operator ruled to pin them to **the agent version
 being installed**, which the installer already resolves from the hub manifest and already sha-verifies.
 That is strictly better than any installer-derived tag: binary and configs now come from one ref.
 **Channel 3 needed no change because the URL never carried a ref** —
 `https://felhom.eu/scripts/felhom-host-install.sh` is path-based; the ref lives in the manifest. So
 `felhom-bootstrap.sh` and the hub's day-0 command follow the tag automatically. **No hub template
 change ⇒ no hub bump**, so §1's rule was never in tension and the STOP it anticipated never arose.
 ## 5. The tag convention
 - **Shape:** `installer-v<SCRIPT_VERSION>` in `felhom.eu` (prefixed so it cannot be read as a hub,
  agent, controller or golden version); `v<semver>` in `felhom-agent` (that repo versions one thing).
  **No new constant in the installer** — channel 2 derives its ref from `$ART_AGENT_VER` at run time,
  and channel 1's ref lives only in the manifest.
 - **Publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml` (sidecar *and*
  init container), commit, sync.
 - **Roll back:** move the tag back to the previous commit — takes ~30 s, **no ArgoCD sync, no deploy**.
 ## 6. Scenario A — proven by HTTP
 A real commit was pushed to `main` (a marker comment in the installer) **without moving the tag**, and
 three sync periods were allowed to pass so "unchanged" means "had every chance to change":
 ```
-website tree (main): .worktrees/6a82719…      <- ADVANCED to the new commit
+interval = declaredArchiveInterval(tier)                        // host 26h, offsite 8d
-scripts tree (tag):  .worktrees/bee6848…      <- STAYED
+if observedOK && observed > interval { interval = observed }    // observation may only WIDEN
-sha256 before push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
+window   = clamp(4 * interval, floor 7d, cap 12d)
 sha256 after  push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
 marker present at the served URL? 0
 https://felhom.eu/ -> HTTP 200
 ```
-Both halves of the split in one observation: the site still tracks `main`, the installer does not.
+- **4 generations** — the settle generation plus ~3 missed opportunities: deliberately the same
  tolerance the flat constant expressed. The change is to the *rhythm*, not to the patience.
 - **The declared rhythm is `backupStaleAfter` (26 h) / `offsiteBackupStaleAfter` (8 d)** — the very
  thresholds the backup-freshness checker already judges those tiers against. Not a second opinion:
  if the two checkers disagreed about how often a tier receives an archive, one would be alarming on
  the other's model.
 - **The observation** comes from reports the hub already holds: `pbs_snapshots[]` plus successful
  `backups[]` attributed by **target type** (slice-A.4 — a PBS-targeted vzdump appears in *both*
  arrays, and classifying by array membership would attribute an offsite archive to the host tier).
  Mean gap over distinct archive times; fewer than two generations ⇒ unobservable.
 - **Floor 7 d** — the old constant kept as a floor, so no tier is judged more tightly than before.
 - **Cap 12 d** — strictly inside the 2-week offsite retention with two days to spare, so a tier is
  never called stale against an archive PBS has already pruned.
-## 7. Scenario B — publish and rollback, both directions
+| tier | interval used | window | proof age in health | verdict |
 |---|---|---|---|---|
 | daily host | 26 h declared | 4×26 h → **floor 168 h** | ≤ ~54 h | OK — numerically unchanged from before |
 | weekly offsite | 8 d declared (7 d observed loses to it) | 4×8 d → **cap 288 h** | ≤ ~198 h | OK, 3.75 d of margin |
 | newborn offsite | 8 d declared | **288 h** | — | UNKNOWN until the anchor passes |
-| act | result |
+## 4. v0.91.1 — a flaw v0.91.0 shipped with, found by checking the live box
 demo-felhom's offsite tier holds two retained snapshots, `2026-07-27T19:55:41Z` and
 `2026-07-28T04:49:43Z` — **8 h 54 m apart**, because one is a healing artefact and the other a real
 weekly run. A mean-gap estimate therefore reads a **weekly** tier as nine-hourly: ×4 = 36 h, the floor
 lifts it to 168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm
 this whole task exists to prevent would have returned within a week, on the box it had just shipped
 to.**
 Fixed by `max(observed, declared)`. A gap *shorter* than the declared rhythm is routine and means
 nothing (a retry, a manual run, a heal, a catch-up after an outage); a gap *longer* than it is real
 information. **Cost, stated rather than hidden:** a tier that truly runs faster than its declared
 rhythm gets a wider window than it needs, i.e. a slower `restore_test_stale`. Right direction for a
 signal meaning *unverified* — *broken now* is `restore_test_failed`, immediate and untouched.
 ## 5. Kept, because it was earned
 - **Absence is UNKNOWN** until an anchored window has passed (R-81's structure, unchanged).
 - **The stale signal stays edge-triggered** (`staleStates` untouched).
 - **`restore_test_failed` and `restore_test_stale` stay DISTINCT** — one says your recovery is broken,
  the other that it is unverified, and the second is the one that quietly becomes the first.
 - **Every reason string now states the window it was judged against** — R-100's corollary: when a
  verdict changes what it counts from, the alarm text must change with it, or an operator reads
  "limit 168h" under a tier judged at 288 h and dismisses a true alarm.
 - **The window READ is unchanged in cost** (14 days) — enough to find proof inside the widest window
  and to see two generations of a weekly tier.
 ## 6. Tests and red-proofs
 | Test | Asserts | Mutation | Observed |
 |---|---|---|---|
 | `TestRestoreTest_HealthyWeeklyTierIsNeverStale` | a healthy weekly tier **with jitter** is never stale across 6 weeks | window pinned flat at `restoreProvenWindowFloor` | **FAIL** — `week 0: … proof age 172h0m0s, window 168h0m0s … "limit 168h0m0s"` |
 | `TestRestoreProvenWindow_Contract` | floor, cap, declared fallback, **observation may only widen** | `observed > interval` → `observed > 0` | **FAIL** — `window(pbs, observed=8h54m0s ok=true) = 168h0m0s, want 288h0m0s` |
 | `TestObservedArchiveIntervals_FromReports` | rhythm observed per tier; a PBS vzdump is **not** counted into the host tier | — | pass |
 | `TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms` | a window that never fires is a deletion, not a fix | — | pass |
 | existing R-85/R-81 suite | anchored UNKNOWN, edge-trigger, distinct events, newborn silence | — | pass, unchanged |
 **A hollow test caught by its own red-proof.** The first Scenario-G fixture used a perfectly regular
 weekly tier and **PASSED under the flat-window mutation**, because a regular tier's proof age lands on
 exactly 168 h and `>` lets it through. It would have shipped Part 1 and its false alarm together while
 looking like proof. The jitter is what makes it a test — and it is also the truth about the old
 constant.
 ## 7. Deployment
 | Step | Evidence |
 |---|---|
-| tag moved `bee6848 → 6a82719` | scripts tree moved in **~40 s**; served `sha256 ea2b4aa9…`; **marker present** |
+| Images built + pushed | `felhom-hub:0.91.0`, then `0.91.1` |
-| tag moved back `→ bee6848` | scripts tree back in **~40 s**; served `sha256 2f859555…` — **exactly** the pre-publish sha; **marker gone** |
+| Manifest bumped in git | `687fedd`, `ff2655c` — never `kubectl set image` |
 | ArgoCD deliberate sync | `sync=Synced health=Healthy` |
 | Running image | `gitea.dooplex.hu/admin/felhom-hub:0.91.1` |
 | Startup log | checkers initialized; `Listening on :8080` |
-`https://felhom.eu/` returned 200 throughout. The marker commit was then reverted, and the tag moved
+## 8. Part 3 — the ep0 ruling, recorded
 to `main`'s head — a **byte no-op**, verified by the served sha not changing.
-## 8. Files, commits, tags
+`runbooks/target-selection.md` had carried *"D-d did not name ep0 either way. Confirm it explicitly"*
 for two days. **Operator ruling, 2026-08-03: ep0 is protected.** Recorded three ways — the tier table
 names it, the standing question is gone, and its per-machine section is retitled **Tier 2, PROTECTED**.
 `CONTEXT.md` S-18 records that this **extends D-d's protected list to three machines** (DooPlex,
 Peti's cluster, ep0) and that it is a *classification*, not new prohibitions: destroying datastores,
 prune jobs, tunnel config or nftables rules was already forbidden by what it would destroy, and **the
 ordinary off-site read a restore-test performs remains permitted** — which this session then exercised.
-**`felhom.eu`** — `bee6848` (installer + gate + manifest), `6a82719` (Scenario A marker), `e79a20b`
+## 9. Registers and docs
 (marker removed), plus the docs commit below.
 `scripts/felhom-host-install.sh` · `scripts/hostinstall_gates.py` · `scripts/CHANGELOG.md` ·
 `manifests/webpage.yaml` · `CLAUDE.md` · `CONTEXT.md` · `STATUS.md` · `REPORT.md` ·
 `documentation/backlog/{OPEN-ITEMS,ROADMAP}.md` · `documentation/architecture/00-capability-map.md`
-**`felhom-agent`** — `dd2d1fe` (release path + gate + CI), `0db7766` (REPORT).
+- **R-86 → CLOSED**, shipped + proven live, with the trap in its own wording recorded on the row.
-`scripts/release-agent.sh` **(new)** · `scripts/check-published-versions.py` **(new)** ·
+- **R-87 → re-ranked UP**: R-86 built most of what it waited for. What remains is restic-specific.
-`scripts/agent_gates.py` · `.gitea/workflows/gates.yml` · `CLAUDE.md` · `CHANGELOG.md` · `REPORT.md`
+- **R-185 / R-186 / R-187 → filed** (grep established 185–187 free; R-184 was the highest in use).
-
+- `ROADMAP.md` R-86 collapsed, keeping the reasoning and **correcting the shape the row itself
-**Tags created:** `felhom.eu/installer-v1.23.0` (the first tag this repo has ever had) and
+  proposed** — which was the never-fires version.
-`felhom-agent/v0.120.0` (retroactive, at `cd6e267` — the commit the published binary was built from;
+- `STATUS.md` rewritten for the operator, trimmed back to one screen.
-`configs/` is byte-identical there and at `main`, so nothing depended on the choice).
+- `07-backup-architecture.md` §3 carries the new contract (S-1); `00-capability-map.md`'s restore-test
-
+  row upgraded with its live citation; `CONTEXT.md` gains S-17 (the rule + the trap + the config key)
-## 9. Tests and red-proofs
+  and S-18 (ep0).
 | Check | Result |
 |---|---|
 | `felhom.eu` `repo_gates.py --fast` | all 5 gates OK |
 | `felhom-agent` `go build ./... && go vet ./...` | OK |
 | `felhom-agent` `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) |
 | `agent_gates.py --fast` | `published` correctly **SKIPPED** (hook must not fail on a network blip) |
 | `agent_gates.py` (full) | both OK |
 **Red-proofs, each demonstrated failing then restored:**
 | # | Mutation | Result |
 |---|---|---|
 | C | one of the sixteen fetches reverted to `raw/branch/main` | **RED** — gate 6a *and* 6b both fired |
 | D | assertions 6a **and** 6b removed (every guard the test covers), same bad installer | **zero** mentions of the regression — the guards are what catch it |
 | 6c | the manifest before the split | **RED** on its own, before I fixed it — the gate was demonstrated red by the real pre-change state |
 | F | `v9.9.9` tagged and not published | **RED**, `binary NOT downloadable (HTTP 404 …)`, rc=1 |
 | F′ | the gate **deregistered** from `agent_gates.py`, same bad state | **rc=0, "all agent gates OK"** — restored → `CONVICTED: published`, rc=1 |
 **Scenario F measured on real CI, not inferred.** Runs **69** and **70** are on the *same commit*
 `0db7766`: **success** before `v9.9.9` existed, **failure** after pushing it. One variable. This also
 retrospectively explains runs 67/68. **One deliberate CI failure email reached the operator — that was
 this proof, not an incident.** I could not read CI's own step log: the jobs endpoint needs a Gitea API
 token, and the only credential available (`~/.docker/config.json`) is a registry password that the API
 rejects — so the controlled before/after replaced the log rather than an assumption standing in for it.
 ## 10. No version bumps, nothing built
 `felhom-agent` **v0.120.0** unchanged (no Go code changed). Hub **v0.89.0** unchanged (no hub file
 touched). The installer's `SCRIPT_VERSION` **did** go 1.22.0 → 1.23.0 — the installer is not in §12's
 no-bump list, its behaviour changed materially, and the tag derives from it. No image, binary or
 golden was built.
 ## 11. Register
 | ID | Outcome |
 |---|---|
 | **R-110** | **CLOSED — SHIPPED** (installer v1.23.0), both-channels condition honoured, though not in the shape the ruling assumed |
 | **R-115** | **CLOSED — SHIPPED** (`release-agent.sh` + `check-published-versions.py`, no bump) |
 | **R-182** | **RE-SCOPED — the direction reversed** by Part 0's measurement; still open, now correctly described |
 | **R-183** | **NEW, and CLOSED the same session** — binary and configs came from two different refs |
 | **R-184** | **NEW, open** — nothing stops the hub vouching a version that was never released |
 **IDs established free:** `^| \*\*R-183\*\*` / `^| \*\*R-184\*\*` in `OPEN-ITEMS.md` → **0 rows** each;
 all other hits are this session's own code and changelogs (forward references I wrote). `R-185` → 0
 hits anywhere and remains free.
 ## 12. Observations — noticed, documented, NOT acted on
 1. **The gate cannot see what is vouched** — filed as R-184 rather than papered over. Closing it needs
   either a hub credential in CI (operator's call) or a check at vouch time in the hub (better: fails
   closed where the mistake is made, needs no new credential).
 2. **A suppressed operator alert leaves no row at all.** The cooldown returns before `LogNotification`,
   so the hub's own records cannot distinguish "never happened" from "held back". Recorded inside
   R-182 because it is what made that row take a day to get the right way round.
 3. **`on: [push]` fires CI for tag pushes too.** Useful (it is how Scenario F was measured), but it
   means a tag push runs the full gate set — worth knowing before anyone adds an expensive gate.
 4. **`felhom.eu` CI still runs `--fast`.** Correct today, since all its gates are network-free; if a
   network gate is ever added there, that workflow needs the same change the agent's just got.
 ## 13. Teardown
 Probe container, probe sync tree and probe tag (`probe-r110-delete-me`) removed; the red-proof tag
 `v9.9.9` deleted (`git ls-remote --tags` → only `v0.120.0`); the Scenario A marker reverted from
 `main` and the installer confirmed byte-identical to the published tag; the throwaway in-cluster curl
 pod removed; the hub DB copy is scratch-only and holds no secret material in any committed file.
@@ -104,7 +104,7 @@
 | `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
 | `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). |
 | `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
-| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. |
+| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. **R-86 (2026-08-03): the window is PER TIER, not one constant.** `restoreProvenWindow(tier, observed, ok)` = `clamp(4 × max(observed, declared), floor 7d, cap 12d)`, where `declared` is that tier's own backup-freshness threshold (`backupStaleAfter` 26 h / `offsiteBackupStaleAfter` 8 d — reuse those, never a second opinion) and `observed` comes from `observedArchiveIntervals` over the retained window. **Observation may only WIDEN**: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. `assessRestoreProven` takes the window as an argument and **every reason string names it** (R-100's corollary). |
 | `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
 | `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
 | `cfg.DRTier` + offsite coupling | hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) | bool | Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) | One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field `dr_tier` (formBool helper). |
@@ -14,114 +14,70 @@ A blank machine boots the Felhom disc, installs itself unattended, and is claime
 who sets their own password. They install apps from a catalogue of fifty-three, share files over the
 home network, and open apps from a launcher or a shared link. Backups run on their own to three
 places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
-restore files and app data from the drive alone. Proven end to end on real hardware.
+restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
-
+box six times, everything returned every time, and an app switched off deliberately stayed off.
-**Apps come back after a power cut.** The machine tells an app the customer switched off from one
+Proven end to end on real hardware.
 that simply did not come back, and waits for the system to finish starting before deciding instead of
 glancing once, five seconds in. Hard-reset the demo box six times in a row: everything came back every
 time, and an app switched off deliberately stayed off every time.
 ## What's broken
-**The off-site copy can be erased by the machine that made it** — the credential that writes it can
+- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
-also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
+  also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
-*(R-95, R-87)*
+  *(R-95, R-87)*
-
+- **Nothing else open that affects a customer.** The three faults that were on this list yesterday —
-**A full disk tells you about ONE app and silently swallows the rest.** Yesterday this was written
+  the reserve watching the wrong step, the last app whose data was never saved, and the alert that
-down the wrong way round — as *too many* emails. Measuring the receiving end reversed it: of nine
+  told you about one app and swallowed the rest — are all fixed and proven on a real machine.
 refusals the machine reported today, **two emails were sent**. When two apps are refused in the same
 second you are told about one of them, and the other leaves no trace anywhere — not an email, not
 even a line in the log saying it was held back. So a second app can be going unbacked-up while you
 have already been told the problem is handled. It is the same fault we fixed once before for
 whole-machine backups, in a second place that never opted into the fix. *(R-182)*
 ## What shipped recently
-**Pushing the installer no longer publishes it.** The script that runs as root on a brand-new
+- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
-machine was copied from the main branch and served within thirty seconds, so pushing it *was*
+  data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
-publishing it, with no staging and no way back but another push. It now comes from a **labelled**
+  restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
-version: publishing is moving the label, and undoing it is moving the label back — about half a
+  day, because the machine correctly refuses to re-prove an archive it has already proven. It is now
-minute, no deploy. The website is untouched by this and still updates in thirty seconds, because a
+  written to disk with the result and survives. This was caught happening, not predicted: a real
-typo fix must never need a release. Proven by actually doing it: a real push changed nothing that
+  14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a
-anyone downloads, moving the label published it, moving it back restored the previous bytes exactly.
+  fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now
  published after the binary, and a new check catches the opposite mistake so nothing is traded away.
  **(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
  approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
 - **Each backup is now proved, instead of the clock being obeyed.** The machine used to re-test a
  backup every twenty-four hours on a timer that restarted whenever the software was updated, so the
  test happened at an arbitrary time and a fresh backup could sit unproven while an old one was
  re-checked. Now a copy is tested once, about a day after it is made, and not again until there is a
  newer one — so the daily copy is proved daily and the weekly off-site copy weekly, each on its own.
  The alert that says "this copy has not been proved lately" learned each copy's own rhythm in the
  same change; without that it would have started complaining every night about a system that is
  working. *(R-86)*
 - **A failed backup now tells you about every app, once**, with every failure written down whether or
  not it is emailed, and a message the machine decides not to send now leaves a record saying so.
  *(R-182)*
 **The catch that would have made it cosmetic was found and covered.** While it runs, the installer
 fetches sixteen more files — not nine, and from the *agent's* repository, not the website's. They now
 come from the same version of the agent the machine is installing. That closed a real fault nobody
 had noticed: a new machine was getting the agent's tested program and its untested settings files, in
 one install, from two different places. *(R-110, R-183)*
 **Releasing the agent now publishes it, in one command.** Putting a built agent where a new machine
 can download it was a step someone had to remember, and it was forgotten three times in five days —
 the last time leaving both demo machines running a version nobody could download, so a rebuild would
 have quietly installed the *older* one and reported success. There is now one command that builds,
 labels, publishes and then **downloads it back to check** — and a check that refuses to stay quiet if
 a released version cannot actually be fetched. Proven by making CI fail on purpose and then go green
 again on the same code. *(R-115)*
 **The backup partition is gone and both demo machines run on the new shape** — wiped, rebuilt and
 taken through the whole customer journey on 3 August, by two deliberately different routes so the disk
 shape and the delivery route are both proven. The space a backup can use went from 19 GB to 65 GB on
 the small machine and 45 GB to 233 GB on the big one. Their previous demo apps and data are gone; that
 was the point of a wipe, and you approved it. *(R-165, R-178)*
 **What replaced the wall now watches the right moment.** The wall was quietly keeping a runaway
 backup from eating the space the machine needs to run. As first built, that replacement was checked
 too late — the big write happened first, unchecked — while still promising your last good copy was
 untouched. Fixed and proven on 3 August: the machine decides once, per app, **before it writes
 anything**, and that one answer covers all three steps, so a refused app writes nothing, is not
 restarted, and the promise is now literally true. It also stopped being blind to size. Nothing is ever
 deleted to make room. *(R-181)*
 **The last of the three apps that never saved their data is fixed.** Installed nowhere, so nothing was
 stranded — checked on both demo machines and in the fleet list rather than assumed. Proven by the check
 that caught it, run in both directions: it clears the fixed version and still convicts the old one.
 *(R-156)*
 **A filling disk warns the customer before anything breaks, and a failed backup reaches you** — the
 customer while there is still room to act, naming the drive and the space left; you when one app's
 backup fails, with the disk figures. The customer is deliberately not told about the second: they can
 free space, but they can do nothing about a failed backup. Both proven by filling a real disk. There
 are two rules and not one because the serious warning fired on free space while the disk was only 91%
 full — a percentage alone would have missed it. *(R-167, R-158)*
 **The checks have two nets and the second emails you.** Every repository has one command that runs all
 its checks, before every push. That one can be skipped, so the build server runs them again and emails
 you on failure. It cannot *stop* a change — everything goes straight to the main copy with no review
 step — but it notices quickly and tells you. *(R-29, R-161, R-168, R-169)*
 ## What we're working on
- **Now:** nothing outstanding from today — the reserve, the last unsaved app, and both of your
+- **Now:** nothing outstanding.
-  decisions are all built and proven.
+- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
- **Next:** the alert that tells you about one app and swallows the second *(R-182)*.
+  tests unattended. Most of the machinery it needed arrived with the restore-test change below.
- **After:** the off-site copy that the machine making it can still erase *(R-95, R-87)*.
+  *(R-87)*
 - **After:** the off-site copy that the machine making it can still erase. *(R-95)*
 ## Waiting on you
 - **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
  session log; nothing suggests anyone else saw it. *(R-132)*
- **Nothing else.** Both decisions you took on 3 August are now built and proven. One small question
+- **One small question, not urgent.** The automatic check cannot see which version you have told
-  will come back later: the automatic check cannot see which version you have told machines to
+  machines to install, only which ones exist. Closing that needs either a password given to the build
-  install, only which ones exist — closing that either needs a password given to the build server or
+  server or a check inside the hub itself. *(R-184)*
-  a check inside the hub itself. Filed, not urgent. *(R-184)*
+- **Nothing else.** The question about whether the off-site endpoint counts as protected is
  **settled — it does**, and the machine list now says so instead of asking.
 ## Changed since last update
- **2026-08-03** — Publishing became something you do rather than something that happens: the
+- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
-  installer and the agent both moved onto labelled versions with a way back, and a check now refuses
+  that could vanish on a restart (seen happening), a release that emailed a failure for a release
-  a release nobody can download. Earlier the same day: the reserve now guards the step that fills the
+  that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
-  disk and its promise is true, and the last app whose data was never saved is fixed. All proven on
+- **2026-08-03** — Backups are now proved one at a time, each about a day after it is made, instead of
-  real machines, not just in tests.
+  on a timer; the "not proved lately" alert learned each copy's own rhythm so it does not complain
-  Earlier the same day: both demo machines wiped and rebuilt from the new base image and taken through
+  about a healthy weekly copy. You settled that the off-site endpoint is protected, and the machine
-  set-up → install an app → back it up → restore it, with the backup space ceiling gone and measured.
+  list records it. One thing found while testing: on the small demo machine the agent cannot see its
-
+  own local backups at all — a permission that was never granted — so that copy has never been
- **2026-08-02** — The false "host offline" warning is fixed. The hub's database was supposed to be in
+  test-restored there. Written down, not yet fixed. *(R-86, R-185)*
  a mode where reading a page cannot block a machine's status update; a one-word difference meant that
  setting had **never taken effect**, for the hub's whole life. Fixed and verified live. **Also found:
  the hub's own database is in no automatic backup** — it holds every machine's emergency password.
  Filed, not yet fixed.
 - **2026-08-02** — Thirteen mechanical checks had built up and nothing ran most of them; two were
  failing quietly. Fixed. Decided the same day: the 20 GB backup partition goes away; and only this
  machine and the tester's box are protected, every other box may be broken or reinstalled freely.
@@ -39,7 +39,7 @@
 | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test |
 | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant  **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). |
 | **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** |
-| **Restore-proof is UNATTENDED — the scheduler covers EVERY tier and a failure is heard** | agent v0.104.0, hub v0.77.0 | **IMPLEMENTED** (not PROVEN-LIVE) | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why NOT PROVEN-LIVE:** rotation has not been observed selecting both tiers across consecutive UNATTENDED cadences. At a 24h cadence that is a multi-day observation window, and it had not elapsed when this shipped. A single passing run proves the code path, not the rotation. → **R-85** |
+| **Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard** | agent v0.104.0 → **v0.121.0**, hub v0.77.0 → **v0.91.0** | **PROVEN-LIVE (2026-08-03)** | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why this was NOT PROVEN-LIVE until now:** rotation had not been observed selecting both tiers across consecutive UNATTENDED cadences — at a 24h cadence a multi-day window — and a single passing run proves the code path, not the schedule. → **R-85**. **R-86 (agent v0.121.0 + hub v0.91.0, 2026-08-03) replaced the schedule and the observation became possible in one afternoon**, because what has to be observed is no longer a multi-day rotation but a RULE: a tier is due when its newest archive that has settled ~24 h has not been proven. **UPGRADE EVIDENCE — a real unattended run on demo-felhom (2026-08-03), triggered by DUE-NESS, not by a timer.** The scheduler's own log: `15:14:38 restore-test tier is DUE … target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z … reason="newest settled archive … has not been proven"` → `proxmox-backup-client restore --crypt-mode=encrypt` under the agent's own token → `15:25:08 gate decision class=guest_destroy guest=990000 allowed=true` → `15:25:14 scratch guest torn down` → **`15:25:14 backup: scheduled restore-test passed archive=felhom-pbs:… duration_s=635.1`**. A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored, booted, verified and destroyed in 635 s, unattended. **The three things a timer could not show**, all verified after it: the state names THAT archive (`{"felhom-pbs":{"archive":"…2026-07-28T04:49:43Z","proven_at":"2026-08-03T13:25:14Z"}}`); a second evaluation reports `due=false … is already proven` and runs nothing; and **an agent restart runs nothing**, which is the defect a person actually noticed — every deploy used to restart the timer. Teardown verified at all three layers: guest absent from `pct list`, **zero** `990000` volumes in `lvs`, and the hub-side `restore_tests[]` entry deliberately RETAINED (it IS the proof the staleness check reads). **R-189 (agent v0.122.0, 2026-08-03) closed the reporting half of this row, and it was a REAL gap in the evidence path:** the proof above reached the hub only because no restart intervened — `restore_tests[]` came solely from an in-memory store, so the 15:25:14 PASS was in fact LOST when the agent restarted 2 m 43 s later for a deploy (`0 restore-tests` on the next two host-reports). Under per-archive due-ness the box would not have repeated the work for a week. The persisted per-tier proof (with the archive, and now the tier) is merged into the report, so a proof survives a restart — one entry per tier, newest wins, and a record that cannot be described honestly is not emitted. **Known gap on this box, filed as R-185 and NOT caused by this change:** the agent's PVE token has no ACL on `/storage/felhom-backup`, so the HOST tier's content listing returns empty and that tier is not restore-testable there — the offsite tier is what this row's live proof covers. Evidence: `felhom-agent/REPORT.md` |
 | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below |
 | **Customer DELETE cascade** (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **UNIT-PROVEN; live leg PENDING** | `hub/internal/web/customer_delete_test.go` — leg ORDER observed from inside leg 2 (hosts already gone, customer row still present, custody still retained); 9 fail-closed gate cases each asserting zero mutations + zero external calls + no journal row; resume-after-external-failure converges; `purgeEscrow` custody semantics; preview leaks no secret. **5 red-proofs** (ack gate, stale-preview gate, ONLINE-host gate, leg order inverted, `purgeEscrow=true`) | Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **v0.70.0 (2026-07-21):** added the **residue** leg — `GetCustomers()` is REPORT-derived, so before it a fully deleted customer stayed on the Customers list and its report stream kept the staleness/offsite checkers alerting (live: `demo-vm-felhom` deleted 07-18, still emailing `offsite_stale` on 07-21). The leg also purges the credential-bearing `appliance_registrations` + `selfbind_tokens`. **Ghost customers (config row already gone) are now deletable** — 404 means "nothing here", not "no config row"; the Hetzner/descriptor legs record `skipped_no_config`. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run |
 | Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 |
@@ -86,7 +86,7 @@
 | An app can be **withdrawn from the catalog without orphaning the customers running it** (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** (2026-07-21) | TASK-F Part 1. Verified on 9201 through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct `POST /api/stacks/plant-it/deploy` → **HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."**; the app page carries the permanent notice and offers no Telepítés button. `felhom-controller/REPORT.md` (2026-07-21) | Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read `Elavult` and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57 |
 | Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped |
 | **The customer is warned BEFORE a filesystem fills** — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller **v0.191.0/.1/.2**, hub **v0.89.0** (R-167, decision D-c) | **PROVEN-LIVE (2026-08-02)** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md` (context) + `felhom-controller/REPORT.md`. Exercised on guest 9201 against a REAL filesystem (`/mnt/sys_drive` filled with `fallocate`): **`disk_warning` at 90% used / 4.7 GB free** → hub `notification_log` `customer | disk_warning | sent` with the dynamic Hungarian rendered; grown to 1.7 GB free → **`disk_critical`** → `customer | sent`; file removed → `critical → ok … cleared silently, re-armed` and the persisted state emptied. **Exactly two events across three boots** — the boot in between produced none, which is the edge trigger holding | **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait |
-| **A failed per-app Tier-1 backup reaches the OPERATOR** (app, error, and the target filesystem's used/free bytes at the moment of failure) | controller **v0.191.0**, hub **v0.89.0** (R-158, closed by R-167) | **PROVEN-LIVE (2026-08-02)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it |
+| **A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed** | controller **v0.194.0**, hub **v0.90.1** (R-158 → R-167 → **R-182**) | **PROVEN-LIVE (2026-08-03)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it. **ROW REWRITTEN 2026-08-03 (R-182) — the 2026-08-02 claim was TRUE OF ONE APP AND FALSE OF THE REST, and it is worth saying which.** The signal existed and worked; what it did not do was scale past the first failing app. Measured: nine per-app events reached the hub in one day and **two** operator mails went out, because the cooldown key is `customerID:eventType(+tier)` and this type carries `app` but no `tier` — so the first refused app took the hour and the rest were dropped **before `LogNotification`**, leaving no row on any channel. The old row said "One event per app, loop continuing", which was true of what the CONTROLLER emitted and not of what the operator received. **Now:** the per-app event is the RECORD (hub `recordOnlyEvents`: stored + logged every time, never mailed) and `backup_run_failures` is the NOTIFICATION — one mail per run listing every failed app, its leg and its reason, with the counts and free space. A suppressed operator event of ANY type now leaves a `suppressed` row naming its key. **Proven live on demo-hp 2026-08-03** by a real 64.6 GiB fill (thin pool held 30.78 → 30.78): `notification_log` shows `recovery_unit_capture_failed | operator | recorded` ×2, `backup_run_failures | operator | sent` ×1 naming BOTH apps, and `backup_run_failures | customer | skipped | operator_only`. A second run in the same hour produced a second digest; after freeing space a run completed with 2 volume dumps and **no** digest. The suppression row proved itself on an unplanned event — `disk_critical | suppressed | key=demo-hp:disk_critical` — a collapse that yesterday would have left nothing at all. **The digest's silence is safe only because** the hub's deadline check raises `expected_backup_missed` from report freshness independently of any mail (`monitor/deadline.go:396,417`); that check is load-bearing for this row |
 | **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.193.1** (R-165 / D-a / B2, completed by R-181) | **PROVEN-LIVE (2026-08-03) — BOTH halves** | `REPORT.md` (R-178 reinstalls) + `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **The row's FIRST clause is now PROVEN-LIVE; its SECOND is not, and they are separated deliberately.** **Proven (R-178, 2026-08-03):** *"a local backup is bounded by the box's FREE SPACE, not by a partition set at build time"* — both demo boxes reinstalled from this golden, by two different supply paths (demo-hp `--golden <local volid>`; demo-felhom the normal manifest route with **`verified sha256 54e2a4c431daf580… matches the hub manifest`**), each showing `mp0` at `/var/lib/felhom` with **no `mp1`**, both consumer paths real mounts on ONE filesystem (`stat -c %d` = `64519` on all three), 3/3 reboots each, and claim → deploy → backup → **restore** with a planted marker returning byte-identical. Space available to a recovery unit measured at **65 GiB / 233 GiB**, against the **19 GiB / 45 GiB** those boxes' `mp1` slices offered. **NOT proven — and measured FALSE in part:** *"a capture that would exhaust it is refused per app rather than allowed to stop the container runtime"*. The floor fired live for the first time (demo-hp 06:40:03) and does refuse per app, delete nothing, and alert — **but it is checked only in `captureAllRecoveryUnits`, while `runVolumeDumps` writes the bulk with no floor check at all**, so the leg that exhausts the volume is the unguarded one; and the refusal's claim that the previous unit is untouched was measured false (a 182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). → **R-181, CLOSED THE SAME DAY (controller v0.193.0 + v0.193.1) and the second half is now PROVEN-LIVE TOO.** The reserve became a **per-app, per-run ADMISSION decision** taken before the app's FIRST write and covering all three legs (DB dump, volume dump, capture) — they write under one per-app root, which is what lets one verdict cover them honestly — and it gained a **size term**, so an app is no longer admitted at 96% and then allowed to write 2 GB. **Re-proven by filling demo-hp deliberately, once for EACH term, using the method that found the defect.** *Headroom @ 08:59:46* (906 MB free / 99%): both apps refused, **the whole `backups/primary` tree byte-identical — `TREE_SHA` 111d1760c18d3440f700634ab325f8b8 before and after**, opengist's tar still at its original 182,272 B; **no `Stopping <app> for safe volume dump` line at all**, which is the positive-by-absence observable that matters because that line IS present in the 08:58 baseline run; 0 volume dumps; one alert per app, HTTP 200. Space freed, re-run @ 09:01:33 → both captured normally. *Size @ 09:03:00*, reproducing the original sequence with a real 2 GiB file in opengist's volume (previous tar **2,147,666,432 B**, the exact figure the defect was measured at) and the filesystem at **91% used / 2.9 GB free — both headroom terms deliberately clear**: opengist refused `(size)` while **privatebin was ADMITTED and dumped normally**, proving the term is per-app rather than a global halt. **The refusal's wording was NOT weakened to fit** — the behaviour moved so the wording became true, and it is verified by tree fingerprint rather than by reading the log line, which is what lied. The `fallocate` instrument was re-proven on the rebuilt box before use (5 GiB step moved guest `df` while thin-pool `data_percent` held **36.83 → 36.83**), and teardown returned the pool to **29.43%**, below its own baseline. The golden **is now VOUCHED** (2026-08-03, hub `Artifact manifest set: … golden=0.192.0`), so fresh installs pick up the merged layout. Every box in the field that has not been reinstalled is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity |
 | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | |
 | **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise |
@@ -132,6 +132,41 @@ executed** (`CAMPAIGN-8…:522`), the host-loss plan **executes nothing by const
 (`felhom-agent/internal/dr/plan.go:1-4`), and **no host has ever been rebuilt as its former self**
 (INV Part D1).
 ### Lane 2's restore-test is scheduled PER ARCHIVE GENERATION (R-86, 2026-08-03)
 **[CONTRACT, changed 2026-08-03 — agent v0.121.0 + hub v0.91.0.]** The scheduled restore-test used to
 fire on an interval started at daemon start. It no longer does. The rule is:
 > Let **A** be the newest archive on a tier that has settled for at least the settle lag (24 h).
 > The tier is **DUE** when **A** exists and **A has not already been proven**.
 So a tier is proved **once per archive**, on its own archive, and the proof follows the backup rather
 than the process's uptime:
 | tier rhythm | what is proved, and when |
 |---|---|
 | daily (host tier) | yesterday's archive, once a day |
 | weekly (offsite tier) | last week's archive, once a week |
 | newborn (no archive yet) | nothing — **UNKNOWN, never a fault** |
 **The trap in the obvious formulation, recorded so it is not reintroduced:** *"due when the newest
 archive is ≥ 24 h old"* is never true on a **daily** tier — a new archive resets the newest-archive
 age to zero long before it reaches the lag — so the literal reading silently switches restore-testing
 off for the tier that matters most.
 What survives unchanged: the restore-test itself (restore → boot → verify → destroy the scratch), its
 journal and crash recovery, the scratch VMID band, the one-heavy-operation gate, proof credit only on
 success, and oldest-proven ordering, which is now the tie-break **between due tiers**. A ticker
 remains, but only as the **evaluation interval** (6 h by default, chosen from a measured cost: one
 due-check is 18 ms on a local dir storage and 392 ms on the PBS tier over the WAN).
 **The hub's half is not optional.** `restoreProvenStaleAfter` was a flat 7 days derived from the very
 cadence this replaced, and a weekly tier proved weekly reaches a proof age of **exactly** one interval
 just before its next proof — 168 h against a 168 h window. It sat ON the line, so any ordinary delay
 tipped a healthy tier into a nightly alarm. The window is now per tier, from that tier's observed
 archive interval, floored at the old 7 days, capped at 12 days (strictly inside the two-week offsite
 retention), and falling back to the tier's declared rhythm when history is too short to observe one.
 ### Why the split is right, stated once
 **[DESIGN]** A customer can reason about "my photos are gone". A customer cannot reason about
@@ -574,6 +609,30 @@ because that was a different filesystem. On a merged box it can. Decision **B2**
 It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could
 only destroy a different app's only local copy.
 **WHAT IS RECORDED, WHAT IS E-MAILED, AND HOW OFTEN (controller v0.194.0 + hub v0.90.x, R-182).**
 The two are deliberately different mechanisms, because conflating them is how seven failures went
 missing on 2026-08-03 without leaving a trace.
 | | Record | Notification |
 |---|---|---|
 | what | `recovery_unit_capture_failed`, one per failed app | `backup_run_failures`, one per RUN |
 | when | every time, unconditionally | at the end of a run, **only if something failed** |
 | gated by | nothing — not cooldowns, preferences or delivery | the hub's operator cooldown |
 | where it lands | the events table **and** `notification_log` (status `recorded`) | the operator's inbox |
 - **A clean run e-mails nothing.** Silence means the run finished and found nothing wrong — and that
  is only safe because the hub's daily deadline check raises `expected_backup_missed` from the box's
  REPORT freshness, independent of any mail the box sends. That check is load-bearing for this
  design; weakening it re-opens a silent-failure path.
 - **A suppressed operator notification leaves a `suppressed` row** naming the key that suppressed it.
  Deciding not to tell someone is itself an event worth recording.
 - **Deliberate skips are not failures** and never appear in the digest — a disconnected or
  decommissioned drive has its own alert, and a nightly digest about an unplugged drive is one the
  operator stops reading.
 - **Cadence:** a nightly run gives at most one mail a day. A manual run always reports, even within
  the hour, because someone pressing the button is actively trying to get a backup. The periodic
  capture sweep is capped by the ordinary hourly cooldown.
 **THE CONTRACT, stated as what the code provides (controller v0.193.0, R-181).** The reserve is a
 **per-app, per-run ADMISSION decision, not a capture check.** It is taken once for an app, immediately
 before that app's FIRST write of the run, and it covers **all three write legs — the database dump, the
@@ -796,7 +855,7 @@ does **not** hold as written. → **R-108**
 | ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` |
 | **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) |
 | R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) |
-| R-86 (open) | Restore-tests are interval-scheduled, not backup-aligned | a tier's proof cadence is unrelated to when its archives are written |
+| ~~R-86~~ | ~~Restore-tests are interval-scheduled, not backup-aligned~~ | **CLOSED 2026-08-03 — agent v0.121.0 + hub v0.91.0.** Restore-testing is now **per archive generation**: a tier is due when its newest archive that has settled ~24 h has not been proven, so a daily tier is proved daily on its own archive and a weekly tier weekly on its own. The ticker survives only as the evaluation interval (6 h, chosen from a measured cost). The hub's staleness window moved with it — per tier, from that tier's observed archive rhythm — because a weekly tier proved weekly sat EXACTLY on the old flat 7-day line (§3, Lane 2's per-archive rule) |
 | R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof |
 ### 10.3 Divergences that are documented elsewhere and are not re-opened here
@@ -862,7 +921,7 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one.
 | whole-guest restore, local and PBS, exact mount parity | **PROVEN-LIVE** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C |
 | corrupted PBS snapshot fails cleanly | **PROVEN-LIVE** | CAMPAIGN-8 fault 17 |
 | the box cannot delete its own **PBS** snapshots | **PROVEN-LIVE** | CAMPAIGN-8, R-89 |
-| unattended restore-test across tiers | **IMPLEMENTED** (rotation not observed across consecutive cadences) | `00-capability-map.md:41`; LIVE per-tier timestamps this session |
+| unattended restore-test across tiers | **IMPLEMENTED**; **per-archive due-ness PROVEN-LIVE 2026-08-03** (agent v0.121.0) | `00-capability-map.md:41`; the due verdict + a real offsite run on demo-felhom (§3, Lane 2's per-archive rule) |
 | guest-power watchdog | **PROVEN-LIVE** | agent v0.107.0, 120 s |
 | quiesce crash recovery | **PROVEN-LIVE** | CAMPAIGN-8 fault 10, 1 s, by SIGKILL |
 | break-glass | **PROVEN-LIVE** | `runbooks/break-glass.md` |
@@ -10,7 +10,7 @@
 **Class:** SPIKE (empirical validation; no product code). **Repos:** felhom.eu (this doc only);
 felhom-agent read-only for grounding (`internal/pbs/{client,pin}.go`, `configs/build-golden.sh`,
 `internal/hub/cloudflared.go`, `internal/escrow/identity.go`).
-**Probe ends:** `felhom-hetzner` = Hetzner CX23, Debian 13.4, public IPv4 `167.233.158.164`,
+**Probe ends:** `felhom-hetzner` = Hetzner CX23 (**note added 2026-08-03: rescaled to a CX33, 8 GB RAM — this spike records the machine as probed and its body is deliberately unchanged**), Debian 13.4, public IPv4 `167.233.158.164`,
 global IPv6 `2a01:4f8:...::/64` (throwaway — NOT the live jarrs.eu box) ⟷ **demo-felhom** =
 the real PVE 9.2.2 host on the operator's home line (One Hungary fixed cable, Budapest), driven
 over the existing LAN SSH path; the tunnel itself always dialed **out**.
@@ -144,7 +144,7 @@
 | R-83 | **Ratify or retire `07-backup-architecture.md`** | S | **DISCHARGED (2026-07-26) — brought current, NOT ratified** | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ **STALE outside the PBS tier** verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names `felhom-pbs` → `felhom-offsite` on ep0 over `wg-felhom`, weekly, keep_last=2 — and §9.1 records the measured capacity. **Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's.** R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. |
 | R-84 | **The agent's cold backup `Store` no longer causes a redundant backup** | S | **SHIPPED (agent v0.103.0, 2026-07-26)** | Observed live: three redundant local backups on demo-felhom in one afternoon of deploys. The `Store` is in-memory ("lost on restart; the cadence re-populates"), so after every restart `/backup/due` said *no successful backup recorded yet* and the controller took another — on the OFFSITE tier a wasted multi-hour WAN upload after every agent deploy. **Resolved by asking the STORAGE rather than persisting the store** (`BackupRunner.NewestArchiveTime` + the optional `localapi.BackupArchiveLister`): ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists; no new on-disk state, no migration. It answers only *when did a backup last land* — the richer fields stay with real records, so the host-report never carries invented numbers. Fail-safes: read error → fall back to memory (never fake freshness, never suppress); genuinely empty → due; old archive → still due. **Proven live on both boxes** immediately after a restart with the store cold: demo-felhom `local age=5298s / felhom-pbs age=14373s`, demo-hp `local age=498s / felhom-pbs age=2346s`, all `due:false`. |
 | R-85 | **The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD** | M | **Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE** | R-82 proved a PBS restore round-trip **by hand**. Three defects kept the unattended path from covering it, each verified at source: **(a)** `buildRestoreTestScheduler` took `cfg.Backup.BackupTarget()` and built ONE runner on it, so the offsite tier's archives were never candidates; **(b)** the `Spec` was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; **(c)** a failed restore-test was a `[WARN]` log line with **no event, no notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. **Selection ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported **so the achieved interval can be MEASURED before anyone decides whether a knob is needed** — measure, then configure, not the reverse. **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS, never fails); persisted `RestoreTestState` (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); `backup.InFlight`, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub `restore_test_failed` / `restore_test_stale`, kept **distinct** and anchored on R-81's verdict structure, operator-tier only. **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). |
-| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — operator ruling 2026-07-27** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. |
+| R-86 | ~~**Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup**~~ | S | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03 (agent v0.121.0 + hub v0.91.0)** | **The reasoning that produced the row still holds and is worth keeping:** R-85 scheduled on a free-running INTERVAL with oldest-proven-first rotation, which cannot express *"the day after the PBS backup"* — any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. The operator's ruling was weekly per tier, ~1 day after that tier produced a new backup, so the gap between an archive being written and being proven stays small and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **What shipped is that ruling, with one correction the row's own wording would have got wrong:** the shape written here — *"test tier X once its newest archive is ~24 h old and not yet proven"* — is NEVER true on a DAILY tier, because a new archive resets the newest-archive age to zero before it reaches 24 h. The rule that works is *the newest archive that HAS settled 24 h, and has not been proven*: `A = newest archive older than the settle lag`, due iff A exists and A ≠ the proven archive. **The interim is retired:** `restore_test_cadence_seconds = 302400` (3.5 d) is no longer the mechanism; that key now only disables (negative) or seeds the settle lag. Alignment is no longer approximated by a cadence — it is the schedule. Full detail: `OPEN-ITEMS.md` R-86, `07-backup-architecture.md` §3 |
 | R-87 | **The restic (app-data offsite) tier is NEVER restore-tested** | M | idea — surfaced 2026-07-27 while closing R-85 | **R-85 covers whole-guest vzdump tiers only** (`local`, `felhom-pbs`); the agent has no restic surface at all. restic is the CONTROLLER's app-data offsite backup to the Hetzner Storage Box, a separate mechanism — so the tier that is arguably most important to a customer is the one nothing verifies. It is the only tier that survives losing the box **and** carries their actual app data: the whole-guest snapshot deliberately excludes the bind-mounted data drives (`/mnt/felhom-drives`). Restore code exists and has been exercised BY HAND (the immich destroy-and-recover drill, PROVEN-LIVE), but nothing tests it unattended — **exactly the state PBS was in before R-85: it works when someone tries it, and nobody would know if it stopped.** Needs its own design: a restic restore-test is controller-side, has no scratch-guest analogue, and would verify into a scratch dir rather than a booted guest, so R-85's machinery does not transfer. |
 | R-88 | **An UNREACHABLE backup target reads as "no backup exists" → the agent fires a doomed backup at it** | S | **Part 1 SHIPPED (controller v0.176.0, 2026-07-27); Part 2 OPEN (agent wire change)** — **Part 1** added the failure breaker: consecutive failures tracked per TARGET, backoff `15m→30m→1h→2h→4h` capped, reset on success, gating the QUIESCE rather than the backup so no stack is stopped for a doomed attempt. Never permanent (the cap bounds the retry interval — a latched breaker is a silent backup outage), never global (a broken offsite tier cannot suppress a healthy local one), never applied to `TriggerNow`, and `stillRunning` is not a failure. In-memory on purpose: forgetting a backoff costs one attempt, persisting one could outlive the fix. **Part 2 is agent-side and needs a wire change** — see the P0 finding below. Live on both boxes; the loop reproduction is UNIT-TEST ONLY (reproducing it live would need PBS taken down on ep0, which has no swap). | **The R-81 class (absence-is-not-failure), reappearing one layer down in the AGENT's due-check.** R-84 replaced the in-memory `Store` with storage ground truth (`NewestArchiveTime`), and fails safe on a read error by *falling back to the in-memory record* — "never fake freshness, never suppress". That fallback is correct in isolation but wrong in the one composition where it actually runs: **a cold store plus an unreachable target.** After a restart the in-memory record is empty, so the fallback answers *no backup has ever been recorded* → tier due → the agent fires a backup at the very storage it just failed to read. **Observed:** `felhom-agent` restarted on demo-felhom at 07:02:57 UTC; `local-api: could not read the backup storage for the due-check — falling back to the in-memory record` (vmid=9201 target=felhom-pbs); 11 s later `vzdump 9201 → felhom-pbs` failed with `could not activate storage 'felhom-pbs' … Can't connect to 10.77.0.1:8007 (Connection refused)`. The offsite PBS service was down at the time — so the trigger was real, not synthetic. **NOT bounded, and it also DISABLES the business-hours protection — an availability fault, not a logging nuisance.** The first reading ("fails in ~0 s, so the cost is one spurious event per restart") was wrong on both halves, and a second reading ("it stopped after three tries, so something limits it") was wrong too — the live run stopped after three only because the storage came back. **Verified in code: there is no limiter.** `internal/quiesce` holds no failure counter, no backoff, no breaker, no attempt budget; `runOnce` keeps no state between ticks, and the driver is a plain 5-minute `time.Ticker` (`quiesce.go:149`, default `quiesce.go:107-109`). Had the target stayed down, the loop would have continued indefinitely.
@@ -175,7 +175,7 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the
 | R-162 | **`docker diff` is the gate's only witness and its failure mode is quiet** | XS | WATCHING — 2026-08-02 | A limitation, not a defect. The gate's power is `docker diff` excluding mounted paths; on a driver where it is unsupported or lies, the gate degrades to mount-occupancy + writability **and would not say so**. It fails closed (the canary self-test stops reporting BROKEN and the gate then refuses to report), but the message blames the prober rather than the driver. Revisit only if a non-overlay driver ships |
 | R-163 | **`mp1` is RETENTION, not staging — and it is sized as if it were neither** | M | **WAITING-ON-OPERATOR** — the ratio is a tier-sizing ruling | Settled by reading: a unit is the KEPT copy on the app's **own** drive (`backup.go:245-255`), falling back to the system SSD for driveless apps — *"the SSD-only system-data fallback"* (`appbackup/paths.go:26-27`); **no post-copy deletion**, the only prune is F5 residue-on-old-drives (`backup.go:1053-1112`). So `mp1` **20 G** retains every driveless app's unit while `mp0` permits **50 G** of volumes, and a DB app's unit is up to **~2×** its data (measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to 0 (`felhom-agent/.../main.go:178`), not derived from the drive; demo-hp's real guest ships 50G/20G. **`mp1` gates the whole chain** — Tier-2 mirrors the unit *"(always)"* (`tier2.go:302,368`) and Tier-3 carries it. Bounded: a unit is volume tars + DB dumps only, **never `mp8` userdata** (`recovery_unit.go:20-25`). **No number proposed.** → `07-backup-architecture.md` §7.5 |
 | R-164 | **C2's chain — the DB volume tar cannot be dropped until a sound dump predicate exists** | S | BLOCKED — on the predicate (2026-08-02) | The unit holds a volume tar **and** a SQL dump and the restore uses both: the dump is authoritative and replayed after the tar so it WINS (F17), with only the DB service up (R-47) — `restore_unit.go:262-266`. Dropping the DB tar would halve DB-app units and close R-127(b)'s initdb-skip trap. **The obvious gate is dead, measured:** `ValidateDump`'s empty-`accounts` warning was **correct** (the DB truly had 0 rows; seeding one stopped the warning and put the row in the dump) — but **a fresh appliance legitimately has zero accounts**, so gating on it blocks every new customer's first backup. Order: sound predicate (dump vs **live** per-table counts) → warn→gate → tar-drop. Pairs with **R-127** |
-| R-90 | **ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86** | S | BLOCKED on Hetzner CX33 availability (operator, 2026-07-27) | Verified read-only: `Swap: 0B`, nothing in `/etc/fstab`, no swap unit, no `/swapfile` — **swap was never added.** On 2026-07-27 06:58:12 a global OOM fired while `proxmox-backup-proxy` held a 3.2 GB peak serving a 14.46 GB restore-test read and the datastore-migration `rsync` held 1.9 GB; the kernel killed the rsync and PBS stopped serving for ~15 minutes. That outage is what exposed **R-88** downstream. Still exposed today: 3.8 GB, no swap, one kill on record. **This gates R-86** — backup-aligned restore-testing puts a multi-GB read on this box *on a schedule*, so the headroom question must be settled before that lands. Interim lever if CX33 stays unavailable: a swapfile, which needs no console |
+| R-90 | ~~**ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86**~~ | S | **CLOSED 2026-08-03 — the operator rescaled ep0 to a CX33.** MEASURED on the box, not read from an invoice: `Mem: 7757` MB (**8 GB**, was 3.8), `nproc` **4**, and the 4 GiB swapfile added 2026-07-27 **survived the resize** and is active (checked because a resize is a stop/start, so "still there" was an assumption until measured). **The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only — so no disk figure in any runbook needed correcting. Unblocks R-86 | Runbooks corrected; SPIKE/evidence documents ANNOTATED not revised. `target-selection.md`'s "D-d did not name ep0 either way" stays open — it is the operator's |
 | R-91 | **The old 13 GB datastore copy is still on ep0's root disk** | XS | WATCHING — gated on demo-felhom's first post-migration PBS backup | The datastore moved to a Hetzner Cloud Volume on 2026-07-27 (`/dev/sdb`, 100 GiB, attached 06:29:40 UTC, now `/mnt/pbs-datastore`, 13 G used of 98 G). The pre-migration copy survives at **`/srv/pbs-felhom`, 13 G**, on `/` (38 G total, 16 G used, 21 G free). **Do not delete yet:** demo-hp has landed two post-migration snapshots (07-27 08:25:47Z, 09:37:29Z) but **demo-felhom's newest is 2026-07-26T12:21:48Z — before the migration**, so the new volume has not yet proven a write for that namespace. Delete once it has. **Doc drift to fix in the same commit:** `CONTEXT.md:1018` still records the datastore at `/srv/pbs-felhom` |
 | R-92 | **The hub's PBS-DR gauge is 0.1 GB-granular, so small deltas are unverifiable** | XS | idea — 2026-07-27 | The PBS-DR box card rounds to 0.1 GB, which is coarser than the changes an operator wants to confirm after a prune or a GC — a successful prune of a small namespace moves the number by less than one displayed digit, so the UI cannot distinguish "it worked" from "nothing happened". Cosmetic today; it becomes load-bearing the moment retention (R-89) is customer-visible and someone needs to see that a policy change took effect |
 | R-93 | **`drill-r50` is both a blocked customer and the only drift fixture** | XS | idea — 2026-07-27 | The drill customer is blocked in the hub (so it stops alarming) yet it is also the only record exercising the endpoint-drift path R-77 added. Blocking hides it from `GetActiveCustomerIDs`, so the fixture it provides is silently inert — a monitor with no live subject reads exactly like a monitor that passes. Decide: retire it and build a synthetic fixture, or unblock it and silence per-customer instead (the operator has a per-alert silencing feature planned). Related to the R-50 drill VM, now shut down |
@@ -3,7 +3,8 @@
 **Class:** supervised operational run. **No repo version bump** — the only commits are this record
 and the capacity note. **Nothing was deleted.**
-**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner CX23, Nuremberg.
+**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner **CX33 (4 vCPU / 8 GB RAM)**, Nuremberg.
 > **Rescaled 2026-08-03** from the CX23 (2 vCPU / 3.8 GB) this runbook was written against. **The 40 GB local disk did NOT change** — this was a CPU/RAM resize — so every disk figure below still stands. The 4 GiB swapfile added on 2026-07-27 survived the resize.
 **Datastore moved:** `felhom-offsite`, `/srv/pbs-felhom` → **`/mnt/pbs-datastore`** (name unchanged).
 **Window:** 06:58 → 07:19 UTC. PBS down 07:00 → 07:17 UTC.
@@ -229,7 +229,9 @@ as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as
 `keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few
 minutes of ep0 CPU per weekly backup.
-> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
+> Watch item (**superseded 2026-08-03**: ep0 is now a **CX33, 8 GB RAM**, and it HAS a 4 GiB swapfile
 > which survived the resize — so the pressure below is much reduced, though the shape of the concern
 > stands). As written: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
 > window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see
 > `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box.
@@ -307,7 +309,7 @@ Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13
   watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here.
 4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to
   it. Not a fault, but it cannot be used as write-proof evidence for small deltas.
-5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM.
+5. ~~**ep0 has no swap** (3.7 GB CX23)~~ — **corrected 2026-08-03: ep0 is a CX33 with 8 GB RAM and an active 4 GiB swapfile.** See the volume runbook's OOM for the original incident.
 ## 11. Observations
@@ -5,7 +5,8 @@
 > firewall, and the hub-driven `felhom-peersync` reconcile surface. Re-running it on a fresh VM
 > re-creates the endpoint from nothing (that is the DR story, step 8).
 >
-> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23, Debian 13,
+> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23 **at the time — rescaled
 > to a CX33, 4 vCPU / 8 GB RAM, on 2026-08-03; the 40 GB local disk is unchanged**, Debian 13,
 > `167.233.158.164` / `2a01:4f8:1c16:7aa1::1`) with hub v0.32.0. The production endpoint is a
 > later re-run of this runbook on a production VM.
 >
@@ -31,7 +32,7 @@ Parameters used throughout (adjust for a new endpoint):
      points at nothing (live-run finding). Home-resolver propagation can lag public DNS by
      minutes — a client-side `wg-quick up` that fails to resolve right after record creation
      just needs a retry.
- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway CX23), not
+- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway box, **CX33 since 2026-08-03**), not
      any production box.
 ## 1. Base (on the box, as root)
@@ -29,20 +29,23 @@ prohibition as covering the act it names and nothing more.
 |---|---|---|
 | **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
 | **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
-| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster** — and, by D-d, **nothing else** |
+| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **`ep0`** (operator ruling 2026-08-03) — and nothing else |
 **DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
 Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
 thing under test, the source of truth for it, and the backups, at once.
-**`ep0` + the Hetzner Storage Boxes were Tier 2 until 2026-08-02 and are no longer** — D-d's protected
+**`ep0` is Tier 2 — PROTECTED. Operator ruling, 2026-08-03.** D-d named two protected machines and did
-list names two machines and ep0 is not one of them. **That does not make them scratch, and the
+not name ep0 either way, so this page carried the question in writing for two days and read it the
-difference is an act, not a tier** (see the rule above the table): ep0 holds the **PBS-DR datastore and
+narrow way meanwhile (not protected, but not wipeable). The ruling settles it and **extends D-d's
-the restic copy of a real customer's data**, which is the only off-premises copy that exists, so
+protected list to three machines**: DooPlex, Peti's cluster, ep0.
-*deleting datastores, prune jobs, tunnel config or nftables rules* remains forbidden by what it would
+
-destroy rather than by what tier it sits in. Reads are fine; it is still never a drill target.
+The reason it was never really in doubt: ep0 holds the **PBS-DR datastore and the restic copy of a
-**Flagged for the operator: D-d did not name ep0 either way.** Confirm it explicitly — this page has
+real customer's data**, which is the only off-premises copy that exists. So *deleting datastores,
-read it the narrow way (not protected, but not wipeable) rather than assume the broad one.
+prune jobs, tunnel config or nftables rules* was already forbidden by what it would destroy; the
 ruling makes the classification say so plainly instead of leaving each session to re-derive it.
 **Reads are fine** — including the ordinary off-site read a restore-test performs (R-86) — and it is
 never a drill target. The Hetzner Storage Boxes ride the same reasoning.
 **Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
 felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
@@ -95,11 +98,15 @@ still shares a device with its guest, so a drive failure is **offsite-only recov
 migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
 enrolled host. No access route from DooPlex, and nothing here needs one.
-### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
+### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2, PROTECTED** (operator ruling 2026-08-03)
 Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
-path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune
+path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a
-jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy —
+4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching — the 8 GB
 is comfortable, not unbounded, and the OOM that started R-90 was a 14.46 GB restore read against
 3.8 GB. Do not delete datastores, prune jobs, tunnel config or nftables rules; never a drill target.
 **The ordinary off-site READ a restore-test performs is permitted and unchanged by the ruling**
 (R-86): the classification forbids destruction, not use. The Storage Boxes hold the restic copy —
 customer documents and photos, on a credential that can still delete (R-95).
 **Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that
 produced a false "unreachable" verdict (standing rule 2).
@@ -139,7 +139,7 @@ is not reachable, for two reasons that are each **already-recorded deliberate po
   `endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).
 So the only two configurations are: *DR tier on* → the campaign's PBS traffic lands on **ep0**, which is
-Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90) and fenced by §3 — or *DR tier off* →
+Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90 — **note added 2026-08-03: ep0 has since been rescaled to a CX33 with 8 GB RAM; this journal records what was true when it was written and is deliberately not revised**) and fenced by §3 — or *DR tier off* →
 no Tier 3 at all. **Chosen: DR tier OFF, offsite OFF**, which is the only option §3 permits.
 Consequence, stated plainly rather than discovered later: the campaign touches **neither ep0 nor the
@@ -1,3 +1,132 @@
 ## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
 **Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's
 offsite tier holds two retained snapshots — `2026-07-27T19:55:41Z` and `2026-07-28T04:49:43Z` —
 **8 h 54 m apart**, because one is a healing artefact and the other a real weekly run. The mean-gap
 estimator therefore reads a **weekly** tier as nine-hourly: ×4 gives 36 h, the 7-day floor lifts it to
 168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm this whole
 task exists to prevent would have returned within a week, on the box it had just shipped to.**
 `restoreProvenWindow` now takes `max(observed, declared)`. Observation refines a tier's rhythm
 **upward** and is ignored downward, which is right on its own terms and not merely cautious: a gap
 SHORTER than the declared rhythm is routine and means nothing — a retry, a manual run, a heal, a
 catch-up after an outage — while a gap LONGER than it is real information, saying this tier genuinely
 receives archives less often than the model assumes and its window must widen or it alarms.
 **The cost, stated rather than hidden:** a tier that truly runs faster than its declared rhythm gets a
 wider window than it strictly needs, i.e. a slower `restore_test_stale` signal. That is the right
 direction for a signal whose message is *"unverified"*. *"Broken now"* is `restore_test_failed`, which
 is immediate and untouched.
 Three live-derived cases added to `TestRestoreProvenWindow_Contract`, including the exact 8 h 54 m
 gap measured on the box; red-proved by restoring the tighten-too branch
 (`window(pbs, observed=8h54m) = 168h, want 288h`).
 ## v0.91.0 — a tier's staleness window learns the tier's own rhythm (2026-08-03, R-86 Part 2)
 **This ships WITH the agent's v0.121.0, not after it.** The agent now proves a tier once per ARCHIVE
 GENERATION rather than on a 24h timer, so a tier backed up weekly is proved weekly — correctly, and
 in perfect health. `restoreProvenStaleAfter` was a flat 7 days, and its own comment derived that
 number from the cadence R-86 removes:
 > *"the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
 > proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."*
 A weekly tier proved weekly reaches a proof age of **exactly one interval** just before its next
 proof — 168h against a 168h window, false by a hair — so it did not merely sit near that line, it sat
 ON it, and any ordinary delay (a late backup, one deferral behind a running backup) tipped it into a
 nightly alarm about a system that is working. Shipping the agent alone would have converted the
 improvement into a false alarm.
 **The window is now per tier**, `restoreProvenWindow(tier, observed, ok)`:
 - **the tier's own archive interval**, observed from the host-reports the hub already holds —
  `pbs_snapshots[]` plus successful `backups[]` attributed by TARGET TYPE (the slice-A.4 rule: a
  PBS-targeted vzdump appears in both arrays, and classifying by array membership would attribute an
  offsite archive to the host tier);
 - **× 4 generations** — the settle generation plus ~3 missed opportunities, deliberately the same
  tolerance the flat constant expressed. The change is to the RHYTHM, not to the patience;
 - **floored at 7 days** — the old constant kept as a floor, so no tier is judged more tightly than
  before;
 - **capped at 12 days** — strictly inside the 2-week offsite retention with two days to spare, so a
  tier is never called stale against an archive PBS has already pruned;
 - **falling back to the DECLARED rhythm** when a box's history is too short to observe one:
  `backupStaleAfter` (26h) for the host tier and `offsiteBackupStaleAfter` (8d) for the offsite tier
  — the very thresholds the backup-freshness checker already judges those tiers against. A fresh box
  with one snapshot has no observable interval, and falling back to the FLOOR there would recreate
  the false alarm on exactly the tier this task is about.
 **Kept, because both were earned:** absence is UNKNOWN until an anchored window has passed (R-81's
 structure, untouched), and the stale signal stays edge-triggered. `restore_test_failed` and
 `restore_test_stale` remain DISTINCT — one says your recovery is broken, the other says it is
 unverified, and the second is the one that quietly becomes the first.
 **Every reason string now states the window it was judged against** — R-100's corollary: when a
 verdict changes what it counts from, the alarm text has to change with it, or an operator reads
 "limit 168h" under a tier actually judged at 288h and dismisses a true alarm.
 **The window READ is unchanged in cost** (14 days), which is both enough to find proof inside the
 widest window and enough to see two generations of a weekly tier.
 ## v0.90.1 — the digest's per-app lines stop repeating the filesystem figures (2026-08-03, R-182)
 **Backfilled 2026-08-03 (R-86 session).** This version was built, deployed and recorded in `REPORT.md`
 and the R-182 row, but never given a CHANGELOG entry — and `REPORT.md` is overwritten every session,
 so the per-repo history under-reported what was actually running. The deployed image has been
 `felhom-hub:0.90.1` since `f21e7ca`.
 Found by reading the first REAL digest, not by design: every app row ended with the same usage clause
 the mail already prints once on its own Filesystem line. On a two-app box that is untidy; down a list
 of a dozen it is the same forty characters twelve times, pushing the part that DIFFERS off a phone
 screen at 07:00 — the only moment that mail has to work.
 The reserve's refusal message is authored for a single-app alert where naming the filesystem is
 right, so the message is unchanged and the DIGEST trims the duplicate when rendering.
 `trimRepeatedUsage` removes ONLY an exact "— <target path>:" suffix, so an unrelated reason is
 untouched and a reason that is nothing but the usage clause is left alone rather than emptied.
 Also inverted the operator half of `TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer`: it
 required the operator to be e-mailed a per-app capture failure, which was correct when that event was
 the only signal and is wrong now that it is the RECORD and the digest is the notification. The
 customer-safety claim is unchanged — R-158's guarantee MOVED, it did not weaken.
 ## v0.90.0 — a dropped notification leaves a trace, and the backup digest arrives (2026-08-03, R-182)
 **The smallest change on the board with the largest effect on trust: `processOperator`'s cooldown no
 longer returns bare.** It used to drop the event *before* `LogNotification`, so a suppressed operator
 alert and an event that never happened were indistinguishable — from the operator's side **and from
 the hub's own records**. Measured 2026-08-03: nine `recovery_unit_capture_failed` events arrived, two
 were mailed, and **seven left no row anywhere**. That is why the defect took a day to get the right
 way round: there was nothing to read.
 A suppressed operator event now writes a `suppressed` row carrying the message and **the key that
 suppressed it**, so the collision is readable without reading code. This applies to **every** operator
 event, not only the one that exposed it. It deliberately does **not** change the cooldown's duration
 or semantics — it makes the drop visible, not absent.
 **`backup_run_failures` — the per-run digest.** One operator mail at the end of a backup run listing
 every app that failed, its leg and its reason, with the counts and the target filesystem's free
 space. Added to `allowedEventTypes` **and** to `operatorOnlyEvents` — allowlisting alone does not make
 an event operator-only, and `FormatCustomerEmail` falls back to the raw English message rather than
 blocking. A test demonstrates a customer with the type in their enabled list receiving nothing.
 **`recordOnlyEvents` — a third routing class.** Types that are STORED and RECORDED but never mailed.
 `recovery_unit_capture_failed` moves here: it is the durable per-failure record, and the digest is the
 notification. Deliberately a register rather than downgrading the severity to `info`, which would have
 had the same routing effect while relabelling a genuine failure as informational in the events table,
 the operator UI and every historical query.
 **`cooldownRunSuffix` — the run discriminator.** A sibling of `cooldownTierSuffix` rather than a
 branch inside it, so `tier` keeps byte-identical semantics and R-97a's tests are untouched. It makes
 the cooldown effectively inert for the digest, **which is the intent**: a digest is already
 rate-limited by construction, one per run and only when something failed, so there is nothing for a
 timer to collapse — while the periodic refresh sweep sends **no** `run_id` and therefore stays under
 the ordinary hourly cooldown.
 **The e-mail is rendered as a list, not a JSON blob** — the one operator mail with a variable-length
 payload, and a dozen apps on one line is unreadable on a phone at 07:00. An absent space reading
 renders as *unavailable*, never as zeros.
 ## v0.89.0 — the two halves of decision D-c (2026-08-02, R-167 · R-158)
 **Decision D-c routes two new signals to two different audiences, and the hub is where that routing
@@ -0,0 +1,40 @@
 package api
 import "testing"
 // R-182 — the backup run digest needs the allowlist half, and NOT the customerMessages half.
 //
 // A new event type is a pair of register entries, and each half fails differently. For this type the
 // pair is unusual and that is the point:
 //
 //   - missing from `allowedEventTypes`            → POST /event returns 400 and the digest VANISHES,
 //     which would rebuild the exact silence R-182 exists to end;
 //   - PRESENT in `customerMessages`               → the customer would be e-mailed, in Hungarian, a
 //     list of which apps' backups failed and why — operator detail they can take no action on. So
 //     this type must be allowlisted and must NOT have a customer message.
 //
 // The customer-facing half of decision D-c is the FILL WARNING, which fires before this and is
 // actionable (free space, delete files, add a drive). This is the operator's half.
 //
 // Operator-only routing itself is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a
 // customerMessages entry — that assumption shipped in v0.78.0 and was wrong, because
 // FormatCustomerEmail falls back to the raw message. It is pinned in
 // `internal/notify/backup_run_digest_test.go`, which demonstrates a customer with the type in their
 // enabled list receiving nothing.
 func TestBackupRunDigestIsAllowlisted(t *testing.T) {
 	if !allowedEventTypes["backup_run_failures"] {
 		t.Fatal("backup_run_failures must be in allowedEventTypes, or POST /event 400s and the " +
 			"whole run digest is dropped at the door — the silence R-182 was filed against")
 	}
 }
 // The per-app event is the RECORD and must not be removed while the digest is the notification.
 // Deleting it would make the digest the only trace, and a digest that fails to send would then take
 // the record with it — the coupling R-182's fix exists to break.
 func TestPerAppCaptureEventStaysAllowlisted(t *testing.T) {
 	if !allowedEventTypes["recovery_unit_capture_failed"] {
 		t.Fatal("recovery_unit_capture_failed was removed from allowedEventTypes — it is the " +
 			"durable per-failure RECORD, and the digest is only the notification; the operator " +
 			"register and every historical query depend on it")
 	}
 }
@@ -1586,6 +1586,11 @@ var allowedEventTypes = map[string]bool{
 	// OPERATOR-ONLY IS ENFORCED BY `notify.operatorOnlyEvents` — see the paragraph above. This entry
 	// alone does NOT make it operator-only.
 	"recovery_unit_capture_failed": true,
 	// R-182. The per-run backup digest: one event at the end of a run, listing every app whose
 	// backup failed or was refused. Allowlisting it is NOT what keeps it away from customers —
 	// `notify.operatorOnlyEvents` is (see the comment there); both entries ship together and
 	// `backup_run_digest_event_test.go` pins the pair.
 	"backup_run_failures": true,
 	// Controller-pushed events
 	"controller_started":      true,
@@ -32,15 +32,104 @@ import (
 // is a NEW monitor written straight after the third, so it copies R-81's verdict structure rather
 // than re-deriving it. A tier never proven on a newborn box is UNKNOWN, never FAILED.
-// restoreProvenStaleAfter is how long a tier may go unproven before it is called stale.
+// ── HOW LONG MAY A TIER GO UNPROVEN? (R-86 Part 2) ───────────────────────────────────────────
 //
-// Derivation, not a guess: the restore-test cadence is 24h and rotation is oldest-first across two
+// This was one flat constant, 7 days, and its comment derived that number like this:
-// tiers, so each tier is proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive
+//
-// missed opportunities before alarming — loud enough to matter, quiet enough not to fire on one
+//	"the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
-// skipped cycle (a deferral behind a long backup is normal, not a fault). It is also comfortably
+//	 proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."
-// inside the 2-week offsite retention (operator ruling 2026-07-26), so a tier is never reported
+//
-// stale against an archive that is about to be pruned anyway.
+// **That premise is exactly what R-86 removed.** The agent no longer tests on an interval at all: a
-const restoreProvenStaleAfter = 7 * 24 * time.Hour
+// tier is tested once per ARCHIVE GENERATION — when it holds a settled archive that has not been
 // proven. A tier backed up weekly is therefore proved weekly, by design and in perfect health, and
 // against a flat 7-day window it would sit on the line and alarm every night about a system that is
 // working. Shipping the agent's half alone would have converted the improvement into a false alarm,
 // which is why the two ship together.
 //
 // The window is now derived from **the tier's own backup rhythm**, which the hub can observe from
 // the reports it already receives, and it keeps everything the constant had earned:
 //
 //   - absence is UNKNOWN until an anchored window has passed (R-81's structure, untouched);
 //   - the signal stays edge-triggered;
 //   - it never exceeds the offsite retention, so a tier is never called stale against an archive
 //     that is about to be pruned;
 //   - and it is never TIGHTER than the 7 days that were already tolerated.
 const (
 	// restoreProvenGenerations is how many archive generations may pass unproven before alarming.
 	// 4 = the settle lag's own generation plus ~3 missed opportunities — deliberately the same
 	// tolerance the flat constant expressed, so the change is to the RHYTHM, not to the patience.
 	restoreProvenGenerations = 4
 	// restoreProvenWindowFloor is the shortest window that may be applied to any tier. It is the
 	// old constant, kept as a FLOOR rather than deleted: a daily tier computes 4 days from its own
 	// rhythm, and tightening a live threshold is not what this task is for. A deferral behind a
 	// long backup is normal, not a fault.
 	restoreProvenWindowFloor = 7 * 24 * time.Hour
 	// restoreProvenWindowCap keeps the window strictly inside the 2-week offsite retention
 	// (operator ruling 2026-07-26) with two days to spare. Beyond it the hub would be judging a
 	// tier against an archive PBS has already pruned — an alarm nobody can act on, and the bound
 	// the old constant respected in its own way.
 	restoreProvenWindowCap = 12 * 24 * time.Hour
 	// restoreWindowRead is how far back the hub reads host-reports for this check: far enough to
 	// find proof anywhere inside the widest window, and to see at least two archive generations of
 	// a WEEKLY tier so its rhythm is observable at all.
 	restoreWindowRead = 2 * restoreProvenWindowFloor
 )
 // declaredArchiveInterval is the rhythm the hub ALREADY attributes to a tier — the same thresholds
 // the backup-freshness checker judges it against (deadline.go / deadline_tiers.go). It is the
 // fallback when a box's history is too short to observe a rhythm, and it is the right fallback
 // precisely because it is not a second opinion: if these two checkers disagreed about how often a
 // tier is expected to receive an archive, one of them would be alarming on the other's model.
 //
 // It is stated per RESTORE tier name ("local"/"pbs" — what the agent reports as source_tier), which
 // is the same split the backup tiers use under different names ("host"/"offsite").
 func declaredArchiveInterval(tier string) time.Duration {
 	if tier == "pbs" {
 		return offsiteBackupStaleAfter // 8 days: the weekly cadence plus a day of headroom
 	}
 	return backupStaleAfter // 26 hours: the daily cadence plus headroom
 }
 // restoreProvenWindow is how long THIS tier may go unproven, given its observed archive interval.
 //
 // observedOK=false means the box's retained history did not contain two archive generations for
 // this tier, so the declared rhythm is used. That fallback matters most for exactly the tier this
 // task is about: a fresh box with a weekly offsite tier has one snapshot and no observable
 // interval, and falling back to the FLOOR there would recreate the false alarm.
 //
 // OBSERVATION MAY ONLY WIDEN, NEVER TIGHTEN — and this is not caution, it is a live measurement.
 // On demo-felhom (2026-08-03) the offsite tier's two retained snapshots are `2026-07-27T19:55:41Z`
 // and `2026-07-28T04:49:43Z`: **8 h 54 m apart**, because one is a healing artefact and the other a
 // real weekly run. A mean-gap estimate therefore reads a WEEKLY tier as nine-hourly, ×4 gives 36 h,
 // the floor lifts it to 7 days — and a weekly tier proved weekly reaches ~8.25 days of proof age, so
 // the false alarm this whole task exists to prevent would have returned within a week, on the very
 // box it shipped to.
 //
 // The asymmetry is right on its own terms too. A gap SHORTER than the declared rhythm is routine and
 // means nothing — a retry, a manual run, a heal, a catch-up after an outage. A gap LONGER than the
 // declared rhythm is real information: this tier genuinely receives archives less often than the
 // model says, and its window must widen or it alarms. So observation refines the rhythm upward and
 // is ignored downward. The cost is stated plainly: a tier that truly runs FASTER than its declared
 // rhythm gets a wider window than it strictly needs, i.e. a slower stale signal. That is the right
 // direction for a signal whose message is "unverified" — "broken NOW" is `restore_test_failed`, and
 // that one is immediate and unaffected.
 func restoreProvenWindow(tier string, observed time.Duration, observedOK bool) time.Duration {
 	interval := declaredArchiveInterval(tier)
 	if observedOK && observed > interval {
 		interval = observed
 	}
 	w := time.Duration(restoreProvenGenerations) * interval
 	if w < restoreProvenWindowFloor {
 		w = restoreProvenWindowFloor
 	}
 	if w > restoreProvenWindowCap {
 		w = restoreProvenWindowCap
 	}
 	return w
 }
 // Event types. Operator-tier only — see the dispatcher note in RestoreTestChecker.
 const (
@@ -156,12 +245,13 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
 		return
 	}
-	rows, err := c.store.GetHostReportsSince(customerID, now.Add(-2*restoreProvenStaleAfter))
+	rows, err := c.store.GetHostReportsSince(customerID, now.Add(-restoreWindowRead))
 	if err != nil {
 		c.logger.Printf("[WARN] restore-test check: window read failed for %s: %v", customerID, err)
 		return
 	}
 	proven := lastProvenPerTier(rows)
 	intervals := observedArchiveIntervals(rows)
 	first, ferr := c.store.GetFirstHostReportAt(customerID)
 	if ferr != nil {
@@ -170,7 +260,9 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
 	}
 	for _, tier := range tiers {
-		v := assessRestoreProven(tier, proven[tier], first, now)
+		observed, observedOK := intervals[tier]
 		window := restoreProvenWindow(tier, observed, observedOK)
 		v := assessRestoreProven(tier, proven[tier], first, now, window)
 		key := customerID + "|" + tier
 		c.mu.Lock()
 		prev := c.staleStates[key]
@@ -192,37 +284,120 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
 	}
 }
-// assessRestoreProven is the per-tier verdict. PURE (now injected) so the policy is unit-tested —
+// assessRestoreProven is the per-tier verdict. PURE (now and the window injected) so the policy is
-// the property that made R-81 provable, kept deliberately.
+// unit-tested — the property that made R-81 provable, kept deliberately.
 //
 //	no proof, anchor NOT elapsed → UNKNOWN (newborn box; never an alarm)
 //	no proof, anchor elapsed     → MISSED
 //	proof older than the window  → MISSED
 //	otherwise                    → OK
-func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time) backupAssessment {
+//
 // `window` is now the TIER'S OWN (R-86 Part 2) rather than one constant for every tier, and every
 // reason string states the window it was judged against. That is R-100's corollary applied here:
 // when a verdict changes what it counts from, the alarm text has to change with it, or an operator
 // reads "limit 168h" under a tier that was actually judged at 288h and dismisses a true alarm.
 func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time, window time.Duration) backupAssessment {
 	if provenAt.IsZero() {
 		if firstReportAt.IsZero() {
 			return backupAssessment{verdict: verdictMissed,
 				reason: fmt.Sprintf("%s tier: never restore-proven, and no first-contact anchor to defer against", tier)}
 		}
 		watched := now.Sub(firstReportAt)
-		if watched <= restoreProvenStaleAfter {
+		if watched <= window {
 			return backupAssessment{verdict: verdictUnknown,
 				reason: fmt.Sprintf("%s tier: not restore-proven yet, but only watching for %s (grace %s since first contact %s) — newborn, not a fault",
-					tier, watched.Round(time.Hour), restoreProvenStaleAfter, firstReportAt.Format(time.RFC3339))}
+					tier, watched.Round(time.Hour), window, firstReportAt.Format(time.RFC3339))}
 		}
 		return backupAssessment{verdict: verdictMissed,
-			reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s) — the tier is unverified, not known-broken",
+			reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
-				tier, watched.Round(time.Hour), restoreProvenStaleAfter)}
+				tier, watched.Round(time.Hour), window)}
 	}
-	if age := now.Sub(provenAt); age > restoreProvenStaleAfter {
+	if age := now.Sub(provenAt); age > window {
 		return backupAssessment{verdict: verdictMissed,
-			reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s) — the tier is unverified, not known-broken",
+			reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
-				tier, age.Round(time.Hour), restoreProvenStaleAfter)}
+				tier, age.Round(time.Hour), window)}
 	}
 	return backupAssessment{verdict: verdictOK}
 }
 // observedArchiveIntervals estimates how often each RESTORE tier actually receives an archive, from
 // the host-reports the hub already holds. Keyed by restore-tier name ("local"/"pbs").
 //
 // Evidence is every distinct archive timestamp in the window: `pbs_snapshots[]` for the offsite
 // tier (PBS enumerates its whole retention in each report, so one report usually settles the
 // question) and successful `backups[]` records attributed by TARGET TYPE for both tiers — the
 // slice-A.4 rule, because a PBS-targeted vzdump appears in BOTH arrays and classifying by array
 // membership would attribute an offsite archive to the host tier.
 //
 // The estimate is the MEAN gap: (newest − oldest) / (generations − 1). It needs two generations;
 // with fewer, ok=false and the caller falls back to the declared rhythm. It is deliberately crude,
 // and can afford to be: restoreProvenWindow clamps the result between a 7-day floor and a 12-day
 // cap, so the only discrimination this has to get right is "roughly daily" versus "several days or
 // slower" — which is exactly the distinction that turns a healthy weekly tier into a false alarm.
 func observedArchiveIntervals(rows []store.HostReportRow) map[string]time.Duration {
 	seen := map[string]map[int64]struct{}{ // tier → set of archive unix times
 		"local": {},
 		"pbs":   {},
 	}
 	add := func(tier string, t time.Time) {
 		if t.IsZero() {
 			return
 		}
 		seen[tier][t.UTC().Unix()] = struct{}{}
 	}
 	for _, r := range rows {
 		var hr hostReportBackups
 		if json.Unmarshal([]byte(r.ReportJSON), &hr) != nil {
 			continue // one malformed retained report must not blind the scan
 		}
 		pbs := pbsTargetSet(hr)
 		for _, ps := range hr.PBSSnapshots {
 			if t, ok := parseBackupTime(ps.BackupTime); ok {
 				add("pbs", t)
 			}
 		}
 		for _, b := range hr.Backups {
 			if !b.Success {
 				continue
 			}
 			t, ok := parseBackupTime(b.StartedAt)
 			if !ok {
 				continue
 			}
 			if pbs[b.TargetID] {
 				add("pbs", t)
 			} else {
 				add("local", t)
 			}
 		}
 	}
 	out := map[string]time.Duration{}
 	for tier, set := range seen {
 		if len(set) < 2 {
 			continue // not observable — the caller uses the declared rhythm
 		}
 		var oldest, newest int64
 		first := true
 		for ts := range set {
 			if first || ts < oldest {
 				oldest = ts
 			}
 			if first || ts > newest {
 				newest = ts
 			}
 			first = false
 		}
 		span := time.Duration(newest-oldest) * time.Second
 		if span <= 0 {
 			continue
 		}
 		out[tier] = span / time.Duration(len(set)-1)
 	}
 	return out
 }
 // expectedRestoreTiers names the tiers this box actually HAS, so a box without an offsite tier is
 // never reported stale for one. Same gate as Slice C's `expected`, and for the same reason: without
 // it every box lacking a tier would alarm once the anchor elapsed — absence-is-not-failure,
@@ -182,26 +182,33 @@ func TestRestoreTest_NewbornDoesNotAlarm(t *testing.T) {
 	}
 }
-// The boundary, pinned by name so a refactor has to delete an obviously-named contract.
+// The boundary, pinned by name so a refactor has to delete an obviously-named contract. R-86 made
 // the limit per-tier, so the anchor is now measured against THE TIER'S OWN window — here the local
 // tier's, which clamps to the 7-day floor and so keeps this contract numerically identical to the
 // one the flat constant expressed.
 func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.T) {
 	now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
 	window := restoreProvenWindow("local", 24*time.Hour, true)
 	if window != restoreProvenWindowFloor {
 		t.Fatalf("precondition: a daily local tier must clamp to the floor; got %s", window)
 	}
 	cases := []struct {
 		name       string
 		watched    time.Duration
 		wantMissed bool
 	}{
 		{"newborn, 1h", time.Hour, false},
-		{"just inside", restoreProvenStaleAfter - time.Minute, false},
+		{"just inside", window - time.Minute, false},
-		{"exactly at the limit", restoreProvenStaleAfter, false},
+		{"exactly at the limit", window, false},
-		{"just outside", restoreProvenStaleAfter + time.Minute, true},
+		{"just outside", window + time.Minute, true},
 		{"long past", 30 * 24 * time.Hour, true},
 	}
 	for _, c := range cases {
 		t.Run(c.name, func(t *testing.T) {
-			got := assessRestoreProven("pbs", time.Time{}, now.Add(-c.watched), now)
+			got := assessRestoreProven("local", time.Time{}, now.Add(-c.watched), now, window)
 			if got.missed() != c.wantMissed {
 				t.Fatalf("CONTRACT VIOLATED: unproven for %s (limit %s) → missed=%v, want %v (reason %q)",
-					c.watched, restoreProvenStaleAfter, got.missed(), c.wantMissed, got.reason)
+					c.watched, window, got.missed(), c.wantMissed, got.reason)
 			}
 			if !c.wantMissed && got.verdict != verdictUnknown {
 				t.Fatalf("a deferred tier must be UNKNOWN (visible), not OK; got verdict=%d", got.verdict)
@@ -215,15 +222,17 @@ func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.
 func TestRestoreTest_StaleIsSeparateFromFailure(t *testing.T) {
 	now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
-	stale := assessRestoreProven("pbs", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now)
+	// A DAILY tier judged on its own rhythm: the window clamps to the 7-day floor.
 	daily := restoreProvenWindow("local", 24*time.Hour, true)
 	stale := assessRestoreProven("local", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
 	if !stale.missed() {
-		t.Fatalf("a tier last proven 9 days ago (limit %s) must be stale; got %q", restoreProvenStaleAfter, stale.reason)
+		t.Fatalf("a daily tier last proven 9 days ago (limit %s) must be stale; got %q", daily, stale.reason)
 	}
 	// The wording must not read as "broken" — that is the other signal.
 	if !strings.Contains(stale.reason, "unverified, not known-broken") {
 		t.Fatalf("staleness must say UNVERIFIED, not broken — merging the two is the thing this avoids; got %q", stale.reason)
 	}
-	fresh := assessRestoreProven("pbs", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now)
+	fresh := assessRestoreProven("local", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
 	if fresh.verdict != verdictOK {
 		t.Fatalf("a tier proven 2 days ago is fine; got verdict=%d reason=%q", fresh.verdict, fresh.reason)
 	}
@@ -283,3 +292,208 @@ func boolStr(b bool) string {
 	}
 	return "false"
 }
 // ── SCENARIO G — a healthy WEEKLY tier is never reported stale (R-86 Part 2) ─────────────────
 //
 // This is the test that pins the false alarm this change would otherwise have CREATED. The agent
 // now proves a tier once per archive generation, so a weekly offsite tier is proved weekly — in
 // perfect health. Against the old flat 7-day window it would sit on the line and alarm every night.
 //
 // COMPANION RED-PROOF (observed 2026-08-03): pin the window flat, as it was —
 //
 //   - window := restoreProvenWindow(tier, observed, observedOK)
 //   - window := restoreProvenWindowFloor   // the pre-R-86 flat 7 days
 //
 // →	--- FAIL: TestRestoreTest_HealthyWeeklyTierIsNeverStale
 //
 //	week 0: a weekly tier proved on its own archive must never be stale (proof age 172h0m0s,
 //	window 168h0m0s); verdict=2 reason="pbs tier: last successful restore-test was 172h0m0s ago
 //	(limit 168h0m0s, this tier's own backup rhythm) — the tier is unverified, not known-broken"
 //
 // Restored. The mutation is one line because the whole of Part 2 is one decision: whose rhythm.
 //
 // NOTE, because it is the finding this test nearly hid: the FIRST version of this fixture had NO
 // jitter, and it PASSED under the mutation. A perfectly regular weekly tier's proof age reaches
 // EXACTLY 168h just before the next proof, and `age > window` is false by a hair — a hollow test
 // that would have shipped Part 1 and its false alarm together. The jitter below is what makes this
 // a test, and it is also the truth about the old constant: a healthy weekly tier did not merely sit
 // near the line, it sat ON it, so any ordinary delay tipped it over.
 func TestRestoreTest_HealthyWeeklyTierIsNeverStale(t *testing.T) {
 	start := time.Date(2026, 6, 1, 3, 0, 0, 0, time.UTC)
 	firstContact := start.Add(-24 * time.Hour)
 	// The observable rhythm of a weekly tier, as the hub would compute it from the reports. No
 	// assertion about the window ITSELF here on purpose: that is the mechanism, and it is pinned in
 	// TestRestoreProvenWindow_Contract. What this test asserts is the CONSEQUENCE — does the alarm
 	// fire? — because R-97b proved a mechanism and shipped a broken consequence anyway.
 	weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
 	// Walk several weeks of a HEALTHY tier, with the jitter a real one has: the backup does not land
 	// to the second, and a restore-test can be deferred one evaluation behind a running backup.
 	//
 	// The jitter is the point. A perfectly regular weekly tier's proof reaches an age of EXACTLY one
 	// interval (168h) just before the next proof, and against a flat 168h window `age > window` is
 	// false by a hair — so a regular fixture would pass against the very constant this task must
 	// change, and prove nothing. That is the brief's "sits exactly on that line": every real-world
 	// delay pushes it over, and the alarm is about a system that is working.
 	settle, evalLatency := 24*time.Hour, 6*time.Hour
 	archiveLate := []time.Duration{0, 4 * time.Hour, 2 * time.Hour, 6 * time.Hour, 0, 3 * time.Hour}
 	deferred := []time.Duration{0, 0, 6 * time.Hour, 0, 0, 6 * time.Hour} // one evaluation behind a backup
 	archiveAt := func(week int) time.Time {
 		return start.AddDate(0, 0, 7*week).Add(archiveLate[week])
 	}
 	provenAt := func(week int) time.Time {
 		return archiveAt(week).Add(settle + evalLatency + deferred[week])
 	}
 	var worst time.Duration
 	for week := 0; week+1 < len(archiveLate); week++ {
 		// The widest the proof's age ever gets: the instant before the NEXT week's proof lands.
 		now := provenAt(week + 1).Add(-time.Second)
 		age := now.Sub(provenAt(week))
 		if age > worst {
 			worst = age
 		}
 		v := assessRestoreProven("pbs", provenAt(week), firstContact, now, weekly)
 		if v.verdict != verdictOK {
 			t.Fatalf("week %d: a weekly tier proved on its own archive must never be stale (proof age %s, window %s); verdict=%d reason=%q",
 				week, age.Round(time.Hour), weekly, v.verdict, v.reason)
 		}
 	}
 	// The fixture must actually EXERCISE the boundary — a jitter-free walk would sit at exactly one
 	// interval and pass against a flat 7-day window, which is the hollow version of this test.
 	if worst <= restoreProvenWindowFloor {
 		t.Fatalf("this fixture never exceeds the old flat window (worst proof age %s) — it cannot detect the defect it exists for", worst)
 	}
 }
 // ...and a weekly tier that genuinely STOPS being proved must still alarm. A window that never
 // fires is not a fix, it is a deletion.
 func TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms(t *testing.T) {
 	now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
 	weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
 	v := assessRestoreProven("pbs", now.Add(-weekly-time.Hour), now.Add(-90*24*time.Hour), now, weekly)
 	if !v.missed() {
 		t.Fatalf("a weekly tier unproven for longer than its own window MUST alarm; got verdict=%d reason=%q", v.verdict, v.reason)
 	}
 	if !strings.Contains(v.reason, weekly.String()) {
 		t.Fatalf("the alarm must state the window it was judged against (R-100's corollary); got %q", v.reason)
 	}
 }
 // The window's own contract: derived from the tier's rhythm, floored, capped, and never dependent
 // on an unobservable history for the tier that would suffer most from a wrong answer.
 func TestRestoreProvenWindow_Contract(t *testing.T) {
 	cases := []struct {
 		name       string
 		tier       string
 		observed   time.Duration
 		observedOK bool
 		want       time.Duration
 	}{
 		{"daily local clamps to the floor", "local", 24 * time.Hour, true, restoreProvenWindowFloor},
 		{"weekly pbs widens", "pbs", 7 * 24 * time.Hour, true, restoreProvenWindowCap},
 		{"3-day tier sits between", "pbs", 72 * time.Hour, true, 12 * 24 * time.Hour},
 		{"unobservable local falls back to its declared rhythm", "local", 0, false, restoreProvenWindowFloor},
 		{"unobservable pbs falls back WIDE, not to the floor", "pbs", 0, false, restoreProvenWindowCap},
 		{"a nonsense zero interval is ignored", "pbs", 0, true, restoreProvenWindowCap},
 		// MEASURED ON THE LIVE BOX, and the reason observation may only WIDEN. demo-felhom's two
 		// retained PBS snapshots sit 8h54m apart (one is a healing artefact), so a mean-gap estimate
 		// reads a WEEKLY tier as nine-hourly. Taking that at face value gives 4x9h = 36h → the 7-day
 		// floor → and a weekly tier proved weekly (~8.25d of proof age) alarms within a week of this
 		// shipping, on the box it shipped to.
 		{"a short observed gap must NOT tighten a weekly tier", "pbs", 8*time.Hour + 54*time.Minute, true, restoreProvenWindowCap},
 		{"a short observed gap must not tighten the host tier either", "local", 30 * time.Minute, true, restoreProvenWindowFloor},
 		// ...but a tier that genuinely runs SLOWER than its declared rhythm still widens.
 		{"a genuinely slower tier widens", "local", 4 * 24 * time.Hour, true, restoreProvenWindowCap},
 	}
 	// The relationship Part 1 depends on: a weekly tier's window must be WIDER than a daily tier's,
 	// or proving weekly (which is now correct behaviour) alarms on itself.
 	if restoreProvenWindow("pbs", 7*24*time.Hour, true) <= restoreProvenWindow("local", 24*time.Hour, true) {
 		t.Fatal("a weekly tier must earn a wider window than a daily one — otherwise R-86's agent half alarms about itself")
 	}
 	for _, c := range cases {
 		t.Run(c.name, func(t *testing.T) {
 			got := restoreProvenWindow(c.tier, c.observed, c.observedOK)
 			if got != c.want {
 				t.Fatalf("window(%s, observed=%s ok=%v) = %s, want %s", c.tier, c.observed, c.observedOK, got, c.want)
 			}
 			if got < restoreProvenWindowFloor || got > restoreProvenWindowCap {
 				t.Fatalf("every window must stay inside [%s, %s]; got %s", restoreProvenWindowFloor, restoreProvenWindowCap, got)
 			}
 		})
 	}
 }
 // The rhythm must be OBSERVED from the reports, not assumed — including the slice-A.4 rule that a
 // PBS-targeted vzdump appears in both arrays and must be attributed by TARGET TYPE.
 func TestObservedArchiveIntervals_FromReports(t *testing.T) {
 	base := time.Date(2026, 7, 1, 2, 0, 0, 0, time.UTC)
 	mk := func(localAt []time.Time, pbsAt []time.Time) string {
 		type stg struct{ Name, Type, Content string }
 		type bk struct {
 			TargetID  string `json:"target_id"`
 			Success   bool   `json:"success"`
 			StartedAt string `json:"started_at"`
 		}
 		type snap struct {
 			BackupTime string `json:"backup_time"`
 		}
 		payload := struct {
 			StorageTargets []struct {
 				Name    string `json:"name"`
 				Type    string `json:"type"`
 				Content string `json:"content"`
 			} `json:"storage_targets"`
 			Backups      []bk   `json:"backups"`
 			PBSSnapshots []snap `json:"pbs_snapshots"`
 		}{}
 		payload.StorageTargets = append(payload.StorageTargets, struct {
 			Name    string `json:"name"`
 			Type    string `json:"type"`
 			Content string `json:"content"`
 		}{"felhom-backup", "dir", "backup"}, struct {
 			Name    string `json:"name"`
 			Type    string `json:"type"`
 			Content string `json:"content"`
 		}{"felhom-pbs", "pbs", "backup"})
 		for _, at := range localAt {
 			payload.Backups = append(payload.Backups, bk{"felhom-backup", true, at.Format(time.RFC3339)})
 		}
 		for _, at := range pbsAt {
 			// The SAME archive appears as a vzdump record AND as a snapshot — slice A.4.
 			payload.Backups = append(payload.Backups, bk{"felhom-pbs", true, at.Format(time.RFC3339)})
 			payload.PBSSnapshots = append(payload.PBSSnapshots, snap{at.Format(time.RFC3339)})
 		}
 		b, err := json.Marshal(payload)
 		if err != nil {
 			t.Fatal(err)
 		}
 		return string(b)
 	}
 	rows := []store.HostReportRow{
 		{ReportJSON: mk(
 			[]time.Time{base, base.AddDate(0, 0, 1), base.AddDate(0, 0, 2)},
 			[]time.Time{base, base.AddDate(0, 0, 7)},
 		)},
 		{ReportJSON: `{{{malformed`}, // must not blind the scan
 	}
 	got := observedArchiveIntervals(rows)
 	if d, ok := got["local"]; !ok || d != 24*time.Hour {
 		t.Fatalf("a daily host tier must be observed as ~24h; got %s ok=%v", d, ok)
 	}
 	if d, ok := got["pbs"]; !ok || d != 7*24*time.Hour {
 		t.Fatalf("a weekly offsite tier must be observed as ~7d — and its vzdump record must NOT be "+
 			"counted into the host tier (slice A.4); got %s ok=%v", d, ok)
 	}
 	// One generation is not a rhythm: unobservable, so the caller falls back to the declared one.
 	single := []store.HostReportRow{{ReportJSON: mk(nil, []time.Time{base})}}
 	if d, ok := observedArchiveIntervals(single)["pbs"]; ok {
 		t.Fatalf("one archive cannot yield an interval; got %s", d)
 	}
 }
@@ -0,0 +1,301 @@
 package notify
 import (
 	"io"
 	"log"
 	"strings"
 	"testing"
 )
 // R-182 — one e-mail per backup run, and nothing dropped without a trace.
 //
 // MEASURED, NOT SUPPOSED. On 2026-08-03 nine `recovery_unit_capture_failed` events reached the hub
 // and TWO operator e-mails went out. The operator cooldown key is
 // `customerID + ":" + eventType + cooldownTierSuffix(details)`, that event carries `app` but no
 // `tier`, so the key held no app identifier: the first refused app took the hour's slot and every
 // other app's failure was discarded — **before `LogNotification`**, so it left no row on any channel
 // and could not be found afterwards.
 //
 // The operator ruled against the obvious fix (putting `app` in the key), because on a full disk that
 // is one e-mail per app. These pin the shape that replaced it.
 // ── Scenario D — a suppressed operator event leaves a trace ───────────────────────────────────────
 // The bare `return` at the cooldown is the whole reason this defect took a day to get the right way
 // round: there was nothing to read. A drop must be as visible in the record as a send.
 //
 // DELIBERATELY EXERCISED ON A DIFFERENT EVENT TYPE than the one that exposed the defect.
 // `recovery_unit_capture_failed` is now record-only and never reaches the cooldown at all, so using
 // it here would prove nothing. `whole_guest_backup_failed` is an ordinary operator event, and using
 // it pins §2.1's actual claim: the suppression row applies to EVERY operator event, not only the one
 // that happened to be measured.
 func TestSuppressedOperatorEvent_LeavesARow(t *testing.T) {
 	st := newDispStore(t)
 	d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
 	sent := captureSeam(d)
 	// Two events of the SAME type with no discriminator — the second must be suppressed.
 	d.ProcessEvent("c1", "whole_guest_backup_failed", "error", "opengist failed", `{"app":"opengist"}`, "controller")
 	d.ProcessEvent("c1", "whole_guest_backup_failed", "error", "privatebin failed", `{"app":"privatebin"}`, "controller")
 	if got := len(mailsFor(*sent, "op@felhom.eu")); got != 1 {
 		t.Fatalf("operator mails = %d, want 1 — the premise of this test is that the second IS suppressed", got)
 	}
 	rows, err := st.GetRecentNotifications("c1", 50)
 	if err != nil {
 		t.Fatal(err)
 	}
 	var sentRows, suppressed []store2Row
 	for _, r := range rows {
 		if r.Channel != "operator" || r.EventType != "whole_guest_backup_failed" {
 			continue
 		}
 		switch r.Status {
 		case "sent":
 			sentRows = append(sentRows, store2Row{r.Status, r.Message, r.ErrorMessage})
 		case "suppressed":
 			suppressed = append(suppressed, store2Row{r.Status, r.Message, r.ErrorMessage})
 		}
 	}
 	if len(sentRows) != 1 {
 		t.Fatalf("want 1 'sent' operator row, got %d", len(sentRows))
 	}
 	if len(suppressed) != 1 {
 		t.Fatalf("want 1 'suppressed' operator row, got %d — a cooldown drop that writes NOTHING is "+
 			"indistinguishable from an event that never happened, which is exactly how seven "+
 			"failures went missing on 2026-08-03", len(suppressed))
 	}
 	// The row must name the app that was dropped, or it records that something was suppressed
 	// without recording WHAT — half a fix.
 	if !strings.Contains(suppressed[0].message, "privatebin") {
 		t.Fatalf("the suppressed row does not name the dropped event: %q", suppressed[0].message)
 	}
 	// And it must carry the key, so the reason it collided is readable without reading code.
 	if !strings.Contains(suppressed[0].errMsg, "key=") {
 		t.Fatalf("the suppressed row does not carry the cooldown key: %q", suppressed[0].errMsg)
 	}
 }
 type store2Row struct{ status, message, errMsg string }
 // ── Scenario E — two runs in a day each report ───────────────────────────────────────────────────
 // The operator ruled explicitly on this: someone pressing the backup button is actively trying to
 // get a backup, and finding out tomorrow would be worse than an extra mail in a rare case.
 func TestTwoRunsInAnHour_BothReport(t *testing.T) {
 	st := newDispStore(t)
 	d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
 	sent := captureSeam(d)
 	nightly := `{"run_id":"run-a","run_kind":"nightly","failed":2,"attempted":5,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"},{"app":"privatebin","leg":"volume dump","reason":"reserve"}]}`
 	manual := `{"run_id":"run-b","run_kind":"manual","failed":2,"attempted":5,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"},{"app":"privatebin","leg":"volume dump","reason":"reserve"}]}`
 	d.ProcessEvent("c1", "backup_run_failures", "error", "2 of 5 apps failed", nightly, "controller")
 	d.ProcessEvent("c1", "backup_run_failures", "error", "2 of 5 apps failed", manual, "controller")
 	if got := len(mailsFor(*sent, "op@felhom.eu")); got != 2 {
 		t.Fatalf("operator mails = %d, want 2 — the 1-hour cooldown swallowed the manual run's "+
 			"digest, which is the fix reappearing one level up: press the button, the run fails, "+
 			"and you are told nothing because the machine already wrote this hour", got)
 	}
 }
 // The run discriminator must be NARROW, exactly like its `tier` sibling — empty unless the producer
 // opts in — or every existing event type's cooldown silently stops collapsing anything.
 func TestCooldownRunSuffix_EmptyForEverythingElse(t *testing.T) {
 	cases := []struct{ name, details string }{
 		{"no details", ""},
 		{"details without run_id", `{"app":"opengist","error":"boom"}`},
 		{"empty run_id value", `{"run_id":""}`},
 		{"malformed json", `{{{nope`},
 		{"run_id mentioned in a STRING, not as a key", `{"error":"the run_id: abc failed"}`},
 		{"null details", `null`},
 	}
 	for _, c := range cases {
 		if got := cooldownRunSuffix(c.details); got != "" {
 			t.Errorf("%s: suffix must be EMPTY so every other type's cooldown is unchanged, got %q", c.name, got)
 		}
 	}
 	if got := cooldownRunSuffix(`{"run_id":"run-a"}`); got != ":run-a" {
 		t.Fatalf("suffix should be the run id, got %q", got)
 	}
 	// The two suffixes must not interfere: a tier event still keys on its tier and nothing else.
 	if got := cooldownTierSuffix(`{"tier":"felhom-pbs"}`) + cooldownRunSuffix(`{"tier":"felhom-pbs"}`); got != ":felhom-pbs" {
 		t.Fatalf("a tier-only event's key changed to %q — R-97a's behaviour must be byte-identical", got)
 	}
 }
 // ── Scenario G — the customer never receives the digest ──────────────────────────────────────────
 // v0.78.0 asserted in a COMMENT that a type with no `customerMessages` entry structurally cannot
 // reach a customer. It can: FormatCustomerEmail falls back to the raw English message and the only
 // customer gate is configuration. So this is demonstrated, not argued.
 func TestDigest_IsOperatorOnly_EvenWithAWideEnabledList(t *testing.T) {
 	st := newDispStore(t)
 	// A customer who has enabled EVERYTHING, including this type by name.
 	if err := st.SaveNotificationPrefs("c1", "cust@example.com",
 		[]string{"backup_run_failures", "node_down", "disk_warning"}, 6); err != nil {
 		t.Fatal(err)
 	}
 	d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
 	sent := captureSeam(d)
 	details := `{"run_id":"run-a","run_kind":"nightly","failed":1,"attempted":3,"apps":[{"app":"opengist","leg":"volume dump","reason":"reserve"}]}`
 	d.ProcessEvent("c1", "backup_run_failures", "error", "1 of 3 apps failed", details, "controller")
 	if got := mailsFor(*sent, "cust@example.com"); len(got) != 0 {
 		t.Fatalf("the CUSTOMER received an operator digest (%d mails) — a list of which apps' "+
 			"backups failed is not something they can act on, and the raw body is English", len(got))
 	}
 	if got := len(mailsFor(*sent, "op@felhom.eu")); got != 1 {
 		t.Fatalf("operator mails = %d, want 1", got)
 	}
 	if !operatorOnlyEvents["backup_run_failures"] {
 		t.Fatal("backup_run_failures is not in operatorOnlyEvents — allowlisting alone does NOT " +
 			"keep it from a customer; that assumption shipped once and was wrong (v0.78.0)")
 	}
 }
 // ── Part 3 — the e-mail a person actually reads ──────────────────────────────────────────────────
 func TestDigestEmail_ListsAppsLegsAndReasons(t *testing.T) {
 	details := `{"run_id":"run-a","run_kind":"nightly","failed":3,"attempted":40,` +
 		`"target_path":"/mnt/sys_drive","used_gb":64.3,"avail_gb":0.9,"total_gb":68.7,` +
 		`"used_percent":94,"space_known":true,"apps":[` +
 		`{"app":"opengist","leg":"volume dump","reason":"refused: below the reserve (headroom)"},` +
 		`{"app":"privatebin","leg":"volume dump","reason":"refused: below the reserve (headroom)"},` +
 		`{"app":"immich","leg":"database dump","reason":"pg_dump: connection refused"}]}`
 	subject, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error",
 		"3 of 40 apps failed to back up", details)
 	// The subject must carry the counts: the operator's first decision is made from it alone.
 	for _, want := range []string{"demo-hp", "3 of 40", "nightly"} {
 		if !strings.Contains(subject, want) {
 			t.Errorf("subject %q missing %q", subject, want)
 		}
 	}
 	// Every app, its leg and its reason.
 	for _, want := range []string{
 		"opengist", "privatebin", "immich",
 		"volume dump", "database dump",
 		"below the reserve", "pg_dump: connection refused",
 	} {
 		if !strings.Contains(body, want) {
 			t.Errorf("body missing %q:\n%s", want, body)
 		}
 	}
 	// The counts and the free space, so "one broken app" and "a full disk" read differently.
 	if !strings.Contains(body, "3 of 40") {
 		t.Errorf("body does not carry the failed-of-attempted count:\n%s", body)
 	}
 	if !strings.Contains(body, "0.9 GB free") {
 		t.Errorf("body does not carry the free space:\n%s", body)
 	}
 	// It must NOT be a JSON blob.
 	if strings.Contains(body, `"apps":[`) {
 		t.Errorf("the digest rendered as raw JSON — unreadable on a phone at 07:00:\n%s", body)
 	}
 }
 // An absent space reading must render as unavailable, never as zeros: "0 GB free" and "we could not
 // look" are opposite diagnoses, and the operator acts differently on each.
 func TestDigestEmail_UnknownSpaceIsNotZero(t *testing.T) {
 	details := `{"run_id":"r","run_kind":"nightly","failed":1,"attempted":2,"target_path":"/mnt/x",` +
 		`"space_known":false,"apps":[{"app":"a","leg":"capture","reason":"boom"}]}`
 	_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "1 of 2 failed", details)
 	if strings.Contains(body, "0.0 GB free") {
 		t.Fatalf("an unreadable filesystem rendered as zeros:\n%s", body)
 	}
 	if !strings.Contains(body, "unavailable") {
 		t.Fatalf("an unreadable filesystem must say so:\n%s", body)
 	}
 }
 // A payload that cannot be parsed must still produce a mail — degraded, never swallowed.
 func TestDigestEmail_UnparseableDetailsStillMails(t *testing.T) {
 	_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "something failed", `{{{`)
 	if body == "" || !strings.Contains(body, "something failed") {
 		t.Fatalf("an unparseable digest lost the mail entirely:\n%s", body)
 	}
 }
 // ── Scenario C — every failure is RECORDED, e-mailed or not ──────────────────────────────────────
 // The per-app event is the record; the digest is the notification. The record must not inherit the
 // notification's conditions — no cooldown, no preferences, no dependence on a mail going out.
 func TestPerAppFailure_IsRecordedButNotMailed(t *testing.T) {
 	st := newDispStore(t)
 	d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
 	sent := captureSeam(d)
 	apps := []string{"opengist", "privatebin", "immich", "homebox", "nextcloud"}
 	for _, a := range apps {
 		d.ProcessEvent("c1", "recovery_unit_capture_failed", "error",
 			"Recovery unit capture FAILED for \""+a+"\"", `{"app":"`+a+`"}`, "controller")
 	}
 	// NOT mailed — the digest is the notification.
 	if got := len(*sent); got != 0 {
 		t.Fatalf("%d mail(s) sent for per-app failures — they are the RECORD; one mail per app on a "+
 			"full disk is the volume problem wearing the correctness problem's clothes, which is "+
 			"exactly what the operator ruled against", got)
 	}
 	// But ALL FIVE recorded — this is the assertion yesterday's defect would have failed: nine
 	// arrived, two were mailed, seven left no row anywhere.
 	rows, err := st.GetRecentNotifications("c1", 50)
 	if err != nil {
 		t.Fatal(err)
 	}
 	seen := map[string]bool{}
 	for _, r := range rows {
 		if r.EventType == "recovery_unit_capture_failed" && r.Status == "recorded" {
 			for _, a := range apps {
 				if strings.Contains(r.Message, a) {
 					seen[a] = true
 				}
 			}
 		}
 	}
 	if len(seen) != len(apps) {
 		t.Fatalf("only %d of %d per-app failures were recorded (%v) — a failure that produced no row "+
 			"anywhere is the measured defect of 2026-08-03", len(seen), len(apps), seen)
 	}
 }
 // The per-app reason must not repeat the filesystem figures the digest already prints once. Reviewed
 // as copy against the first real digest, not designed in the abstract.
 func TestDigestEmail_ReasonDoesNotRepeatTheUsageLine(t *testing.T) {
 	reason := "refused: below the reserve (reserve: 97% used or 1.0 GiB free) — /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free"
 	details := `{"run_id":"r","run_kind":"nightly","failed":1,"attempted":2,"target_path":"/mnt/sys_drive",` +
 		`"used_gb":65,"avail_gb":0.2,"total_gb":68.7,"used_percent":95,"space_known":true,` +
 		`"apps":[{"app":"opengist","leg":"whole app","reason":"` + reason + `"}]}`
 	_, body := FormatOperatorEmail("demo-hp", "backup_run_failures", "error", "1 of 2 failed", details)
 	// The figures appear ONCE, on the Filesystem line — not again on every app row.
 	if strings.Count(body, "65.0/68.7 GB used") != 1 {
 		t.Fatalf("the usage clause appears %d times; it must appear once, on its own line — repeated "+
 			"down a list of a dozen apps it pushes the part that DIFFERS off a phone screen:\n%s",
 			strings.Count(body, "65.0/68.7 GB used"), body)
 	}
 	// But the reason itself survives — trimming must not eat the diagnosis.
 	if !strings.Contains(body, "below the reserve") {
 		t.Fatalf("the reason was trimmed away entirely:\n%s", body)
 	}
 }
 // A reason naming a DIFFERENT path, or none, must be left completely alone.
 func TestTrimRepeatedUsage_LeavesUnrelatedReasonsAlone(t *testing.T) {
 	for _, c := range []struct{ reason, target string }{
 		{"pg_dump: connection refused", "/mnt/sys_drive"},
 		{"tar failed — /mnt/other: 1/2 GB used (50%), 1 GB free", "/mnt/sys_drive"},
 		{"boom", ""},
 		{"", "/mnt/sys_drive"},
 	} {
 		if got := trimRepeatedUsage(c.reason, c.target); got != c.reason {
 			t.Errorf("reason %q (target %q) was altered to %q", c.reason, c.target, got)
 		}
 	}
 }
@@ -116,6 +116,18 @@ func (d *Dispatcher) ProcessEvent(customerID, eventType, severity, message, deta
 		return
 	}
 	// R-182: record-only types are written down and never mailed. Placed BEFORE the severity gate
 	// so the row is written whatever the severity — the record must not inherit the notification's
 	// conditions, which is the coupling this whole finding is about.
 	if recordOnlyEvents[eventType] {
 		if err := d.store.LogNotification(customerID, eventType, severity, message, "recorded",
 			"record-only: the per-run digest (backup_run_failures) carries the notification", "operator"); err != nil {
 			d.logger.Printf("[WARN] Failed to record %s for %s: %v", eventType, customerID, err)
 		}
 		d.logger.Printf("[INFO] Recorded (not mailed) %s for %s — the run digest is the notification", eventType, customerID)
 		return
 	}
 	// warning / error / critical trigger notifications. "info" is an intentional non-notify (status/
 	// recovery events). Anything else is UNRECOGNIZED — log it (don't silently drop), so a bad severity
 	// surfaces instead of vanishing (the felhom-pve-class lesson: a critical event must never be lost).
@@ -260,15 +272,67 @@ func cooldownTierSuffix(detailsJSON string) string {
 	return ":" + d.Tier
 }
 // cooldownRunSuffix returns ":"+run_id when the event's details carry a non-empty `run_id`, else "".
 //
 // R-182. `cooldownTierSuffix`'s sibling, and deliberately a SEPARATE function rather than an extra
 // branch inside it: `tier` keeps byte-identical semantics for every type that uses it, so R-97a's
 // behaviour and its tests are untouched by this.
 //
 // WHY A BACKUP RUN NEEDS ONE. The run digest describes ONE RUN, and a box can have two in a day —
 // the nightly one and a manual one the operator triggered *because* something looked wrong. With no
 // run-scoped discriminator the 1-hour cooldown would swallow the second, which is the failure this
 // row exists to fix, reappearing one level up: the operator presses the button, the run fails, and
 // they are told nothing because the machine already wrote that hour.
 //
 // IT MAKES THE COOLDOWN EFFECTIVELY INERT FOR THIS TYPE, AND THAT IS THE INTENT, NOT AN OVERSIGHT.
 // A digest is already rate-limited by construction — one per run, emitted only when something
 // failed — so there is nothing for a timer to collapse. The cooldown protects against a repeating
 // identical alert; a digest cannot repeat, because each run is a different run.
 //
 // NARROW, LIKE ITS SIBLING: empty unless the producer opts in by sending a `run_id`, so no existing
 // event type's cooldown behaviour changes.
 func cooldownRunSuffix(detailsJSON string) string {
 	if detailsJSON == "" || !strings.Contains(detailsJSON, "\"run_id\"") {
 		return ""
 	}
 	var d struct {
 		RunID string `json:"run_id"`
 	}
 	if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || d.RunID == "" {
 		return ""
 	}
 	return ":" + d.RunID
 }
 func (d *Dispatcher) processOperator(customerID, eventType, severity, message, detailsJSON, source string) {
 	if !d.operatorOn || d.operatorEmail == "" {
 		return
 	}
-	cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON)
+	cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON) + cooldownRunSuffix(detailsJSON)
 	d.mu.Lock()
 	if last, ok := d.opCooldowns[cooldownKey]; ok && time.Since(last) < 1*time.Hour {
 		d.mu.Unlock()
 		// R-182: RECORD THE SUPPRESSION. This used to be a bare `return` — the event was dropped
 		// before any LogNotification, so a cooldown drop and an event that never happened were
 		// indistinguishable from the operator's side AND from the hub's own records.
 		//
 		// Measured 2026-08-03: nine `recovery_unit_capture_failed` events arrived, two emails were
 		// sent, and the other seven left NO ROW ON ANY CHANNEL. The defect that hid was serious —
 		// the cooldown key carries no app identifier, so the first refused app took the slot and
 		// every other app's failure that hour was discarded — but the reason it took a day to find
 		// the right way round is this line: there was nothing to read.
 		//
 		// "We chose not to e-mail you" and "nothing happened" must never look identical. This
 		// applies to EVERY operator event, not only the one that exposed it. It makes the drop
 		// visible; it deliberately does NOT change the cooldown's duration or semantics.
 		if err := d.store.LogNotification(customerID, eventType, severity, message,
 			"suppressed", "operator cooldown 1h, key="+cooldownKey, "operator"); err != nil {
 			d.logger.Printf("[WARN] Failed to record suppressed operator notification for %s/%s: %v",
 				customerID, eventType, err)
 		}
 		d.logger.Printf("[INFO] Operator email suppressed for %s/%s — cooldown (key=%s)",
 			customerID, eventType, cooldownKey)
 		return
 	}
 	d.opCooldowns[cooldownKey] = time.Now()
@@ -285,6 +349,33 @@ func (d *Dispatcher) processOperator(customerID, eventType, severity, message, d
 	d.store.LogNotification(customerID, eventType, severity, message, "sent", "", "operator")
 }
 // recordOnlyEvents are STORED and RECORDED but never e-mailed, on either channel.
 //
 // R-182. The distinction this register exists to make is the whole of that finding: **the record and
 // the notification are different things.** A per-app backup failure must always be written down —
 // every time, unconditionally, regardless of cooldowns, preferences or whether any mail went out —
 // and it must NOT compete for an e-mail slot, because the per-run digest
 // (`backup_run_failures`) is what a person is meant to read.
 //
 // Before this, `recovery_unit_capture_failed` was both at once, and it did neither well: on
 // 2026-08-03 nine of them arrived, two were e-mailed, and the other seven were dropped by the
 // 1-hour cooldown BEFORE anything was written down. So the operator was told about one app, the
 // other apps' failures were discarded, and nothing anywhere recorded that a choice had been made.
 //
 // WHY A REGISTER AND NOT severity "info". Downgrading the severity would have the same routing
 // effect — `severityNotifies` drops info — but it would also relabel a genuine failure as
 // informational in the events table, the operator UI and every historical query, and it would
 // silently drop the X-Priority handling if the type were ever promoted back. This says what it
 // means: not silent, not urgent, RECORDED.
 //
 // IT IS NOT A WAY TO MUTE THINGS. A type belongs here only when something else carries its
 // notification. Adding one with no digest behind it rebuilds the silence R-182 was filed against.
 var recordOnlyEvents = map[string]bool{
 	// The per-app Tier-1 capture/refusal failure. Its notification is the run digest, which lists
 	// every failed app in one mail; this row is the durable per-failure record behind it.
 	"recovery_unit_capture_failed": true,
 }
 // operatorOnlyEvents are event types that must NEVER reach a customer, whatever their preferences say.
 //
 // R-97c. This register exists because the guarantee it provides was previously ASSERTED IN A COMMENT
@@ -321,6 +412,15 @@ var operatorOnlyEvents = map[string]bool{
 	// figures, the raw error). The customer's half of D-c is the FILL WARNING, which fires BEFORE
 	// this and is actionable: free space, delete files, add a drive.
 	"recovery_unit_capture_failed": true,
 	// R-182. The per-run backup digest. It is the same class as the line above and for the same
 	// reason — a customer can act on a full disk (that is the fill warning, which fires first and
 	// IS customer-facing) but not on a list of which apps' backups failed and why. It also carries
 	// operator-grade detail: per-app leg names, raw refusal reasons and byte figures.
 	//
 	// Listed here rather than relying on the absence of a `customerMessages` entry, which is NOT a
 	// block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped
 	// once (v0.78.0) and the comment above records it.
 	"backup_run_failures": true,
 }
 // IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
@@ -39,33 +39,45 @@ func TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer(t *testing.T) {
 		}
 	}
-	// The operator must still get it: the register mutes the customer channel, not the signal.
+	// R-182 CHANGED WHAT THIS ASSERTS, DELIBERATELY, AND THE OLD ASSERTION IS WORTH KEEPING IN VIEW.
-	gotOperator := false
+	//
 	// Until 2026-08-03 this test required the OPERATOR to be e-mailed here, on the grounds that "the
 	// alert is the whole point of R-158". That was right when this event was the only signal, and it
 	// is wrong now: measured, nine of these arrived at the hub and two were mailed, because the
 	// operator cooldown key carries no app identifier — so as an alarm it told the operator about one
 	// app and threw the rest away.
 	//
 	// The type is now RECORD-ONLY: written down every time, never mailed. R-158's guarantee — the
 	// operator learns WHICH app failed and WHY — is not weakened, it MOVED: the per-run digest
 	// `backup_run_failures` carries every failed app in one mail, and is pinned by
 	// backup_run_digest_test.go. The customer safety claim below is untouched and is the reason this
 	// test still exists.
 	for _, to := range rec.to {
 		if to == "operator@felhom.eu" {
-			gotOperator = true
+			t.Fatal("the operator was e-mailed a PER-APP capture failure — this type is the record " +
 				"now, not the alarm. One mail per failing app on a full disk is a dozen mails, which " +
 				"is the volume problem the operator ruled against; the digest is the notification")
 		}
 	}
 	if !gotOperator {
 		t.Fatal("the operator was not notified of a recovery-unit capture failure — the alert is the " +
 			"whole point of R-158 and it went nowhere")
 	}
-	// The skip must be VISIBLE. An absent log row is equally consistent with "correctly skipped" and
+	// The RECORD must exist, always. It is what makes the digest trustworthy: if the digest is ever
-	// "the dispatcher never ran" — the positive observable is the row itself (standing rule 3).
+	// lost, delayed or suppressed, the failures are still individually written down. An absent row is
 	// equally consistent with "correctly not mailed" and "the dispatcher never ran" — the positive
 	// observable is the row itself (standing rule 3).
 	logs, err := st.GetRecentNotifications("c1", 20)
 	if err != nil {
 		t.Fatalf("GetRecentNotifications: %v", err)
 	}
 	found := false
 	for _, l := range logs {
-		if l.Channel == "customer" && l.Status == "skipped" && strings.Contains(l.ErrorMessage, "operator_only") {
+		if l.EventType == "recovery_unit_capture_failed" && l.Status == "recorded" &&
 			strings.Contains(l.ErrorMessage, "record-only") {
 			found = true
 		}
 	}
 	if !found {
-		t.Fatalf("the customer skip is not logged as skipped/operator_only — it is indistinguishable "+
+		t.Fatalf("the per-app failure left no 'recorded' row — a failure that is neither mailed nor "+
-			"from a delivery that never happened; got %d row(s)", len(logs))
+			"written down is exactly the 2026-08-03 defect, rebuilt; got %d row(s)", len(logs))
 	}
 }
@@ -1,6 +1,7 @@
 package notify
 import (
 	"encoding/json"
 	"fmt"
 	"strings"
 	"time"
@@ -41,6 +42,17 @@ Severity: %s
 Time:     %s
 Message:  %s`, customerID, eventType, severity, now, message)
 	// R-182: the backup run digest gets a rendered list instead of a raw JSON blob. It is the one
 	// operator mail that carries a VARIABLE-LENGTH payload, and a dozen apps as one line of JSON is
 	// unreadable on a phone at 07:00, which is the only time it matters.
 	if eventType == "backup_run_failures" {
 		if rendered, sub, ok := renderBackupRunFailures(customerID, detailsJSON); ok {
 			return sub, body + rendered + fmt.Sprintf("\n\nDashboard: https://hub.felhom.eu/customers/%s", customerID)
 		}
 		// Unparseable details fall through to the raw form below rather than losing the mail. A
 		// digest that renders badly still tells the operator something; a swallowed one does not.
 	}
 	if detailsJSON != "" && detailsJSON != "{}" {
 		body += fmt.Sprintf("\nDetails:  %s", detailsJSON)
 	}
@@ -282,3 +294,115 @@ Ha nem te kérted ezt, hagyd figyelmen kívül ezt az e-mailt.
 Felhom.eu`, link)
 	return subject, body
 }
 // ──────────────────────────────────────────────────────────────────────
 // R-182 — the backup run digest
 // ──────────────────────────────────────────────────────────────────────
 // backupRunFailure is one app's failed leg within a run.
 type backupRunFailure struct {
 	App    string `json:"app"`
 	Leg    string `json:"leg"`
 	Reason string `json:"reason"`
 }
 // backupRunDetails is the digest payload the controller sends.
 type backupRunDetails struct {
 	RunID       string             `json:"run_id"`
 	RunKind     string             `json:"run_kind"`
 	Failed      int                `json:"failed"`
 	Attempted   int                `json:"attempted"`
 	TargetPath  string             `json:"target_path"`
 	UsedGB      float64            `json:"used_gb"`
 	AvailGB     float64            `json:"avail_gb"`
 	TotalGB     float64            `json:"total_gb"`
 	UsedPercent float64            `json:"used_percent"`
 	SpaceKnown  bool               `json:"space_known"`
 	Apps        []backupRunFailure `json:"apps"`
 }
 // renderBackupRunFailures turns the digest details into an operator-readable block and a subject
 // that says the count without being opened. Returns ok=false when the payload cannot be parsed or
 // names no apps, so the caller can fall back to the raw rendering rather than mail an empty list.
 //
 // THE SUCCESS COUNT IS NOT DECORATION. "3 of 4 apps failed" is a catastrophe and "3 of 40" is a bad
 // night; the list alone cannot tell them apart, and the operator's first decision — get up now, or
 // look after coffee — depends entirely on which it is.
 func renderBackupRunFailures(customerID, detailsJSON string) (string, string, bool) {
 	if detailsJSON == "" {
 		return "", "", false
 	}
 	var d backupRunDetails
 	if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || len(d.Apps) == 0 {
 		return "", "", false
 	}
 	kind := d.RunKind
 	if kind == "" {
 		kind = "backup"
 	}
 	subject := fmt.Sprintf("[Felhom] 🔴 %s: %d of %d apps failed to back up (%s run)",
 		customerID, d.Failed, d.Attempted, kind)
 	// Column-align the app names so the leg and reason line up and the block scans vertically.
 	width := 0
 	for _, a := range d.Apps {
 		if len(a.App) > width {
 			width = len(a.App)
 		}
 	}
 	legWidth := 0
 	for _, a := range d.Apps {
 		if len(a.Leg) > legWidth {
 			legWidth = len(a.Leg)
 		}
 	}
 	var b strings.Builder
 	fmt.Fprintf(&b, "\n\nFAILED: %d of %d apps attempted in this %s run.\n\n", d.Failed, d.Attempted, kind)
 	for _, a := range d.Apps {
 		reason := trimRepeatedUsage(a.Reason, d.TargetPath)
 		if reason == "" {
 			reason = "(no reason recorded)"
 		}
 		fmt.Fprintf(&b, "  %-*s  %-*s  %s\n", width, a.App, legWidth, a.Leg, reason)
 	}
 	// The space figures answer "is this one broken app or a full disk" before the reasons are read.
 	// An absent reading renders as unavailable, never as zeros — "0 GB free" and "we could not look"
 	// are opposite diagnoses (the UnitSpace rule, same reasoning, other side of the wire).
 	if d.SpaceKnown {
 		fmt.Fprintf(&b, "\nFilesystem: %s — %.1f/%.1f GB used (%.0f%%), %.1f GB free\n",
 			d.TargetPath, d.UsedGB, d.TotalGB, d.UsedPercent, d.AvailGB)
 	} else {
 		fmt.Fprintf(&b, "\nFilesystem: %s — usage unavailable (the filesystem could not be read)\n", d.TargetPath)
 	}
 	b.WriteString("\nEvery failure above is also recorded individually in the notification log,\n")
 	b.WriteString("whether or not this mail was sent.")
 	return b.String(), subject, true
 }
 // trimRepeatedUsage strips the trailing "— /path: X/Y GB used (Z%), W GB free" clause from a per-app
 // reason, because the digest prints those figures ONCE for the whole run on its own line.
 //
 // This is a copy fix, and it was made after reading the first real digest rather than from the
 // design. The reserve's refusal message is authored for a single-app alert, where naming the
 // filesystem is exactly right; repeated down a list of a dozen apps it is the same forty characters
 // twelve times, and it pushes the part that differs off the right-hand edge of a phone screen at
 // 07:00 — which is the only moment this mail has to work.
 //
 // It trims ONLY an exact "— <target path>:" suffix, so a reason that mentions a different path, or
 // none, is left completely alone. A reason that is nothing but the usage clause is left alone too:
 // removing everything would turn a bad line into an empty one.
 func trimRepeatedUsage(reason, targetPath string) string {
 	if reason == "" || targetPath == "" {
 		return reason
 	}
 	marker := " — " + targetPath + ":"
 	i := strings.LastIndex(reason, marker)
 	if i <= 0 {
 		return reason
 	}
 	return strings.TrimSpace(reason[:i])
 }
@@ -815,7 +815,7 @@ type NotificationLogEntry struct {
 	EventType    string
 	Severity     string
 	Message      string
-	Status       string // "sent", "skipped", "failed"
+	Status       string // "sent", "skipped", "failed", "suppressed" (R-182: a cooldown drop, recorded rather than silent)
 	ErrorMessage string
 	Channel      string // "operator" or "customer"
 	CreatedAt    time.Time
@@ -125,7 +125,7 @@ spec:
    spec:
      containers:
        - name: hub
-          image: gitea.dooplex.hu/admin/felhom-hub:0.89.0
+          image: gitea.dooplex.hu/admin/felhom-hub:0.91.1
          ports:
            - containerPort: 8080
              name: http
@@ -1,3 +1,32 @@
 ## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185)
 **`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the
 storage and calls `felhom-backup-target-apply grant` in the same breath — a box that builds its own
 target has always been fine. The **Scenario-F arm** — *"the target already exists, leave it exactly as
 it is"* — returned without ever granting.
 So a box whose `felhom-backup` pre-dated the install (created by the vzdump-target-move runbook, or
 surviving a reinstall) ended up with `local_backup_target: felhom-backup` while its token held
 `FelhomAgentStore` on only `local`, `local-lvm` and `felhom-pbs`. Measured on **both** demo boxes
 2026-08-03: the content API answers `{"data":[]}` through the agent's token while root lists three
 archives. That tier was invisible to the agent and never restore-tested — and nothing said so,
 because an empty listing is also what a brand-new tier returns.
 The reuse arm now ensures the ACL through the same guarded wrapper, so both arms leave the box in the
 same state. **Scenario F is unviolated:** the storage DEFINITION is still untouched — granting the
 role the agent is supposed to have on the target this script is about to write into `agent.json` is
 finishing the job, not retargeting the box. `pveum acl modify` is idempotent, so a box that already
 has the grant is unchanged and a box whose token was rotated gets it back.
 `$BACKUP_TARGET_ID` is deliberately **not** added to `PVE_STORAGES`, and the comment now says why: that
 list is granted in step 4/5, before the target has been resolved in step 6, and `--acl-storages`
 entries are preflight-checked for existence. The grant belongs with the resolution, which is where it
 already was for a newly created target.
 **A gate now asserts it** (`hostinstall_gates.py`): every arm of `configure_backup_target` that
 resolves the target must also grant on it. Red-proved by reverting the reuse arm — `resolves the
 backup target in 2 place(s) but grants in only 1`.
 ## v1.23.0 — the installer is published, not pushed (2026-08-03, R-110 + R-183)
 **Two channels moved off `main` in the same change, because either one left behind makes the other
@@ -184,7 +184,7 @@
 set -euo pipefail
-SCRIPT_VERSION="1.23.0"   # the SINGLE version source (F-1): -h and the run banners follow it.
+SCRIPT_VERSION="1.24.0"   # the SINGLE version source (F-1): -h and the run banners follow it.
                          # The hub used to carry a copy for its Setup tab; R-94 DELETED it
                          # (2026-08-02) because the hub cannot know which version a box runs —
                          # the Setup command fetches this script at run time. scripts/
@@ -307,6 +307,14 @@ PVE_PRIVS_STORE="Datastore.Allocate Datastore.AllocateSpace"
 PVE_PRIVS_BASE="Sys.Audit SDN.Use Datastore.Audit"
 # Storages the agent reads/writes (archive+dump=local, restore=local-lvm, offsite DR=felhom-pbs). The
 # offsite felhom-pbs MUST be included or the agent's DR backup 403s (SPIKE residual #1). --acl-storages overrides.
 #
 # `$BACKUP_TARGET_ID` is deliberately NOT in this list, and the reason matters because the obvious
 # tidy-up is to add it (R-185, 2026-08-03). This list is granted in STEP 4/5, before
 # configure_backup_target has run in STEP 6 — so at this point the target may not exist yet, and
 # --acl-storages entries are preflight-checked for existence. The target's grant therefore belongs
 # with the target's RESOLUTION, where it already is for a freshly created one, and now also for a
 # pre-existing one. Adding it here would grant on a storage that may not exist and would still leave
 # the resolution path as the single owner of that decision, split across two places.
 PVE_STORAGES=(local local-lvm felhom-pbs)
 # E-2: the whole-guest backup target storage id, and what configure_backup_target resolved to.
 # BACKUP_TARGET_RESOLVED feeds agent.json's local_backup_target; "local" means DEGRADED (Case B).
@@ -652,8 +660,31 @@ configure_backup_target() {
    # existing id is an error, and repointing a live target is exactly the silent retarget this
    # whole arc closes.
    if pvesm status --storage "$BACKUP_TARGET_ID" >/dev/null 2>&1; then
-        log_skip "  backup target '$BACKUP_TARGET_ID' already exists — leaving it exactly as it is (Scenario F)"
+        log_skip "  backup target '$BACKUP_TARGET_ID' already exists — leaving its DEFINITION exactly as it is (Scenario F)"
        BACKUP_TARGET_RESOLVED="$BACKUP_TARGET_ID"
        # R-185: …but STILL ensure the ACL. "The storage already exists" says nothing about whether
        # the agent may READ it, and this early return is where the two came apart.
        #
        # THE DEFECT THIS CLOSES, measured on both demo boxes 2026-08-03. The CASE A path below
        # creates the storage and grants in the same breath, so a box that built its own target is
        # fine. A box whose target ALREADY existed — created by the vzdump-target-move runbook, or
        # surviving a reinstall — returned here and never granted. The result: `local_backup_target`
        # pointed at `felhom-backup` while the token held FelhomAgentStore only on local, local-lvm
        # and felhom-pbs, so the API answered `{"data":[]}` for that storage while root saw three
        # archives. The tier was invisible to the agent and never restore-tested, and nothing said so
        # — because an empty listing is also what a brand-new tier returns.
        #
        # Scenario F is UNVIOLATED: the storage definition is still untouched. Granting a role the
        # agent is supposed to have on the target this same script is about to write into
        # agent.json is not "touching the box's target", it is finishing the job. `pveum acl modify`
        # is idempotent, so a box that already has the grant is unchanged, and a box that had its
        # token rotated gets it back.
        if $DRY_RUN; then
            log_dry "felhom-backup-target-apply grant $BACKUP_TARGET_ID   # R-185: ACL on a pre-existing target"
        else
            /usr/local/sbin/felhom-backup-target-apply grant "$BACKUP_TARGET_ID" \
                || die "backup target grant failed on the pre-existing target — the agent could not read its own backup tier (R-185)"
        fi
        return 0
    fi
    local mp
@@ -202,6 +202,37 @@ if wp is not None:
        fail("manifests/webpage.yaml no longer tracks main for the website — pinning the SITE to "
             "the installer tag turns every copy edit into a release")
 # ── R-185: every path that RESOLVES the backup target must also grant on it ──────────────────
 #
 # THE DEFECT THIS WOULD HAVE CAUGHT, measured on both demo boxes 2026-08-03. `configure_backup_target`
 # has two arms. The CASE A arm creates the storage and grants in the same breath. The Scenario-F arm —
 # "the target already exists, leave it alone" — returned WITHOUT granting, so a box whose target
 # pre-dated the install pointed `local_backup_target` at a storage its own token could not read. The
 # API answered `{"data":[]}` while root saw three archives, and nothing said so, because an empty
 # listing is also what a brand-new tier returns.
 #
 # The assertion is deliberately about the FUNCTION, not about PVE_STORAGES: the target's grant belongs
 # with the target's resolution (PVE_STORAGES is granted a step earlier, before the target exists), so
 # what must hold is that no arm of that function can resolve a target and skip the grant.
 fn = re.search(r'^configure_backup_target\(\)\s*\{(.*?)^\}', src, re.S | re.M)
 if not fn:
    fail("cannot find configure_backup_target() — the backup-target ACL assertion cannot run, and a "
         "check that cannot run must never report OK (R-185)")
 else:
    body = fn.group(1)
    resolutions = len(re.findall(r'BACKUP_TARGET_RESOLVED="\$BACKUP_TARGET_ID"', body))
    grants = len(re.findall(r'felhom-backup-target-apply grant', body))
    if resolutions == 0:
        fail("configure_backup_target no longer resolves BACKUP_TARGET_ID anywhere — re-read it")
    elif grants >= resolutions:
        ok("every arm that resolves the backup target also grants on it (%d resolution(s), %d grant(s))"
           % (resolutions, grants))
    else:
        fail("configure_backup_target resolves the backup target in %d place(s) but grants in only %d "
             "— an arm resolves a target the agent may not READ. That is R-185: the tier's archives are "
             "invisible to the agent, it is never restore-tested, and an empty listing looks exactly "
             "like a brand-new tier." % (resolutions, grants))
 print()
 if fails:
    print("hostinstall gates: %d FAILURE(S)" % len(fails))
Author	SHA1	Message	Date
admin	688470c945	installer 1.24.0: a PRE-EXISTING backup target is granted too (R-185) gates / gates (push) Successful in 7s Details configure_backup_target has two arms and only one granted. Case A creates the storage and grants in the same breath; the Scenario-F arm ('the target already exists') returned without granting. A box whose felhom-backup pre-dated the install therefore pointed local_backup_target at a storage its own token could not read — measured on BOTH demo boxes: {"data":[]} through the token while root lists three archives. That tier was never restore-tested and nothing said so, because an empty listing is also what a brand-new tier returns. The reuse arm now ensures the ACL through the same guarded wrapper. Scenario F is unviolated: the storage DEFINITION is untouched, and pveum acl modify is idempotent. BACKUP_TARGET_ID is deliberately NOT added to PVE_STORAGES — that list is granted a step before the target is resolved, and --acl-storages entries are preflight-checked for existence; the comment now says so. A gate asserts it: every arm that resolves the target must also grant on it. Red-proved by reverting the arm.	2026-08-03 18:58:49 +02:00
admin	7a5694341d	REPORT: point at the agent report for the later 2026-08-03 session (docs-only here) gates / gates (push) Successful in 9s Details	2026-08-03 16:49:17 +02:00
admin	81d4438630	docs: R-189, R-188 and R-186 closed (agent v0.122.0); R-185 stays open gates / gates (push) Successful in 8s Details - OPEN-ITEMS: all three rows closed with their measurements. R-189 records the migration consequence seen live (a pre-R-189 record has no tier and is not reportable, so upgrading does not retroactively make an old proof visible). R-188 records both directions demonstrated and the fixture cleaned up. R-186 carries the before/after sha values including the CGO_ENABLED discrepancy. The priority list states explicitly that R-185 is the one left open from that group and that this session did not touch it. - capability map: the restore-proof row now says the evidence path itself had a gap — the proof reached the hub only because no restart intervened — and what closed it. - CONTEXT: S-19 (a proof is durable and reportable, a failure is neither, and why) and S-20 (the release ordering, what each step protects, and how a binary is verified). - STATUS: rewritten for the operator, back to one screen.	2026-08-03 16:46:49 +02:00
admin	c9a3e48b21	R-189: a passing restore-test can be invisible to the hub — observed live gates / gates (push) Successful in 8s Details The agent reports restore_tests[] from an IN-MEMORY store ('lost on restart; the cadence re-populates'), and the hub report interval is 900s. A restart inside that window drops the result. That used to self-heal within 24h because the next cadence re-tested the tier; under per-archive due-ness the agent will NOT re-test a proven archive, so the hub can stay ignorant until the next archive generation — a week on the offsite tier. Observed today: the 15:25:14 PASS on felhom-pbs (a real 14.5 GB offsite restore) reached no host-report at all, because the agent was restarted 2m43s later for the v0.121.1 deploy. The persisted proof already exists — RestoreTestState.Snapshot() even carries the comment 'for the host-report gauge' and has NO production caller: a seam built and never wired, and an invariant asserted in a comment with no test, in one method.	2026-08-03 15:35:17 +02:00
admin	0779c5db85	REUSE: the restore-proven window is per tier, and observation may only widen it (R-86) gates / gates (push) Successful in 7s Details	2026-08-03 15:33:19 +02:00
admin	0f8f084817	R-188: a correct agent release emails a CI failure ~50% of the time gates / gates (push) Successful in 8s Details on: [push] fires the gates workflow on the TAG push too, and release-agent.sh pushes the tag before publishing (deliberately). CI can therefore run the published-versions gate inside the window where the tag exists and the package does not, and correctly reports 'every released agent version must be INSTALLABLE' for a release that completes seconds later. Measured across two releases in one session: v0.121.0 -> runs 12/13 (success, failure) on the same sha; v0.121.1 -> runs 17/18 (failure, success) on the same sha; and one pair both green — a race, not a rule. It matters because R-168 made CI email on failure so a red gate cannot be missed. A signal that cries wolf on every second correct release is how that mail becomes something you archive unread.	2026-08-03 15:29:59 +02:00
admin	c102832892	docs: R-86 closed and proven live; ep0 recorded as protected; R-185/186/187 filed gates / gates (push) Successful in 8s Details - OPEN-ITEMS: R-86 CLOSED with the trap in its own wording recorded (the literal reading is never true on a daily tier); R-87 re-ranked UP because R-86 built most of what it waited for; R-185 (the agent cannot list demo-felhom's host backup tier — a missing storage ACL, pre-existing), R-186 (a released binary's sha is not reproducible from its tag), R-187 (R-115's publish leg had never actually run) filed. R-184 was the highest ID in use. - ROADMAP: R-86 collapsed, keeping the reasoning and correcting the shape the row itself proposed — which would have been the never-fires version. - 07-backup-architecture: new contract section — restore-testing is per ARCHIVE GENERATION, with the trap and what did not change (S-1). - 00-capability-map: the unattended restore-proof row upgraded to PROVEN-LIVE on the 635 s due-triggered offsite run, with the restart and teardown evidence. - CONTEXT: S-17 (the rule, the trap, the config key, the hub's derivation) and S-18 (ep0 is Tier 2 — extends D-d's protected list to three machines). Numbered 17/18 because S-14 and S-15 were already duplicated in the file. - STATUS: rewritten for the operator, trimmed back to one screen.	2026-08-03 15:28:32 +02:00
admin	ff2655cf19	manifests: hub 0.91.0 -> 0.91.1 (R-86: observation may only widen a tier's window) gates / gates (push) Successful in 7s Details	2026-08-03 15:18:00 +02:00
admin	046df303b6	hub v0.91.1 — observation may only WIDEN a tier's window, never tighten it (R-86) gates / gates (push) Successful in 7s Details Found by checking v0.91.0 against the live box, not by review. demo-felhom's two retained PBS snapshots sit 8h54m apart (one is a healing artefact), so the mean-gap estimator reads a WEEKLY tier as nine-hourly: x4 = 36h, the 7-day floor lifts it to 168h, and a weekly tier proved weekly reaches ~8.25d of proof age. The false alarm this task exists to prevent would have returned within a week, on the box it had just shipped to. restoreProvenWindow now takes max(observed, declared). A gap SHORTER than the declared rhythm is routine and means nothing (a retry, a manual run, a heal, a catch-up); a gap LONGER than it is real information. Cost stated: a tier running faster than its declared rhythm gets a slower stale signal — the right direction for a signal that means 'unverified', since 'broken now' is a different event.	2026-08-03 15:16:59 +02:00
admin	687fedd8ee	manifests: hub 0.90.1 -> 0.91.0 (R-86 Part 2, per-tier restore-proven window) gates / gates (push) Successful in 7s Details	2026-08-03 15:07:52 +02:00
admin	323f45a5ef	hub v0.91.0 — the staleness window learns each tier's own rhythm (R-86 Part 2) gates / gates (push) Successful in 7s Details Ships WITH agent v0.121.0, not after it. The agent now proves a tier once per ARCHIVE GENERATION, so a weekly tier is proved weekly — in perfect health. The flat 7-day restoreProvenStaleAfter derived its number from the 24h cadence R-86 removes, and a healthy weekly tier's proof age reaches EXACTLY 168h just before its next proof: it sat ON the line, so any ordinary delay tipped it into a nightly alarm about a working system. restoreProvenWindow(tier, observed, ok): - the tier's own archive interval, OBSERVED from reports the hub already holds (pbs_snapshots + successful backups attributed by TARGET TYPE, slice A.4) - x4 generations = the same tolerance the flat constant expressed - floored at 7d (never tighter than before), capped at 12d (strictly inside the 2-week offsite retention) - falls back to the DECLARED rhythm (26h host / 8d offsite — the thresholds the backup-freshness checker already uses) when history is too short to observe one; falling back to the FLOOR would recreate the false alarm on a fresh box Kept: absence is UNKNOWN until the anchored window passes; the signal stays edge-triggered; failed and stale remain distinct events. Every reason string now states the window it was judged against (R-100's corollary). Also backfills the missing v0.90.1 CHANGELOG entry (deployed since `f21e7ca`), and records the operator's 2026-08-03 ruling that ep0 is Tier 2 / protected.	2026-08-03 15:03:35 +02:00
admin	e34b614e5b	docs: R-182 closed, R-90 closed on measurement, R-86 unblocked, ep0 record corrected gates / gates (push) Successful in 7s Details R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The hub's notification_log for the run reads: two per-app failures RECORDED, one digest SENT naming both, and the customer channel SKIPPED with operator_only. Against the measured previous behaviour — two failures, one email naming one app, one leaving no trace anywhere. Scenario D proved itself on an event I had not planned: disk_critical alarmed on two filesystems, the second was collapsed by the cooldown, and that collapse is now visible WITH ITS KEY. Yesterday it would have left nothing at all. A gap the spec did not anticipate is recorded with its fix: the per-app event also fires from the periodic sweep, outside any run, so making it record-only would have created a NEW silence. The sweep emits a digest too, with no run_id, so it stays under the ordinary hourly cooldown. ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile SURVIVED the resize and is active (checked, because a resize is a stop/start). The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere. Five documents corrected — three of which the task's list did not name, found by searching. Two audit/evidence documents ANNOTATED, body untouched: they record what was true when written and that is their value. R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable, not unbounded — the original OOM was a 14.46 GB restore — so the restore-test cadence should still be paced, just not by fear of the endpoint. target-selection.md's "D-d did not name ep0 either way" is deliberately left standing. It is the operator's question, not CC's. STATUS.md 127 -> 83 lines, items rather than sentences.	2026-08-03 14:00:54 +02:00
admin	f21e7caed1	hub v0.90.1 — the digest's per-app lines stop repeating the filesystem figures (R-182) gates / gates (push) Successful in 7s Details Found by reading the first REAL digest, not by design. Every app row ended with the same usage clause the mail already prints once on its own Filesystem line. On a two-app box that is untidy; down a list of a dozen it is the same forty characters twelve times, pushing the part that DIFFERS off a phone screen at 07:00 — the only moment this mail has to work. The reserve's refusal message is authored for a single-app alert where naming the filesystem is right, so the message is unchanged; the digest trims the duplicate when rendering. trimRepeatedUsage removes ONLY an exact "— <target path>:" suffix, so an unrelated reason is untouched and a reason that is nothing but the usage clause is left alone rather than emptied. Also updates TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer, which required the OPERATOR to be emailed a per-app capture failure. That was correct when the event was the only signal and is wrong now that it is the record and the digest is the notification. Its customer-safety claim is unchanged and is why the test still exists; the operator assertion is inverted with the reasoning written in place, and R-158's guarantee is shown to have MOVED, not weakened.	2026-08-03 13:54:02 +02:00
admin	dd40f85bb8	hub v0.90.0 — a dropped notification leaves a trace, and the backup digest arrives (R-182) gates / gates (push) Successful in 7s Details processOperator's cooldown no longer returns bare. It dropped the event BEFORE LogNotification, so a suppressed operator alert and an event that never happened were indistinguishable — from the operator's side and from the hub's own records. Measured 2026-08-03: nine recovery_unit_capture_failed events arrived, two were mailed, seven left no row anywhere. That is why the defect took a day to get the right way round: there was nothing to read. A suppressed operator event now writes a `suppressed` row carrying the message and the key that suppressed it. This applies to EVERY operator event, not only the one that exposed it. It does NOT change the cooldown's duration or semantics. backup_run_failures: the per-run digest. In allowedEventTypes AND in operatorOnlyEvents — allowlisting alone does not make an event operator-only, and FormatCustomerEmail falls back to the raw English message rather than blocking. A test demonstrates a customer with the type enabled receiving nothing. recordOnlyEvents: a third routing class — stored and recorded, never mailed. recovery_unit_capture_failed moves here: it is the record, the digest is the notification. A register rather than downgrading severity to info, which would relabel a genuine failure as informational everywhere it is queried. cooldownRunSuffix: a sibling of cooldownTierSuffix, not a branch inside it, so tier keeps byte-identical semantics and R-97a's tests are untouched. It makes the cooldown effectively inert for the digest, which is the intent — a digest is already rate-limited by construction; the refresh sweep sends no run_id and so stays under the ordinary hourly cooldown. The email renders as a list, not a JSON blob. An absent space reading renders as unavailable, never as zeros.	2026-08-03 13:46:48 +02:00
admin	7dc1744eec	Merge branch 'main' of https://gitea.dooplex.hu/admin/felhom.eu gates / gates (push) Successful in 7s Details	2026-08-03 13:14:17 +02:00
admin	a5cd480280	reports deleted	2026-08-03 13:14:15 +02:00