RUNBOOK: the host tier is restore-PROVEN unattended on both demo boxes

Four SCHEDULED runs, none triggered by hand: demo-felhom host 83.8s / offsite
540.4s; demo-hp host 109.3s / offsite 300.1s. Each restored into a scratch guest,
booted, verified and destroyed itself; zero 990000 guests or volumes afterwards
and both local-lvm figures returned to their pre-run values.

Both boxes had BOTH tiers due at once, so R-86's ordering was observed live for
the first time: never-proven sorted first, each box took its HOST tier, deferred
the offsite one, and picked it up on the next evaluation six hours later. The
host-tier proofs reached the hub through R-189's merge — demo-felhom's report
carries two tiers, and the local one can only have come from disk.

The capability map's optimistic half is cashed, with its scope stated: these two
boxes, not the fleet.

Surfaced and filed rather than fixed:
- R-190: a storage ACL that demonstrably worked at 04:44 was gone by 09:24, with
  a reinstall, any logged pveum activity and any cluster-log entry ruled out.
- R-191: every weekly offsite backup uploads successfully and then fails the job
  on a prune the box is deliberately not allowed to do (R-89 moved it
  server-side; both boxes still arm keep_last=2).

Two corrections to yesterday's record: the R-185 drift DID surface as 403s on the
write path (six, with the hub raising whole_guest_backup_failed at the first), and
my earlier "no restore_test_* events" was produced by grepping a 404 page.

CONTEXT.md

@@ -17,6 +17,27 @@
 
 ## Standing rulings
 
+**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
+(2026-08-04). Scope: those two boxes, not the fleet.**
+
+Four SCHEDULED runs overnight, none triggered by hand: demo-felhom host **83.8 s** / offsite
+**540.4 s**; demo-hp host **109.3 s** / offsite **300.1 s**. Every one restored into a scratch guest,
+booted, verified and destroyed itself.
+
+*What this closes.* Until yesterday every live restore-proof this project held was on the OFFSITE
+tier. The on-box tier — the one an ordinary recovery uses — had never been proven on either box, and
+not because it failed: the agent could not read the storage it lives on (R-185), so it never saw an
+archive there to test.
+
+*What was observed for the first time.* Both boxes had BOTH tiers due simultaneously. Never-proven
+sorts first, so each took its host tier, deferred the offsite one, and picked that up on the next
+evaluation six hours later — R-86's ordering and the one-heavy-operation gate, working together,
+unsupervised. The host-tier proof then reached the hub through R-189's merge, which is that path
+carrying a host-tier entry for the first time.
+
+*The asymmetry worth remembering:* a host-tier restore is **83–109 s**; an offsite one is
+**300–540 s**. The tier that matters for an ordinary recovery is also the cheapest to prove.
+
 **S-21 — an empty listing cannot distinguish FORBIDDEN from NEWBORN, so the box asks the permission
 question directly (2026-08-03, R-185; agent v0.123.0 + installer 1.24.0).**
 

REPORT.md

@@ -1,137 +1,173 @@
-# REPORT — R-86 Part 2: the staleness window learns each tier's rhythm (2026-08-03)
+# REPORT — RUNBOOK: the first host-tier restore-test, on both boxes
 
-> **A LATER SESSION THE SAME DAY (R-189 · R-188 · R-186) touched this repo for REGISTERS AND DOCS
-> ONLY** — `OPEN-ITEMS.md`, `STATUS.md`, `CONTEXT.md` (S-19, S-20) and the capability map. **No hub
-> code changed and the hub was not bumped**: the defect was the agent no longer sending
-> `restore_tests[]`, not the hub failing to read them. That session's report is
-> `felhom-agent/REPORT.md`. What follows is the last hub IMPLEMENTATION, v0.91.1, which is what this
-> file is for.
+**Date:** 2026-08-03 → 2026-08-04 · **Repos:** `felhom.eu` docs + registers only. **Nothing was built
+and no version was bumped.** The only binary that moved is the already-published `v0.123.0`, onto the
+box that did not have it (P1). Baselines re-read and matched: `felhom-agent` `72161f6cf010` /
+`v0.123.0`; `felhom.eu` `e3187c86d58d` / hub `v0.91.1`, installer `1.24.0`. Constants re-confirmed at
+source: `defaultRestoreTestEvalInterval` **6 h**, `defaultRestoreTestSettle` **24 h**.
 
-**Date:** 2026-08-03 · **Repo:** `felhom.eu` hub **v0.90.1 → v0.91.0 → v0.91.1**
-(`323f45a`, `046df30`), manifests `687fedd`, `ff2655c`. Ships **with** `felhom-agent` v0.121.0, not
-after it. Ruling recorded: **ep0 is Tier 2, protected**.
+**Outcome: four scheduled runs, all passed, nothing triggered by hand.**
 
-## 1. Baselines, re-read on arrival
+---
 
-| Repo | `main` @ commit | Version | Matched §1? |
-|---|---|---|---|
-| `felhom.eu` | `e34b614e5b65` | hub CHANGELOG top entry `v0.90.0`; `manifests/hub.yaml` ran **`0.90.1`** | **Yes — and the discrepancy was real** |
+## 1. Preconditions
 
-The missing `v0.90.1` CHANGELOG entry is backfilled. It was built, deployed and recorded in
-`REPORT.md` + the R-182 row on 2026-08-03 (`f21e7ca`) but never given a CHANGELOG entry — and
-`REPORT.md` is overwritten every session, so the per-repo history under-reported what was running.
+| # | demo-felhom | demo-hp |
+|---|---|---|
+| **P1** agent | `v0.123.0` ✓ | **`0.120.0` — below the 0.121.0 floor, so it could not become due at all.** Remediated with the published `v0.123.0` (sha `74910135…`, deployed sha identical) |
+| **P2** tiers | host `felhom-backup` + offsite `felhom-pbs` (weekly) | **the same — the runbook expected demo-hp to have no offsite tier, and it has one**, active with 2 snapshots |
+| **P3** grant | `ok=70 total=70 degraded=0` | `ok=70 total=70 degraded=0` (once the probe existed) |
+| **P4** storage | `/dev/sdb → /mnt/hdd_1`, ext4, on the N100 | `/mnt/nvme-1tb` on the t740 — **different hosts, different disks ⇒ INDEPENDENT ⇒ parallel is safe** |
+| **P5** space | target 889 GB free; restore pool `local-lvm` 358 GB | target 925 GB free; restore pool `data` **53.9 G at 30.79 %** (~37 GB free) against a 2.35 GB archive — adequate, and measured *because* that pool is the over-subscribed one |
+| **P6** candidate | `…2026_08_02-04_42_14.tar.zst` (08-02) | `…2026_08_02-04_49_29.tar.zst` (08-02) — both correctly the settled archive, not the day's |
+| **P7** other heavy work | daily backup ~04:44, outside the window | daily backup ~04:49, outside the window |
 
-## 2. Why Part 2 could not ship later
-
-The agent now proves a tier **once per archive generation**, so a tier backed up weekly is proved
-weekly — correctly, in perfect health. `restoreProvenStaleAfter` was a flat 7 days and its own comment
-derived that number *from the cadence R-86 removes*.
-
-The brief said a weekly tier would "sit exactly on that line". That is literally true, and the
-arithmetic is worth stating: proofs land at `archive + settle`, so just before the next proof the
-current one is **exactly one interval old** — 168 h against a 168 h window, and `age > window` is false
-by a hair. It did not sit near the line, it sat **on** it. Any ordinary delay — a backup landing late,
-one evaluation deferred behind a running backup — tips a healthy tier into a nightly alarm.
-
-## 3. The window as derived
+## 2. The due verdicts before the run, quoted
 
 ```
-interval = declaredArchiveInterval(tier)                        // host 26h, offsite 8d
-if observedOK && observed > interval { interval = observed }    // observation may only WIDEN
-window   = clamp(4 * interval, floor 7d, cap 12d)
+demo-felhom  tier=felhom-backup due=true  archive="…2026_08_02-04_42_14.tar.zst"
+               reason: newest settled archive … has not been proven; nothing proven on this tier yet
+             tier=felhom-pbs    due=true  archive="…2026-07-28T04:49:43Z"
+               reason: … has not been proven (last proven archive was a different one)
+
+demo-hp      tier=felhom-backup due=true  archive="…2026_08_02-04_49_29.tar.zst"
+             tier=felhom-pbs    due=true  archive="…2026-07-28T19:19:45Z"
 ```
 
-- **4 generations** — the settle generation plus ~3 missed opportunities: deliberately the same
-  tolerance the flat constant expressed. The change is to the *rhythm*, not to the patience.
-- **The declared rhythm is `backupStaleAfter` (26 h) / `offsiteBackupStaleAfter` (8 d)** — the very
-  thresholds the backup-freshness checker already judges those tiers against. Not a second opinion:
-  if the two checkers disagreed about how often a tier receives an archive, one would be alarming on
-  the other's model.
-- **The observation** comes from reports the hub already holds: `pbs_snapshots[]` plus successful
-  `backups[]` attributed by **target type** (slice-A.4 — a PBS-targeted vzdump appears in *both*
-  arrays, and classifying by array membership would attribute an offsite archive to the host tier).
-  Mean gap over distinct archive times; fewer than two generations ⇒ unobservable.
-- **Floor 7 d** — the old constant kept as a floor, so no tier is judged more tightly than before.
-- **Cap 12 d** — strictly inside the 2-week offsite retention with two days to spare, so a tier is
-  never called stale against an archive PBS has already pruned.
+**Both boxes had BOTH tiers due**, which made §4's ordering question live rather than theoretical.
 
-| tier | interval used | window | proof age in health | verdict |
+## 3. The runs — all four SCHEDULED, none triggered
+
+| box | tier | due at | archive | result |
 |---|---|---|---|---|
-| daily host | 26 h declared | 4×26 h → **floor 168 h** | ≤ ~54 h | OK — numerically unchanged from before |
-| weekly offsite | 8 d declared (7 d observed loses to it) | 4×8 d → **cap 288 h** | ≤ ~198 h | OK, 3.75 d of margin |
-| newborn offsite | 8 d declared | **288 h** | — | UNKNOWN until the anchor passes |
+| demo-felhom | **host** | 00:55:21 | `…2026_08_02-04_42_14.tar.zst` | **passed, 83.8 s**, scratch torn down 00:56:45 |
+| demo-felhom | offsite | 06:55:21 | `…2026-07-28T04:49:43Z` | **passed, 540.4 s**, torn down 07:04:21 |
+| demo-hp | **host** | 02:05:39 | `…2026_08_02-04_49_29.tar.zst` | **passed, 109.3 s**, torn down 02:07:28 |
+| demo-hp | offsite | 08:05:39 | `…2026-07-28T19:19:45Z` | **passed, 300.1 s**, torn down 08:10:39 |
 
-## 4. v0.91.1 — a flaw v0.91.0 shipped with, found by checking the live box
+**No box failed to fire, so Phase C was not entered and no `--selftest` was used as a proof.** The
+only selftest invocations in this session were the read-only `restore-test-due` verdict prints in §2,
+which start nothing.
 
-demo-felhom's offsite tier holds two retained snapshots, `2026-07-27T19:55:41Z` and
-`2026-07-28T04:49:43Z` — **8 h 54 m apart**, because one is a healing artefact and the other a real
-weekly run. A mean-gap estimate therefore reads a **weekly** tier as nine-hourly: ×4 = 36 h, the floor
-lifts it to 168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm
-this whole task exists to prevent would have returned within a week, on the box it had just shipped
-to.**
+**§4's question, answered live:** each box took its **host** tier first — never-proven sorts ahead of
+proven, and ahead on the id tie-break — deferred the offsite one, and picked it up on the **following
+evaluation six hours later**. One heavy operation at a time, per box, with nobody sequencing it. That
+is R-86's oldest-proven ordering and the heavy-operation gate observed together for the first time.
 
-Fixed by `max(observed, declared)`. A gap *shorter* than the declared rhythm is routine and means
-nothing (a retry, a manual run, a heal, a catch-up after an outage); a gap *longer* than it is real
-information. **Cost, stated rather than hidden:** a tier that truly runs faster than its declared
-rhythm gets a wider window than it needs, i.e. a slower `restore_test_stale`. Right direction for a
-signal meaning *unverified* — *broken now* is `restore_test_failed`, immediate and untouched.
+**The asymmetry worth keeping:** a host-tier restore is **83–109 s**; an offsite one **300–540 s**. The
+tier an ordinary recovery uses is also the cheapest to prove.
 
-## 5. Kept, because it was earned
+## 4. What the runs left behind
 
-- **Absence is UNKNOWN** until an anchored window has passed (R-81's structure, unchanged).
-- **The stale signal stays edge-triggered** (`staleStates` untouched).
-- **`restore_test_failed` and `restore_test_stale` stay DISTINCT** — one says your recovery is broken,
-  the other that it is unverified, and the second is the one that quietly becomes the first.
-- **Every reason string now states the window it was judged against** — R-100's corollary: when a
-  verdict changes what it counts from, the alarm text must change with it, or an operator reads
-  "limit 168h" under a tier judged at 288 h and dismisses a true alarm.
-- **The window READ is unchanged in cost** (14 days) — enough to find proof inside the widest window
-  and to see two generations of a weekly tier.
+**Persisted state — v3, naming the archive, the tier and what was verified:**
 
-## 6. Tests and red-proofs
+```json
+demo-felhom  felhom-backup → {archive …2026_08_02-04_42_14.tar.zst, tier local, verified boot+running,
+                              proven_at 2026-08-03T22:56:45Z}
+             felhom-pbs    → {archive …2026-07-28T04:49:43Z,        tier pbs,   proven_at 2026-08-04T05:04:21Z}
+demo-hp      felhom-backup → {archive …2026_08_02-04_49_29.tar.zst, tier local, proven_at 2026-08-04T00:07:28Z}
+             felhom-pbs    → {archive …2026-07-28T19:19:45Z,        tier pbs,   proven_at 2026-08-04T06:10:39Z}
+```
 
-| Test | Asserts | Mutation | Observed |
-|---|---|---|---|
-| `TestRestoreTest_HealthyWeeklyTierIsNeverStale` | a healthy weekly tier **with jitter** is never stale across 6 weeks | window pinned flat at `restoreProvenWindowFloor` | **FAIL** — `week 0: … proof age 172h0m0s, window 168h0m0s … "limit 168h0m0s"` |
-| `TestRestoreProvenWindow_Contract` | floor, cap, declared fallback, **observation may only widen** | `observed > interval` → `observed > 0` | **FAIL** — `window(pbs, observed=8h54m0s ok=true) = 168h0m0s, want 288h0m0s` |
-| `TestObservedArchiveIntervals_FromReports` | rhythm observed per tier; a PBS vzdump is **not** counted into the host tier | — | pass |
-| `TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms` | a window that never fires is a deletion, not a fix | — | pass |
-| existing R-85/R-81 suite | anchored UNKNOWN, edge-trigger, distinct events, newborn silence | — | pass, unchanged |
+**The hub received the host-tier proofs — R-189's path carrying one for the first time.** demo-felhom's
+latest report holds **two** entries, one per tier; the `local` one can only have come from the
+persisted state, because the in-memory store held only that morning's offsite run:
 
-**A hollow test caught by its own red-proof.** The first Scenario-G fixture used a perfectly regular
-weekly tier and **PASSED under the flat-window mutation**, because a regular tier's proof age lands on
-exactly 168 h and `>` lets it through. It would have shipped Part 1 and its false alarm together while
-looking like proof. The jitter is what makes it a test — and it is also the truth about the old
-constant.
+```
+demo-felhom  tier=local pass=True archive=felhom-backup:…2026_08_02-04_42_14.tar.zst tested_at=2026-08-03T22:56:45Z
+             tier=pbs   pass=True archive=felhom-pbs:…2026-07-28T04:49:43Z          tested_at=2026-08-04T05:04:21Z
+demo-hp      tier=local pass=True archive=felhom-backup:…2026_08_02-04_49_29.tar.zst tested_at=2026-08-04T00:07:28Z
+```
 
-## 7. Deployment
+**A subsequent evaluation runs nothing on a proven tier**, quoted from demo-felhom now:
 
-| Step | Evidence |
-|---|---|
-| Images built + pushed | `felhom-hub:0.91.0`, then `0.91.1` |
-| Manifest bumped in git | `687fedd`, `ff2655c` — never `kubectl set image` |
-| ArgoCD deliberate sync | `sync=Synced health=Healthy` |
-| Running image | `gitea.dooplex.hu/admin/felhom-hub:0.91.1` |
-| Startup log | checkers initialized; `Listening on :8080` |
+```
+tier=felhom-pbs  due=false  proven="…2026-07-28T04:49:43Z"
+  reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
+```
 
-## 8. Part 3 — the ep0 ruling, recorded
+…while its **host** tier reads `due=true` again — on the **08-03** archive, which has now settled 24 h.
+That is not a defect: it is "proved daily, on its own archive", visible one day later.
 
-`runbooks/target-selection.md` had carried *"D-d did not name ep0 either way. Confirm it explicitly"*
-for two days. **Operator ruling, 2026-08-03: ep0 is protected.** Recorded three ways — the tier table
-names it, the standing question is gone, and its per-machine section is retitled **Tier 2, PROTECTED**.
-`CONTEXT.md` S-18 records that this **extends D-d's protected list to three machines** (DooPlex,
-Peti's cluster, ep0) and that it is a *classification*, not new prohibitions: destroying datastores,
-prune jobs, tunnel config or nftables rules was already forbidden by what it would destroy, and **the
-ordinary off-site read a restore-test performs remains permitted** — which this session then exercised.
+## 5. Teardown — three layers, per box
 
-## 9. Registers and docs
+| layer | demo-felhom | demo-hp |
+|---|---|---|
+| the machine | `pct list` → **0** entries for 990000 | **0** |
+| the host | `lvs` → **0** volumes for 990000; `local-lvm` **1.95 % → 1.95 %** | **0**; `local-lvm` 30.79 % → 40.86 % during the offsite run → **30.83 %** after |
+| the hub | the `restore_tests[]` entries are **RETAINED DELIBERATELY** — they *are* the proof the staleness check reads, so deleting them would delete the result | same |
 
-- **R-86 → CLOSED**, shipped + proven live, with the trap in its own wording recorded on the row.
-- **R-87 → re-ranked UP**: R-86 built most of what it waited for. What remains is restic-specific.
-- **R-185 / R-186 / R-187 → filed** (grep established 185–187 free; R-184 was the highest in use).
-- `ROADMAP.md` R-86 collapsed, keeping the reasoning and **correcting the shape the row itself
-  proposed** — which was the never-fires version.
-- `STATUS.md` rewritten for the operator, trimmed back to one screen.
-- `07-backup-architecture.md` §3 carries the new contract (S-1); `00-capability-map.md`'s restore-test
-  row upgraded with its live citation; `CONTEXT.md` gains S-17 (the rule + the trap + the config key)
-  and S-18 (ep0).
+Nothing else was created: no scratch customer, no fixture storage, no probe tag, no package version.
+
+## 6. What the run surfaced — three findings, two of them corrections to my own record
+
+### R-190 (new) — a storage ACL that worked in the morning was gone by mid-morning
+
+A `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50** on
+2026-08-03; the first `403 … missing privilege Datastore.Allocate` on that path is **09:24:56**, and by
+~14:50 `pveum acl list` held **no row at all** for it. Ruled out by measurement: a host reinstall
+(uptime 12 days), any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00, any cluster-log ACL entry.
+Correlated but not established: guest 9201 was reprovisioned at 09:15–09:19, nine minutes before.
+**A permission that can vanish silently makes every ACL-based guarantee on these hosts provisional**,
+and v0.123.0's probe detects the *state* but says nothing about the *transition*.
+
+### R-191 (new) — every weekly offsite backup reports FAILED although it worked
+
+demo-felhom, 06:49–06:53 today: the upload **succeeded** (223 s, 629 MiB of 1.874 GiB, 67.2 % reused
+incrementally) and the job then failed on the prune —
+`missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `TASK ERROR: job
+errors`, and the hub raised `whole_guest_backup_failed`. **The token behaves exactly as R-89 designed**
+(box tokens are write-only; ep0 prunes). What did not follow is the config: **both** boxes still arm
+the offsite tier with `keep_last=2 prune_pbs_allowed=true`, so every weekly run asks for a prune that
+must fail. The data is safe; the verdict and the weekly e-mail are wrong, which is the R-100 corollary
+— an alarm whose text is true and whose trigger is not the thing you would act on. **Not fixed here**
+(§6 rule 2), and the fix needs one check first: whether ep0's prune jobs actually cover these two
+namespaces.
+
+### Two corrections to yesterday's record
+
+1. **The R-185 drift was NOT silent on the write path.** demo-felhom's local-api backup jobs 403'd
+   **six times** (09:24 → 17:34) on that storage and privilege, and the hub raised
+   `whole_guest_backup_failed` at the first with edge-triggering suppressing the rest. My annotation
+   said backups kept landing because writes go through a root path — wrong, and now corrected in the
+   runbook and on the row.
+2. **My "no `restore_test_*` events at all" was an instrument error.** The hub has no `/events` route;
+   I grepped a **404 page**. Read from the events table: five such events exist (2026-07-27/28), none
+   since the R-86 work — and one **more** since, below.
+
+**One further event, correctly raised and worth stating:** `restore_test_stale` for demo-felhom at
+2026-08-03 22:33:42 UTC — **22 minutes before** the host-tier run. It was **true**: at that moment the
+host tier had never been proven, and the hub said so on its own, without being asked. It has not
+re-fired; the signal is edge-triggered, so the return to healthy is silent by design.
+
+## 7. The capability map
+
+The unattended restore-proof row now reads **PROVEN-LIVE for the host tier, unattended, on both demo
+boxes**, with the four runs, their durations, the deferred-tier ordering, the hub-side proof and the
+teardown cited — and it states its **scope explicitly**: `demo-felhom` and `demo-hp`. The tester's box
+is untested and untouched, and one box proving something does not make it a fleet property.
+
+## 8. Registers
+
+- **R-185** — its consequence is now demonstrated; the row already closed yesterday, and carries the
+  correction in §6.
+- **R-190**, **R-191** — filed. `grep` established R-190 and R-191 were free before minting (R-189 was
+  the highest in use).
+- `ROADMAP.md` holds none of these rows, so nothing to collapse.
+- `STATUS.md` rewritten for the operator and kept to one screen (85 lines); R-191 appears under
+  "What's broken" because it produces a weekly e-mail you would otherwise learn to ignore.
+
+## 9. Observations — noticed, NOT acted on
+
+- **demo-hp's `local-lvm` thin pool reached 40.86 %** during its offsite restore (from 30.79 %,
+  returning to 30.83 %). Comfortable, but that is the over-subscribed pool the target-selection notes
+  warn about, and the offsite archive is the larger of the two. A materially bigger guest would want
+  the restore pointed at `/mnt/nvme-1tb` instead.
+- **demo-hp has an offsite tier**, contrary to the runbook's §2 premise and to the note that it "has
+  none". Nothing depends on that assumption now, but the operations notes still carry it.
+- **The two boxes' daily archives are ~6.3 GB (demo-felhom) vs ~2.35 GB (demo-hp)** for the same guest
+  role — a 2.7× difference worth understanding before either is used to size anything.
+- **Both waiters this session produced no output** despite the runs completing; the evidence was
+  gathered by direct query afterwards. A watcher that silently produces nothing is exactly the
+  instrument class this project distrusts — the conclusions here rest on the boxes' own journals and
+  the hub's database, not on the waiters.

STATUS.md

@@ -20,12 +20,23 @@ Proven end to end on real hardware.
 
 ## What's broken
 
+- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
+  trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
+  you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
+  setting per machine fixes it. *(R-191)*
 - **The off-site copy can be erased by the machine that made it.** The credential that writes it can
-  also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
+  also delete it. A daily snapshot is armed as a stopgap.
   *(R-95, R-87)*
 
 ## What shipped recently
 
+- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both
+  demo machines restored their own on-machine backup into a throwaway machine overnight, booted it,
+  checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we
+  had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had
+  never been tested on either machine. Both also proved their off-site copy on the same night, one
+  after the other rather than at once, which is the machine deciding for itself what to do first.
+  *(closes the last open half of R-86/R-185)*
 - **A backup copy the machine was never allowed to read — and could not tell you about.** One demo
   machine kept its whole-machine backups on a dedicated storage area the agent had never been granted
   permission to read. Asked what was there it was told "nothing", while an administrator saw three
@@ -45,9 +56,6 @@ Proven end to end on real hardware.
   published after the binary, and a new check catches the opposite mistake so nothing is traded away.
   **(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
   approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
-- **Each backup is now proved, instead of the clock being obeyed** — tested once, about a day after
-  it is made, and not again until there is a newer one; the "not proved lately" alert learned each
-  copy's own rhythm in the same change. *(R-86)*
 
 
 ## What we're working on
@@ -69,12 +77,9 @@ Proven end to end on real hardware.
 
 ## Changed since last update
 
-- **2026-08-03** — Found and fixed a backup copy the machine was never permitted to read, on both demo
-  machines. The permission was one line; what mattered was that the machine now says so instead of
-  treating "I am not allowed" and "there is nothing here yet" as the same answer. *(R-185)*
+- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
+  overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
+  weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
 - **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
   that could vanish on a restart (seen happening), a release that emailed a failure for a release
   that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
-- **2026-08-03** — Backups are now proved one at a time, each about a day after it is made, instead of
-  on a timer; the "not proved lately" alert learned each copy's own rhythm. You settled that the
-  off-site endpoint is protected. *(R-86)*

documentation/backlog/OPEN-ITEMS.md

@@ -34,6 +34,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
 | **R-188** | **Every agent release has a ~50 % chance of emailing the operator a CI failure for a release that is correct.** `on: [push]` fires the gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag BEFORE publishing the package (deliberately — a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it). So CI can run the published-versions gate inside the seconds-long window where `v<ver>` exists and the package does not, and it correctly reports *"every released agent version must be INSTALLABLE"* — for a release that completes normally a moment later | **CLOSED — SHIPPED 2026-08-03** (agent **v0.122.0**) | — | **MEASURED, 2026-08-03, across two releases in one session:** v0.121.0 → runs #12 (id 79) **success** and #13 (id 81) **failure** on the same sha `4618169`; v0.121.1 → #17 (id 88) **failure** and #18 (id 89) **success** on the same sha `cd1b087`; and v0.121.0's second pair (#14/#15, `4d82591`) both **success** — i.e. it is a race, not a rule. Local gate afterwards: all three versions verify. **Why it matters more than it looks:** R-168 made CI email the operator on failure precisely so a red gate cannot be missed. A signal that cries wolf on every second correct release is how that mail becomes something you archive unread — the R-29 failure mode one layer up, and R-100's corollary in a new place (an alarm whose text is true but whose trigger is not the thing you would act on). **Fix shapes, cheapest first:** have the workflow skip tag refs (`if: !startsWith(github.ref, 'refs/tags/')` — the commit push already covers the content, and the tag adds no new code); or make `check-published-versions.py` tolerate a tag younger than ~2 minutes; or publish before tagging, which the release script's own comment argues against. **Do not "fix" it by quieting the alarm** **FIXED by moving ONLY the tag push** (build → tag locally → publish → push tag). The tag is still created before anything is published, so the build and the tag still describe the same commit; it becomes VISIBLE — to CI, and to any `raw/tag/…` fetch — only once the package is downloadable. **The invariant the old order protected was not traded away:** `check-published-versions.py` now asserts the converse directly — *no published version may be missing its tag* — as a BOUNDED probe of the frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set on every run because a check whose coverage is invisible reads as a guarantee it is not making. The package listing api still answers **401** without a token (re-measured 2026-08-03), so absence still cannot be enumerated and the script says so. **Both directions demonstrated:** a real published-but-untagged fixture (0.121.2, published then deleted — HTTP 201 then 204, absence re-verified by a 404) made the gate FAIL and print the one-line recovery; removing the converse block made it pass over that same orphan. A tag with no package still fails, shown against a local stand-in rather than by pushing a probe tag — pushing one would have emailed the operator a TRUE alarm about a fixture, which is the same attention cost this row removes. **A half-done release is now loud:** publish-succeeded-then-push-failed dies naming the exact recovery command, and a FAILED publish removes the local-only tag so the release can simply be retried instead of colliding with step 2's re-release guard. **Scenario E, live:** v0.122.0's own release produced CI runs **21 (task id 96)** and **22 (task id 97)** on the same sha — **both green**, where the same shape produced one red run on each of the two previous releases. | CC |
 | **R-189** | **A passing restore-test can be invisible to the hub forever — and R-86 made that window a week instead of a day.** The agent reports `restore_tests[]` from `internal/backup.Store`, which is **in-memory** (its own comment: *"lost on restart; the cadence re-populates"*), and the hub's report interval is 900 s. An agent restart inside that window drops the result. That used to self-heal within 24 h because the next cadence re-tested the same tier; **under per-archive due-ness the agent will NOT re-test a proven archive**, so the hub can stay ignorant of a proof until the NEXT archive generation — a week on the offsite tier. **Observed live, 2026-08-03:** the 15:25:14 PASS on `felhom-pbs` (a real 14.5 GB offsite restore) reached **no** host-report — the hub logged `0 restore-tests` at 15:28:00 and 15:32:14 — because the agent was restarted at 15:27:57 for the v0.121.1 deploy, 2 m 43 s after the run and before the next report | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.122.0**) | — | **The persisted proof already exists and is not reported.** `RestoreTestState` (on disk, per tier, R-86: with the archive id) survives exactly this, and `RestoreTestState.Snapshot()` carries the doc comment *"for the host-report gauge"* — **and has no production caller at all** (grep, 2026-08-03). That is this project's own *seam built but never wired* class, fifth instance, and a comment asserting an invariant with no test pinning it, both in one method. **Severity is bounded, and worth stating so it is not over-ranked:** the hub scans its retained window, archives keep arriving, and R-86's per-tier window for the offsite tier is 12 days against a 7-day archive rhythm — so ONE lost report is tolerated. What is not tolerable is that the loss is **silent** and that the box will not repeat the work. **Fix shape:** report the persisted per-tier proof (target, archive, proven_at) alongside the in-memory latest run, and let the hub's `lastProvenPerTier` read it — a report-contract change, so it touches `internal/hub/report.go` + `contract_test.go` + the hub's checker, and it is a spec, not a one-liner **FIXED.** `RestoreTestState` now stores the **tier** and what was **verified** beside the archive (recorded at proof time, from the run's own result — deriving them later would need a storage lookup that can fail on the one path where failing means mislabelling a proof), exposes `ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it with the in-memory result. **Merge rule: one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news and lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (two entries would read at the hub as two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **It refuses to lie:** a record missing the archive OR the tier produces NO entry, and run mechanics (scratch VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be. **The asymmetry is now written where it will be read:** only successes are persisted, because a success SUPPRESSES future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it believes) while a failure CAUSES future work and heals itself at the next evaluation. **`Store`'s comment was corrected in place** — *"lost on restart; the cadence re-populates"* was true under a timer and became false the moment R-86 shipped; leaving it is how the next reader concludes this is handled. **The seam that had no caller is now pinned by an AST test** (`TestMainWiresTheDurableRestoreTestProof`), because `Snapshot()` claimed a *"host-report gauge"* in its doc comment for weeks with nothing calling it. **MIGRATION, stated because it is visible on the live box:** a pre-R-189 record carries an archive but no tier and is therefore NOT reportable — upgrading does not retroactively make an old proof visible to the hub; the tier's next real proof fills it in. Observed on demo-felhom immediately after the upgrade: still `0 restore-tests` until a fresh run. | CC |
 | **R-190** | **A storage ACL that demonstrably WORKED in the morning was gone by mid-morning, and nothing recorded its removal.** On demo-felhom, a `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50 CEST 2026-08-03** (task log read in full). From **09:24:56** the same path returned `HTTP 403 … missing privilege Datastore.Allocate at /storage/felhom-backup`, six times through the day, until the grant was re-applied by hand at 18:54. By ~14:50 `pveum acl list` showed **no row at all** for that path | **OPEN** | — | **Why this is not just R-185 restated:** R-185's mechanism (the installer's Scenario-F arm resolves a pre-existing target without granting) explains a box that NEVER had the grant. This box HAD it and lost it, inside five hours, with the machine up throughout. **Ruled out, each by measurement:** a host reinstall (`uptime` = 12 days); any `pveum`/ACL/`user.cfg` activity in syslog between 04:00 and 10:00 (none); any ACL entry in `/cluster/log` (none). **Correlated, not established:** `host_leaf_changed` at 09:15 and `controller_started` at 09:19 — guest 9201 was reprovisioned nine minutes before the first 403. PVE removes ACLs at `/vms/<vmid>` when a guest is destroyed (`AccessControl::remove_vm_access`, the F-LEAK mechanism); whether any path can take a `/storage/<id>` row with it has NOT been established and is the first thing to check. **Why it matters more than the grant did:** a permission that can vanish silently makes every ACL-based guarantee on these hosts provisional, and the agent's new store-grant probe (v0.123.0) now detects the STATE but says nothing about the TRANSITION. **Worth pairing with:** whether the probe should report a grant it once had and no longer has as a distinct, louder signal than one it never had | CC |
+| **R-191** | **Every weekly offsite backup UPLOADS successfully and then FAILS the job on a prune the box is deliberately not allowed to do — on both demo boxes.** Measured on demo-felhom 2026-08-04 06:49–06:53: the upload completed (223 s, 629 MiB of 1.874 GiB, **67.2 % reused incrementally**), then `ERROR: prune 'ct/9201': proxmox-backup-client failed: Error: permission check failed - missing Datastore.Modify\|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `ERROR: Backup of VM 9201 failed - error pruning backups` → `TASK ERROR: job errors`. The hub raised `whole_guest_backup_failed` | **OPEN** | — | **This is R-89's rule not reaching the config.** R-89 moved PBS pruning SERVER-SIDE — *"boxes set `keep_last: 0`, ep0 runs prune jobs; box tokens stay write-only, never widen the grant"*. The token behaves exactly as designed: it refuses. But **both** demo boxes still arm the offsite tier with `keep_last=2 prune_pbs_allowed=true` (`backup_targets: [{target_id: felhom-pbs, cadence_seconds: 604800, keep_last: 2}]`), so every run asks for a prune that must fail. **The data is SAFE and that is why this is not a P1:** the snapshot lands before the prune is attempted; what is wrong is the job's VERDICT and the weekly operator e-mail it produces. **But it is corrosive in the specific way this project keeps finding:** a backup that reports FAILED while succeeding trains the operator to discount `whole_guest_backup_failed`, which is the same alert that would carry a real one — and it is exactly the failure the R-100 corollary warns about, an alarm whose text is true and whose trigger is not the thing you would act on. **Fix is one config line per box** (`keep_last: 0` on the PBS tier) plus whatever writes it on a fresh install; **deliberately NOT applied in this session** — the session was a runbook with an explicit "change nothing, and if a change appears necessary, stop and report" rule, and a retention field on a live backup tier is not a change to slip into an observation run. **Check before fixing:** whether ep0's prune jobs actually cover these two namespaces, or the snapshots simply accumulate once the box stops asking | CC |
 | — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC |
 | — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator |
 | **R-90** | ~~ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged~~ | **CLOSED — the operator rescaled ep0 to a CX33 on 2026-08-03** | — | **MEASURED ON THE BOX, not read from an invoice:** `felhom-hetzner` reports `Mem: 7757` MB total (**8 GB**, was 3.8) and `nproc` **4**. **The interim lever survived and was checked rather than assumed** — a resize is a stop/start, so "the swapfile is still there" was an assumption until measured: `/swapfile`, 4 GiB, dated `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`), 0 B in use on an idle box. **THE 40 GB LOCAL DISK DID NOT CHANGE** and must not be "corrected" alongside the RAM: `/` is 38 G, 58% used. This was a CPU/RAM resize only, so every disk figure in the runbooks still stands — the separate 98 G volume at `/mnt/pbs-datastore` (R-82 P0.3) is unaffected. **Why this was BLOCKED and no longer is:** the row recorded CX33 as *"confirmed unavailable even powered OFF"* — the Cost-Optimized line's limited availability, not a power-state problem. It became available and the operator took it. **Documentation corrected** (`RUNBOOK-ep0-datastore-volume`, `RUNBOOK-pbs-prune-serverside` ×2, `runbooks/offsite-endpoint.md` ×2, `runbooks/target-selection.md`) and **audit/evidence documents ANNOTATED, not revised** (`SPIKE-connectivity-wireguard-2026-07-03`, campaign-10 `phaseA-journal`) — they record what was true when written and that is their value. **Still open and still the operator's, deliberately untouched:** `target-selection.md`'s *"D-d did not name ep0 either way. Confirm it explicitly."* | — |