diff --git a/CONTEXT.md b/CONTEXT.md index 80f1a56..07a4fd1 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -17,6 +17,33 @@ ## Standing rulings +**S-24 — offsite retention is ep0's, and the box asks for none (2026-08-04, R-191; installer 1.25.0).** +R-89 moved offsite pruning server-side and box tokens stay write-only. The 2026-07-26 "two weeks" +ruling was not reversed — **where it is ENFORCED moved, and the installer's `keep_last: 2` did not +follow**, so every weekly run uploaded fine and then failed the job on a refused prune. The offsite +tier now writes `keep_last: 0`, which the agent's existing guard already reads as *never prune from +the box*; the local tier is untouched. A gate asserts it. **Before switching it off, ep0 was read:** +both namespaces have a prune job at 03:30 keep-last 2 that has run daily since 2026-07-27 (18 tasks, +all OK). **If that ever stops, `keep_last: 0` is unbounded growth** — check ep0's prune jobs before +assuming the offsite tier is retained. + +**S-25 — a lost storage grant repairs itself, and the repair is REPORTED (2026-08-04, R-190; agent +v0.124.1).** On a missing grant the agent runs the existing root wrapper (`felhom-backup-target-apply +grant `, already sudoers-permitted for any id) and re-reads once — the pbsdr R-22 shape. Bounded +at one attempt per tier per hour. + +*The record is the point, and getting it wrong is easy.* A repair that leaves only `ok` behind +destroys the only evidence a permission vanished. v0.124.0 reported degraded for "one cycle" and +**that failed in production while every unit test passed**: `probeAll` is called independently by the +self-check log and by the collector, the repairing call was the log's, and the host-report three +seconds later sent `ok`. The fix is a latch on **time** (20 min > the 900 s report interval), so at +least one report must carry it. The explanation rides `Feature`, because that is the field the hub +interpolates into the operator's e-mail — `Reason` does not travel. **Nothing new was built:** the +hub's existing ok→degraded→ok edge is the channel. + +*Caveat measured live:* **PVE caches permissions** (~40 s and ~16 min observed), so detection lags the +loss and a single permission read is a lagging indicator → R-194. + **S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes (2026-08-04). Scope: those two boxes, not the fleet.** diff --git a/STATUS.md b/STATUS.md index 4e8d911..994d7ce 100644 --- a/STATUS.md +++ b/STATUS.md @@ -20,36 +20,30 @@ Proven end to end on real hardware. ## What's broken -- **Rebuilding a machine silently takes away its off-site app-data backup.** `demo-hp` was rebuilt on - 3 August and came up without one, and stayed that way for a day. **Fixed on 4 August** — re-issued, - the machine picked the new password up in 15 seconds and reattached to the same repository, and you - escrowed the key. **The underlying fault is not fixed:** the off-site password is delivered exactly - once and a rebuilt machine cannot ask for another, so this will happen again on the next rebuild. - The other machine survived the same rebuild only because an unused password happened to be waiting - for it. *(R-193)* -- **The daily email about it tells you the wrong story**, and the automatic repair that exists for - this declines without saying why. The message says the password was never applied; it was, on - 23 July, and worked for eleven days. *(R-192)* -- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then - trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and - you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one - setting per machine fixes it. *(R-191)* +- **Rebuilding a machine silently takes away its off-site app-data backup.** Fixed on demo-hp on + 4 August (re-issued, picked up in 15 seconds, same repository, you escrowed the key), but the + underlying fault stands: the password is delivered once and a rebuilt machine cannot ask again. + The daily email about it also tells the wrong story. *(R-193, R-192)* +- *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine + and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer + asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing + it every day since 27 July. Both machines and all future ones. *(R-191)* - **The off-site copy can be erased by the machine that made it.** The credential that writes it can also delete it. A daily snapshot is armed as a stopgap. *(R-95, R-87)* ## What shipped recently -- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both - demo machines restored their own on-machine backup into a throwaway machine overnight, booted it, - checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we - had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had - never been tested on either machine. Both also proved their off-site copy on the same night, one - after the other rather than at once, which is the machine deciding for itself what to do first. - *(closes the last open half of R-86/R-185)* -- **A backup copy the machine was never allowed to read — and could not tell you about**, on both - demo machines. The permission was one command; the silence was the real fault, and the machine now - checks whether it may read each copy it depends on and says so when it may not. *(R-185)* +- **A permission the machine needs can now repair itself — and it tells you it had to.** Yesterday we + found a permission that vanished with no reinstall and no recorded action; the machine now notices, + restores it through the same controlled helper that grants it in the first place, and **raises an + alert saying it was missing and was restored**. The alert is the point: a permission that silently + fixes itself is one that can be lost over and over and never investigated. Proven twice on a demo + machine by taking the permission away by hand. *(R-190 — the repair; why it vanished is still open)* + +- **The on-machine backup copy has now been proved to restore — by the machines themselves**, both of + them, overnight and unasked (84 and 109 seconds). Every restore proof before this was of the + off-site copy. *(R-86, R-185)* - **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index dfe3e25..b453a03 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -278,7 +278,10 @@ The tiers are **inputs to recovery**, not recovery routes. §7 and §8 say what | **Tier-2** cross-drive | `/backups/secondary//` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/` + `userdata/` | nightly at W+60m | mirror (rsync) | no | | **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) | | **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** | -| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` | **yes** (per-customer key) | +| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` — **and the box asks for none** (R-191) | **yes** (per-customer key) | + +> **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does. + **[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 8d4335b..6b08f38 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -33,8 +33,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-187** | **R-115's one-command release had never actually run its publish leg — the first real use died there.** `scripts/publish-agent.sh` has been mode `0644` since it was created (2026-06-28), because every earlier caller invoked it as `bash scripts/publish-agent.sh`; `release-agent.sh` (written 2026-08-03) called it directly and got `Permission denied` on v0.121.0's release | **CLOSED — SHIPPED 2026-08-03** (`felhom-agent`) | — | **Fixed both ways in one commit:** the executable bit restored, and the caller changed to `bash "$REPO_ROOT/scripts/publish-agent.sh"` so the release no longer depends on a file mode — the kind of thing a checkout, an archive or a copy silently loses again. **The lesson is R-115's own, one level up:** the mechanism written to make a step unforgettable was itself never exercised end-to-end, so it failed the first time it mattered. A mechanism that has not been RUN is a note with better formatting | CC | | **R-188** | **Every agent release has a ~50 % chance of emailing the operator a CI failure for a release that is correct.** `on: [push]` fires the gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag BEFORE publishing the package (deliberately — a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it). So CI can run the published-versions gate inside the seconds-long window where `v` exists and the package does not, and it correctly reports *"every released agent version must be INSTALLABLE"* — for a release that completes normally a moment later | **CLOSED — SHIPPED 2026-08-03** (agent **v0.122.0**) | — | **MEASURED, 2026-08-03, across two releases in one session:** v0.121.0 → runs #12 (id 79) **success** and #13 (id 81) **failure** on the same sha `4618169`; v0.121.1 → #17 (id 88) **failure** and #18 (id 89) **success** on the same sha `cd1b087`; and v0.121.0's second pair (#14/#15, `4d82591`) both **success** — i.e. it is a race, not a rule. Local gate afterwards: all three versions verify. **Why it matters more than it looks:** R-168 made CI email the operator on failure precisely so a red gate cannot be missed. A signal that cries wolf on every second correct release is how that mail becomes something you archive unread — the R-29 failure mode one layer up, and R-100's corollary in a new place (an alarm whose text is true but whose trigger is not the thing you would act on). **Fix shapes, cheapest first:** have the workflow skip tag refs (`if: !startsWith(github.ref, 'refs/tags/')` — the commit push already covers the content, and the tag adds no new code); or make `check-published-versions.py` tolerate a tag younger than ~2 minutes; or publish before tagging, which the release script's own comment argues against. **Do not "fix" it by quieting the alarm** **FIXED by moving ONLY the tag push** (build → tag locally → publish → push tag). The tag is still created before anything is published, so the build and the tag still describe the same commit; it becomes VISIBLE — to CI, and to any `raw/tag/…` fetch — only once the package is downloadable. **The invariant the old order protected was not traded away:** `check-published-versions.py` now asserts the converse directly — *no published version may be missing its tag* — as a BOUNDED probe of the frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set on every run because a check whose coverage is invisible reads as a guarantee it is not making. The package listing api still answers **401** without a token (re-measured 2026-08-03), so absence still cannot be enumerated and the script says so. **Both directions demonstrated:** a real published-but-untagged fixture (0.121.2, published then deleted — HTTP 201 then 204, absence re-verified by a 404) made the gate FAIL and print the one-line recovery; removing the converse block made it pass over that same orphan. A tag with no package still fails, shown against a local stand-in rather than by pushing a probe tag — pushing one would have emailed the operator a TRUE alarm about a fixture, which is the same attention cost this row removes. **A half-done release is now loud:** publish-succeeded-then-push-failed dies naming the exact recovery command, and a FAILED publish removes the local-only tag so the release can simply be retried instead of colliding with step 2's re-release guard. **Scenario E, live:** v0.122.0's own release produced CI runs **21 (task id 96)** and **22 (task id 97)** on the same sha — **both green**, where the same shape produced one red run on each of the two previous releases. | CC | | **R-189** | **A passing restore-test can be invisible to the hub forever — and R-86 made that window a week instead of a day.** The agent reports `restore_tests[]` from `internal/backup.Store`, which is **in-memory** (its own comment: *"lost on restart; the cadence re-populates"*), and the hub's report interval is 900 s. An agent restart inside that window drops the result. That used to self-heal within 24 h because the next cadence re-tested the same tier; **under per-archive due-ness the agent will NOT re-test a proven archive**, so the hub can stay ignorant of a proof until the NEXT archive generation — a week on the offsite tier. **Observed live, 2026-08-03:** the 15:25:14 PASS on `felhom-pbs` (a real 14.5 GB offsite restore) reached **no** host-report — the hub logged `0 restore-tests` at 15:28:00 and 15:32:14 — because the agent was restarted at 15:27:57 for the v0.121.1 deploy, 2 m 43 s after the run and before the next report | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.122.0**) | — | **The persisted proof already exists and is not reported.** `RestoreTestState` (on disk, per tier, R-86: with the archive id) survives exactly this, and `RestoreTestState.Snapshot()` carries the doc comment *"for the host-report gauge"* — **and has no production caller at all** (grep, 2026-08-03). That is this project's own *seam built but never wired* class, fifth instance, and a comment asserting an invariant with no test pinning it, both in one method. **Severity is bounded, and worth stating so it is not over-ranked:** the hub scans its retained window, archives keep arriving, and R-86's per-tier window for the offsite tier is 12 days against a 7-day archive rhythm — so ONE lost report is tolerated. What is not tolerable is that the loss is **silent** and that the box will not repeat the work. **Fix shape:** report the persisted per-tier proof (target, archive, proven_at) alongside the in-memory latest run, and let the hub's `lastProvenPerTier` read it — a report-contract change, so it touches `internal/hub/report.go` + `contract_test.go` + the hub's checker, and it is a spec, not a one-liner **FIXED.** `RestoreTestState` now stores the **tier** and what was **verified** beside the archive (recorded at proof time, from the run's own result — deriving them later would need a storage lookup that can fail on the one path where failing means mislabelling a proof), exposes `ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it with the in-memory result. **Merge rule: one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news and lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (two entries would read at the hub as two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **It refuses to lie:** a record missing the archive OR the tier produces NO entry, and run mechanics (scratch VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be. **The asymmetry is now written where it will be read:** only successes are persisted, because a success SUPPRESSES future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it believes) while a failure CAUSES future work and heals itself at the next evaluation. **`Store`'s comment was corrected in place** — *"lost on restart; the cadence re-populates"* was true under a timer and became false the moment R-86 shipped; leaving it is how the next reader concludes this is handled. **The seam that had no caller is now pinned by an AST test** (`TestMainWiresTheDurableRestoreTestProof`), because `Snapshot()` claimed a *"host-report gauge"* in its doc comment for weeks with nothing calling it. **MIGRATION, stated because it is visible on the live box:** a pre-R-189 record carries an archive but no tier and is therefore NOT reportable — upgrading does not retroactively make an old proof visible to the hub; the tier's next real proof fills it in. Observed on demo-felhom immediately after the upgrade: still `0 restore-tests` until a fresh run. | CC | -| **R-190** | **A storage ACL that demonstrably WORKED in the morning was gone by mid-morning, and nothing recorded its removal.** On demo-felhom, a `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50 CEST 2026-08-03** (task log read in full). From **09:24:56** the same path returned `HTTP 403 … missing privilege Datastore.Allocate at /storage/felhom-backup`, six times through the day, until the grant was re-applied by hand at 18:54. By ~14:50 `pveum acl list` showed **no row at all** for that path | **OPEN** | — | **Why this is not just R-185 restated:** R-185's mechanism (the installer's Scenario-F arm resolves a pre-existing target without granting) explains a box that NEVER had the grant. This box HAD it and lost it, inside five hours, with the machine up throughout. **Ruled out, each by measurement:** a host reinstall (`uptime` = 12 days); any `pveum`/ACL/`user.cfg` activity in syslog between 04:00 and 10:00 (none); any ACL entry in `/cluster/log` (none). **Correlated, not established:** `host_leaf_changed` at 09:15 and `controller_started` at 09:19 — guest 9201 was reprovisioned nine minutes before the first 403. PVE removes ACLs at `/vms/` when a guest is destroyed (`AccessControl::remove_vm_access`, the F-LEAK mechanism); whether any path can take a `/storage/` row with it has NOT been established and is the first thing to check. **Why it matters more than the grant did:** a permission that can vanish silently makes every ACL-based guarantee on these hosts provisional, and the agent's new store-grant probe (v0.123.0) now detects the STATE but says nothing about the TRANSITION. **Worth pairing with:** whether the probe should report a grant it once had and no longer has as a distinct, louder signal than one it never had | CC | -| **R-191** | **Every weekly offsite backup UPLOADS successfully and then FAILS the job on a prune the box is deliberately not allowed to do — on both demo boxes.** Measured on demo-felhom 2026-08-04 06:49–06:53: the upload completed (223 s, 629 MiB of 1.874 GiB, **67.2 % reused incrementally**), then `ERROR: prune 'ct/9201': proxmox-backup-client failed: Error: permission check failed - missing Datastore.Modify\|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `ERROR: Backup of VM 9201 failed - error pruning backups` → `TASK ERROR: job errors`. The hub raised `whole_guest_backup_failed` | **OPEN** | — | **This is R-89's rule not reaching the config.** R-89 moved PBS pruning SERVER-SIDE — *"boxes set `keep_last: 0`, ep0 runs prune jobs; box tokens stay write-only, never widen the grant"*. The token behaves exactly as designed: it refuses. But **both** demo boxes still arm the offsite tier with `keep_last=2 prune_pbs_allowed=true` (`backup_targets: [{target_id: felhom-pbs, cadence_seconds: 604800, keep_last: 2}]`), so every run asks for a prune that must fail. **The data is SAFE and that is why this is not a P1:** the snapshot lands before the prune is attempted; what is wrong is the job's VERDICT and the weekly operator e-mail it produces. **But it is corrosive in the specific way this project keeps finding:** a backup that reports FAILED while succeeding trains the operator to discount `whole_guest_backup_failed`, which is the same alert that would carry a real one — and it is exactly the failure the R-100 corollary warns about, an alarm whose text is true and whose trigger is not the thing you would act on. **Fix is one config line per box** (`keep_last: 0` on the PBS tier) plus whatever writes it on a fresh install; **deliberately NOT applied in this session** — the session was a runbook with an explicit "change nothing, and if a change appears necessary, stop and report" rule, and a retention field on a live backup tier is not a change to slip into an observation run. **Check before fixing:** whether ep0's prune jobs actually cover these two namespaces, or the snapshots simply accumulate once the box stops asking | CC | +| **R-190** | **A storage ACL that demonstrably WORKED in the morning was gone by mid-morning, and nothing recorded its removal.** On demo-felhom, a `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50 CEST 2026-08-03** (task log read in full). From **09:24:56** the same path returned `HTTP 403 … missing privilege Datastore.Allocate at /storage/felhom-backup`, six times through the day, until the grant was re-applied by hand at 18:54. By ~14:50 `pveum acl list` showed **no row at all** for that path | **MITIGATION SHIPPED 2026-08-04** (agent **v0.124.0 → v0.124.1**) — **MECHANISM STILL OPEN** | — | **Why this is not just R-185 restated:** R-185's mechanism (the installer's Scenario-F arm resolves a pre-existing target without granting) explains a box that NEVER had the grant. This box HAD it and lost it, inside five hours, with the machine up throughout. **Ruled out, each by measurement:** a host reinstall (`uptime` = 12 days); any `pveum`/ACL/`user.cfg` activity in syslog between 04:00 and 10:00 (none); any ACL entry in `/cluster/log` (none). **Correlated, not established:** `host_leaf_changed` at 09:15 and `controller_started` at 09:19 — guest 9201 was reprovisioned nine minutes before the first 403. PVE removes ACLs at `/vms/` when a guest is destroyed (`AccessControl::remove_vm_access`, the F-LEAK mechanism); whether any path can take a `/storage/` row with it has NOT been established and is the first thing to check. **Why it matters more than the grant did:** a permission that can vanish silently makes every ACL-based guarantee on these hosts provisional, and the agent's new store-grant probe (v0.123.0) now detects the STATE but says nothing about the TRANSITION. **Worth pairing with:** whether the probe should report a grant it once had and no longer has as a distinct, louder signal than one it never had **THE ROW NOW REFLECTS THE MITIGATION, NOT THE CAUSE — stated plainly because the two are different things.** The box is resilient; the loss is still unexplained. **Mitigation:** when the store-grant probe finds the grant absent, the agent runs the EXISTING root wrapper (`felhom-backup-target-apply grant `) and **re-reads once** to confirm — the pbsdr R-22 self-grant shape, including its restraint. **No new privileged surface:** the sudoers vector `grant *` already covers any storage id (confirmed in `configs/felhom-agent.sudoers`, not assumed), and the verb already grants BOTH user and token. The verb existed, was permitted, and had only ever been called at storage CREATION — the *built but never wired* shape in a verb rather than a seam, this project's seventh instance. Bounded at one attempt per tier per hour (a storage can be unreadable for reasons an ACL cannot fix; re-granting every cycle is a repair loop wearing a fix's clothes). **THE RECORD IS THE HALF THIS ROW IS ABOUT, and v0.124.0 got it wrong in production while every unit test passed.** It reported degraded for *one cycle* — meaning the probe call that repaired. But `probeAll` is invoked INDEPENDENTLY by the self-check log and by the collector building a host-report: on the box the repairing call was the log's (`09:39:34`, journal shows the repair and `degraded=1`) and the report three seconds later found the grant present and sent **`ok`**. The agent's journal had the record, the hub had nothing, and the operator would have learned nothing — the exact silence this row exists for, re-created inside its own mitigation. **v0.124.1** replaces it with a latch on TIME (20 min > the 900 s report interval), so at least one report must carry it. **PROVEN LIVE, twice, on demo-felhom** (grant deleted by hand, both rows): agent logs `store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190) target=felhom-backup privilege=Datastore.AllocateSpace action="felhom-backup-target-apply grant felhom-backup" confirmed_by=re-read`; the ACL rows return; and on v0.124.1 the **host-report at 08:00:30Z carried `status=degraded`** with the explanation and the hub raised `agent_capability_degraded` and **e-mailed the operator** at 08:00:40. Nothing new was built to carry it — the hub's existing ok→degraded→ok edge is the channel, and the text rides `Feature` because that is the field the hub interpolates into the e-mail (`Reason` does not travel). **PART 3 — the single bounded pass at the mechanism, with the negatives named.** The lead §8.6 nominated is real as a CLASS and is documented in our own installer: *"`pveum user token remove` purges the token's ACL, so re-applying post-rotate is mandatory"*. **It does NOT fit this box.** A rotation purges ALL of the token's ACLs and mints a NEW secret; demo-felhom's token still authenticates with the same secret (`--selftest` OK), it retained its other three storage grants throughout, and only `felhom-backup` was refused. No installer run is evidenced (no 2026-08-03 install log; host uptime 12 days at the time). Previously ruled out and unchanged: a host reinstall, any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00, any cluster-log ACL entry. **Ruled out on THIS box; NOT ruled out fleet-wide** — any installer run still purges and re-grants only the hardcoded `PVE_STORAGES` set, though installer 1.24.0's reuse-arm fix now re-grants the backup target on that path. **A NEW OBSERVATION FROM THE LIVE RUNS, relevant to the timeline:** PVE **caches permissions** — after deleting both ACL rows the probe still read the privilege as present for ~40 s in one run and ~16 min in another. Detection is only as prompt as that cache, and a cache expiry could equally explain why a box kept working for hours after a grant was removed. → **R-194**. | CC | +| **R-191** | **Every weekly offsite backup UPLOADS successfully and then FAILS the job on a prune the box is deliberately not allowed to do — on both demo boxes.** Measured on demo-felhom 2026-08-04 06:49–06:53: the upload completed (223 s, 629 MiB of 1.874 GiB, **67.2 % reused incrementally**), then `ERROR: prune 'ct/9201': proxmox-backup-client failed: Error: permission check failed - missing Datastore.Modify\|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `ERROR: Backup of VM 9201 failed - error pruning backups` → `TASK ERROR: job errors`. The hub raised `whole_guest_backup_failed` | **CLOSED — SHIPPED 2026-08-04** (installer **1.25.0**; both live boxes corrected) | — | **This is R-89's rule not reaching the config.** R-89 moved PBS pruning SERVER-SIDE — *"boxes set `keep_last: 0`, ep0 runs prune jobs; box tokens stay write-only, never widen the grant"*. The token behaves exactly as designed: it refuses. But **both** demo boxes still arm the offsite tier with `keep_last=2 prune_pbs_allowed=true` (`backup_targets: [{target_id: felhom-pbs, cadence_seconds: 604800, keep_last: 2}]`), so every run asks for a prune that must fail. **The data is SAFE and that is why this is not a P1:** the snapshot lands before the prune is attempted; what is wrong is the job's VERDICT and the weekly operator e-mail it produces. **But it is corrosive in the specific way this project keeps finding:** a backup that reports FAILED while succeeding trains the operator to discount `whole_guest_backup_failed`, which is the same alert that would carry a real one — and it is exactly the failure the R-100 corollary warns about, an alarm whose text is true and whose trigger is not the thing you would act on. **Fix is one config line per box** (`keep_last: 0` on the PBS tier) plus whatever writes it on a fresh install; **deliberately NOT applied in this session** — the session was a runbook with an explicit "change nothing, and if a change appears necessary, stop and report" rule, and a retention field on a live backup tier is not a change to slip into an observation run. **Check before fixing:** whether ep0's prune jobs actually cover these two namespaces, or the snapshots simply accumulate once the box stops asking **THE GATE WAS RUN FIRST, AND IT MATTERED.** Before disabling anything, ep0 was read (read-only, Tier 2): prune jobs `prune-demo-felhom` and `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, `schedule 03:30`, `keep-last 2`, comment *"R-82 retention keep-last=2, server-side (box tokens are write-only)"* — and they have run **every day since 2026-07-27: 18 tasks, all `status=OK`**. The newest task log reads `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` / `keep ct/9201/2026-07-27…` / `keep ct/9201/2026-07-28…` / `TASK OK`. Retention happens, and it happens there. **A METHODOLOGICAL WARNING WORTH MORE THAN THE FIX.** Three separate queries said the OPPOSITE — *no prune jobs have ever run* — and **all three were broken instruments**: `worker-type` where the field is `worker_type`; the value `prune` where the worker type is `prunejob`; and `journalctl -u proxmox-backup` where the unit is `proxmox-backup-proxy`. A fourth reading (3 snapshots under keep-last 2) was mis-framed by CC and self-corrected — the third snapshot had landed AFTER that day's 03:30 window. Acting on any of them would have disabled the only pruning ATTEMPT while reporting that nothing prunes: a weekly false alarm traded for unbounded growth on the protected endpoint, invisible for months. **The gate is what caught it, and only because it demanded evidence rather than a verdict.** **Shipped:** installer **1.25.0** writes `keep_last: 0` on the offsite tier (the agent's existing guard `allowPBSPrune = !primary && keep_last > 0` already reads that as *never prune from the box* — no agent change), the justifying paragraph is rewritten to say where retention lives and cite R-89, and `hostinstall_gates.py` asserts it (red-proved: pinning `keep_last: 2` back fails the gate). **Both live boxes corrected in their own config** — `backup tier armed target=felhom-pbs … keep_last=0 … prune_pbs_allowed=false` on demo-felhom and demo-hp, with the local tier untouched at `keep_last=3`. Served over HTTPS at `1.25.0` with `"keep_last":0` in the served bytes. **STILL TO OBSERVE:** the next weekly offsite run completing OK end-to-end. The change removes the failing step; the *schedule* proving it is next week's event, and this row should carry that line when it happens. | CC | +| **R-194** | **PVE's permission cache delays every grant-state verdict by an unknown amount, so "the agent can read it" and "the ACL exists" are not the same measurement.** Observed twice while validating R-190's self-repair on demo-felhom 2026-08-04: both ACL rows for `/storage/felhom-backup` were deleted, and `GET /access/permissions` continued to report `Datastore.AllocateSpace` present — for **~40 s** in one run and **~16 minutes** in another. During that window the capability probe reads healthy and the self-repair does not fire | **OPEN** | — | **Why it matters beyond the delay:** it puts a floor under how fast a lost grant can be noticed, it makes any single permission read a lagging indicator, and — the interesting part — **it is a candidate contributor to R-190's own timeline**: a grant removed at an unknown moment could keep working until a cache expiry, which is exactly the shape of *worked at 04:44, refused at 09:24*. That does not explain what removed it, but it may explain when the refusal SURFACED, and the two have been treated as the same instant. **Not a defect in our code** — it is PVE behaviour, and the mitigation already tolerates it (the repair fires on the next probe after the cache clears). **What is worth deciding:** whether the store-grant probe should ALSO consult the storage content listing as a second signal, since that appeared to reflect the loss immediately (`{"data":[]}` while the permission read still said present) — two signals disagreeing is itself information, and today only one of them is read | CC | | **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **OPEN** | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. | CC | | **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **OPEN** | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. **The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained. Record the answer here. | CC | | — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC |