docs: R-191 closed, R-190 mitigation shipped (mechanism open), R-194 filed
gates / gates (push) Successful in 7s

R-191: the gate was run first and it mattered — ep0's prune jobs have run daily
since 2026-07-27 (18 tasks, all OK). THREE of my own queries said the opposite and
all three were broken instruments (worker-type vs worker_type, prune vs prunejob,
proxmox-backup vs proxmox-backup-proxy); acting on any would have disabled the only
pruning attempt while reporting that nothing prunes.

R-190: the row now reflects the MITIGATION, not the cause, and says so. Includes
that v0.124.0's transition record failed in production while every unit test
passed, and Part 3's single pass — the token-rotation class is real and documented
in our own installer but does not fit this box (same token still authenticates, the
other three grants survived, no installer run evidenced); ruled out on this box, not
fleet-wide.

R-194: PVE caches permissions (~40s and ~16min measured), so a single permission
read is a lagging indicator and may explain when R-190's refusal surfaced.

07-backup-architecture: the offsite retention row was RIGHT and the configuration
disagreed with it — noted, since a doc that states a contract does not enforce it.
This commit is contained in:
2026-08-04 10:04:25 +02:00
parent 71662336aa
commit a684a81796
4 changed files with 52 additions and 27 deletions
+27
View File
@@ -17,6 +17,33 @@
## Standing rulings ## Standing rulings
**S-24 — offsite retention is ep0's, and the box asks for none (2026-08-04, R-191; installer 1.25.0).**
R-89 moved offsite pruning server-side and box tokens stay write-only. The 2026-07-26 "two weeks"
ruling was not reversed — **where it is ENFORCED moved, and the installer's `keep_last: 2` did not
follow**, so every weekly run uploaded fine and then failed the job on a refused prune. The offsite
tier now writes `keep_last: 0`, which the agent's existing guard already reads as *never prune from
the box*; the local tier is untouched. A gate asserts it. **Before switching it off, ep0 was read:**
both namespaces have a prune job at 03:30 keep-last 2 that has run daily since 2026-07-27 (18 tasks,
all OK). **If that ever stops, `keep_last: 0` is unbounded growth** — check ep0's prune jobs before
assuming the offsite tier is retained.
**S-25 — a lost storage grant repairs itself, and the repair is REPORTED (2026-08-04, R-190; agent
v0.124.1).** On a missing grant the agent runs the existing root wrapper (`felhom-backup-target-apply
grant <id>`, already sudoers-permitted for any id) and re-reads once — the pbsdr R-22 shape. Bounded
at one attempt per tier per hour.
*The record is the point, and getting it wrong is easy.* A repair that leaves only `ok` behind
destroys the only evidence a permission vanished. v0.124.0 reported degraded for "one cycle" and
**that failed in production while every unit test passed**: `probeAll` is called independently by the
self-check log and by the collector, the repairing call was the log's, and the host-report three
seconds later sent `ok`. The fix is a latch on **time** (20 min > the 900 s report interval), so at
least one report must carry it. The explanation rides `Feature`, because that is the field the hub
interpolates into the operator's e-mail — `Reason` does not travel. **Nothing new was built:** the
hub's existing ok→degraded→ok edge is the channel.
*Caveat measured live:* **PVE caches permissions** (~40 s and ~16 min observed), so detection lags the
loss and a single permission read is a lagging indicator → R-194.
**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes **S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
(2026-08-04). Scope: those two boxes, not the fleet.** (2026-08-04). Scope: those two boxes, not the fleet.**
+18 -24
View File
@@ -20,36 +20,30 @@ Proven end to end on real hardware.
## What's broken ## What's broken
- **Rebuilding a machine silently takes away its off-site app-data backup.** `demo-hp` was rebuilt on - **Rebuilding a machine silently takes away its off-site app-data backup.** Fixed on demo-hp on
3 August and came up without one, and stayed that way for a day. **Fixed on 4 August** — re-issued, 4 August (re-issued, picked up in 15 seconds, same repository, you escrowed the key), but the
the machine picked the new password up in 15 seconds and reattached to the same repository, and you underlying fault stands: the password is delivered once and a rebuilt machine cannot ask again.
escrowed the key. **The underlying fault is not fixed:** the off-site password is delivered exactly The daily email about it also tells the wrong story. *(R-193, R-192)*
once and a rebuilt machine cannot ask for another, so this will happen again on the next rebuild. - *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine
The other machine survived the same rebuild only because an unused password happened to be waiting and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer
for it. *(R-193)* asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing
- **The daily email about it tells you the wrong story**, and the automatic repair that exists for it every day since 27 July. Both machines and all future ones. *(R-191)*
this declines without saying why. The message says the password was never applied; it was, on
23 July, and worked for eleven days. *(R-192)*
- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
setting per machine fixes it. *(R-191)*
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can - **The off-site copy can be erased by the machine that made it.** The credential that writes it can
also delete it. A daily snapshot is armed as a stopgap. also delete it. A daily snapshot is armed as a stopgap.
*(R-95, R-87)* *(R-95, R-87)*
## What shipped recently ## What shipped recently
- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both - **A permission the machine needs can now repair itself — and it tells you it had to.** Yesterday we
demo machines restored their own on-machine backup into a throwaway machine overnight, booted it, found a permission that vanished with no reinstall and no recorded action; the machine now notices,
checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we restores it through the same controlled helper that grants it in the first place, and **raises an
had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had alert saying it was missing and was restored**. The alert is the point: a permission that silently
never been tested on either machine. Both also proved their off-site copy on the same night, one fixes itself is one that can be lost over and over and never investigated. Proven twice on a demo
after the other rather than at once, which is the machine deciding for itself what to do first. machine by taking the permission away by hand. *(R-190 — the repair; why it vanished is still open)*
*(closes the last open half of R-86/R-185)*
- **A backup copy the machine was never allowed to read — and could not tell you about**, on both - **The on-machine backup copy has now been proved to restore — by the machines themselves**, both of
demo machines. The permission was one command; the silence was the real fault, and the machine now them, overnight and unasked (84 and 109 seconds). Every restore proof before this was of the
checks whether it may read each copy it depends on and says so when it may not. *(R-185)* off-site copy. *(R-86, R-185)*
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked - **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
@@ -278,7 +278,10 @@ The tiers are **inputs to recovery**, not recovery routes. §7 and §8 say what
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no | | **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) | | **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
| **Plane-2** whole-guest, local | `local:``/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** | | **Plane-2** whole-guest, local | `local:``/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** |
| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` | **yes** (per-customer key) | | **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30`**and the box asks for none** (R-191) | **yes** (per-customer key) |
> **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does.
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed **[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
File diff suppressed because one or more lines are too long