docs: R-191 closed, R-190 mitigation shipped (mechanism open), R-194 filed
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
R-191: the gate was run first and it mattered — ep0's prune jobs have run daily since 2026-07-27 (18 tasks, all OK). THREE of my own queries said the opposite and all three were broken instruments (worker-type vs worker_type, prune vs prunejob, proxmox-backup vs proxmox-backup-proxy); acting on any would have disabled the only pruning attempt while reporting that nothing prunes. R-190: the row now reflects the MITIGATION, not the cause, and says so. Includes that v0.124.0's transition record failed in production while every unit test passed, and Part 3's single pass — the token-rotation class is real and documented in our own installer but does not fit this box (same token still authenticates, the other three grants survived, no installer run evidenced); ruled out on this box, not fleet-wide. R-194: PVE caches permissions (~40s and ~16min measured), so a single permission read is a lagging indicator and may explain when R-190's refusal surfaced. 07-backup-architecture: the offsite retention row was RIGHT and the configuration disagreed with it — noted, since a doc that states a contract does not enforce it.
This commit is contained in:
+27
@@ -17,6 +17,33 @@
|
|||||||
|
|
||||||
## Standing rulings
|
## Standing rulings
|
||||||
|
|
||||||
|
**S-24 — offsite retention is ep0's, and the box asks for none (2026-08-04, R-191; installer 1.25.0).**
|
||||||
|
R-89 moved offsite pruning server-side and box tokens stay write-only. The 2026-07-26 "two weeks"
|
||||||
|
ruling was not reversed — **where it is ENFORCED moved, and the installer's `keep_last: 2` did not
|
||||||
|
follow**, so every weekly run uploaded fine and then failed the job on a refused prune. The offsite
|
||||||
|
tier now writes `keep_last: 0`, which the agent's existing guard already reads as *never prune from
|
||||||
|
the box*; the local tier is untouched. A gate asserts it. **Before switching it off, ep0 was read:**
|
||||||
|
both namespaces have a prune job at 03:30 keep-last 2 that has run daily since 2026-07-27 (18 tasks,
|
||||||
|
all OK). **If that ever stops, `keep_last: 0` is unbounded growth** — check ep0's prune jobs before
|
||||||
|
assuming the offsite tier is retained.
|
||||||
|
|
||||||
|
**S-25 — a lost storage grant repairs itself, and the repair is REPORTED (2026-08-04, R-190; agent
|
||||||
|
v0.124.1).** On a missing grant the agent runs the existing root wrapper (`felhom-backup-target-apply
|
||||||
|
grant <id>`, already sudoers-permitted for any id) and re-reads once — the pbsdr R-22 shape. Bounded
|
||||||
|
at one attempt per tier per hour.
|
||||||
|
|
||||||
|
*The record is the point, and getting it wrong is easy.* A repair that leaves only `ok` behind
|
||||||
|
destroys the only evidence a permission vanished. v0.124.0 reported degraded for "one cycle" and
|
||||||
|
**that failed in production while every unit test passed**: `probeAll` is called independently by the
|
||||||
|
self-check log and by the collector, the repairing call was the log's, and the host-report three
|
||||||
|
seconds later sent `ok`. The fix is a latch on **time** (20 min > the 900 s report interval), so at
|
||||||
|
least one report must carry it. The explanation rides `Feature`, because that is the field the hub
|
||||||
|
interpolates into the operator's e-mail — `Reason` does not travel. **Nothing new was built:** the
|
||||||
|
hub's existing ok→degraded→ok edge is the channel.
|
||||||
|
|
||||||
|
*Caveat measured live:* **PVE caches permissions** (~40 s and ~16 min observed), so detection lags the
|
||||||
|
loss and a single permission read is a lagging indicator → R-194.
|
||||||
|
|
||||||
**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
|
**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
|
||||||
(2026-08-04). Scope: those two boxes, not the fleet.**
|
(2026-08-04). Scope: those two boxes, not the fleet.**
|
||||||
|
|
||||||
|
|||||||
@@ -20,36 +20,30 @@ Proven end to end on real hardware.
|
|||||||
|
|
||||||
## What's broken
|
## What's broken
|
||||||
|
|
||||||
- **Rebuilding a machine silently takes away its off-site app-data backup.** `demo-hp` was rebuilt on
|
- **Rebuilding a machine silently takes away its off-site app-data backup.** Fixed on demo-hp on
|
||||||
3 August and came up without one, and stayed that way for a day. **Fixed on 4 August** — re-issued,
|
4 August (re-issued, picked up in 15 seconds, same repository, you escrowed the key), but the
|
||||||
the machine picked the new password up in 15 seconds and reattached to the same repository, and you
|
underlying fault stands: the password is delivered once and a rebuilt machine cannot ask again.
|
||||||
escrowed the key. **The underlying fault is not fixed:** the off-site password is delivered exactly
|
The daily email about it also tells the wrong story. *(R-193, R-192)*
|
||||||
once and a rebuilt machine cannot ask for another, so this will happen again on the next rebuild.
|
- *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine
|
||||||
The other machine survived the same rebuild only because an unused password happened to be waiting
|
and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer
|
||||||
for it. *(R-193)*
|
asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing
|
||||||
- **The daily email about it tells you the wrong story**, and the automatic repair that exists for
|
it every day since 27 July. Both machines and all future ones. *(R-191)*
|
||||||
this declines without saying why. The message says the password was never applied; it was, on
|
|
||||||
23 July, and worked for eleven days. *(R-192)*
|
|
||||||
- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
|
|
||||||
trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
|
|
||||||
you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
|
|
||||||
setting per machine fixes it. *(R-191)*
|
|
||||||
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
||||||
also delete it. A daily snapshot is armed as a stopgap.
|
also delete it. A daily snapshot is armed as a stopgap.
|
||||||
*(R-95, R-87)*
|
*(R-95, R-87)*
|
||||||
|
|
||||||
## What shipped recently
|
## What shipped recently
|
||||||
|
|
||||||
- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both
|
- **A permission the machine needs can now repair itself — and it tells you it had to.** Yesterday we
|
||||||
demo machines restored their own on-machine backup into a throwaway machine overnight, booted it,
|
found a permission that vanished with no reinstall and no recorded action; the machine now notices,
|
||||||
checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we
|
restores it through the same controlled helper that grants it in the first place, and **raises an
|
||||||
had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had
|
alert saying it was missing and was restored**. The alert is the point: a permission that silently
|
||||||
never been tested on either machine. Both also proved their off-site copy on the same night, one
|
fixes itself is one that can be lost over and over and never investigated. Proven twice on a demo
|
||||||
after the other rather than at once, which is the machine deciding for itself what to do first.
|
machine by taking the permission away by hand. *(R-190 — the repair; why it vanished is still open)*
|
||||||
*(closes the last open half of R-86/R-185)*
|
|
||||||
- **A backup copy the machine was never allowed to read — and could not tell you about**, on both
|
- **The on-machine backup copy has now been proved to restore — by the machines themselves**, both of
|
||||||
demo machines. The permission was one command; the silence was the real fault, and the machine now
|
them, overnight and unasked (84 and 109 seconds). Every restore proof before this was of the
|
||||||
checks whether it may read each copy it depends on and says so when it may not. *(R-185)*
|
off-site copy. *(R-86, R-185)*
|
||||||
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
||||||
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
||||||
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
||||||
|
|||||||
@@ -278,7 +278,10 @@ The tiers are **inputs to recovery**, not recovery routes. §7 and §8 say what
|
|||||||
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
|
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
|
||||||
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
|
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
|
||||||
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** |
|
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** |
|
||||||
| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` | **yes** (per-customer key) |
|
| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` — **and the box asks for none** (R-191) | **yes** (per-customer key) |
|
||||||
|
|
||||||
|
> **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does.
|
||||||
|
|
||||||
|
|
||||||
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
|
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
|
||||||
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
|
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user