RUNBOOK: the host tier is restore-PROVEN unattended on both demo boxes
gates / gates (push) Successful in 7s

Four SCHEDULED runs, none triggered by hand: demo-felhom host 83.8s / offsite
540.4s; demo-hp host 109.3s / offsite 300.1s. Each restored into a scratch guest,
booted, verified and destroyed itself; zero 990000 guests or volumes afterwards
and both local-lvm figures returned to their pre-run values.

Both boxes had BOTH tiers due at once, so R-86's ordering was observed live for
the first time: never-proven sorted first, each box took its HOST tier, deferred
the offsite one, and picked it up on the next evaluation six hours later. The
host-tier proofs reached the hub through R-189's merge — demo-felhom's report
carries two tiers, and the local one can only have come from disk.

The capability map's optimistic half is cashed, with its scope stated: these two
boxes, not the fleet.

Surfaced and filed rather than fixed:
- R-190: a storage ACL that demonstrably worked at 04:44 was gone by 09:24, with
  a reinstall, any logged pveum activity and any cluster-log entry ruled out.
- R-191: every weekly offsite backup uploads successfully and then fails the job
  on a prune the box is deliberately not allowed to do (R-89 moved it
  server-side; both boxes still arm keep_last=2).

Two corrections to yesterday's record: the R-185 drift DID surface as 403s on the
write path (six, with the hub raising whole_guest_backup_failed at the first), and
my earlier "no restore_test_* events" was produced by grepping a 404 page.
This commit is contained in:
2026-08-04 08:13:43 +02:00
parent b93ee06abc
commit d692c46db0
5 changed files with 185 additions and 122 deletions
+15 -10
View File
@@ -20,12 +20,23 @@ Proven end to end on real hardware.
## What's broken
- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
setting per machine fixes it. *(R-191)*
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
also delete it. A daily snapshot is armed as a stopgap.
*(R-95, R-87)*
## What shipped recently
- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both
demo machines restored their own on-machine backup into a throwaway machine overnight, booted it,
checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we
had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had
never been tested on either machine. Both also proved their off-site copy on the same night, one
after the other rather than at once, which is the machine deciding for itself what to do first.
*(closes the last open half of R-86/R-185)*
- **A backup copy the machine was never allowed to read — and could not tell you about.** One demo
machine kept its whole-machine backups on a dedicated storage area the agent had never been granted
permission to read. Asked what was there it was told "nothing", while an administrator saw three
@@ -45,9 +56,6 @@ Proven end to end on real hardware.
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
- **Each backup is now proved, instead of the clock being obeyed** — tested once, about a day after
it is made, and not again until there is a newer one; the "not proved lately" alert learned each
copy's own rhythm in the same change. *(R-86)*
## What we're working on
@@ -69,12 +77,9 @@ Proven end to end on real hardware.
## Changed since last update
- **2026-08-03** — Found and fixed a backup copy the machine was never permitted to read, on both demo
machines. The permission was one line; what mattered was that the machine now says so instead of
treating "I am not allowed" and "there is nothing here yet" as the same answer. *(R-185)*
- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
that could vanish on a restart (seen happening), a release that emailed a failure for a release
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
- **2026-08-03** — Backups are now proved one at a time, each about a day after it is made, instead of
on a timer; the "not proved lately" alert learned each copy's own rhythm. You settled that the
off-site endpoint is protected. *(R-86)*