RUNBOOK: the host tier is restore-PROVEN unattended on both demo boxes
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
Four SCHEDULED runs, none triggered by hand: demo-felhom host 83.8s / offsite 540.4s; demo-hp host 109.3s / offsite 300.1s. Each restored into a scratch guest, booted, verified and destroyed itself; zero 990000 guests or volumes afterwards and both local-lvm figures returned to their pre-run values. Both boxes had BOTH tiers due at once, so R-86's ordering was observed live for the first time: never-proven sorted first, each box took its HOST tier, deferred the offsite one, and picked it up on the next evaluation six hours later. The host-tier proofs reached the hub through R-189's merge — demo-felhom's report carries two tiers, and the local one can only have come from disk. The capability map's optimistic half is cashed, with its scope stated: these two boxes, not the fleet. Surfaced and filed rather than fixed: - R-190: a storage ACL that demonstrably worked at 04:44 was gone by 09:24, with a reinstall, any logged pveum activity and any cluster-log entry ruled out. - R-191: every weekly offsite backup uploads successfully and then fails the job on a prune the box is deliberately not allowed to do (R-89 moved it server-side; both boxes still arm keep_last=2). Two corrections to yesterday's record: the R-185 drift DID surface as 403s on the write path (six, with the hub raising whole_guest_backup_failed at the first), and my earlier "no restore_test_* events" was produced by grepping a 404 page.
This commit is contained in:
@@ -20,12 +20,23 @@ Proven end to end on real hardware.
|
||||
|
||||
## What's broken
|
||||
|
||||
- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
|
||||
trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
|
||||
you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
|
||||
setting per machine fixes it. *(R-191)*
|
||||
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
||||
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
|
||||
also delete it. A daily snapshot is armed as a stopgap.
|
||||
*(R-95, R-87)*
|
||||
|
||||
## What shipped recently
|
||||
|
||||
- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both
|
||||
demo machines restored their own on-machine backup into a throwaway machine overnight, booted it,
|
||||
checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we
|
||||
had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had
|
||||
never been tested on either machine. Both also proved their off-site copy on the same night, one
|
||||
after the other rather than at once, which is the machine deciding for itself what to do first.
|
||||
*(closes the last open half of R-86/R-185)*
|
||||
- **A backup copy the machine was never allowed to read — and could not tell you about.** One demo
|
||||
machine kept its whole-machine backups on a dedicated storage area the agent had never been granted
|
||||
permission to read. Asked what was there it was told "nothing", while an administrator saw three
|
||||
@@ -45,9 +56,6 @@ Proven end to end on real hardware.
|
||||
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
|
||||
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
|
||||
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
|
||||
- **Each backup is now proved, instead of the clock being obeyed** — tested once, about a day after
|
||||
it is made, and not again until there is a newer one; the "not proved lately" alert learned each
|
||||
copy's own rhythm in the same change. *(R-86)*
|
||||
|
||||
|
||||
## What we're working on
|
||||
@@ -69,12 +77,9 @@ Proven end to end on real hardware.
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-03** — Found and fixed a backup copy the machine was never permitted to read, on both demo
|
||||
machines. The permission was one line; what mattered was that the machine now says so instead of
|
||||
treating "I am not allowed" and "there is nothing here yet" as the same answer. *(R-185)*
|
||||
- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
|
||||
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
|
||||
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
|
||||
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
|
||||
that could vanish on a restart (seen happening), a release that emailed a failure for a release
|
||||
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
|
||||
- **2026-08-03** — Backups are now proved one at a time, each about a day after it is made, instead of
|
||||
on a timer; the "not proved lately" alert learned each copy's own rhythm. You settled that the
|
||||
off-site endpoint is protected. *(R-86)*
|
||||
|
||||
Reference in New Issue
Block a user