RUNBOOK: the host tier is restore-PROVEN unattended on both demo boxes
gates / gates (push) Successful in 7s

Four SCHEDULED runs, none triggered by hand: demo-felhom host 83.8s / offsite
540.4s; demo-hp host 109.3s / offsite 300.1s. Each restored into a scratch guest,
booted, verified and destroyed itself; zero 990000 guests or volumes afterwards
and both local-lvm figures returned to their pre-run values.

Both boxes had BOTH tiers due at once, so R-86's ordering was observed live for
the first time: never-proven sorted first, each box took its HOST tier, deferred
the offsite one, and picked it up on the next evaluation six hours later. The
host-tier proofs reached the hub through R-189's merge — demo-felhom's report
carries two tiers, and the local one can only have come from disk.

The capability map's optimistic half is cashed, with its scope stated: these two
boxes, not the fleet.

Surfaced and filed rather than fixed:
- R-190: a storage ACL that demonstrably worked at 04:44 was gone by 09:24, with
  a reinstall, any logged pveum activity and any cluster-log entry ruled out.
- R-191: every weekly offsite backup uploads successfully and then fails the job
  on a prune the box is deliberately not allowed to do (R-89 moved it
  server-side; both boxes still arm keep_last=2).

Two corrections to yesterday's record: the R-185 drift DID surface as 403s on the
write path (six, with the hub raising whole_guest_backup_failed at the first), and
my earlier "no restore_test_* events" was produced by grepping a 404 page.
This commit is contained in:
2026-08-04 08:13:43 +02:00
parent b93ee06abc
commit d692c46db0
5 changed files with 185 additions and 122 deletions
+21
View File
@@ -17,6 +17,27 @@
## Standing rulings
**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
(2026-08-04). Scope: those two boxes, not the fleet.**
Four SCHEDULED runs overnight, none triggered by hand: demo-felhom host **83.8 s** / offsite
**540.4 s**; demo-hp host **109.3 s** / offsite **300.1 s**. Every one restored into a scratch guest,
booted, verified and destroyed itself.
*What this closes.* Until yesterday every live restore-proof this project held was on the OFFSITE
tier. The on-box tier — the one an ordinary recovery uses — had never been proven on either box, and
not because it failed: the agent could not read the storage it lives on (R-185), so it never saw an
archive there to test.
*What was observed for the first time.* Both boxes had BOTH tiers due simultaneously. Never-proven
sorts first, so each took its host tier, deferred the offsite one, and picked that up on the next
evaluation six hours later — R-86's ordering and the one-heavy-operation gate, working together,
unsupervised. The host-tier proof then reached the hub through R-189's merge, which is that path
carrying a host-tier entry for the first time.
*The asymmetry worth remembering:* a host-tier restore is **83109 s**; an offsite one is
**300540 s**. The tier that matters for an ordinary recovery is also the cheapest to prove.
**S-21 — an empty listing cannot distinguish FORBIDDEN from NEWBORN, so the box asks the permission
question directly (2026-08-03, R-185; agent v0.123.0 + installer 1.24.0).**