R-193: a guest rebuild silently drops the offsite tier; R-192 cause established
gates / gates (push) Successful in 8s

Operator confirms no hub-side offsite config change, so the regression was not an
action. Evidence: demo-hp's controller went 0.187.0 -> 0.192.0 at 06:12:18 with a
new config hash and the agent re-keyed its leaf three minutes earlier — a guest
rebuild. The last pre-rebuild report shows the tier fully healthy: escrowed, last
success 02:16:39Z, 15 snapshots, 40.9 MB. No offsite object in the 108 reports
since.

Mechanism: the restic credential is delivered once. demo-hp consumed its secret on
2026-07-23; the rebuilt controller has no copy and no way to request another.
demo-felhom survived the SAME rebuild only because its secret was still unconsumed
— it consumed it four seconds after its config hash changed and was reporting
offsite again 76 seconds later. That difference was luck, not design.

Also sharpens R-192: the self-heal's guard refuses when any report since the
consume carried an offbox target, but that query reads the OLDEST 500 reports —
all of which predate the rebuild. Healthy history before a rebuild is not evidence
the credential still works, which is why the automation that exists for this case
declined to act.
This commit is contained in:
2026-08-04 09:05:53 +02:00
parent 185661062a
commit 2584dfb938
2 changed files with 12 additions and 8 deletions
+10 -7
View File
@@ -20,13 +20,16 @@ Proven end to end on real hardware.
## What's broken
- **One demo machine has no off-site copy of its app data, and has not since yesterday morning.** The
second demo machine (`demo-hp`) stopped reporting an off-site backup at 08:12 on 3 August and has
not reported one in 108 check-ins since. The daily email you are getting about it is right that
something is wrong, but **its explanation is wrong** — it says the password was never applied, when
in fact it was applied on 23 July and then stopped working eleven days later. The automatic repair
declines, correctly, because it can see it is a different problem — but says nothing about why.
*(R-192)*
- **One demo machine has no off-site copy of its app data, and rebuilding it is what took it away.**
`demo-hp` was rebuilt on 3 August; before that its off-site backup was healthy and had run
successfully at 04:16 that morning (15 snapshots). The rebuilt machine came up without it and has
not had it in 108 check-ins since. **The cause is that the off-site password is delivered exactly
once and a rebuilt machine cannot ask for another** — the other demo machine survived the same
rebuild only because it happened to have an unused password waiting for it, and recovered in 76
seconds. Nothing about that difference was designed. *(R-193)*
- **The daily email about it tells you the wrong story**, and the automatic repair that exists for
this declines without saying why. The message says the password was never applied; it was, on
23 July, and worked for eleven days. *(R-192)*
- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one