R-193: a guest rebuild silently drops the offsite tier; R-192 cause established
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
Operator confirms no hub-side offsite config change, so the regression was not an action. Evidence: demo-hp's controller went 0.187.0 -> 0.192.0 at 06:12:18 with a new config hash and the agent re-keyed its leaf three minutes earlier — a guest rebuild. The last pre-rebuild report shows the tier fully healthy: escrowed, last success 02:16:39Z, 15 snapshots, 40.9 MB. No offsite object in the 108 reports since. Mechanism: the restic credential is delivered once. demo-hp consumed its secret on 2026-07-23; the rebuilt controller has no copy and no way to request another. demo-felhom survived the SAME rebuild only because its secret was still unconsumed — it consumed it four seconds after its config hash changed and was reporting offsite again 76 seconds later. That difference was luck, not design. Also sharpens R-192: the self-heal's guard refuses when any report since the consume carried an offbox target, but that query reads the OLDEST 500 reports — all of which predate the rebuild. Healthy history before a rebuild is not evidence the credential still works, which is why the automation that exists for this case declined to act.
This commit is contained in:
@@ -20,13 +20,16 @@ Proven end to end on real hardware.
|
||||
|
||||
## What's broken
|
||||
|
||||
- **One demo machine has no off-site copy of its app data, and has not since yesterday morning.** The
|
||||
second demo machine (`demo-hp`) stopped reporting an off-site backup at 08:12 on 3 August and has
|
||||
not reported one in 108 check-ins since. The daily email you are getting about it is right that
|
||||
something is wrong, but **its explanation is wrong** — it says the password was never applied, when
|
||||
in fact it was applied on 23 July and then stopped working eleven days later. The automatic repair
|
||||
declines, correctly, because it can see it is a different problem — but says nothing about why.
|
||||
*(R-192)*
|
||||
- **One demo machine has no off-site copy of its app data, and rebuilding it is what took it away.**
|
||||
`demo-hp` was rebuilt on 3 August; before that its off-site backup was healthy and had run
|
||||
successfully at 04:16 that morning (15 snapshots). The rebuilt machine came up without it and has
|
||||
not had it in 108 check-ins since. **The cause is that the off-site password is delivered exactly
|
||||
once and a rebuilt machine cannot ask for another** — the other demo machine survived the same
|
||||
rebuild only because it happened to have an unused password waiting for it, and recovered in 76
|
||||
seconds. Nothing about that difference was designed. *(R-193)*
|
||||
- **The daily email about it tells you the wrong story**, and the automatic repair that exists for
|
||||
this declines without saying why. The message says the password was never applied; it was, on
|
||||
23 July, and worked for eleven days. *(R-192)*
|
||||
- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
|
||||
trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
|
||||
you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
|
||||
|
||||
Reference in New Issue
Block a user