R-82: restore round-trip PASSED + multi-tier quiesce proven live -> PROVEN-LIVE

Restore round-trip on demo-hp: pass=true, verified=boot+running,
mount_parity=ok, source_tier=pbs (the v0.100.0 fix — the earlier attempt said
'local' and died at 600s), 4m5s restore+boot+verify+teardown, clean teardown
with no 403 and no leak. That last point confirms 06's reading that the
teardown 403 was a phantom, and corrects my earlier framing of it as a standing
privilege gap.

Multi-tier quiesce driven through the REAL UI endpoint (authed+CSRF):
exactly ONE stop/start pair with BOTH backups inside it, local-first/PBS-last,
app quiesced through the non-last tier, early resume on the last tier's
snapshot. Total downtime 1m27s for both tiers; app healthy after.

Capability map row upgraded IMPLEMENTED -> PROVEN-LIVE, kept distinct from the
DR-tier row above which proves ACTIVATION not ARRIVAL. Remaining gaps recorded:
the SCHEDULED restore-test still only selects the primary tier (manual path
proven, unattended not), and the hub infers cadence from storage type.
This commit is contained in:
Claude Code
2026-07-26 19:04:47 +02:00
parent a0bdd8eed6
commit c74d752a9a
2 changed files with 44 additions and 12 deletions
+43 -11
View File
@@ -83,9 +83,44 @@ takes the backup**.
and it **restored cleanly** (`vzrestore: stopped OK`, all volumes back). Both tiers armed and
verified over the real local API; the untargeted response confirmed byte-identical.
**demo-hp** — reached via the documented break-glass path; binary and config backed up first;
v0.102.0 deployed, tier armed, and its **first ever** PBS backup triggered against a namespace that
was verifiably empty. *(In flight at the time of writing.)*
**demo-hp** — reached via the documented break-glass path; binary and config backed up first.
Its **first ever** PBS backup **landed**: 4.25 GB into a namespace that was verifiably empty.
**THE RESTORE ROUND-TRIP PASSED** — the bar for calling a tier real:
```
source_archive : felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z
source_tier : pbs <- the v0.100.0 fix; the earlier attempt said "local" and died at 600s
pass : true
verified : boot+running
mount_parity : ok <- mp0=/var/lib/docker 50G, mp1=/mnt/sys_drive 20G, mp8/mp9 stand-ins
duration : 4m5s (restore + boot + verify + teardown)
```
`mount_parity` is the non-hollow half: a boot-only verify cannot see a missing data volume. The
scratch guest tore down **cleanly** — no 403, no leak — which confirms `06-offsite-connectivity.md`'s
reading that the teardown 403 was a **phantom** (a consequence of the short timeout, not an ACL gap),
and corrects my earlier framing of it as a standing privilege gap. Afterwards: scratch band empty,
thin pool back to its exact pre-restore figure, live guest running, snapshot intact.
**THE MULTI-TIER QUIESCE RAN LIVE**, through the real UI endpoint (`POST /api/guest-backup/trigger`,
session auth + CSRF — the exact call the "Mentés most" button makes):
```
17:01:39 manual backup requested — quiescing now
17:01:39 backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx] <- ONE stop
17:01:46 tier local: backup job ... started
17:02:56 tier local: ... done — next tier may start (app still quiesced) <- app stays DOWN
17:02:56 tier felhom-pbs: backup job backup-9201-felhom-pbs-... started
17:03:06 tier felhom-pbs: ... snapshotted — resuming app early (8B.2)
17:03:06 unquiescing (snapshotted (early resume, last tier)): restarting 1 stack(s) <- ONE start
```
**Exactly one stop/start pair with both backups inside it** — the assertion that matters, since
"both backups ran" would also pass against an implementation that quiesces twice. Tier order was
local-first/PBS-last as designed, the app stayed quiesced through the non-last tier (app-consistency
preserved on the DR tier), and it resumed at the **last** tier's snapshot rather than waiting for the
upload. **Total app downtime 1m27s for both tiers**; paperless came back healthy.
**Hub Slice C replayed against the live DB before deploying:**
@@ -135,19 +170,16 @@ exact exit-code trap recorded twice earlier in this arc.
## 9. NOT done — explicitly
1. **The offsite tier is never AUTOMATICALLY restore-tested.** The scheduled restore-test picks
candidates from a runner built on the primary target, so it can never select a PBS archive. This
is arguably the more important half of "is the DR tier real?" and it needs a per-tick spec.
candidates from a runner built on the primary target, so it can never select a PBS archive. The
**manual/selftest** path is now proven end-to-end; the **unattended** one is not. Needs a per-tick
spec.
2. **The hub infers "PBS ⇒ weekly" from storage TYPE.** `defaultBackupTarget` is `felhom-pbs`, so a
box that never sets `local_backup_target` would run PBS as its **daily** tier and be judged
against 8 days — seven days of blindness. No box is in that shape today. The real fix is the agent
reporting each tier's actual cadence.
3. **Every agent restart re-triggers an offsite backup** (the in-memory store, R-84). On the local
tier that is wasted minutes; on the offsite tier it is a wasted multi-hour upload after every
deploy. R-84 is now closer to a prerequisite than a tidy-up.
4. **The installer-default fleet flip** (Slice D step 4) waits on a full weekly cycle holding — a
3. **The installer-default fleet flip** (Slice D step 4) waits on a full weekly cycle holding — a
genuine gate, not an oversight.
5. **The boot+verify half of the restore round-trip has not run under the fixed code.**
6. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header;
5. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header;
ratification is Viktor's review of the §10 list.
## 10. Observations