R-82: restore round-trip PASSED + multi-tier quiesce proven live -> PROVEN-LIVE
Restore round-trip on demo-hp: pass=true, verified=boot+running, mount_parity=ok, source_tier=pbs (the v0.100.0 fix — the earlier attempt said 'local' and died at 600s), 4m5s restore+boot+verify+teardown, clean teardown with no 403 and no leak. That last point confirms 06's reading that the teardown 403 was a phantom, and corrects my earlier framing of it as a standing privilege gap. Multi-tier quiesce driven through the REAL UI endpoint (authed+CSRF): exactly ONE stop/start pair with BOTH backups inside it, local-first/PBS-last, app quiesced through the non-last tier, early resume on the last tier's snapshot. Total downtime 1m27s for both tiers; app healthy after. Capability map row upgraded IMPLEMENTED -> PROVEN-LIVE, kept distinct from the DR-tier row above which proves ACTIVATION not ARRIVAL. Remaining gaps recorded: the SCHEDULED restore-test still only selects the primary tier (manual path proven, unattended not), and the hub infers cadence from storage type.
This commit is contained in:
@@ -83,9 +83,44 @@ takes the backup**.
|
||||
and it **restored cleanly** (`vzrestore: stopped OK`, all volumes back). Both tiers armed and
|
||||
verified over the real local API; the untargeted response confirmed byte-identical.
|
||||
|
||||
**demo-hp** — reached via the documented break-glass path; binary and config backed up first;
|
||||
v0.102.0 deployed, tier armed, and its **first ever** PBS backup triggered against a namespace that
|
||||
was verifiably empty. *(In flight at the time of writing.)*
|
||||
**demo-hp** — reached via the documented break-glass path; binary and config backed up first.
|
||||
Its **first ever** PBS backup **landed**: 4.25 GB into a namespace that was verifiably empty.
|
||||
|
||||
**THE RESTORE ROUND-TRIP PASSED** — the bar for calling a tier real:
|
||||
|
||||
```
|
||||
source_archive : felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z
|
||||
source_tier : pbs <- the v0.100.0 fix; the earlier attempt said "local" and died at 600s
|
||||
pass : true
|
||||
verified : boot+running
|
||||
mount_parity : ok <- mp0=/var/lib/docker 50G, mp1=/mnt/sys_drive 20G, mp8/mp9 stand-ins
|
||||
duration : 4m5s (restore + boot + verify + teardown)
|
||||
```
|
||||
|
||||
`mount_parity` is the non-hollow half: a boot-only verify cannot see a missing data volume. The
|
||||
scratch guest tore down **cleanly** — no 403, no leak — which confirms `06-offsite-connectivity.md`'s
|
||||
reading that the teardown 403 was a **phantom** (a consequence of the short timeout, not an ACL gap),
|
||||
and corrects my earlier framing of it as a standing privilege gap. Afterwards: scratch band empty,
|
||||
thin pool back to its exact pre-restore figure, live guest running, snapshot intact.
|
||||
|
||||
**THE MULTI-TIER QUIESCE RAN LIVE**, through the real UI endpoint (`POST /api/guest-backup/trigger`,
|
||||
session auth + CSRF — the exact call the "Mentés most" button makes):
|
||||
|
||||
```
|
||||
17:01:39 manual backup requested — quiescing now
|
||||
17:01:39 backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx] <- ONE stop
|
||||
17:01:46 tier local: backup job ... started
|
||||
17:02:56 tier local: ... done — next tier may start (app still quiesced) <- app stays DOWN
|
||||
17:02:56 tier felhom-pbs: backup job backup-9201-felhom-pbs-... started
|
||||
17:03:06 tier felhom-pbs: ... snapshotted — resuming app early (8B.2)
|
||||
17:03:06 unquiescing (snapshotted (early resume, last tier)): restarting 1 stack(s) <- ONE start
|
||||
```
|
||||
|
||||
**Exactly one stop/start pair with both backups inside it** — the assertion that matters, since
|
||||
"both backups ran" would also pass against an implementation that quiesces twice. Tier order was
|
||||
local-first/PBS-last as designed, the app stayed quiesced through the non-last tier (app-consistency
|
||||
preserved on the DR tier), and it resumed at the **last** tier's snapshot rather than waiting for the
|
||||
upload. **Total app downtime 1m27s for both tiers**; paperless came back healthy.
|
||||
|
||||
**Hub Slice C replayed against the live DB before deploying:**
|
||||
|
||||
@@ -135,19 +170,16 @@ exact exit-code trap recorded twice earlier in this arc.
|
||||
## 9. NOT done — explicitly
|
||||
|
||||
1. **The offsite tier is never AUTOMATICALLY restore-tested.** The scheduled restore-test picks
|
||||
candidates from a runner built on the primary target, so it can never select a PBS archive. This
|
||||
is arguably the more important half of "is the DR tier real?" and it needs a per-tick spec.
|
||||
candidates from a runner built on the primary target, so it can never select a PBS archive. The
|
||||
**manual/selftest** path is now proven end-to-end; the **unattended** one is not. Needs a per-tick
|
||||
spec.
|
||||
2. **The hub infers "PBS ⇒ weekly" from storage TYPE.** `defaultBackupTarget` is `felhom-pbs`, so a
|
||||
box that never sets `local_backup_target` would run PBS as its **daily** tier and be judged
|
||||
against 8 days — seven days of blindness. No box is in that shape today. The real fix is the agent
|
||||
reporting each tier's actual cadence.
|
||||
3. **Every agent restart re-triggers an offsite backup** (the in-memory store, R-84). On the local
|
||||
tier that is wasted minutes; on the offsite tier it is a wasted multi-hour upload after every
|
||||
deploy. R-84 is now closer to a prerequisite than a tidy-up.
|
||||
4. **The installer-default fleet flip** (Slice D step 4) waits on a full weekly cycle holding — a
|
||||
3. **The installer-default fleet flip** (Slice D step 4) waits on a full weekly cycle holding — a
|
||||
genuine gate, not an oversight.
|
||||
5. **The boot+verify half of the restore round-trip has not run under the fixed code.**
|
||||
6. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header;
|
||||
5. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header;
|
||||
ratification is Viktor's review of the §10 list.
|
||||
|
||||
## 10. Observations
|
||||
|
||||
Reference in New Issue
Block a user