R-82: restore round-trip PASSED + multi-tier quiesce proven live -> PROVEN-LIVE

Restore round-trip on demo-hp: pass=true, verified=boot+running,
mount_parity=ok, source_tier=pbs (the v0.100.0 fix — the earlier attempt said
'local' and died at 600s), 4m5s restore+boot+verify+teardown, clean teardown
with no 403 and no leak. That last point confirms 06's reading that the
teardown 403 was a phantom, and corrects my earlier framing of it as a standing
privilege gap.

Multi-tier quiesce driven through the REAL UI endpoint (authed+CSRF):
exactly ONE stop/start pair with BOTH backups inside it, local-first/PBS-last,
app quiesced through the non-last tier, early resume on the last tier's
snapshot. Total downtime 1m27s for both tiers; app healthy after.

Capability map row upgraded IMPLEMENTED -> PROVEN-LIVE, kept distinct from the
DR-tier row above which proves ACTIVATION not ARRIVAL. Remaining gaps recorded:
the SCHEDULED restore-test still only selects the primary tier (manual path
proven, unattended not), and the hub infers cadence from storage type.

REPORT.md

@@ -83,9 +83,44 @@ takes the backup**.
 and it **restored cleanly** (`vzrestore: stopped OK`, all volumes back). Both tiers armed and
 verified over the real local API; the untargeted response confirmed byte-identical.
 
-**demo-hp** — reached via the documented break-glass path; binary and config backed up first;
-v0.102.0 deployed, tier armed, and its **first ever** PBS backup triggered against a namespace that
-was verifiably empty. *(In flight at the time of writing.)*
+**demo-hp** — reached via the documented break-glass path; binary and config backed up first.
+Its **first ever** PBS backup **landed**: 4.25 GB into a namespace that was verifiably empty.
+
+**THE RESTORE ROUND-TRIP PASSED** — the bar for calling a tier real:
+
+```
+source_archive : felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z
+source_tier    : pbs          <- the v0.100.0 fix; the earlier attempt said "local" and died at 600s
+pass           : true
+verified       : boot+running
+mount_parity   : ok           <- mp0=/var/lib/docker 50G, mp1=/mnt/sys_drive 20G, mp8/mp9 stand-ins
+duration       : 4m5s         (restore + boot + verify + teardown)
+```
+
+`mount_parity` is the non-hollow half: a boot-only verify cannot see a missing data volume. The
+scratch guest tore down **cleanly** — no 403, no leak — which confirms `06-offsite-connectivity.md`'s
+reading that the teardown 403 was a **phantom** (a consequence of the short timeout, not an ACL gap),
+and corrects my earlier framing of it as a standing privilege gap. Afterwards: scratch band empty,
+thin pool back to its exact pre-restore figure, live guest running, snapshot intact.
+
+**THE MULTI-TIER QUIESCE RAN LIVE**, through the real UI endpoint (`POST /api/guest-backup/trigger`,
+session auth + CSRF — the exact call the "Mentés most" button makes):
+
+```
+17:01:39  manual backup requested — quiescing now
+17:01:39  backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx]      <- ONE stop
+17:01:46  tier local: backup job ... started
+17:02:56  tier local: ... done — next tier may start (app still quiesced)      <- app stays DOWN
+17:02:56  tier felhom-pbs: backup job backup-9201-felhom-pbs-... started
+17:03:06  tier felhom-pbs: ... snapshotted — resuming app early (8B.2)
+17:03:06  unquiescing (snapshotted (early resume, last tier)): restarting 1 stack(s)   <- ONE start
+```
+
+**Exactly one stop/start pair with both backups inside it** — the assertion that matters, since
+"both backups ran" would also pass against an implementation that quiesces twice. Tier order was
+local-first/PBS-last as designed, the app stayed quiesced through the non-last tier (app-consistency
+preserved on the DR tier), and it resumed at the **last** tier's snapshot rather than waiting for the
+upload. **Total app downtime 1m27s for both tiers**; paperless came back healthy.
 
 **Hub Slice C replayed against the live DB before deploying:**
 
@@ -135,19 +170,16 @@ exact exit-code trap recorded twice earlier in this arc.
 ## 9. NOT done — explicitly
 
 1. **The offsite tier is never AUTOMATICALLY restore-tested.** The scheduled restore-test picks
-   candidates from a runner built on the primary target, so it can never select a PBS archive. This
-   is arguably the more important half of "is the DR tier real?" and it needs a per-tick spec.
+   candidates from a runner built on the primary target, so it can never select a PBS archive. The
+   **manual/selftest** path is now proven end-to-end; the **unattended** one is not. Needs a per-tick
+   spec.
 2. **The hub infers "PBS ⇒ weekly" from storage TYPE.** `defaultBackupTarget` is `felhom-pbs`, so a
    box that never sets `local_backup_target` would run PBS as its **daily** tier and be judged
    against 8 days — seven days of blindness. No box is in that shape today. The real fix is the agent
    reporting each tier's actual cadence.
-3. **Every agent restart re-triggers an offsite backup** (the in-memory store, R-84). On the local
-   tier that is wasted minutes; on the offsite tier it is a wasted multi-hour upload after every
-   deploy. R-84 is now closer to a prerequisite than a tidy-up.
-4. **The installer-default fleet flip** (Slice D step 4) waits on a full weekly cycle holding — a
+3. **The installer-default fleet flip** (Slice D step 4) waits on a full weekly cycle holding — a
    genuine gate, not an oversight.
-5. **The boot+verify half of the restore round-trip has not run under the fixed code.**
-6. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header;
+5. **`07-backup-architecture.md` is NOT ratified** — brought current with an honest staleness header;
    ratification is Viktor's review of the §10 list.
 
 ## 10. Observations

documentation/architecture/00-capability-map.md

@@ -37,7 +37,7 @@
 | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends |
 | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test |
 | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant  **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). |
-| **Recurring offsite (PBS) whole-guest backups actually LAND** — local daily + offsite weekly, as scheduled work rather than a one-off | agent v0.97–0.102, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **IMPLEMENTED** (not PROVEN-LIVE) | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs; live on demo-felhom: first real PBS-targeted backup **`TASK OK`, 41 min, 14.46 GB snapshot**, restored cleanly (`vzrestore: stopped OK`) | **This row exists because the row above proves ACTIVATION, not ARRIVAL.** The DR tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (2026-07-18, a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. R-82 makes the split expressible at all: `backup_targets[]` with per-tier cadence/retention (agent), ONE quiesce window for both tiers (controller), per-tier thresholds (hub, host 26h / offsite 8d), and a fresh-install default (host-install). **Why NOT PROVEN-LIVE:** the restore round-trip has not completed under the fixed code. The 2026-07-26 attempt restored the data correctly but its boot+verify half never ran — the restore-test derived its tier from the configured target rather than the archive, took the 10-minute LOCAL bound against a 14.46 GB WAN restore, and failed at 600s (fixed, agent v0.100.0). **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only ever selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested; the hub infers "PBS ⇒ weekly" from storage TYPE, not from a reported cadence; and every agent restart currently re-triggers an offsite backup (→ R-84). → **R-82** |
+| **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** |
 | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below |
 | **Customer DELETE cascade** (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **UNIT-PROVEN; live leg PENDING** | `hub/internal/web/customer_delete_test.go` — leg ORDER observed from inside leg 2 (hosts already gone, customer row still present, custody still retained); 9 fail-closed gate cases each asserting zero mutations + zero external calls + no journal row; resume-after-external-failure converges; `purgeEscrow` custody semantics; preview leaks no secret. **5 red-proofs** (ack gate, stale-preview gate, ONLINE-host gate, leg order inverted, `purgeEscrow=true`) | Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **v0.70.0 (2026-07-21):** added the **residue** leg — `GetCustomers()` is REPORT-derived, so before it a fully deleted customer stayed on the Customers list and its report stream kept the staleness/offsite checkers alerting (live: `demo-vm-felhom` deleted 07-18, still emailing `offsite_stale` on 07-21). The leg also purges the credential-bearing `appliance_registrations` + `selfbind_tokens`. **Ghost customers (config row already gone) are now deletable** — 404 means "nothing here", not "no config row"; the Hetzner/descriptor legs record `skipped_no_config`. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run |
 | Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 |