From f45b1f6761ff580ba1a805937df7e45f4288d6ba Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 5 Aug 2026 12:56:55 +0200 Subject: [PATCH] docs: R-193 CLOSED (the recovery screen); R-213 minted for the put-back MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - OPEN-ITEMS: R-193 CLOSED with both 2026-08-05 rulings (unlocking and restoring are separate; 'I do not want the old data' moves the store aside after a double confirmation), and the shape-(b) reasoning — WriteOffboxSecrets auto-generates a repository password on re-apply, so the literal 'fresh data area' trigger would have opened a window that closes by itself. - R-213 MINTED (R-212 was and still is the highest, re-checked for the second writer): putting files back in place, with the live-versus-backup comparison named as its requirement. Not started, deliberately. - capability map: the 'needs someone who knows to look' qualifier is GONE; what remains is stated narrowly — no correct-code run through the page, the put-back is out of scope, and the journey has not been re-walked end to end. - 07-backup-architecture 7.0: a fifth row, and where the screen deliberately stops. - CONTEXT: standing ruling S-34. - STATUS: the headline change and the two things still owed as proof. No hub change and no hub bump. --- CONTEXT.md | 42 +++++++++++++++++++ STATUS.md | 31 +++++++++----- .../architecture/00-capability-map.md | 2 +- .../architecture/07-backup-architecture.md | 25 +++++++---- documentation/backlog/OPEN-ITEMS.md | 3 +- documentation/backlog/ROADMAP.md | 1 + 6 files changed, 85 insertions(+), 19 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index b34947f..5bac63a 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -17,6 +17,48 @@ ## Standing rulings +**S-34 — UNLOCKING AND RESTORING ARE SEPARATE. The recovery screen shipped (2026-08-05, controller +v0.200.0, R-193 CLOSED). Read with S-32 and S-33; together they close the whole customer journey up to +the listing.** + +**THE RULINGS (operator, 2026-08-05), recorded as given:** + +1. **Unlocking and restoring are separate.** This screen unlocks only. Restore stays per-app, in the + backups area. **A screen that unlocks and then offers to overwrite is two decisions wearing one + button.** Putting files back is its own item (→ **R-213**), and its stated requirement is a + live-versus-backup comparison — the customer must see what would change before anything is + overwritten. +2. **"I do not want the old data" MOVES THE STORE ASIDE**, after double-checking with the customer. + +*Mechanism facts a future session needs:* + +- **ONE CORE, TWO CALLERS.** `backup.RecoverInstallCore` is the only fetch→unseal→compare→install path. + `RecoverAndInstall` is now a thin CLI wrapper (exit codes and printed lines byte-identical — every + pre-existing CLI test passed unchanged) and the web handler calls the same function. **Asserted from + source by AST on BOTH sides**, plus a third test that the routes and the landing-page interception + exist. Two implementations of the one operation that can permanently lose a customer's data would + drift, and only one would be tested. +- **THE TRIGGER HAS TWO SHAPES, and the second is the one that matters in practice.** + `backup.OffsiteRecoveryOffer` = the hub holds a package **AND** (no repository password **OR** the + tier is orphaned). The literal "no repository password" alone is a window that CLOSES BY ITSELF: + `WriteOffboxSecrets` auto-generates a password on re-apply — R-193's own orphaning mechanism — and + since hub v0.96.0's credential self-heal the re-apply happens within ~15–30 minutes. Shape (b) is + also the state the shipped move-aside requires, which is what lets the discard choice reach the + existing handler instead of needing a new one. +- **CLAIMED IS PART OF THE PREDICATE.** A legacy-open box (no password anywhere) reaches `ServeHTTP` + through `RequireAuth`'s pass-through, so without an explicit `authEnabled()` check the interception + fires for an unauthenticated visitor. Caught by a test, not by review. +- **"Most nem" suppresses the FULL PAGE ONLY.** The backups-area entry point is bound to + `recoveryOffer`, never to the postpone flag. A one-shot notice a flustered person clicks past is a + notice that never happened. +- **The code is handled no more loosely than on the CLI:** POST body only (`PostFormValue`), never + logged, never persisted, never echoed, cleared on every path, `no-store`, `autocomplete="off"`. + **NO LOCKOUT, deliberately** — the code is a ten-word phrase and locking a customer out of their own + data for a typo is worse than anything a lockout prevents; failures are logged locally without the code. +- **NOT YET PROVEN LIVE: a CORRECT code through the page.** No recovery code was kept for demo-felhom's + orphaned history (`RECON-offsite-dr-chain-2026-08-04.md` §12.3) and demo-hp's is operator-held. The + live run exercised handler → agent → hub fetch → age KDF and stopped at the unseal. + **S-33 — THE BOX DECLARES, THE HUB ANSWERS. R-204 item 4 / R-193's credential half closed (2026-08-05, controller v0.199.0 + hub v0.96.0). Read with S-32; together they close all four of the drill's manual interventions.** diff --git a/STATUS.md b/STATUS.md index b7c4fe6..e74ddaa 100644 --- a/STATUS.md +++ b/STATUS.md @@ -22,16 +22,19 @@ over; and the file was restored **byte for byte identical**. *(R-201)* ## What's broken -- **All four recovery steps are now automatic — but a customer still would not know to start.** The - four steps that stood between "the key is recoverable" and "the file is back" are closed *(R-204)*: - the reset-code tool works first time; re-issuing the storage credential no longer falsely marks the - recovery key "stale"; the everyday restore says in plain Hungarian that it returned the app's - settings and database and **not** your documents; and, as of today, **a rebuilt machine asks for its - storage credential itself and the hub answers** — no Re-issue click. **What is missing is the - offer:** there is no screen that meets the owner of a rebuilt machine, tells them a sealed package - is waiting, takes their recovery code and shows what would come back. So nothing needs *you* any - more, but it still needs someone who knows to look. **And the whole journey has not been re-run end - to end since these fixes** — the four are proved one at a time, not as a single walk. *(R-193)* +- **A customer can now get their backups open on their own — but not yet put the files back.** Every + step from a rebuilt machine to an open backup store is done, and today the last piece landed: a + **full screen** meets the owner of a rebuilt machine, explains that the backups are still there, + says plainly that **nobody can replace a lost recovery code**, takes the code, and shows what is in + the store — which apps, from when, how big. Nothing needs you, and nothing needs a command line. + **What it deliberately does not do is put files back.** That is per-app, in the backups area, and + the piece that would guide it — showing what would change before anything is overwritten — is not + built yet. *(R-193 closed; the put-back is R-213)* +- **Two things still owed as proof.** The final unlock has never been done with a **correct** code + through the new screen: no recovery code was kept for the N100 machine's orphaned history, and the + HP machine's is in your hands, not ours — so the live test ran the whole chain and stopped at the + last step. **And the whole journey has not been re-run end to end since these fixes** — the pieces + are proved one at a time, not as a single walk. That re-run is one more drill. *(R-201)* - **Rebuilding a machine still throws away its off-site backup HISTORY.** The machine invents the key that encrypts its own off-site backups, and a rebuilt machine invents a brand-new one. Both demo machines did this on 3–4 August — 51 backups (~1.2 GB) between them. The old key now survives the @@ -47,6 +50,14 @@ over; and the file was restored **byte for byte identical**. *(R-201)* ## What shipped recently +- **2026-08-05** — **The recovery screen: a customer whose machine was rebuilt is now told, and shown + how.** Until today they had everything needed to get their data back and no way to find out — the + only route was a command line. The screen unlocks the backups and lists what is in them; it does + **not** restore anything, because unlocking and restoring are two different decisions and mixing + them would turn one clear moment into a wizard. Three ways out, none of them a dismiss button — and + the „most nem" option keeps the route to the data permanently visible in the backups area, because + a notice someone clicks past once is a notice that never happened. *(R-193)* + - **2026-08-05** — **The orphaned backups are deleted — and the list you were given was wrong, which is why you were asked again.** You had approved "about 1.2 GB in two set-aside stores". Measured before touching anything: there were **three** set-aside stores totalling **~1.45 GB** — and the diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 3e13d0f..81b2148 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -40,7 +40,7 @@ | **The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code** | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE (2026-08-04)** | On demo-felhom, through the real endpoints end to end: the box fetched its own sealed blob from the hub with its own per-host credential (hub log: *escrow blob SERVED … 572 opaque bytes, self_scope=true*), the agent unsealed it with the customer's recovery code, and the extracted repository password's sha256 was **byte-identical** to the one on disk — `c60c8bc737a6…`, which is ALSO the hash the hub had independently stored, so three sources agree. Five minutes earlier the same path with a WRONG code failed closed at age's KDF with nothing written, which proves links 6 and 7 ran independently of the success. R was searched for afterwards and found in 0 log lines and 0 files, with a positive control confirming the search would have found it. Evidence: per-repo CHANGELOGs; `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 links 6–8 | **WHAT THIS ROW DOES NOT CLAIM, stated because the previous over-claim here was struck out four hours earlier.** It covers the KEY, not the DATA. **R-199 BACK-POINTER (omitted when this row was written): the recovery chain's link inventory and the per-link status live in `audits/RECON-offsite-dr-chain-2026-08-04.md` §3; links 1–8 are walked, 9–11 are not.** **Re-confirmed 2026-08-04 evening by an attempt to prove the DATA half:** the R-201 drill was prepared on demo-hp and **halted before the wipe** — the sentinel file was not in the off-site snapshot (R-203), so the wipe would have destroyed it and proven nothing. **No file has still ever been restored from an off-site backup after a wipe** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). The install half of the chain (controller v0.196.0 `--recover-offsite-install`, R-200) is likewise unit-proven only — it has never run against a live recovery. A recovered password has never been **installed** (the diagnostic compares and refuses to write, by design), no existing repository has ever been **reopened** under one, and **no file has ever been restored** from an off-site history via a recovered key. Links 9–11 of the chain are open (R-200's remaining half, R-201). The proof also used a box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is exactly what the drill covers and it has not run || DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). | -| **A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end** | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE (2026-08-04 night drill)** | demo-hp's controller data volume was destroyed and the sentinel deleted from disk. The customer's recovery code then produced `8a9e33aa4da6…` — **byte-identical** to the pre-wipe on-disk key and to the hub's independent record — and installed cleanly on the bare box. `identity_blob` was unchanged across the wipe (572 B, `updated_at` still 11:11:37). Evidence: `audits/DRILL-r201-night-run-2026-08-04.md` §2 | **WHAT IT DOES AND DOES NOT CLAIM.** PROVEN: after a real rebuild the key recovers byte-identical, the EXISTING repository opens (3 snapshots, 42 026 B — the pre-wipe size; not a fresh history), and the customer restore flow returns the file byte-identical. **THE FOUR MANUAL INTERVENTIONS ARE NOW ALL CLOSED (2026-08-05) — AND THE ROW STILL KEEPS A QUALIFIER, for a different reason. State it precisely.** The 2026-08-04 drill needed four interventions between "the key is recoverable" and "the file is back", none of them in any design document (R-204). **Items 1–3 closed in controller v0.198.0 + hub v0.95.0:** the reset code works first time; a credential Re-issue no longer marks a healthy escrow stale; a unit restore states that it returned the app's definition and database and NOT the customer's files. **Item 4 closed in controller v0.199.0 + hub v0.96.0:** a rebuilt box DECLARES `offsite.state=needs_credential` (a declaration, not an inference — an absence has four meanings and the hub cannot tell them apart) and the hub's `internal/offsiteheal` re-arms the stored credential before minting. Proven live on demo-felhom 9201: report id=16743 carried the declaration and the single occurrence was absorbed by the debounce. **WHAT STILL NEEDS A PERSON, stated because removing the qualifier outright would over-claim:** the recovery journey has no CUSTOMER-FACING SCREEN. Every step is now automatic or self-service, but a customer meeting a rebuilt box is not yet OFFERED the recovery — there is no page that says a sealed package is waiting, takes the recovery code and previews what would come back (R-193's remaining half). So the row's honest reading today is: **the machinery no longer requires an operator; the EXPERIENCE still requires someone who knows to look.** The qualifier goes when that screen ships and is walked by a customer. **A second drill has not re-run end to end since the fixes** — the four closures are proven individually, not as one uninterrupted customer journey. Scope: `demo-hp`, a controller-data-volume rebuild — **NOT** a total host loss, and **NOT** a guest reprovision. R-198's retention is still unit-proven: nothing has superseded a key in production | +| **A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end** | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE (2026-08-04 night drill)** | demo-hp's controller data volume was destroyed and the sentinel deleted from disk. The customer's recovery code then produced `8a9e33aa4da6…` — **byte-identical** to the pre-wipe on-disk key and to the hub's independent record — and installed cleanly on the bare box. `identity_blob` was unchanged across the wipe (572 B, `updated_at` still 11:11:37). Evidence: `audits/DRILL-r201-night-run-2026-08-04.md` §2 | **WHAT IT DOES AND DOES NOT CLAIM.** PROVEN: after a real rebuild the key recovers byte-identical, the EXISTING repository opens (3 snapshots, 42 026 B — the pre-wipe size; not a fresh history), and the customer restore flow returns the file byte-identical. **ALL FOUR MANUAL INTERVENTIONS ARE CLOSED, AND THE CUSTOMER IS NOW OFFERED THE RECOVERY (2026-08-05). ONE QUALIFIER REMAINS AND IT IS NOT THE OLD ONE — state it precisely.** The 2026-08-04 drill needed four interventions between "the key is recoverable" and "the file is back", none of them in any design document (R-204). **Items 1–3 closed in controller v0.198.0 + hub v0.95.0** (the reset code works first time; a Re-issue no longer marks a healthy escrow stale; a unit restore states it returned the app's definition and database and NOT the customer's files). **Item 4 closed in controller v0.199.0 + hub v0.96.0** (a rebuilt box DECLARES `offsite.state=needs_credential` and `internal/offsiteheal` re-arms the stored credential before minting). **AND THE "needs someone who knows to look" HALF IS NOW GONE TOO (controller v0.200.0, R-193):** a full-page recovery screen takes over the landing pages while the hub holds a sealed package this box cannot open, explains that nobody can replace a lost recovery code, takes the code, opens the repository and lists what is in it — proven live on demo-felhom, which is genuinely in that state (`/launcher` → 302 `/recovery`; a wrong code refused with nothing written, three times, no lockout; the code found in no file, no container log and no debug ring, with a planted-copy positive control that first exposed a mis-aimed sweep). **WHAT THE QUALIFIER IS NOW, and it is narrower:** (1) **the final unlock has never been driven with a CORRECT code through the page** — no recovery code was kept for demo-felhom's orphaned history and demo-hp's is operator-held out of band, so the live run exercised the whole chain (handler → agent → hub fetch → age KDF) and stopped at the unseal; the install and listing rest on tests. (2) **putting files back in place is deliberately NOT part of this** — restore stays per-app, and the step after the listing is R-213. (3) **the journey has still not been re-walked end to end since these fixes** — the closures are proven individually, not as one uninterrupted run. That re-walk is one more drill and it is what is owed. Scope: `demo-hp`, a controller-data-volume rebuild — **NOT** a total host loss, and **NOT** a guest reprovision. R-198's retention is still unit-proven: nothing has superseded a key in production | | **Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts** | controller v0.197.0 | **PROVEN-LIVE (2026-08-04) — and this row was OPTIMISTIC before it** | On demo-hp, an app on the **system-data fallback** bound `/mnt/sys_drive/userdata/media/books` while the capture set looked in `/mnt/sys_drive/felhom-data/userdata/media/books`: the declared-mandatory directory was in **no** snapshot and the run reported `ok` (R-203). After v0.197.0 the capture log reads `1 mandatory path(s)` and the file is **listed inside the snapshot** — `restic ls -l latest --tag calibre-web` → `-rw-r--r-- 1000 1000 181 … /DRILL-SENTINEL.txt`. Evidence: `audits/DRILL-r201-offsite-recovery-2026-08-04.md` §2 and the v0.197.0 CHANGELOG | **WHAT IT DOES NOT CLAIM.** It covers CAPTURE, not RESTORE: **no file has ever been restored from an off-site backup after a wipe** (R-201, ready to resume). It also does not claim the enrolled-drive layout was ever wrong — it was not, which is precisely why this survived. And a run that still misses a mandatory directory now reports **`incomplete`** rather than `ok`, so this row's guarantee is one the status can express || **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** | | **Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard** | agent v0.104.0 → **v0.121.0**, hub v0.77.0 → **v0.91.0** | **PROVEN-LIVE (2026-08-03)** | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why this was NOT PROVEN-LIVE until now:** rotation had not been observed selecting both tiers across consecutive UNATTENDED cadences — at a 24h cadence a multi-day window — and a single passing run proves the code path, not the schedule. → **R-85**. **R-86 (agent v0.121.0 + hub v0.91.0, 2026-08-03) replaced the schedule and the observation became possible in one afternoon**, because what has to be observed is no longer a multi-day rotation but a RULE: a tier is due when its newest archive that has settled ~24 h has not been proven. **UPGRADE EVIDENCE — a real unattended run on demo-felhom (2026-08-03), triggered by DUE-NESS, not by a timer.** The scheduler's own log: `15:14:38 restore-test tier is DUE … target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z … reason="newest settled archive … has not been proven"` → `proxmox-backup-client restore --crypt-mode=encrypt` under the agent's own token → `15:25:08 gate decision class=guest_destroy guest=990000 allowed=true` → `15:25:14 scratch guest torn down` → **`15:25:14 backup: scheduled restore-test passed archive=felhom-pbs:… duration_s=635.1`**. A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored, booted, verified and destroyed in 635 s, unattended. **The three things a timer could not show**, all verified after it: the state names THAT archive (`{"felhom-pbs":{"archive":"…2026-07-28T04:49:43Z","proven_at":"2026-08-03T13:25:14Z"}}`); a second evaluation reports `due=false … is already proven` and runs nothing; and **an agent restart runs nothing**, which is the defect a person actually noticed — every deploy used to restart the timer. Teardown verified at all three layers: guest absent from `pct list`, **zero** `990000` volumes in `lvs`, and the hub-side `restore_tests[]` entry deliberately RETAINED (it IS the proof the staleness check reads). **R-189 (agent v0.122.0, 2026-08-03) closed the reporting half of this row, and it was a REAL gap in the evidence path:** the proof above reached the hub only because no restart intervened — `restore_tests[]` came solely from an in-memory store, so the 15:25:14 PASS was in fact LOST when the agent restarted 2 m 43 s later for a deploy (`0 restore-tests` on the next two host-reports). Under per-archive due-ness the box would not have repeated the work for a week. The persisted per-tier proof (with the archive, and now the tier) is merged into the report, so a proof survives a restart — one entry per tier, newest wins, and a record that cannot be described honestly is not emitted. **THE HOST TIER'S HALF OF THIS ROW WAS OPTIMISTIC UNTIL 2026-08-03, and it is worth saying plainly.** Every live restore-test cited here is on the OFFSITE tier. The HOST tier was not merely unproven on demo-felhom — it was **unprovable**, and on demo-hp equally: the agent's token had no ACL on `/storage/felhom-backup`, the storage both boxes configure as `local_backup_target`, so the content API answered `{"data":[]}` through the token while root listed three archives. The scheduler skipped it as *"no settled archive yet"* — which is exactly what a brand-new tier reports — so nothing ever said so (**R-185**). **CLOSED 2026-08-03 — agent v0.123.0 + installer 1.24.0.** The grant is applied on both demo boxes (the token now lists 3 and 4 archives), the installer's reuse arm grants on a pre-existing target, and the agent now **asks whether it may read each tier it depends on** instead of inferring it from an empty list: a critical degraded capability naming the storage and the missing role, which the hub alerted and emailed on **while the box was still blind**. **THE HOST TIER IS NOW PROVEN-LIVE, UNATTENDED, ON BOTH DEMO BOXES (2026-08-04) — this row's optimistic half is cashed.** Four SCHEDULED runs, nothing triggered by hand: **demo-felhom** host tier `…2026_08_02-04_42_14.tar.zst` **passed in 83.8 s** at 00:55, offsite `…2026-07-28T04:49:43Z` **passed in 540.4 s** at 06:55; **demo-hp** host tier `…2026_08_02-04_49_29.tar.zst` **passed in 109.3 s** at 02:05, offsite `…2026-07-28T19:19:45Z` **passed in 300.1 s** at 08:05. Each restored into a scratch guest, booted, verified and destroyed itself; `pct list` and `lvs` show zero `990000` afterwards on both, and both boxes' `local-lvm` returned to their pre-run figures (1.95 % and 30.83 %). **Both boxes had BOTH tiers due at once and the ordering was observed live for the first time under R-86's rule:** never-proven sorted first, so each box took its HOST tier, deferred the offsite one, and picked it up on the following evaluation six hours later — one heavy operation at a time, per box, without anyone sequencing it. **The proofs reached the hub**, which is R-189 carrying a host-tier entry for the first time: demo-felhom's report holds **two** entries, one per tier, and the `local` one can only have come from the persisted state because the in-memory store held only that morning's offsite run. **SCOPE, stated because one box proving something does not make it a fleet property:** this covers `demo-felhom` and `demo-hp`. The tester's box is untested and untouched. Evidence: `felhom-agent/REPORT.md`, `felhom.eu/REPORT.md` | | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below | diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 4af1b8c..c2aee2b 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -367,13 +367,23 @@ here — §8 has rows where it is the actual state. | 3 | **The restore's default returned the wrong thing, silently** | `mode=unit` restores the recovery unit — the app's definition, configuration and database dumps — and **not the customer's files**; the userdata in the same snapshot is excluded by `--include`. The outcome message was one sentence for both modes and named neither scope. On the last step of a disaster recovery, the default quietly did not do what the person asked. | **CLOSED — controller v0.198.0** (R-204 item 3). The unit outcome names what came back, what did not, and the step that gets it; the wizard's intent card states its scope before the choice. **The size gate on `mode=full` is untouched**, and the default stays `unit`. | | 4 | **A rebuilt box cannot obtain an off-site credential unaided** | The one-time provider password was spent by its predecessor, so the rebuilt guest has nothing to authenticate with and an **operator Re-issue** was required. | **CLOSED — controller v0.199.0 + hub v0.96.0** (2026-08-05, operator ruling: automate it, and **the trigger is a state the BOX DECLARES**). The box now reports `offsite.state=needs_credential` when two local facts hold together — a fresh data area AND a hub-held recovery package — and the hub's `internal/offsiteheal` re-arms the stored one-time secret, minting only when there is nothing to re-arm. **Deliberately NOT automated: the escrow ceremony.** A credential is replaceable; the recovery code is not. | -> **[FACT] The honest current answer, updated 2026-08-05: all four are gone, and a customer alone -> still does not have a guided recovery — for a different reason, which is worth keeping straight.** -> No step now REQUIRES an operator. What is missing is the OFFER: there is no customer-facing screen -> that tells a rebuilt box's owner a sealed package is waiting, takes their recovery code and previews -> what would come back (R-193's remaining half). The machinery is self-service; the experience is not -> yet built. **And the whole journey has not been re-walked end to end since these fixes** — the four -> closures are proven individually, not as one uninterrupted run. +> **[FACT] The honest current answer, updated 2026-08-05 (controller v0.200.0): all four are gone AND +> the customer is now offered the recovery.** A full-page screen meets the owner of a rebuilt box while +> the hub holds a sealed package the box cannot open: it explains the situation, says plainly that +> **nobody can replace a lost recovery code**, takes the code, opens the repository and **lists what is +> in it** — apps, dates, sizes. No step requires an operator, and no step requires a command line. +> +> **WHERE THAT STOPS, and it is a real stop.** The screen **unlocks and only unlocks**. It restores +> nothing. Putting files back is per-app, lives in the backups area, and the step after the listing — +> a customer seeing what would change before anything is overwritten — is **not built** (→ R-213). A +> screen that unlocks and then offers to overwrite is two decisions wearing one button, which is why +> the operator ruled them separate. +> +> **What is still owed as evidence:** the final unlock has never been driven with a CORRECT code +> through the page (no code was kept for demo-felhom's orphaned history; demo-hp's is operator-held), +> so the live run exercised handler → agent → hub fetch → age KDF and stopped at the unseal. **And the +> whole journey has not been re-walked end to end since these fixes** — the closures are proven +> individually, not as one uninterrupted run. That re-walk is one more drill. > > **WHY THE TRIGGER FOR STEP 4 IS A DECLARATION, recorded here because it is the design and not an > implementation detail:** from the hub, an ABSENT off-site object means *never configured*, @@ -979,3 +989,4 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one. - It does not estimate a single RTO or RPO. Every blank in §8 is a real gap. - It does not answer §11. Those are the operator's. - It does not claim ratification. +| 5 | **A customer had no way to BEGIN — the only route was a command line** | Everything above was self-service, and nothing told the owner of a rebuilt box that a sealed package was waiting or how to open it. | **CLOSED — controller v0.200.0** (R-193). A full-page recovery screen, shown while the hub holds a package this box cannot open; it unlocks and lists, and **restores nothing** (→ R-213 for the put-back). | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index d62b749..7030fc7 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -57,6 +57,7 @@ drill needed — the fourth is R-193, and until it goes the recovery is still op | ID | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---| +| **R-213** | **Putting files back in place — the half the recovery screen deliberately does not do.** The screen (R-193, controller v0.200.0) unlocks the repository and LISTS what is in it; restoring is per-app and lives in the backups area, and the operator ruled the two separate on 2026-08-05: *a screen that unlocks and then offers to overwrite is two decisions wearing one button*. What is missing is the step after the listing — a customer who can now SEE their files still has to work out, per app, which restore to choose. **The operator named its requirement: a live-versus-backup comparison** — the customer must be able to see what would change before anything is overwritten | **OPEN — not started, deliberately** | the comparison design (nothing exists for it yet) | Design the live-vs-backup comparison, then the put-back flow on top of it. Do NOT fold it into the recovery screen | Operator + CC | | **R-212** | **The orphaned-ciphertext deletion HALTED: the stores on the storage box do not match this register's record.** Ruling 2 (2026-08-04) says *"delete the orphaned ciphertext, ~1.2 GB across the two demo boxes, in set-aside stores nothing prunes"*. **Measured 2026-08-05 over SFTP with each box's own credential, read-only:** demo-felhom (`u629488-sub1`) holds `felhom-repo` **1.2 G**, `felhom-repo.orphaned-20260717` **1.4 G**, `felhom-repo.orphaned-20260718` **3.0 M**; demo-hp (`u629488-sub3`) holds `felhom-repo` **582 K**, `felhom-repo.orphaned-20260804` **43 M**. So there are **THREE** set-aside stores totalling **~1.45 GB**, not two totalling ~1.2 GB — and **the figure that DOES match ~1.2 GB is demo-felhom's LIVE `felhom-repo`**, which is the configured `repo_path` and must not be deleted (it is unopenable by the box, which is R-193's point, but it is not a set-aside store) | **CLOSED 2026-08-05 — all three deleted after the operator confirmed the corrected list** | an exact path list, confirmed by the operator | Confirm which of the three `*.orphaned-*` paths to delete. **Nothing was deleted**; the listing above is the complete measured state | **DONE 2026-08-05.** The operator was shown the measured listing, confirmed *delete all three*, and all three `*.orphaned-*` stores were removed: demo-felhom `felhom-repo.orphaned-20260717` (1.4 G) + `felhom-repo.orphaned-20260718` (3.0 M), demo-hp `felhom-repo.orphaned-20260804` (43 M). **~1.45 GB freed.** **Both LIVE `felhom-repo` stores are untouched** (demo-felhom 1.2 G, demo-hp 582 K) — confirmed by a full listing before AND after on each account. **Proof nothing live was caught:** a REAL off-site run on demo-hp immediately afterwards returned `status: ok`, `orphaned: false`, `last_error: ""`, `last_run 2026-08-05T09:14:02Z`, 6 snapshots. **METHOD NOTE for the next session:** the Hetzner storage box has a RESTRICTED shell — no shell operators, no `test`, no GNU long flags. `rm -r ` as ONE simple command works; `test -d X && rm -rf -- X` returns *"Command not found"* and silently does nothing, which is how the first attempt failed CLOSED (verified by an unchanged listing). | | **R-88a** | ~~Failing backup re-quiesces every 5 min, no backoff~~ | **SHIPPED** (controller v0.176.0, 2026-07-27) | — | Live on both boxes; breaker 15m→4h, per-tier, never permanent | — | | **R-88b** | ~~`/backup/due` cannot say *unknown*~~ | **SHIPPED + PROVEN-LIVE** (agent v0.105.0 + controller v0.178.0, 2026-07-27) | — | `age_state=unknown` captured on real hardware during a deliberate ep0 outage; controller deferred, **zero app stacks stopped** | — | @@ -96,7 +97,7 @@ drill needed — the fourth is R-193, and until it goes the recovery is still op | **R-203** | **A customer-declared MANDATORY data directory was silently absent from the off-site snapshot while the run reported `ok`.** Measured live on demo-hp 2026-08-04: `calibre-web` declares `userdata: media/books class: mandatory`; its live bind is `/mnt/sys_drive/userdata/media/books` (the sentinel file was there), while the off-site capture set looked for `/mnt/sys_drive/felhom-data/userdata/media/books`, which does not exist. Result: `[WARN] mandatory data path missing on disk, skipped from offsite`, then `backed up calibre-web (…, 0 mandatory path(s))` and `backup OK: 3 app(s), 3 snapshot(s)` — `last_status: ok`, `last_success` stamped, nothing customer-visible, nothing hub-visible | **SHIPPED + PROVEN-LIVE 2026-08-04** (controller **v0.197.0**) | — | **THE MECHANISM, from source.** `NamespaceRoot(drivePath, inGuestDrive)` (`appbackup/paths.go:28-33`) appends the `felhom-data` segment **when the drive IS the system data path** (`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`, `backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is `/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an app deployed at `HDD_PATH=/mnt/sys_drive`, the two produce different directories. **The same compose used BOTH roots**, from one deploy: `${IMPORT_PATH}` → `/mnt/sys_drive/felhom-data/userdata/import/calibre` (with the segment), `${USERDATA_PATH}` → `/mnt/sys_drive/userdata/media/books` (without). **WHAT IS MEASURED vs NOT, because it changes the fix.** MEASURED: the paths disagree, the mandatory directory is absent from the snapshot, the run says `ok`, and the only signal is a container-log WARN. **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a SUPPORTED choice — it was used because demo-hp's only registered drive (`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working as designed), while `/mnt/sys_drive` was **accepted** (HTTP 202). **EITHER BRANCH IS A DEFECT:** if the system drive is a supported app namespace, userdata resolution is wrong for every app on it and their mandatory directories are silently unprotected; if it is not supported, the deploy accepted a namespace it should have refused one call after refusing the NAS. **NOT a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths (everything of theirs is in named volumes), so they are unaffected and their snapshots are real. **Fix shape:** make the two roots one function, whichever is right — and make a skipped MANDATORY path a customer/hub-visible signal rather than a WARN, because `ok` with a missing mandatory directory is this project's own *a path the customer thinks is protected is not in the snapshot* shape. Source: `audits/DRILL-r201-offsite-recovery-2026-08-04.md` §2 **BOTH HALVES SHIPPED.** **(1) The paths.** `appbackup`'s helpers take a NAMESPACE ROOT; the census found **FIVE** bare-drive-path callers, not the four the spec named — the fifth is the **FileBrowser mount builder** (`web/handlers.go`), i.e. the customer's own file browser would have shown the wrong directory on a non-enrolled path (latent: the system drive is deliberately never a registered `StoragePath`). The rule now has **ONE expression** (`appbackup.NamespaceRootFor` / `IsEnrolledDrive`); there were already **two** copies and **they differed** — `backup.Manager.namespaceRoot` compared without `filepath.Clean`, `stacks.Manager.inGuest` with it, so a trailing slash from config would have flipped the mode in one package and not the other. `ComputeFabBuckets` now receives the namespace root, which is what `ComputeCaptureSet` has always received, so the export and the backup describe the same directories by construction. **(2) The verdict.** `last_status` gains **`incomplete`** — minted, because `ok`|`error`|`running` had nothing meaning *"it ran, and this app is not fully protected"*. **Not `error`:** the rest of the run worked, so `SnapshotCount` and the `LastSuccess` anchor still record what WAS captured. It reaches the operator through the **existing** per-run digest (`backup_run_failures`) — a new event type would be a two-repo change and the hub drops anything outside `allowedEventTypes`. **§8.4's narrowing is a NO-OP and no customer warning disappears:** `TierOffsite`'s `tierKeeps()` already admits mandatory only, demonstrated by widening the tier filter alone and watching the class check hold the line. **THE SPEC'S §8.3 RISK DOES NOT EXIST, and this is the correction owed:** `ExportDataMounts` lives in `delete.go` but is **export-only** — its single production caller is the `.fab` adapter, nothing deletes on its result, and the delete path's own guard `ProtectedHDDPaths` is layout-agnostic by construction (it protects BOTH `/…` and `/felhom-data/…`). It shipped as its own commit anyway. **PROVEN LIVE on demo-hp:** the bind moved `/mnt/sys_drive/userdata/media/books` → `/mnt/sys_drive/felhom-data/userdata/media/books`, the capture log went `0 mandatory path(s)` → **`1 mandatory path(s)`**, and **the sentinel is in the snapshot's own file listing** — `-rw-r--r-- 1000 1000 181 … /mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt` — not a green status. **Red-proofs:** the bare-path call makes the two paths differ; inverting the drive-kind comparison breaks every enrolled row; leaving the export site bare emits the short path; and the verdict fails under both an unreachable gap-recording and an unconditional `ok`. **One red-proof PASSED and the test was wrong, not the code** — the first Scenario-C test only reached `offboxCaptureSet` while the mutation lives in `runOffboxInternal`; a run-level test replaced it | CC | | **R-204** | **A rebuilt box can recover its off-site key and still cannot use it: the remedy that reconfigures the tier is the thing that blocks the recovery.** Measured end to end on demo-hp during the 2026-08-04 night drill, after a real controller-data wipe | **ALL FOUR ITEMS CLOSED 2026-08-05** (items 1–3 controller v0.198.0 + hub v0.95.0; item 4 controller v0.199.0 + hub v0.96.0) | — | **THE CHAIN, each link measured.** (1) A rebuilt controller **cannot configure its off-site target at all**: `offsite-apply: consume one-time password: no unconsumed offsite password` — the previous controller consumed it (ledger: created `07:11:51`, consumed `07:12:06`). That is R-193, reconfirmed live. (2) The documented remedy is an operator **Re-issue**, which works — and **sets `stale_at` on the escrow** (measured: `2026-08-04 20:15:49`) **while `restic_pw_sha256` is unchanged**, i.e. R-196's false staleness. (3) A stale escrow makes the hub **withhold the hash from the report ACK**, so `EscrowAutoConfirmer` can never flip `pending → escrowed`, and `OffboxRunnable` (`configured && escrowed`) **refuses every off-site run**. (4) The only documented way to clear a stale escrow is **a fresh ceremony — which supersedes the identity blob and destroys the key being recovered.** **So the recovery and its precondition are mutually exclusive as built.** The key comes back (proven — `8a9e33aa4da6…` recovered byte-identical after the wipe, and installed) and then cannot be used to open the repository. **(5) A FOURTH link, undocumented anywhere:** a rebuilt box is **unclaimed**, and the claim gate correctly intercepts every non-claim route (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`), so **no controller endpoint responds at all** until the customer re-claims. That step appears in no design document and comes first. **Fix shape, not decided here:** either the Re-issue stops marking the escrow stale on evidence it does not have (R-196's own fix), or a recovered-and-verified key is allowed to confirm the escrow without a ceremony — the hash comparison that `EscrowAutoConfirmer` already performs is exactly the evidence needed, and it is being withheld precisely when it would be conclusive. **Do NOT fix by widening `OffboxRunnable`** — the atomicity guarantee it enforces (no un-recoverable ciphertext) is the reason the escrow exists. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3 **ALL FOUR WALKED AND MEASURED during the passing 2026-08-04 drill, so this row is now a known-good manual runbook AND the gap list.** (1) **R-193:** a rebuilt controller cannot configure its off-site tier — `no unconsumed offsite password` (ledger: created `07:11:51`, consumed `07:12:06` by its predecessor). Remedy: operator Re-issue. (2) **The claim gate:** a rebuilt box is unclaimed, so the gate intercepts EVERY controller endpoint (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`) — a first step of every recovery that appears in no design document. **And the local escape hatch does not work unaided:** `--print-reset-code` writes the new hash to `settings.json` while the RUNNING controller keeps its old copy in memory, so `effectiveClaimCode()` never sees it and the claim fails with *"Hibás vagy lejárt kód"*. **The controller must be restarted between minting and claiming** — two attempts failed before this was diagnosed. (3) **R-196:** the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged → the hub withholds the hash → auto-confirm can never fire → `OffboxRunnable` refuses every run. Cleared here with the **manual** confirm (`/backup/offbox/confirm-escrow`) — NOT a ceremony, which would have superseded the identity blob and destroyed the recovered key. (4) **`mode=unit` is the restore default and returns the recovery unit, NOT the userdata leg.** A customer told to "restore from off-site" gets their app definition back and not their documents, and nothing in that outcome says so. **Fix priorities, in the order they hurt:** (4) is a customer-facing trap on the last step; (3) is R-196's fix; (2) needs the escape hatch to reload settings (or the claim state to survive a rebuild); (1) is R-193's restage. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3–§4 | CC + operator **OUTCOME 2026-08-05 — controller v0.198.0 + hub v0.95.0.** **Item 1 (the reset code needs a restart) — CLOSED, proven live.** `effectiveClaimCode` reads through to the persisted claim state, so a code minted by the separate `--print-reset-code` process is seen without a restart; the precedence rule between settings and config is unchanged. Read-through, not a TTL: a TTL leaves a window in which a superseded code still works, and that is the mutation `TestClaimCode_SupersededByASecondMint_RefusedImmediately` kills. Fails closed on an unreadable state. **Live on demo-felhom 9201, nothing restarted (`restarts=0`, container older than both mints): the superseded code returned „Hibás vagy lejárt kód" and the current one was accepted first time.** **Item 2 (a re-issue marks a healthy escrow stale) — CLOSED, → R-196.** Test-proven; deliberately NOT fired live on demo-hp. **Item 3 (the restore's default silently returns the wrong thing) — CLOSED, proven live.** A `mode=unit` restore now names what came back, what did not and the step that gets it; the wizard's intent card states its scope BEFORE the choice; the full-restore size gate is untouched and pinned as unchanged. **The default stays `unit`** — all three wizard forms set `mode` explicitly, so changing it would alter nothing the customer sees while silently changing a mode-less POST. **ITEM 4 REMAINS AND IS THE WHOLE OF WHAT IS LEFT HERE: a rebuilt box cannot obtain an off-site credential unaided**, because the one-time password was spent by its predecessor, so an operator Re-issue is still required. **Its dependency is the one-shot credential design decision — it needs an operator ruling and belongs to → R-193.** Not begun in this session, deliberately. **ITEM 4 CLOSED 2026-08-05 — controller v0.199.0 + hub v0.96.0.** The box now DECLARES that it needs a credential (`offsite.state=needs_credential`) instead of reporting an absence the hub cannot interpret; the hub's new `internal/offsiteheal` answers it. **Operator ruling, recorded because a ruling that lives only in a conversation binds nobody (R-96): the trigger is a state the BOX DECLARES, not an inference.** An absent off-site object has FOUR meanings — never configured, mid-restart, a transient config read failure, rebuilt-and-stranded — and the hub cannot tell them apart; the box can, from two local facts (a fresh data area AND a hub-held recovery package). **Both halves are required:** freshness alone would make every un-configured box in the fleet ask for a credential, which is what `TestOffsiteDeclare_NeverHadOffsiteSaysNothing` exists to catch. The reconciler mirrors `pbsdrheal`: declared states only, a two-DISTINCT-REPORT debounce (derived from the ~15-min report cadence), **restage before mint**, an event per remediation, and a healthy box is a pure no-op. **PROVEN LIVE:** demo-felhom 9201 arranged (reversibly) into the stranded shape produced report id=16743 carrying `{enabled:false, state:needs_credential, quota_gb:0, repo_size_bytes:0}`, and the single declaration was **absorbed by the debounce** — no self-heal event fired — with the box restored the same minute. **What is deliberately NOT automated: the escrow ceremony.** A credential is replaceable; the recovery code is not. **Credential automatic, key customer-present.** **Second ruling recorded: the dashboard-password exposure on the recovery preview is METADATA (backup dates, app names), not content, and is ACCEPTED.**| | **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **CLOSED 2026-08-05 — the guard's scoping half closed BY REPLACEMENT** (hub v0.96.0) | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. **SPIKE 2026-08-04 — BOTH HALVES CONFIRMED WITH NUMBERS, still OPEN, still not fixed here** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q7). The query is quoted at source (`store.go:987`): `... ORDER BY id LIMIT 500` = **the oldest 500**. Reproduced against the live hub DB with demo-hp's real consume anchor `2026-07-23 09:53:41`: the guard sees `total=500, withOffsite=500`, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** — the entire evidence set ends **six days before** the 2026-08-03 rebuild. The true window totals are `1174 / 1063` (⇒ 111 without, matching the 111 offsite-less reports). So the e-mail's *"500 report(s) since carry no offbox target"* interpolates `ReportsSinceConsume` while `OffsiteReportsSinceConsume` was **500** — the message states the precise negation of its own measurement. `offsite_credential_restaged` has **never fired for any customer** (zero rows of that type in the DB — checked, not assumed). **A NEW REASON NOT TO FIX THIS IN ISOLATION, from the same spike:** under R-193's Q2 finding a successful auto-restage would have restored demo-hp's TRANSPORT while the box minted a new repo password anyway — the heal can protect the plumbing and **cannot** protect the data, and had it fired on 2026-08-04 both boxes would have looked healthy while their snapshots were orphaned. **That is strictly worse than the current loud failure.** Whatever shape the fix takes must say so in its message. **RECON 2026-08-04 adds two inputs and changes no verdict** (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **(1) The guard's own conclusion — that a fix here protects the plumbing and not the data — is now stronger, not weaker:** even a perfect restage leaves the rebuilt box minting a fresh repo password, and per **R-198** the old one is destroyed by the re-ceremony the box is pushed into. Whatever shape the message takes must say that in the same breath, or it will read as an all-clear. **(2) The recency-bounded discriminator this row asks for has a ready anchor the hub already receives:** the report ACK's `escrow{identity_blob_present, restic_pw_sha256}` moves when a box re-keys, so "evidence since the newest re-key" is computable from data already stored — the same observation R-197 makes, aimed at this guard's time window. **BOTH HONESTY HALVES SHIPPED 2026-08-04 (hub v0.93.0); THE GUARD'S LOGIC IS DELIBERATELY UNTOUCHED.** **(a) The message now describes what was measured.** The one stuck state is reported as the two situations it actually covers — **burned** (`OffsiteReportsSinceConsume == 0`) and **regressed** (> 0, the demo-hp shape) — each stating its own measurement and carrying its own recommendation; the regressed text explicitly WITHDRAWS Re-issue and points at what removes an offbox target (a guest rebuild, R-193). `offsite_reports_since_consume` rides the details for the first time. **(b) Every refusal to self-heal leaves a record** — a `notification_log` row on the operator channel, status `refused`, with its reason (the R-182 suppressed-e-mail precedent), riding the stuck event's 24 h cadence so it sits beside the e-mail it explains rather than accumulating per tick. The two conditions were split into separate branches solely so each can name its own reason; **the set of situations in which the heal fires is byte-for-byte what it was**. **(c) Not in the spec and done anyway, narrowing only:** `offsite_delivery_stuck` and `offsite_credential_restaged` are added to `operatorOnlyEvents`. Neither was ever registered and neither has a `customerMessages` entry — which is not a block — so a customer with a configured recipient was in line for an English e-mail about one-time passwords being *"likely burned"*. Measured live: `notification_log` holds operator rows for demo-hp and no customer rows, which is NOT evidence the leg was blocked (equally consistent with no configured recipient), so the register makes it structural. **WHAT STAYS OPEN, and it is this row now:** `CountReportsOffsiteSince` reads `ORDER BY id LIMIT 500` — the OLDEST 500 reports after the consume — so the counts describe the start of the window, not the present. Its correct shape (recency-bounded, rebuild-aware) depends on the recovery chain that is not yet assembled (R-199/R-200/R-201), so it was NOT fixed here. **The window is named inside the alert text** so the limitation travels with the number instead of being laundered into a confident sentence. **Red-proofs observed:** restoring the single hardcoded sentence fails both message scenarios (the first mutation attempt left the default branch in place and only the burned scenario failed — recorded because a mutation that does not remove every guard is not a red-proof); replacing the regressed branch with a bare `return` fails the refusal record and its cadence test. | CC **CLOSED 2026-08-05 (hub v0.96.0), by REPLACEMENT rather than repair.** The guard inferred the shape by counting how many of the **OLDEST 500** reports after a consume carried an offbox target — on demo-hp all 500 predated the rebuild, so the checker reported the *regressed* shape with confidence and declined to heal for 108 reports while the box sat stranded. A DECLARED state needs no window, no count and no inference, so `shapeDeclared` now **outranks both inferred shapes**, the stuck e-mail states what the box declared instead of prescribing Re-issue, and `maybeHeal` **stands down with a record** because `internal/offsiteheal` owns the remediation — two mechanisms healing one customer would double-issue, and this one can only mint, so it would also skip the cheap restage path. The counting fields remain for boxes on a pre-v0.199.0 controller, where there is still nothing better to go on.| -| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **CREDENTIAL HALF CLOSED 2026-08-05** (controller v0.199.0 + hub v0.96.0); the recovery SCREEN and the ciphertext deletion stay OPEN | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTP access key…)"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the one a rebuild destroys, and **nothing automatic ever restages it**: `WriteOffboxSecrets` mints a fresh 256-bit password whenever `/offbox/repo_password` is absent (`offbox.go:392`), and the only recovery path, `InjectOffboxPassword`, has **exactly one caller in the whole repo** — a web form a human pastes into (`web/offbox_handlers.go:189`). **(2) MEASURED, WITHOUT TOUCHING A BOX:** the hub already stores `restic_pw_sha256` on both the live and the superseded escrow, so the question is a hash comparison. demo-hp `8e03eddf…` → `8a9e33aa…`; demo-felhom `48741892…` → `c60c8bc7…`. **Both boxes minted a new repository password.** **(3) THE CONTRAST IN THIS ROW IS FALSE FOR THE DATA.** demo-felhom's 76-second "lucky" recovery restored **delivery only** — its pre-rebuild object carried **36 snapshots / 1.14 GB** (`repo_size_bytes 1136685919`) and it has reported `snapshot_count: 0` in all 109 reports since, with a changed repo password and **nothing marking its escrow stale for 13 h**. Both boxes lost repository continuity; one loudly, one silently, and **the silent one is worse**. **(4) Q3 IS UNMEASURED AND THE BINARY WAS WRONG.** Neither box could run on 2026-08-04 02:15 UTC (demo-hp had no target at all until 07:15; demo-felhom's target was `escrow_state: pending`, which `OffboxRunnable` blocks) — **the decisive run is 2026-08-05 ~02:15 UTC**. Source predicts a **third** outcome, neither 15 nor 1: same account + same repo path (`u629488-sub3:/home/felhom-repo`, and `repoPath` is a compile-time constant) + new password + `claimed: 1` ⇒ `ensureOffboxRepo` classifies **orphaned** and returns `ErrOffboxOrphaned` — **the run refuses and shows the orphan card**, it does not start a silent fresh history. **Record which of the three actually occurs; a prediction from source is not a measurement.** **(5) CANDIDATE (b) IS NOT IMPLEMENTABLE AS STATED** — the escrow is R-wrapped/zero-knowledge and the hub has no recovery code, so "recoverable from escrow at re-bootstrap" describes a customer-present ceremony, i.e. the manual form that already exists. **(6) CANDIDATE (a) ALREADY EXISTS AND IS WIRED TO THE WRONG EVENT:** `reissueOnReenroll`'s **F3 leg** does exactly this (`api/handler.go:1051-1084`) but sits behind `handleHostEnroll`'s mint-once-reuse short-circuit (`if existing != nil { return }`), and a **guest** rebuild leaves the `hosts` row intact — so F3 is never reached. **(7) A NEW CANDIDATE (c), not previously named and recommended second:** the **agent survives a guest rebuild**, already receives the repo password over the pinned local API (`POST /escrow/stage-secret`) and already writes it to a fixed 0600 path — it merely **wipes** it after the ceremony. Retaining it and serving it back is the only candidate that addresses the irreplaceable secret, and every seam it needs exists. Its cost is one deliberate trade the operator must make (a copy of the data key at rest on the Proxmox host — see D6). **SPIKE RECOMMENDATION: ship the honesty pass (R-196 + R-197) now; then decide (c). Do NOT ship (a) first — it would have hidden this.** **NOT CLOSED — WAITING-ON-OPERATOR for the (c)-vs-accept-it decision, stated at the end of the spike.** **RECON 2026-08-04 — the chain was traced link by link, and the picture is worse than the spike's** (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **(A) THE SPIKE'S CANDIDATE (b) IS OVERTURNED IN PART.** "Recoverable from escrow" is not blocked by zero-knowledge — the *hub* cannot open the blob, the *customer* can, with R. What is genuinely impossible is an **unattended** recovery. A **customer-present** one is a real design, and the operator has now ruled on its shape (below). **(B) THE CHAIN IS NOT ASSEMBLED — eleven links, and the automation stops at four.** Mint → stage → seal → store on the hub are PROVEN-LIVE. Then: the hub's blob-serving endpoints have **no client anywhere** (**R-199**); unsealing's only production caller is a `--selftest` mode reading R from an env var; nothing extracts `restic_repo_password` from the recovered bundle; the injection seam **has no form** (**R-200**); and reopening an existing repo with a recovered password has **never happened** (**R-201**). **(C) THE LOAD-BEARING NEW FACT, and it retires this row's own "hub v0.60.0 retains superseded escrow" premise: the retention does NOT retain the restic repo password** — `host_escrow_superseded` has no `identity_blob` column (**R-198**). Both demo boxes' old passwords were destroyed by the 2026-08-04 re-ceremonies, so the orphaned snapshots are unrecoverable for a **second, independent** reason; keeping R would not have helped. **(D) A FAIL-CLOSED REFUSAL IS IMPLEMENTABLE — this is the most useful thing settled.** The hub already tells every box, on every report ACK, `escrow{identity_blob_present, restic_pw_sha256, created_at}` (`hub/internal/api/handler.go:504-510`) — and the controller **discards it** whenever no offbox target exists (`report/escrow_confirm.go:75-84`). Persisting it (the `ClaimSync` set-only pattern, `report/claim_sync.go:39-53`) and refusing to mint when a blob covers a password we do not have needs **no new hub API and no new secret**. **(E) OPERATOR RULING, 2026-08-04, recorded verbatim:** *"If a node is a fresh install AND the hub has a recovery blob, then the controller should yell that recovery is available, and provide a form for the customer to enter the recovery key. After unlocking the blob, the controller should show what will be recovered before proceeding."* Priced row by row in the recon §9: fresh-install signal **exists** (the mint branch's own `os.Stat`); hub-has-a-blob **exists on the wire, S to persist**; the yell **S**; an **R** form **does not exist** (the UI has only ever *emitted* R) **S**; unsealing must cross agent→controller because the controller image ships no `age` — **M**, one new agent local-API endpoint mirroring `/escrow/ceremony/claim`, plus a narrow hub `GET /hosts//escrow`; the **preview is cheap and read-only** — `restic snapshots --json` + `stats --mode raw-data --json` are already how the box counts snapshots (`offbox.go:1234-1265`), so count, dates, sizes, app tags and paths are all knowable before committing **S**. **Security question put to the operator, not answered:** the form sits behind the dashboard password (bcrypt + CSRF + 5/min lockout); the preview exposes backup cadence and app names; the form is an **oracle** for a stolen R and must fail as generically as `UnwrapIdentity` already does; and R transits the agent, which is the same trade as option (c) in a smaller, time-bounded form. **(F) THE DRILL IS DESIGNED AND NOT RUN** (recon §10): demo-hp, ~3–4 h, R kept deliberately, sentinel file sha256 before and after, **pass = byte-identical sha256, NOT "the repository opened"**, fail = a snapshot count of 1. **Run R-198's fix first.** **(G) Q3 STILL UNMEASURED:** neither box has run since (`last_run` absent on both, 2026-08-04 09:49/09:56 reports) — the decisive run remains **2026-08-05 ~02:15 UTC**. **OPERATOR RULINGS 2026-08-04, and one of them changes what the other items are for.** (1) **Candidate (c) is REFUSED — the risk is accepted:** no repository password is retained on the Proxmox host. **That makes the customer-present recovery path the ONLY way back from a rebuild**, which is why R-198 was shipped the same day as a load-bearing fix rather than a tidy-up: with no host-retained copy, everything runs through the retained identity blob, and until hub v0.93.0 the ceremony destroyed it. (2) **Run the drill, after R-198** — R-198 has shipped, so the drill is the next session (design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10; tracked as R-201). (3) **Delete the orphaned ciphertext** — ~1.2 GB across the two demo boxes; **STILL OWED**, deliberately not done in v0.93.0 (a destructive act on a protected endpoint does not ride a schema-change release). **What v0.93.0 delivers against this row:** the key now SURVIVES a supersession (R-198) and a changed key is now REPORTED on the day (R-197). **What it does NOT:** the chain that hands the key back is still unassembled at three links — R-199 (no client for the hub's blob-serving endpoints), R-200 (no form for the injection seam), R-201 (never exercised end to end). This row stays open until the drill returns a byte-identical sentinel file. | CC + operator **THE CREDENTIAL HALF IS CLOSED (2026-08-05).** A rebuilt box no longer needs an operator Re-issue: it declares, and `internal/offsiteheal` re-arms the stored one-time secret (or mints if there is nothing to re-arm). See R-204 item 4. **STILL OPEN under this ID:** (a) the customer-facing recovery SCREEN — a freshly installed box that finds a sealed package waiting should say so and offer a recovery-code box with a preview; the operator ruling on its dashboard-password exposure is recorded (metadata, not content — accepted), but nothing is built; (b) ~~the orphaned-ciphertext deletion~~ — **DONE 2026-08-05**, ~1.45 GB across three set-aside stores, after the operator confirmed a CORRECTED list (the register's "two stores, ~1.2 GB" was wrong in both numbers, and the 1.2 GB figure matched a LIVE repo). See R-212.| +| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **CLOSED 2026-08-05 — controller v0.200.0** (credential half v0.199.0/v0.96.0; the recovery SCREEN v0.200.0) | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTP access key…)"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the one a rebuild destroys, and **nothing automatic ever restages it**: `WriteOffboxSecrets` mints a fresh 256-bit password whenever `/offbox/repo_password` is absent (`offbox.go:392`), and the only recovery path, `InjectOffboxPassword`, has **exactly one caller in the whole repo** — a web form a human pastes into (`web/offbox_handlers.go:189`). **(2) MEASURED, WITHOUT TOUCHING A BOX:** the hub already stores `restic_pw_sha256` on both the live and the superseded escrow, so the question is a hash comparison. demo-hp `8e03eddf…` → `8a9e33aa…`; demo-felhom `48741892…` → `c60c8bc7…`. **Both boxes minted a new repository password.** **(3) THE CONTRAST IN THIS ROW IS FALSE FOR THE DATA.** demo-felhom's 76-second "lucky" recovery restored **delivery only** — its pre-rebuild object carried **36 snapshots / 1.14 GB** (`repo_size_bytes 1136685919`) and it has reported `snapshot_count: 0` in all 109 reports since, with a changed repo password and **nothing marking its escrow stale for 13 h**. Both boxes lost repository continuity; one loudly, one silently, and **the silent one is worse**. **(4) Q3 IS UNMEASURED AND THE BINARY WAS WRONG.** Neither box could run on 2026-08-04 02:15 UTC (demo-hp had no target at all until 07:15; demo-felhom's target was `escrow_state: pending`, which `OffboxRunnable` blocks) — **the decisive run is 2026-08-05 ~02:15 UTC**. Source predicts a **third** outcome, neither 15 nor 1: same account + same repo path (`u629488-sub3:/home/felhom-repo`, and `repoPath` is a compile-time constant) + new password + `claimed: 1` ⇒ `ensureOffboxRepo` classifies **orphaned** and returns `ErrOffboxOrphaned` — **the run refuses and shows the orphan card**, it does not start a silent fresh history. **Record which of the three actually occurs; a prediction from source is not a measurement.** **(5) CANDIDATE (b) IS NOT IMPLEMENTABLE AS STATED** — the escrow is R-wrapped/zero-knowledge and the hub has no recovery code, so "recoverable from escrow at re-bootstrap" describes a customer-present ceremony, i.e. the manual form that already exists. **(6) CANDIDATE (a) ALREADY EXISTS AND IS WIRED TO THE WRONG EVENT:** `reissueOnReenroll`'s **F3 leg** does exactly this (`api/handler.go:1051-1084`) but sits behind `handleHostEnroll`'s mint-once-reuse short-circuit (`if existing != nil { return }`), and a **guest** rebuild leaves the `hosts` row intact — so F3 is never reached. **(7) A NEW CANDIDATE (c), not previously named and recommended second:** the **agent survives a guest rebuild**, already receives the repo password over the pinned local API (`POST /escrow/stage-secret`) and already writes it to a fixed 0600 path — it merely **wipes** it after the ceremony. Retaining it and serving it back is the only candidate that addresses the irreplaceable secret, and every seam it needs exists. Its cost is one deliberate trade the operator must make (a copy of the data key at rest on the Proxmox host — see D6). **SPIKE RECOMMENDATION: ship the honesty pass (R-196 + R-197) now; then decide (c). Do NOT ship (a) first — it would have hidden this.** **NOT CLOSED — WAITING-ON-OPERATOR for the (c)-vs-accept-it decision, stated at the end of the spike.** **RECON 2026-08-04 — the chain was traced link by link, and the picture is worse than the spike's** (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **(A) THE SPIKE'S CANDIDATE (b) IS OVERTURNED IN PART.** "Recoverable from escrow" is not blocked by zero-knowledge — the *hub* cannot open the blob, the *customer* can, with R. What is genuinely impossible is an **unattended** recovery. A **customer-present** one is a real design, and the operator has now ruled on its shape (below). **(B) THE CHAIN IS NOT ASSEMBLED — eleven links, and the automation stops at four.** Mint → stage → seal → store on the hub are PROVEN-LIVE. Then: the hub's blob-serving endpoints have **no client anywhere** (**R-199**); unsealing's only production caller is a `--selftest` mode reading R from an env var; nothing extracts `restic_repo_password` from the recovered bundle; the injection seam **has no form** (**R-200**); and reopening an existing repo with a recovered password has **never happened** (**R-201**). **(C) THE LOAD-BEARING NEW FACT, and it retires this row's own "hub v0.60.0 retains superseded escrow" premise: the retention does NOT retain the restic repo password** — `host_escrow_superseded` has no `identity_blob` column (**R-198**). Both demo boxes' old passwords were destroyed by the 2026-08-04 re-ceremonies, so the orphaned snapshots are unrecoverable for a **second, independent** reason; keeping R would not have helped. **(D) A FAIL-CLOSED REFUSAL IS IMPLEMENTABLE — this is the most useful thing settled.** The hub already tells every box, on every report ACK, `escrow{identity_blob_present, restic_pw_sha256, created_at}` (`hub/internal/api/handler.go:504-510`) — and the controller **discards it** whenever no offbox target exists (`report/escrow_confirm.go:75-84`). Persisting it (the `ClaimSync` set-only pattern, `report/claim_sync.go:39-53`) and refusing to mint when a blob covers a password we do not have needs **no new hub API and no new secret**. **(E) OPERATOR RULING, 2026-08-04, recorded verbatim:** *"If a node is a fresh install AND the hub has a recovery blob, then the controller should yell that recovery is available, and provide a form for the customer to enter the recovery key. After unlocking the blob, the controller should show what will be recovered before proceeding."* Priced row by row in the recon §9: fresh-install signal **exists** (the mint branch's own `os.Stat`); hub-has-a-blob **exists on the wire, S to persist**; the yell **S**; an **R** form **does not exist** (the UI has only ever *emitted* R) **S**; unsealing must cross agent→controller because the controller image ships no `age` — **M**, one new agent local-API endpoint mirroring `/escrow/ceremony/claim`, plus a narrow hub `GET /hosts//escrow`; the **preview is cheap and read-only** — `restic snapshots --json` + `stats --mode raw-data --json` are already how the box counts snapshots (`offbox.go:1234-1265`), so count, dates, sizes, app tags and paths are all knowable before committing **S**. **Security question put to the operator, not answered:** the form sits behind the dashboard password (bcrypt + CSRF + 5/min lockout); the preview exposes backup cadence and app names; the form is an **oracle** for a stolen R and must fail as generically as `UnwrapIdentity` already does; and R transits the agent, which is the same trade as option (c) in a smaller, time-bounded form. **(F) THE DRILL IS DESIGNED AND NOT RUN** (recon §10): demo-hp, ~3–4 h, R kept deliberately, sentinel file sha256 before and after, **pass = byte-identical sha256, NOT "the repository opened"**, fail = a snapshot count of 1. **Run R-198's fix first.** **(G) Q3 STILL UNMEASURED:** neither box has run since (`last_run` absent on both, 2026-08-04 09:49/09:56 reports) — the decisive run remains **2026-08-05 ~02:15 UTC**. **OPERATOR RULINGS 2026-08-04, and one of them changes what the other items are for.** (1) **Candidate (c) is REFUSED — the risk is accepted:** no repository password is retained on the Proxmox host. **That makes the customer-present recovery path the ONLY way back from a rebuild**, which is why R-198 was shipped the same day as a load-bearing fix rather than a tidy-up: with no host-retained copy, everything runs through the retained identity blob, and until hub v0.93.0 the ceremony destroyed it. (2) **Run the drill, after R-198** — R-198 has shipped, so the drill is the next session (design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10; tracked as R-201). (3) **Delete the orphaned ciphertext** — ~1.2 GB across the two demo boxes; **STILL OWED**, deliberately not done in v0.93.0 (a destructive act on a protected endpoint does not ride a schema-change release). **What v0.93.0 delivers against this row:** the key now SURVIVES a supersession (R-198) and a changed key is now REPORTED on the day (R-197). **What it does NOT:** the chain that hands the key back is still unassembled at three links — R-199 (no client for the hub's blob-serving endpoints), R-200 (no form for the injection seam), R-201 (never exercised end to end). This row stays open until the drill returns a byte-identical sentinel file. | CC + operator **THE CREDENTIAL HALF IS CLOSED (2026-08-05).** A rebuilt box no longer needs an operator Re-issue: it declares, and `internal/offsiteheal` re-arms the stored one-time secret (or mints if there is nothing to re-arm). See R-204 item 4. **STILL OPEN under this ID:** (a) the customer-facing recovery SCREEN — a freshly installed box that finds a sealed package waiting should say so and offer a recovery-code box with a preview; the operator ruling on its dashboard-password exposure is recorded (metadata, not content — accepted), but nothing is built; (b) ~~the orphaned-ciphertext deletion~~ — **DONE 2026-08-05**, ~1.45 GB across three set-aside stores, after the operator confirmed a CORRECTED list (the register's "two stores, ~1.2 GB" was wrong in both numbers, and the 1.2 GB figure matched a LIVE repo). See R-212. **THE RECOVERY SCREEN SHIPPED 2026-08-05 (controller v0.200.0) — R-193 IS CLOSED.** A full page (not a banner) takes over the landing pages while the hub holds a sealed package this box cannot open; afterwards it stays reachable from the backups area for as long as the situation lasts. **It UNLOCKS AND ONLY UNLOCKS** (operator ruling): it explains, takes the recovery code, opens the repository and lists what is in there — apps, dates, sizes — and restores NOTHING. **Three ways out, none a dismiss button:** recover; „most nem" (the full page stops interrupting, the entry point stays PERMANENTLY — bound to the offer, never to the postpone flag); and „nem kérem vissza a korábbi adatokat", confirmed TWICE and reaching the SHIPPED move-aside, which sets aside and never deletes. **ONE CORE, TWO CALLERS:** `RecoverInstallCore` is split out of the CLI, whose exit codes and printed lines are byte-identical (every pre-existing CLI test passes unchanged); both directions asserted from source by AST. **A SHAPE THE TASK DID NOT NAME had to be added:** `WriteOffboxSecrets` auto-generates a repository password on re-apply — R-193's own orphaning mechanism — and since hub v0.96.0 the re-apply happens by itself within ~15–30 min, so the literal "no repository password" trigger would have opened a window that closes on its own before the customer next logs in. The predicate is therefore *the hub holds a package AND (no repository password OR the tier is orphaned)*. **Two defects the tests caught before shipping:** an UNCLAIMED (legacy-open) box would have been shown the page, and the inventory nil-dereferenced in exactly the pristine rebuilt shape. **Still owed under other IDs:** putting files back in place (→ R-213), and one end-to-end re-walk of the whole journey.| | — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC | | — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator | | **R-90** | ~~ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged~~ | **CLOSED — the operator rescaled ep0 to a CX33 on 2026-08-03** | — | **MEASURED ON THE BOX, not read from an invoice:** `felhom-hetzner` reports `Mem: 7757` MB total (**8 GB**, was 3.8) and `nproc` **4**. **The interim lever survived and was checked rather than assumed** — a resize is a stop/start, so "the swapfile is still there" was an assumption until measured: `/swapfile`, 4 GiB, dated `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`), 0 B in use on an idle box. **THE 40 GB LOCAL DISK DID NOT CHANGE** and must not be "corrected" alongside the RAM: `/` is 38 G, 58% used. This was a CPU/RAM resize only, so every disk figure in the runbooks still stands — the separate 98 G volume at `/mnt/pbs-datastore` (R-82 P0.3) is unaffected. **Why this was BLOCKED and no longer is:** the row recorded CX33 as *"confirmed unavailable even powered OFF"* — the Cost-Optimized line's limited availability, not a power-state problem. It became available and the operator took it. **Documentation corrected** (`RUNBOOK-ep0-datastore-volume`, `RUNBOOK-pbs-prune-serverside` ×2, `runbooks/offsite-endpoint.md` ×2, `runbooks/target-selection.md`) and **audit/evidence documents ANNOTATED, not revised** (`SPIKE-connectivity-wireguard-2026-07-03`, campaign-10 `phaseA-journal`) — they record what was true when written and that is their value. **Still open and still the operator's, deliberately untouched:** `target-selection.md`'s *"D-d did not name ep0 either way. Confirm it explicitly."* | — | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index d5948e5..7db6960 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -67,6 +67,7 @@ | R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 | | R-203 | **The app and its backup looked in different directories, and a run that skipped a mandatory folder still said `ok`** | M | **SHIPPED + PROVEN-LIVE (controller v0.197.0, 2026-08-04)** | Adds the capability-map row *"off-site app-data capture covers MANDATORY paths on both drive layouts"* as PROVEN-LIVE, and corrects that row's predecessor, which was optimistic. Unblocks R-201 | | R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **ALL FOUR ITEMS SHIPPED (controller v0.199.0 + hub v0.96.0, 2026-08-05)** | Collapsed per the coupling rule. The fix taken was R-196's — remove the guess, keep the two MEASURED signals (the controller's per-ACK hash re-check, which the stale flag was blinding, and R-197's supersession detector). Plus the claim-code read-through and the restore-scope wording. **Flips the capability-map recovery row from four crutches to one**, and that one is R-193's operator Re-issue, which keeps the row's *with a person present* qualifier. Does NOT touch R-198's retention, which is still unit-proven and needs the second drill **Item 4 shipped 2026-08-05 by DECLARATION, per the operator ruling: the box states its condition, the hub answers; an absence has four meanings and the hub cannot tell them apart. `internal/offsiteheal` mirrors `pbsdrheal` (declared states only, two-distinct-report debounce, restage before mint, healthy = no-op). Closes R-193's credential half and R-192's guard half (by replacement). **The capability-map row keeps a qualifier for a NEW reason** — no step needs an operator any more, but there is no customer-facing recovery SCREEN, so the experience still requires someone who knows to look. | +| R-193 | **A rebuilt box's off-site history was unreachable, and later reachable only from a command line** | L | **CLOSED (2026-08-05)** — credential half controller v0.199.0 + hub v0.96.0; the customer-facing SCREEN controller v0.200.0 | Collapsed per the coupling rule. Three legs: (a) the hub's retention keeps the sealed key (R-198); (b) a rebuilt box gets its credential back by DECLARING it needs one (R-204 item 4); (c) **the recovery screen** — a full page that unlocks the repository and lists what is in it, and **restores nothing**. Operator ruling: unlocking and restoring are SEPARATE, and the put-back needs a live-versus-backup comparison first (→ R-213). Flips the capability-map row's *needs someone who knows to look* qualifier; what remains there is one end-to-end re-walk and a correct-code run through the page | | R-201 | **The wipe-and-recover drill** | L | **PASSED + PROVEN-LIVE (2026-08-04 night)** | Flips the off-site app-data row to *"a customer's file survives a machine rebuild and comes back"* — PROVEN-LIVE, scoped to demo-hp and to a controller-data rebuild. The journey took four undocumented manual steps (R-204); the capability is real, the customer experience is not yet | | R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE | | R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |