diff --git a/CONTEXT.md b/CONTEXT.md index c07f37e..e77b63c 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -17,6 +17,27 @@ ## Standing rulings +**S-30 — the R-201 drill was PREPARED and HALTED BEFORE THE WIPE (2026-08-04). Nothing was wiped.** + +It stopped at step 4 because the sentinel file was **not in the off-site snapshot** while the run +reported `ok` (**R-203**): the deploy-time `${USERDATA_PATH}` root and the backup-time `NamespaceRoot` +disagree for an app whose `HDD_PATH` equals `system_data_path` — and the same compose used both roots. +Wiping would have destroyed the only copy of the sentinel and proven nothing. + +*Established live on the way, all firsts, and all worth carrying:* +- **A rebuilt box's off-site run REFUSES** — `repo_state: orphaned`, the orphan card, and + `offbox_repo_orphaned` at the hub. It does **not** silently start a fresh history. Closes R-193's Q3. +- **The orphan reset works**: move-aside to `/home/felhom-repo.orphaned-20260804`, never delete. +- **demo-hp's pre-rebuild off-site history is permanently unrecoverable** — its key is in superseded + row id 3 with `identity_blob` NULL, superseded **four hours before** v0.93.0 fixed the retention. +- **Neither pre-existing off-site-toggled app has a restorable file leg** — both are named-volume-only, + which the tier tars but the customer restore never unpacks. Any drill needing a recoverable file + needs an app with a **mandatory userdata path**; `calibre-web` is now deployed on demo-hp as that + fixture, with the sentinel in place. + +*Still not established, unchanged:* **no file has ever been restored from an off-site backup after a +wipe**, and Part 0's install path (controller v0.196.0) has never run against a live recovery. + **S-29 — a box may fetch its OWN sealed recovery blob with its OWN credential; the operator-driven DR path is a separate thing and stays gated (2026-08-04, R-199; hub v0.94.0 + agent v0.125.0 + controller v0.195.0).** diff --git a/REPORT.md b/REPORT.md index a71b167..c0a72a7 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,221 +1,168 @@ -# REPORT — the offsite backup key comes back (2026-08-04, R-199 closed, R-200 half) +# REPORT — R-201 drill: PREPARED, HALTED BEFORE THE WIPE (2026-08-04) -**Class:** implementation across three repos + one decisive live proof. Nothing wiped, deleted, -rotated or reconfigured; no customer-facing change in any language. +**Nothing was wiped. Nothing irreversible happened.** The drill stopped at step 4 because the sentinel +file was not in the off-site snapshot — wiping would have destroyed the only copy and proven nothing. --- -## 1. SCENARIO A — THE RESULT, FIRST, BECAUSE IT IS THE SESSION +## 1. THE VERDICT — not reached, and that is the correct outcome -**On `demo-felhom`, 2026-08-04 13:49 CEST, through the real endpoints end to end:** +The pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established the sentinel +is not in the off-site snapshot at all**, so there was nothing to recover and the verdict is +unobtainable. The runbook's own rule applies: *a precondition was wrong, and the finding outranks the +drill.* + +**Sentinel (step 3), still on the box:** +`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` +at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt`, 181 B. + +## 2. Snapshot count at step 9 — N/A; at step 4 it was 3, and the sentinel was in none of them ``` -=== offsite key recovery check (R-200) — compares, never installs === - on-disk sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb - recovered sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb - [MATCH] the offsite repository password IS recoverable from the sealed escrow. - Nothing was written: this check compares and never installs. -EXIT=0 +last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped +snapshot_count 3 · repo_size_bytes 30 636 +[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite: + /mnt/sys_drive/felhom-data/userdata/media/books +[INFO] [offbox] backed up calibre-web (…, 0 mandatory path(s)) +[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s ``` -**The hashes matched.** And they match a **third, independent** source: `host_escrow.restic_pw_sha256` -on the hub reads `c60c8bc737a6…` for `demo-felhom-8363b5`, recorded at the ceremony and never involved -in this comparison. The box's file, the sealed bundle and the hub's record all agree. +**The run reported `ok`.** → **R-203**, §9. -**The offsite repository password is recoverable from the sealed escrow with the customer's recovery -code.** That had never been demonstrated — the one live round-trip on record (2026-06-10) predates the -field by a month. +## 3. Preconditions P1–P6 -## 2. Part 0 — the recovery code +| # | Result | +|---|---| +| **P1** recovery code held | **PASS** — `R_DEMO-HP`, recorded by the operator in the DooPlex credentials file. The code appears nowhere in this report and **was never used** — no unseal ran on demo-hp. | +| **P2** versions | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed published v0.125.0 (sha `f7d8339b53d9…`, verified against the release output) + controller v0.196.0. `age` present at `/usr/bin/age`. | +| **P3** the blob seals the repo under test | **PASS with the caveat that reshaped the drill** — `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`; but **no repository existed under that key** (§5). | +| **P4** verified rollback archive | **NOT TAKEN, deliberately** — only needed for the wipe, and the wipe did not happen. | +| **P5** demo-felhom untouched | **PASS** — not touched at any point. | +| **P6** space | **PASS** — `felhom-backup` 927 GB free, `local-lvm` 30.9 %, guest 64 GB free. | -**Answered by the operator before the session began:** the 2026-08-04 07:16/07:20 codes had not been -kept, so **the operator ran a fresh ceremony on both nodes and saved the codes** (supplied to this -session as `R_DEMO-FELHOM` in the DooPlex credentials file). No ceremony was run by CC. +## 4. Every step's observable -Measured hub-side, and worth recording because it exercised two of yesterday's behaviours live: -the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z` (demo-felhom), and -`restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated them as -same-password re-ceremonies: **no superseded row was created** and **no `offsite_repo_key_changed` -fired**. That is v0.93.0's Scenario E, live. It also means yesterday's `identity_blob` retention is -still unit-proven and structurally verified, **not yet exercised live** — no supersession has happened. +| step | observable | +|---|---| +| 1 | `identity_blob` 572 B; `restic_pw_sha256` `8a9e33aa4da6…`; `snapshot_count: 0`, **no `last_run`/`last_status` at all** — never ran an off-site backup in this lifetime | +| 1b | **repository ORPHANED** — see §5 | +| 1c | **reset** (operator-authorised) — move-aside to `/home/felhom-repo.orphaned-20260804`, fresh repo initialised, `offbox_repo_reset` pushed | +| 2 | recovery codes for both boxes created and saved by the operator earlier the same day; escrow rows stamped `11:11:37Z` / `11:13:06Z` with **`restic_pw_sha256` unchanged** on both — correctly treated as same-password re-ceremonies (**no superseded row, no `offsite_repo_key_changed`**): v0.93.0's Scenario E, live | +| 3 | `calibre-web` deployed through the real API (HTTP 202), toggled off-site, Tier-1 recovery unit captured; sentinel written + `sync`ed + hashed | +| 4 | `ok`, 3 snapshots, 30 636 B — **and the mandatory directory missing** | +| 5–11 | **NOT RUN.** The §7 STOP was never reached: the drill failed its own precondition first | -Subject box: **demo-felhom** (Tier 0, reachable with a baked key, current escrow). Not demo-hp -(break-glass only), not DooPlex, not ep0. +## 5. Step 1b — the first live proof of a prediction, and it is the good news -## 3. The §8.2 decision as built, and where the gate is +The 2026-08-04 spike predicted from source, and could not measure, that a rebuilt box's next off-site +run would hit a **third** outcome — neither reattaching the old snapshots nor silently starting fresh, +but **refusing**. Measured two ways: -**Built as §8.2:** `GET /api/v1/hosts/{host_id}/escrow`, authenticated with the box's own per-host key, -self-scoped (global may read any — the same asymmetry the PUT has). +**Read-only probe** (`restic cat config` with the current key, writes nothing): +`Fatal: wrong password or no key found` — the exact string `classifyResticProbe` maps to `orphaned`. -**The single gate point is `api.escrowSelfServiceRetrieval`** (`hub/internal/api/handler.go`), a named -constant. `false` re-imposes the recovery-mode requirement on this endpoint and changes nothing else — -the same unseal, the same extraction, the same proof, minus the self-service property. The trade is -written above the handler in `dr.go`'s style: what it gives out, why it is safe, and **what it changes -about who is required** — obtaining the blob used to need an operator to arm recovery mode and now -needs only that box's credential. They still cannot open it. The mitigation is the audit row. +**Then the real customer path** (`POST /backup/offbox/run`): -## 4. Files, commits, deployed versions +``` +[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous, + no-longer-available key; runs will skip until reset +[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset) +[INFO] Event pushed: offbox_repo_orphaned (warning) … +``` -| Repo | Commits | Version | Deployed | -|---|---|---|---| -| `felhom.eu` | `435f4a5` (code) · `dd08926` (manifest) · docs commit below | hub **v0.94.0** | ArgoCD Synced/Healthy, `felhom-hub:0.94.0` | -| `felhom-agent` | `6d79047` | **v0.125.0** | published sha `f7d8339b53d9…`, **verified by independent download**, installed on demo-felhom (`felhom-agent --version` → 0.125.0, `active`). **NOT vouched** | -| `felhom-controller` | `9640e51` | **v0.195.0** | `felhom-controller:0.195.0` on guest 9201, `Up (healthy)` | +`repo_state: orphaned`, orphan card rendered, event at the hub (HTTP 200). +**The system stopped and said so. It did not quietly start a new history.** Closes R-193's Q3. -**Hub:** `internal/api/handler.go` (route + `handleHostEscrowGet` + `escrowSelfServiceRetrieval`), -`internal/notify/dispatcher.go` (`escrow_blob_served` → `operatorOnlyEvents`), -`internal/api/escrow_get_test.go` (new). -**Agent:** `internal/escrow/recover.go` (new — `OffsiteKeyRecoverer`, distinct errors), -`internal/hub/client.go` (`FetchIdentityEscrow`), `internal/localapi/escrow_recover.go` (new route), -`internal/localapi/server.go` (seam + registration), `cmd/felhom-agent/main.go` (production wiring + -the §8.6 selftest message fix), plus `recover_test.go` and `escrow_recover_wiring_test.go` (new). -**Controller:** `internal/backup/offbox_recovery_check.go` + `offbox_recovery_cli.go` (new), -`internal/agentapi/escrow.go` (`RecoverOffsiteRepoPassword`), `cmd/controller/main.go` -(`--recover-offsite-check`), plus `offbox_recovery_check_test.go` (new). +**Why it was orphaned:** the 15 snapshots / 40.9 MB were written under key `8e03eddf9ff7…` before the +2026-08-03 rebuild. That key survives only in superseded escrow row **id 3**, whose `identity_blob` is +**NULL** — superseded at `2026-08-04 07:15:36`, **four hours before hub v0.93.0 fixed the retention**. +Permanently unrecoverable, with or without a recovery code. -**Green gate in each repo after each phase** (`go build && go vet && go test ./...`, rc=0), plus -`repo_gates.py` / `agent_gates.py` / `controller_gates.py` — all OK. **No test run was combined with a -commit.** +## 6. Step 7's assertion (`identity_blob` unchanged across the rebuild) — NOT REACHED -## 5. Tests and every red-proof +There was no rebuild. Unmeasured, and stated as such. -| Scenario | Result | Red-proof — mutation → outcome | +## 7. Did the built chain carry steps 8–9? — NOT REACHED + +Neither the check nor the install ran against a live recovery on demo-hp. The manual substitute was not +needed and was not used. **Part 0 remains unit-proven only.** + +## 8. Part 0 — files, commit, tests, deployment + +**Commit `1b1366b`**, controller **v0.196.0**, deployed to demo-hp guest 9201 (`Up (healthy)`). +`internal/backup/offbox_recovery_cli.go` (`RecoverAndInstall`), `cmd/controller/main.go` +(`--recover-offsite-install` + `--confirm-install`), `internal/backup/offbox_recovery_check_test.go`. + +`go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK. Test run and commit were +never combined. + +**Red-proof observed:** removing the confirmation gate makes the dry run write the password — +`TestRecoverAndInstall_InstallsOnABareBox` fails with *"the DRY RUN wrote the password"*. + +## 9. R-203 — the finding that halted the drill + +| what | path | exists? | |---|---|---| -| **A** recovered key == on-disk key | PASS (unit, real `age`) + **PROVEN LIVE** | return `bundle.PBSToken` instead of `ResticRepoPassword` → **FAIL**: *"the recovered key is not the sealed repository password (len 24 vs 64)"*. This is the mutation that mattered: every field of that bundle is a non-empty string that looks like a secret | -| **B** wrong R fails closed, nothing written | PASS + **PROVEN LIVE** | — (the property is age's KDF, asserted rather than guarded, so there is no guard to remove) | -| **C** a box fetches only its own blob | PASS | delete the `!isGlobal && authHostID != pathHostID` check → **FAIL**: host A was served host B's blob, body and all | -| **D** no blob → a clean "none" | PASS | — | -| **E** every retrieval leaves a record | PASS + **PROVEN LIVE** | `if false &&` on the audit block → **FAIL**: *"a sealed recovery blob was served and NOTHING recorded it"* | -| **F** R persists nowhere | PASS + **PROVEN LIVE** | write R to a temp file inside the unseal → **FAIL** (see the correction below) | -| **G** the operator DR path is untouched | PASS | — | -| **H** the seam is wired (AST) | PASS | `_ = escrowRecoverer` with the `Options` field removed → **FAIL**: *"localapi.Options … carries no EscrowRecovery field"* | -| §8.5 compare-never-install | PASS + **PROVEN LIVE** | add `InjectOffboxPassword` to the check → **FAIL**: *"the check INSTALLED the recovered password"* | +| the app's live bind (where the sentinel is) | `/mnt/sys_drive/userdata/media/books` | **YES** | +| what the off-site capture set looked for | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** | -**TWO MUTATIONS DID NOT REMOVE EVERY GUARD ON THE FIRST ATTEMPT, AND BOTH ARE RECORDED RATHER THAN -QUIETLY REDONE** (§9.13 — two of the last four sessions found a red-proof that passed): +**Mechanism, from source:** `NamespaceRoot` (`appbackup/paths.go:28-33`) appends `felhom-data` **when +the drive IS the system data path** (`backup/backup.go:331`); the deploy-time `${USERDATA_PATH}` is +`/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an +app at `HDD_PATH=/mnt/sys_drive`, they differ. **The same compose used both roots** — +`${IMPORT_PATH}` → `…/felhom-data/userdata/import/calibre`, `${USERDATA_PATH}` → +`/mnt/sys_drive/userdata/media/books`. -1. **Scenario F.** The planted leak wrote R into the walked directory and **the test still passed** — - because a later failing call OVERWROTE the leak file with the wrong code, and the test only scanned - for the first one. Instrumenting the mutation (printing where it wrote) is what found it. **The test - was wrong, not the mutation.** It now asserts **emptiness** of the TMPDIR tree as its primary check — - nothing under it is created by the test, so any survivor is a leak regardless of content — with the - content scan kept as defence in depth over all three secrets. Re-run: **FAIL**, *"the unseal left - 1 file(s) behind under TMPDIR"*. -2. **Scenario H.** Commenting the wiring line out failed to **compile** (`declared and not used`), which - is not a red-proof. The mutation that compiles keeps the variable alive (`_ = escrowRecoverer`) and - drops the struct field. Re-run: **FAIL** as quoted above. +**Measured vs not, because it changes the fix.** MEASURED: the disagreement, the absence from the +snapshot, the `ok` status, and that the only signal is a container-log WARN. **NOT ESTABLISHED:** +whether `HDD_PATH=/mnt/sys_drive` is supported — it was chosen because demo-hp's only registered drive +(`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working), while +`/mnt/sys_drive` was **accepted** (202). **Either branch is a defect** and the row says so. -**Every `-run` filter was verified to have matched** (`-v`, `=== RUN` lines counted) before any result -was read as a proof. +**Not a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths, so +they are unaffected and their snapshots are real. -## 6. The live validation trail, in order +## 10. Teardown — three layers, nothing torn down that proves anything -1. **Wrong code first, deliberately** (13:44:44) — hub: `escrow blob SERVED to host demo-felhom-8363b5 - (572 opaque bytes, self_scope=true) — recovery path in use`; agent: `offsite key recovery FAILED … - err="escrow: the recovery code did not unwrap the identity escrow (wrong recovery code, or a corrupt - blob): exit status 1"`; CLI: `[FAIL] … nothing was written.`, exit 1. - **This is Scenario B live AND it proves links 6 and 7 ran independently of the success** — the blob - was fetched and the unseal was attempted and refused. -2. **The real code** (13:49:47) — §1's output, exit 0. -3. **Nothing was written** — `offbox/repo_password` mtime on the live box is still - `2026-08-03 07:18:02`, unchanged by a check that ran at `11:49` UTC. The whole `offbox/` directory is - byte-for-byte the pre-check listing. +| layer | state | +|---|---| +| the guest | `calibre-web` deployed, running, off-site-toggled, sentinel in place. **Kept** — it is the only app on either demo box with a restorable file leg, and the resumed drill needs it | +| the off-site store | fresh, working, 3 snapshots, 30 636 B. Old 15-snapshot history **set aside, not deleted**, at `/home/felhom-repo.orphaned-20260804` (≈41 MB) | +| the host | agent v0.125.0, controller v0.196.0; `pvesm` unchanged beyond normal usage | +| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` recorded for `demo-hp`. **No scratch customer records were created — nothing was reinstalled** | -## 7. The retrieval record as it appears at the hub +**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — ruled, still owed, and §8.3 forbids +riding it along with a drill. -``` -customer_id severity created_at details_json -demo-felhom warning 2026-08-04 11:44:44 {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true} -demo-felhom warning 2026-08-04 11:49:47 {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true} -``` +## 11. The capability-map row as it now reads -And the routing, which is the half that shows the register works: +The R-199 row (the KEY is recoverable, PROVEN-LIVE) keeps its scope and gains the back-pointer the +previous update omitted, plus: the attempt to prove the DATA half was **halted before the wipe**, and +**no file has still ever been restored from an off-site backup after a wipe**. The install half +(v0.196.0) is likewise unit-proven only. -``` -customer_id channel status created_at -demo-felhom operator sent 2026-08-04 11:44:44 -demo-felhom customer skipped 2026-08-04 11:44:44 ← operator_only -demo-felhom operator suppressed 2026-08-04 11:49:47 ← 1h operator cooldown, itself recorded (R-182) -demo-felhom customer skipped 2026-08-04 11:49:47 -``` +## 12. Registers -**Both retrievals raised an event; the operator was mailed for the first; the second's mail was -cooldown-suppressed and that suppression is written down.** The audit is per-retrieval and complete -even where the mail is not. +- **R-203 NEW** — the missing mandatory directory. Highest ID before this session was R-202. +- **R-201** — prepared, halted, with everything it established and exactly what resuming needs. +- **R-200** — plumbing complete (v0.196.0); the form is not built; the install has never run live. +- **R-202** stays open. **The ciphertext deletion is still owed.** +- **The v0.93.0 `identity_blob` retention is still unit-proven only** — nothing here superseded a key. -## 8. Evidence that R persisted nowhere — the search, not a claim +## 13. CI -Searched on the subject host with the pattern read from a file (never in argv), across the agent -journal, the controller's container log, and `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`: +Run numbers and task ids in the session summary; **`--no-verify` was not used**. -``` -agent journal lines containing R: 0 -controller log lines containing R: 0 -host files containing R: 0 -leftover felhom-idesc-* staging dirs: 0 -staged escrow secret file: 0 entries -``` +## 14. Observations — noticed, NOT acted on -**With a positive control, because an absent match is not a measurement unless the instrument is shown -to work:** a copy of R was planted under `/tmp`, the same sweep found **1**, the copy was shredded, and -the sweep returned **0**. The instrument detects R when R is there. - -R reached the box only as: the credentials-file value on DooPlex → a pipe → the container's stdin → -the request body → the agent's memory. It was never an argument, never a file on either machine, and -appears in no log line at any level. - -## 9. Register - -- **R-199 → SHIPPED + PROVEN-LIVE.** Links 6, 7 and 8 assembled and walked. -- **R-200 → plumbing shipped; the FORM is not built** and the row stays open for it and for link 9. -- **R-201 → OPEN, and the distinction is deliberate:** the KEY is proven recoverable; a recovered - password has never been **installed**, no repository has been **reopened** under one, and **no file - has ever been restored**. The drill's pass condition is unchanged — a byte-identical sentinel file, - not "the store opened". -- **R-202 → OPEN and untouched.** The orphan card still promises recoverability unconditionally. -- **The orphaned-ciphertext deletion (~1.2 GB) is STILL OWED** — operator ruling 2 of 2026-08-04, - deliberately not ridden along with a code release. -- Capability map, `03-host-agent.md` §8a, `CONTEXT.md` **S-29**, `REUSE.md` and `ROADMAP.md` updated; - `STATUS.md` rewritten for the operator. - -## 10. The capability-map row, and what it deliberately does not claim - -Added: *"The offsite repository password can be RECOVERED from the sealed escrow with the customer's -recovery code"* — **PROVEN-LIVE (2026-08-04)**, with the evidence above. - -**What the row explicitly refuses to claim**, stated in the row itself because the previous over-claim -on this subject was struck out four hours earlier: it covers the **key**, not the **data**. No -recovered password has been installed, no repository reopened, no file restored. And the proof used a -box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is -exactly what the drill covers and it has not run. - -## 11. CI - -Run numbers, task ids and conclusions for all commits are quoted in the session summary. -**`--no-verify` was NOT used** — every push ran its repo's pre-push gate and passed. - -## 12. Teardown - -**Nothing was provisioned.** The temporary pattern file used by the R sweep and its positive control -were shredded on the host by the same command that created them (verified: 0 files remain). No -diagnostic artifact persists on either machine. The agent's previous binary is retained as -`/usr/local/bin/felhom-agent.bak-0.124.1`, per the documented deploy. - -## 13. Observations — noticed, NOT acted on - -1. **The `escrow_blob_served` mail is subject to the 1-hour operator cooldown** — visible above, where - the second retrieval's mail was suppressed. The EVENT rows are per-retrieval so the audit is - complete, but if retrieval ever becomes routine, a burst would produce one mail. Worth revisiting - **when** the customer-facing flow lands, not before. -2. **`ListSupersededEscrow` still has no production caller.** Recovery reads the CURRENT blob; a - customer needing a PREVIOUS generation's key (which is what R-198's retention exists for) has no - path at all. That is the next gap in this area after the drill. -3. **The controller binary is at `/usr/local/bin/felhom-controller`, not `/app/…`** — the first - diagnostic invocation failed on a guessed path. Worth a line in the runbook when one is written. -4. **`age` is at the hardcoded `/usr/bin/age` on both demo hosts** (1.2.1 on demo-felhom) — the recon - listed this as unestablished; it is now established for demo-felhom. `ageBinary` is a package - variable with no config override, so a host that installs `age` elsewhere would fail the unseal at - the last step. -5. **Yesterday's `identity_blob` retention has still not been exercised live** — the operator's fresh - ceremonies sealed the *same* password, so no supersession occurred. It remains unit-proven with a - verified schema; the first real supersession will be its live proof. +1. **`${IMPORT_PATH}` and `${USERDATA_PATH}` resolved to different roots in one deploy** — the sharpest + single expression of R-203, and the place a fix should probably start. +2. **The controller's CSRF form field is `_csrf`, not `csrf_token`** — the login page carries one name + and the protected forms another; an hour went into that. Worth a line in the headless-access memory. +3. **`/api/stacks//deploy` accepted `HDD_PATH=/mnt/sys_drive`** one call after refusing the NAS + path with a clear Hungarian error. If the system drive is not a valid app namespace, that refusal is + missing. +4. **demo-hp has no registered drive usable as an app namespace** — its only drive is a NAS. Any drill + needing a file-leg app on that box is constrained by this. +5. **The `felhom-r-leak`-style positive control is now used in three separate absence checks** across + two sessions. It is cheap and it has already caught one wrong test; worth making a documented habit. diff --git a/STATUS.md b/STATUS.md index 9c3a546..d61d459 100644 --- a/STATUS.md +++ b/STATUS.md @@ -46,6 +46,16 @@ Proven end to end on real hardware. recovered key back*, reopens the old backup store with it, or **restores a single file**. Today proves the key survives and returns; it does not prove the backups do. *(R-199 closed; R-200 half; R-201 open)* +- **A backup reported success while leaving out a folder the customer was told is protected.** Found + this evening on the HP machine, while setting up the wipe-and-restore proof. We deployed an app whose + book folder is declared *mandatory* — the strongest protection class — put a marked file in it, and + ran an off-site backup. **The backup said OK. Three snapshots. The folder was not in any of them.** + The machine knew: one warning line inside the container says the folder was skipped. Nothing else + does — not the card, not the counters, not the hub, not you. This is the same shape as everything + else we have been fixing this month: *a path the customer thinks is protected is not in the copy.* + It is the reason the proof stopped before the wipe — wiping would have destroyed the marked file and + proven nothing. **The two apps that were already backing up off-site are unaffected** — they declare + no such folders. *(R-203)* - **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card says the old backups may later be restorable with the matching recovery code. From today that is true for machines that re-seal from now on and **false for anything already orphaned** — and the machine @@ -94,10 +104,9 @@ Proven end to end on real hardware. ## What we're working on -- **Now:** the proof exercise — wipe a demo machine and recover it with a saved recovery code. It is - designed, and it is now a much better bet than it was this morning: the first half of the path was - walked live today, so if the drill fails we will know *which* step failed instead of just "recovery - did not work". Waiting on your go-ahead. Both honesty fixes shipped today: a changed backup key now +- **Now:** fixing the folder-left-out-of-the-backup problem above. The wipe-and-restore proof is + **staged and waiting on it** — the machine, the code, the working off-site store, the app and the + marked file are all in place; only the missing folder blocks it. Nothing was wiped. Both honesty fixes shipped today: a changed backup key now raises an alarm on the day, and the email that stated the opposite of what it measured now describes what it actually saw. - **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to @@ -149,6 +158,12 @@ Proven end to end on real hardware. ## Changed since last update +- **2026-08-04 (late)** — Set up the wipe-and-restore proof on the HP machine and **stopped before the + wipe**: a folder marked as protected was missing from the off-site backup while the backup reported + success. Three things were proved on the way, all firsts: a rebuilt machine's off-site backup now + **refuses and says so** instead of quietly starting over; the "start a new store" repair works and + keeps the old data aside; and the HP machine's pre-3-August history is gone for good — its key was + destroyed four hours before the fix that would have kept it. *(R-203, R-201, R-193)* - **2026-08-04 (evening)** — **Proved the backup key comes back.** A demo machine fetched its own sealed package, opened it with the saved recovery code, and produced a key identical to the one it uses. A wrong code was refused and wrote nothing. The recovery code left no trace anywhere on the diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 2379ac8..58b1805 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -38,7 +38,7 @@ | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends | | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention) — ⚠ the claim as written was FALSE for the offsite tier for two months; **FIXED in hub v0.93.0 (2026-08-04), and the row below states what ships TODAY:** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`) — and **since v0.93.0 the retained row carries `identity_blob` as well as the K-escrow blob**, so a previous passphrase does now stay recoverable with the recovery code that sealed it. **What was wrong until v0.93.0, recorded because it is the ninth entry in `CLAUDE.md`'s comment-vs-code table and the first that was also customer-facing copy:** `host_escrow_superseded` had no `identity_blob` column and `demoteCurrentEscrowTx` did not copy one, so what survived a supersession was the **PBS datastore key only** — never the restic repository password, which lives in `identity_blob`. The destroying act was the **escrow ceremony** a rebuilt box asks its customer to run. Measured live 2026-08-04, before the fix: both current rows held `blob`=383 B **and** `identity_blob`=572 B; both retained rows held `blob`=383 B only. → **R-198** (SHIPPED), evidence `audits/RECON-offsite-dr-chain-2026-08-04.md` §7. **THREE SCOPE LIMITS THIS ROW MUST NOT BE READ PAST.** (1) **Nothing was backfilled and nothing could be** — rows superseded before v0.93.0 were written without the blob and their source rows are already overwritten; both demo boxes' pre-2026-08-04 repository passwords are gone permanently. (2) **A retained key is not a restore — but as of 2026-08-04 evening it IS a recovered key.** See the row below. (3) The customer-facing orphan card still promises recoverability unconditionally (R-202, gate hit, card untouched). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test | -| **The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code** | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE (2026-08-04)** | On demo-felhom, through the real endpoints end to end: the box fetched its own sealed blob from the hub with its own per-host credential (hub log: *escrow blob SERVED … 572 opaque bytes, self_scope=true*), the agent unsealed it with the customer's recovery code, and the extracted repository password's sha256 was **byte-identical** to the one on disk — `c60c8bc737a6…`, which is ALSO the hash the hub had independently stored, so three sources agree. Five minutes earlier the same path with a WRONG code failed closed at age's KDF with nothing written, which proves links 6 and 7 ran independently of the success. R was searched for afterwards and found in 0 log lines and 0 files, with a positive control confirming the search would have found it. Evidence: per-repo CHANGELOGs; `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 links 6–8 | **WHAT THIS ROW DOES NOT CLAIM, stated because the previous over-claim here was struck out four hours earlier.** It covers the KEY, not the DATA. A recovered password has never been **installed** (the diagnostic compares and refuses to write, by design), no existing repository has ever been **reopened** under one, and **no file has ever been restored** from an off-site history via a recovered key. Links 9–11 of the chain are open (R-200's remaining half, R-201). The proof also used a box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is exactly what the drill covers and it has not run || DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). | +| **The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code** | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE (2026-08-04)** | On demo-felhom, through the real endpoints end to end: the box fetched its own sealed blob from the hub with its own per-host credential (hub log: *escrow blob SERVED … 572 opaque bytes, self_scope=true*), the agent unsealed it with the customer's recovery code, and the extracted repository password's sha256 was **byte-identical** to the one on disk — `c60c8bc737a6…`, which is ALSO the hash the hub had independently stored, so three sources agree. Five minutes earlier the same path with a WRONG code failed closed at age's KDF with nothing written, which proves links 6 and 7 ran independently of the success. R was searched for afterwards and found in 0 log lines and 0 files, with a positive control confirming the search would have found it. Evidence: per-repo CHANGELOGs; `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 links 6–8 | **WHAT THIS ROW DOES NOT CLAIM, stated because the previous over-claim here was struck out four hours earlier.** It covers the KEY, not the DATA. **Re-confirmed 2026-08-04 evening by an attempt to prove the DATA half:** the R-201 drill was prepared on demo-hp and **halted before the wipe** — the sentinel file was not in the off-site snapshot (R-203), so the wipe would have destroyed it and proven nothing. **No file has still ever been restored from an off-site backup after a wipe** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). The install half of the chain (controller v0.196.0 `--recover-offsite-install`, R-200) is likewise unit-proven only — it has never run against a live recovery. A recovered password has never been **installed** (the diagnostic compares and refuses to write, by design), no existing repository has ever been **reopened** under one, and **no file has ever been restored** from an off-site history via a recovered key. Links 9–11 of the chain are open (R-200's remaining half, R-201). The proof also used a box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is exactly what the drill covers and it has not run || DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). | | **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** | | **Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard** | agent v0.104.0 → **v0.121.0**, hub v0.77.0 → **v0.91.0** | **PROVEN-LIVE (2026-08-03)** | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why this was NOT PROVEN-LIVE until now:** rotation had not been observed selecting both tiers across consecutive UNATTENDED cadences — at a 24h cadence a multi-day window — and a single passing run proves the code path, not the schedule. → **R-85**. **R-86 (agent v0.121.0 + hub v0.91.0, 2026-08-03) replaced the schedule and the observation became possible in one afternoon**, because what has to be observed is no longer a multi-day rotation but a RULE: a tier is due when its newest archive that has settled ~24 h has not been proven. **UPGRADE EVIDENCE — a real unattended run on demo-felhom (2026-08-03), triggered by DUE-NESS, not by a timer.** The scheduler's own log: `15:14:38 restore-test tier is DUE … target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z … reason="newest settled archive … has not been proven"` → `proxmox-backup-client restore --crypt-mode=encrypt` under the agent's own token → `15:25:08 gate decision class=guest_destroy guest=990000 allowed=true` → `15:25:14 scratch guest torn down` → **`15:25:14 backup: scheduled restore-test passed archive=felhom-pbs:… duration_s=635.1`**. A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored, booted, verified and destroyed in 635 s, unattended. **The three things a timer could not show**, all verified after it: the state names THAT archive (`{"felhom-pbs":{"archive":"…2026-07-28T04:49:43Z","proven_at":"2026-08-03T13:25:14Z"}}`); a second evaluation reports `due=false … is already proven` and runs nothing; and **an agent restart runs nothing**, which is the defect a person actually noticed — every deploy used to restart the timer. Teardown verified at all three layers: guest absent from `pct list`, **zero** `990000` volumes in `lvs`, and the hub-side `restore_tests[]` entry deliberately RETAINED (it IS the proof the staleness check reads). **R-189 (agent v0.122.0, 2026-08-03) closed the reporting half of this row, and it was a REAL gap in the evidence path:** the proof above reached the hub only because no restart intervened — `restore_tests[]` came solely from an in-memory store, so the 15:25:14 PASS was in fact LOST when the agent restarted 2 m 43 s later for a deploy (`0 restore-tests` on the next two host-reports). Under per-archive due-ness the box would not have repeated the work for a week. The persisted per-tier proof (with the archive, and now the tier) is merged into the report, so a proof survives a restart — one entry per tier, newest wins, and a record that cannot be described honestly is not emitted. **THE HOST TIER'S HALF OF THIS ROW WAS OPTIMISTIC UNTIL 2026-08-03, and it is worth saying plainly.** Every live restore-test cited here is on the OFFSITE tier. The HOST tier was not merely unproven on demo-felhom — it was **unprovable**, and on demo-hp equally: the agent's token had no ACL on `/storage/felhom-backup`, the storage both boxes configure as `local_backup_target`, so the content API answered `{"data":[]}` through the token while root listed three archives. The scheduler skipped it as *"no settled archive yet"* — which is exactly what a brand-new tier reports — so nothing ever said so (**R-185**). **CLOSED 2026-08-03 — agent v0.123.0 + installer 1.24.0.** The grant is applied on both demo boxes (the token now lists 3 and 4 archives), the installer's reuse arm grants on a pre-existing target, and the agent now **asks whether it may read each tier it depends on** instead of inferring it from an empty list: a critical degraded capability naming the storage and the missing role, which the hub alerted and emailed on **while the box was still blind**. **THE HOST TIER IS NOW PROVEN-LIVE, UNATTENDED, ON BOTH DEMO BOXES (2026-08-04) — this row's optimistic half is cashed.** Four SCHEDULED runs, nothing triggered by hand: **demo-felhom** host tier `…2026_08_02-04_42_14.tar.zst` **passed in 83.8 s** at 00:55, offsite `…2026-07-28T04:49:43Z` **passed in 540.4 s** at 06:55; **demo-hp** host tier `…2026_08_02-04_49_29.tar.zst` **passed in 109.3 s** at 02:05, offsite `…2026-07-28T19:19:45Z` **passed in 300.1 s** at 08:05. Each restored into a scratch guest, booted, verified and destroyed itself; `pct list` and `lvs` show zero `990000` afterwards on both, and both boxes' `local-lvm` returned to their pre-run figures (1.95 % and 30.83 %). **Both boxes had BOTH tiers due at once and the ordering was observed live for the first time under R-86's rule:** never-proven sorted first, so each box took its HOST tier, deferred the offsite one, and picked it up on the following evaluation six hours later — one heavy operation at a time, per box, without anyone sequencing it. **The proofs reached the hub**, which is R-189 carrying a host-tier entry for the first time: demo-felhom's report holds **two** entries, one per tier, and the `local` one can only have come from the persisted state because the in-memory store held only that morning's offsite run. **SCOPE, stated because one box proving something does not make it a fleet property:** this covers `demo-felhom` and `demo-hp`. The tester's box is untested and untouched. Evidence: `felhom-agent/REPORT.md`, `felhom.eu/REPORT.md` | | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below | diff --git a/documentation/audits/DRILL-r201-offsite-recovery-2026-08-04.md b/documentation/audits/DRILL-r201-offsite-recovery-2026-08-04.md new file mode 100644 index 0000000..f8bcb36 --- /dev/null +++ b/documentation/audits/DRILL-r201-offsite-recovery-2026-08-04.md @@ -0,0 +1,281 @@ +# DRILL RECORD — R-201, the wipe-and-recover proof: **PREPARED, HALTED BEFORE THE WIPE** + +**Date:** 2026-08-04 · **Box:** `demo-hp` (Tier 0, the designated drill host) · **Nothing was wiped.** +**Outcome:** the drill did not reach its verdict. It was halted at step 4 by a defect that makes the +verdict unobtainable — and that defect is worth more than the drill. + +> **THE HEADLINE.** A customer-declared **mandatory** data directory was **silently absent from the +> off-site snapshot**, while the backup reported `ok` with three snapshots. The only trace is one +> `[WARN]` line inside the controller container. The hub, the card and the counters all say the backup +> succeeded. → **R-203** +> +> **Nothing irreversible happened.** The wipe never ran. `demo-hp` is left in a BETTER state than it +> started: it now has a working off-site repository (it had an orphaned one), three snapshots, and a +> sentinel file on disk. + +--- + +## 1. The verdict — not reached, and why that is the correct outcome + +The drill's pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established that +the sentinel is not in the off-site snapshot at all.** Wiping the box would therefore have: + +- destroyed the sentinel, which exists only on that box; +- proven nothing about recovery, because there would be nothing to recover; +- and done so *after* the point of no return. + +The runbook's own rule applies: *"If this session finds itself writing code beyond Part 0, stop. That +means a precondition was wrong, and the finding outranks the drill."* A precondition was wrong. It was +one the runbook's P1–P6 table did not contain, because nobody knew to look for it. + +**Sentinel sha256 (step 3), recorded and still on the box:** +`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` +at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt` (181 B). + +--- + +## 2. R-203 — the defect that halted the drill + +**Measured, twice, on the live box.** + +| what | path | exists? | +|---|---|---| +| the app's live bind, where the customer's files actually land | `/mnt/sys_drive/userdata/media/books` | **YES** (the sentinel is here) | +| the path the off-site capture set treats as the mandatory directory | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** | + +The controller's own log, verbatim: + +``` +[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite: + /mnt/sys_drive/felhom-data/userdata/media/books +[INFO] [offbox] backed up calibre-web (…/backups/primary/calibre-web, 0 mandatory path(s)) +[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s +``` + +**The run reported `ok`.** `last_status: ok`, `last_success` stamped, `snapshot_count: 3`. Nothing +customer-visible, nothing hub-visible and nothing in any counter says the mandatory directory was +dropped. This is `CLAUDE.md`'s recurring shape — *a path the customer thinks is protected is not in the +snapshot* — and the code even has the right words for it in a WARN nobody reads. + +**The mechanism, from source.** `NamespaceRoot(drivePath, inGuestDrive)` +(`appbackup/paths.go:28-33`) appends `felhom-data` **when the drive IS the system data path** +(`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`, +`backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is documented and computed +as **`/userdata`** (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` +and an app deployed at `HDD_PATH=/mnt/sys_drive`, those two produce different directories. + +**And the same compose file used BOTH roots.** From `docker inspect calibre-web`: + +``` +bind /mnt/sys_drive/felhom-data/userdata/import/calibre -> /cwa-book-ingest ← felhom-data root +bind /mnt/sys_drive/userdata/media/books -> /calibre-library ← NO felhom-data root +``` + +`${IMPORT_PATH}` resolved *with* the segment; `${USERDATA_PATH}` resolved *without* it. One deploy, one +template, two roots. + +### What is measured and what is not — stated because the scope changes the fix + +- **MEASURED:** the two paths disagree; the mandatory directory is absent from the snapshot; the run + reports `ok`; the only signal is a container-log WARN. +- **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a *supported* choice. It was chosen because + demo-hp's only registered drive, `Felhom-Share`, is a NAS and was **correctly refused** as an app + namespace (R-108's `RefuseAsAppNamespace`, working as designed). `/mnt/sys_drive` was **accepted** + (HTTP 202) rather than refused. + +**Either branch is a defect, which is why this is filed regardless:** + +- if the system drive **is** a supported app namespace → the userdata path resolution is wrong for + every app deployed on it, and their mandatory directories are silently unprotected; +- if it is **not** supported → the deploy accepted a namespace it should have refused, exactly as it + refused the NAS one call earlier, and the refusal that exists is not reaching this case. + +**What must NOT be concluded from this drill:** that off-site backups are broken generally. The two +pre-existing apps (`opengist`, `privatebin`) declare **no** mandatory userdata paths — everything they +own is in named volumes — so they are unaffected, and their snapshots are real. + +--- + +## 3. Preconditions, each measured + +| # | Precondition | Result | +|---|---|---| +| **P1** | operator holds the recovery code | **PASS** — `R_DEMO-HP`, recorded by the operator in the DooPlex credentials file (`~/.config/credentials`). *The code itself appears nowhere in this record.* | +| **P2** | demo-hp on agent v0.125.0 + controller v0.196.0 | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed the published v0.125.0 (sha `f7d8339b53d9…`, verified against the release) and controller v0.196.0. `age` present at `/usr/bin/age`. | +| **P3** | the current `identity_blob` seals the repository under test | **PASS with a caveat that reshaped the drill** — `identity_blob` = 572 B, `restic_pw_sha256` = `8a9e33aa4da6…`. But **no repository existed under that key** (see §4). | +| **P4** | a deliberate, verified whole-guest archive as rollback | **NOT TAKEN** — deliberately. It is only needed for the wipe, and the wipe did not happen. | +| **P5** | demo-felhom untouched and healthy | **PASS** — not touched at any point. | +| **P6** | space | **PASS** — `felhom-backup` 927 GB free, `local-lvm` 30.9 % used, guest 64 GB free. | + +--- + +## 4. Step-by-step, with every observable + +### Step 1 — starting state (hub, read-only) + +`demo-hp-bb76ea`: `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`, created `2026-08-04T11:11:37Z`, `escrow_state: escrowed`. +Report: `snapshot_count: 0`, `repo_size_bytes: 0`, **no `last_run`/`last_status` at all** — the shape of +a controller that has never run an off-site backup in this lifetime. + +### Step 1b — the repository was ORPHANED, and this is the first live proof of a prediction + +The 2026-08-04 spike predicted from source — and could not measure — that a rebuilt box's next +off-site run would hit a **third** outcome: neither reattaching the old snapshots nor silently starting +a fresh history, but **refusing**. Measured here, twice over. + +**Read-only probe first** (`restic cat config` with the current key, writes nothing): + +``` +Fatal: wrong password or no key found +``` + +— the exact string `classifyResticProbe` maps to `"orphaned"`. + +**Then the real run, through the customer UI endpoint** (`POST /backup/offbox/run`): + +``` +[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous, + no-longer-available key; runs will skip until reset +[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset) +[INFO] Event pushed: offbox_repo_orphaned (warning) — A távoli mentési tároló elárvult … +``` + +`repo_state: "orphaned"`, `orphaned_at: 2026-08-04T12:37:30Z`, `last_status: "error"`, the orphan card +rendered on `/backups/remote`, and the event reached the hub (HTTP 200). + +> **The system stopped and said so. It did not quietly start a new history over the old one.** +> That closes R-193's open Q3 — and it is the good half of this month's story. + +**Why the repository was orphaned:** the 15 snapshots / 40.9 MB at `/home/felhom-repo` were written +under key `8e03eddf9ff7…` before the 2026-08-03 rebuild. That key survives only in superseded escrow +row **id 3**, which carries **`identity_blob` = NULL** — it was superseded at `2026-08-04 07:15:36`, +**four hours before** hub v0.93.0 fixed the retention. Unrecoverable, permanently, with or without a +recovery code. + +### Step 1c — the reset (operator-authorised) + +The orphan card's own reset, confirmed by the operator during the session: + +``` +[WARN] [offbox] resetting orphaned repo (operator-confirmed (claimed)): + move-aside /home/felhom-repo -> /home/felhom-repo.orphaned-20260804, then re-init +[INFO] [offbox] orphaned repo reset complete — old history set aside at + /home/felhom-repo.orphaned-20260804 (move-aside, not deleted); fresh repo initialized +[INFO] Event pushed: offbox_repo_reset (info) +``` + +**Nothing was deleted.** The reset path had never run in anger before; it works. + +### Steps 2–3 — the recovery code and the sentinel + +The operator ran the ceremonies for **both** boxes earlier the same day and saved the codes. Measured +hub-side: the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z` +(demo-felhom), and `restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated +them as same-password re-ceremonies: **no superseded row, no `offsite_repo_key_changed`**. That is +v0.93.0's Scenario E, live. + +**A file-leg app had to be deployed**, and this is a precondition the runbook did not anticipate: +neither off-site-toggled app on the box (`opengist`, `privatebin`) has a restorable file leg — both +keep everything in **named volumes**, which the off-site tier backs up as tars but the customer restore +flow **never unpacks** (`offbox_reconstitute.go`). A sentinel in either would have been unrecoverable +by design. + +`calibre-web` was chosen: it declares `userdata: media/books class: mandatory`, and it is a +single-container app. Deployed through the real API (`POST /api/stacks/calibre-web/deploy`, HTTP 202), +toggled for off-site, and a Tier-1 recovery unit captured +(`Recovery unit captured for calibre-web → …/backups/primary/calibre-web`). + +Sentinel written, `sync`ed, hashed: **`643166269103a25c…`**, 181 B. + +### Step 4 — the pre-wipe off-site backup: **`ok`, and wrong** + +``` +last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped +snapshot_count 3 · repo_size_bytes 30 636 · repo_state null +``` + +Three snapshots, three apps, 37 s — and **the sentinel is in none of them**, per §2. + +### Steps 5–11 — NOT RUN + +Step 5 (verify the pre-wipe snapshot contents) would have confirmed the absence a second way; it is +moot given §2. Steps 6–11 (archive, wipe, reinstall, recover, install, restore, compare) were **not +started**. The §7 STOP was never reached, because the drill failed its own precondition first. + +--- + +## 5. Part 0 — shipped, tested, deployed (R-200's plumbing half) + +`--recover-offsite-install` (controller **v0.196.0**, commit `1b1366b`): same fetch → unseal → extract +path as `--recover-offsite-check`, same STDIN discipline for R, but it **places** the recovered +password via `InjectOffboxPassword`. + +- **Confirmation is a second invocation.** Without `--confirm-install` it prints both hashes and writes + nothing. A single interactive prompt would have had to share stdin with R. +- **Three outcomes, named distinctly:** *installed* (no local password — the rebuilt-box shape), + *unchanged* (identical key already present, nothing written), *refused* (a DIFFERENT key present; + installing would clobber the key the current repository is encrypted under — exit 2, no force offered). +- It re-reads the file after writing rather than trusting the call's return. + +**Tests + red-proof.** `go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK. +Removing the confirmation gate makes the dry run write the password and fails +`TestRecoverAndInstall_InstallsOnABareBox` with *"the DRY RUN wrote the password"*. The R-persistence +test carries a **positive control** — a planted copy of the code is found by the sweep, then removed +and not found — because an absence check is worth only what its sensitivity is. + +**It was deployed to demo-hp and never exercised against a live recovery**, because the drill halted +before step 9. Its unit proof stands; its live proof does not exist. + +--- + +## 6. What this drill did and did not establish + +**Established, live, for the first time:** + +1. A rebuilt box's off-site repository is **orphaned and the run refuses** — the spike's predicted + third outcome, measured. It does not silently start a fresh history. +2. The **orphan reset works**: move-aside, never delete, fresh repo initialised, event pushed. +3. **A customer-declared mandatory data directory can be silently absent from the off-site snapshot + while the run reports `ok`** (R-203). +4. demo-hp's pre-rebuild off-site history is **permanently unrecoverable** — its key was destroyed four + hours before the fix that would have kept it. + +**NOT established — and unchanged from before this session:** + +- **No file has ever been restored from an off-site backup after a wipe.** R-201's question is still + open, and its pass condition is unchanged. +- Part 0's install path has never run against a live recovery. +- The v0.93.0 `identity_blob` retention is **still unit-proven only** — nothing in this session + superseded a key, so nothing exercised it. + +--- + +## 7. State left behind, and teardown + +**Deliberately not torn down** — this is evidence, and the box is better off than it was: + +| layer | state | +|---|---| +| the guest | `calibre-web` deployed and running, off-site-toggled, sentinel file in place. **Kept** as the drill fixture for the resumed run — it is the only app on either demo box with a restorable file leg. | +| the off-site repository | fresh, working, 3 snapshots, 30 636 B. The old 15-snapshot history is **set aside**, not deleted, at `/home/felhom-repo.orphaned-20260804`. | +| the host | agent v0.125.0, controller v0.196.0, `pvesm` unchanged apart from normal usage. | +| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` events recorded for `demo-hp`. No scratch customer records were created — **nothing was reinstalled**. | + +**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — that act is ruled and still owed, +and §8.3 of the runbook forbids riding it along with a drill. The reset added +`/home/felhom-repo.orphaned-20260804` (≈41 MB) to what is set aside. + +**No `R` persisted anywhere** — the recovery code was never used in this session. It was read only to +confirm the key exists in the credentials file; no unseal was performed on demo-hp. + +--- + +## 8. To resume the drill + +1. **Fix or scope R-203.** Until the mandatory userdata path lands in the snapshot, no sentinel can + survive a wipe and the drill cannot reach its verdict. +2. Re-run steps 4–5 and confirm the sentinel IS in the snapshot — by listing it, not by a green status. +3. Then P4 (the deliberate archive), the §7 STOP, and steps 6–11 as written. + +Everything else is already in place: the code, the versions, the recovery code, the working repository, +the file-leg app and the sentinel. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 2eca475..4728992 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -75,9 +75,10 @@ file with it.** | **R-197** | **The hub holds both halves of the evidence that a box's offsite DATA key changed, and reads neither.** `restic_pw_sha256` is stored on `host_escrow` and carried to `host_escrow_superseded` on every re-escrow. Comparing the two is what let this spike answer its hardest question in one query — and **nothing in the hub does it** | **SHIPPED** (hub **v0.93.0**, 2026-08-04) | — | **Why this is the cheapest real fix on the table.** A changed repo password means the previous offsite repository is now unopenable by the box, i.e. the customer's off-site history is orphaned. That is the single most consequential state change in the backup system, it is **already fully observable from data the hub owns**, and today it produces **no event, no e-mail, no card and no log line** — demo-felhom's went unremarked for 13 hours and would have gone unremarked indefinitely had this spike not run. **Two-line verdict:** on `SaveHostEscrow`, if the incoming `restic_pw_sha256` differs from the row being superseded, emit a distinct operator event naming the orphaned generation. **Deliberately an EVENT, not a heal** — nothing should act on this automatically until R-193's (c)-vs-accept decision is taken; the point is that the operator learns on the day. **Pair with R-196**, which is the same signal aimed at the right trigger. **Generalises past this row:** *a comparison the system could be making from data it already stores, and is not, is a silence with no cost of entry* — cf. R-190's store-grant probe, where the state was read and the TRANSITION was not. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q8 option (d) **SHIPPED.** `SaveHostEscrow` returns the hash it replaced; `handleHostEscrowPut` raises **`offsite_repo_key_changed`** when both hashes are known and differ. **Edge-triggered** (once per supersession, never per report — the dispatcher owns cooldown), **operator-only** (registered in `notify.operatorOnlyEvents` in the same commit that mints the type, because a missing `customerMessages` entry is NOT a block — the v0.78.0 defect), and **no hash value travels** in the message or the details. The in-between shapes (a first-ever hash, a hash-less supersession) are LOGGED rather than dropped, so *"we chose not to alarm"* and *"the check did not run"* never look identical. **Severity = warning, chosen for the world v0.93.0 creates:** before R-198 a changed key meant the previous history was unopenable by anyone ever, which would have argued for `error`; from v0.93.0 the superseding ceremony retains the old identity blob, so the fact is *"this customer's off-site history now depends on an older recovery code"* — operator-actionable, not a loss. `warning` also routes (the dispatcher treats `info` as an intentional non-notify). Driven through the real endpoint in test, not by calling the emitter. **Red-proof observed:** removing the comparison from the escrow PUT → the changed-key scenario fails with *"the repository key demonstrably changed and NO signal was raised"* while the unchanged-key scenario still passes. | CC | | **R-198** | **The hub's superseded-escrow retention does NOT retain the offsite repository password — and the ceremony the system tells the customer to run is what destroys the last copy.** `host_escrow_superseded` has **no `identity_blob` column**, and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`) copies only `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256`. `blob` is the **K-escrow** (the PBS datastore key, PBS-native scrypt); the **restic repo password lives in `identity_blob`** (`felhom-agent/internal/escrow/identity.go:34-39`, age-wrapped `IdentityBundle`). Measured live: both hosts' current rows hold `blob`=383 B **and** `identity_blob`=572 B; both superseded rows hold `blob`=383 B and nothing else | **SHIPPED** (hub **v0.93.0**, 2026-08-04) | — | **This is the NINTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does not provide, and the first that is ALSO customer-facing copy.** The claim appears three times: the schema comment (`store.go:370-375`, *"so the old passphrase stays customer-R-recoverable … turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable'"*), the capability map's escrow row, and — in Hungarian, to the customer, on the orphan card — `controller/internal/web/templates/backups_remote.html:66,69` (*„a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók lehetnek"*). **For the offsite restic repository, the incident it names, all three are false.** **Why it is worse than a missing column:** a rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`, `offbox.go:570`), the card says „Helyreállítási kód szükséges", and the controller's own detector logs *"run the escrow ceremony"* (`report/escrow_confirm.go:100`) — so **the prescribed remedy is the act that overwrites `host_escrow.identity_blob` and loses the old password forever**. Both demo boxes crossed that line on **2026-08-04 at 07:15:36 (demo-hp) and 07:20:08 (demo-felhom)**. **This is an independent, stronger reason the 51 orphaned demo snapshots are unrecoverable than "nobody kept the recovery codes" — keeping R would not have helped.** **Fix shape (small):** add `identity_blob` to the superseded table and to `demoteCurrentEscrowTx`'s SELECT list; pin it with a test that asserts the CONSEQUENCE (a superseded row can still yield a repo password) rather than the mechanism. **Then correct all three claims in the same commit** — including the Hungarian card, which must not promise what the system cannot do. **Prerequisite for R-193's (d)-alone branch** and for the drill. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §7 **SHIPPED.** `host_escrow_superseded` gains `identity_blob` (CREATE + additive `ALTER TABLE`) and `demoteCurrentEscrowTx` carries it, so **both** callers — re-escrow and host-delete demotion — are fixed by one change to the routine its own comment calls *"THE ONE escrow row-copy routine"*. `ListSupersededEscrow` reads it back and `store.HostEscrow` gains `IdentityBlob`, so a retained blob is reachable from Go at all. The table comment now records that the ruling stated above it was not met and what that cost. **Tests assert the CONSEQUENCE, which is why the existing one stayed green:** `TestSaveHostEscrow_RetainsSuperseded` asserted that a retained row exists carrying the old K-blob and passed throughout; `TestSaveHostEscrow_RetainsIdentityBlob` asserts the retained row can still yield a repository password, and pins the load-bearing ordering (the identity blob is written by `SaveHostDRBundle` AFTER `SaveHostEscrow`, so the demote sees the PREVIOUS generation — if that inverts, the retained bytes would be the new blob filed under the old hash, recoverable-looking and wrong). `TestDeleteHost_DemotesIdentityBlob` proves the shared routine through its other caller. **Red-proofs, both observed failing:** dropping `identity_blob` from the copy (production behaviour ≤ v0.92.0) fails BOTH scenarios; fixing only the re-escrow caller fails the delete scenario while the re-escrow one passes — the §8.2 mistake, demonstrated rather than asserted. **Nothing was backfillable and it was CHECKED, not deduced:** rows superseded before v0.93.0 were written without the blob and their source rows are already overwritten; the live database holds exactly 2 retained rows, both from 2026-08-04, both `identity_blob` NULL. **Who the fix protects, measured live:** 2 of 2 hosts with a current escrow carry an identity blob (`demo-felhom-8363b5`, `demo-hp-bb76ea`) — their NEXT ceremony now retains a recoverable off-site key instead of destroying one. | CC | | **R-199** | **The hub serves recovery blobs on two endpoints that have no client anywhere in the system.** `handleReEnroll` (`hub/internal/api/dr.go:101`) and `handleGetRestoreDirective` (`:155`) return `identity_escrow_b64` + `k_escrow_b64`, gated on operator-armed recovery mode. Census: **zero** callers in `felhom-agent` (no `ReEnroll` symbol at all; `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`), **zero** in the hub UI or any template, **zero** in `scripts/` or any runbook | **SHIPPED + PROVEN-LIVE 2026-08-04** (hub **v0.94.0**, agent **v0.125.0**) | — | **The documented retrieval path is a human with `sqlite3`:** `SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …` on a `kubectl cp`-ed `hub.db` — recorded in project memory as the 2026-07-04 S5 prep, where hub-side blob serving was explicitly deemed *"Part 3 NOT needed"*. **This is the built-but-never-wired class at the DR capstone**, and it is why the chain from a dead node to an open repository has no automatable middle. **What a recovery flow actually needs is smaller than what exists:** a narrow `GET /hosts//escrow` authed with the box's own per-host key, serving opaque bytes to the box that owns them — zero-knowledge untouched, and far lighter than `re-enroll`, which **rotates the host API key** and returns the new key in the response body (`dr.go:130,148`). **Decide before building:** whether `re-enroll`/`restore-directive` should get a client, be replaced by the narrow GET, or be retired. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 6 **LINKS 6, 7 AND 8 ARE ASSEMBLED AND WALKED.** **Link 6:** `GET /api/v1/hosts/{host_id}/escrow` — the box-authenticated MIRROR of the PUT that stores the blob, self-scoped by the per-host key (global may read any, the same asymmetry the PUT has). The operator-driven DR endpoints are UNTOUCHED and pinned by a test that exercises them with recovery mode off and on. **Link 7:** `POST /escrow/recover-offsite-password` on the agent's pinned local API gives `UnwrapIdentityBundle` its first production caller in two months. **Link 8:** it extracts and returns **only** the repository password (not the tunnel token, not the PBS token, not the WG key — the controller is a trust tier down). **PROVEN ON HARDWARE, demo-felhom, 2026-08-04 13:49 CEST:** on-disk `c60c8bc737a6…` vs recovered `c60c8bc737a6…` — **MATCH**, and the same hash the hub independently stores as `restic_pw_sha256`, so three sources agree. **Scenario B proven live 5 minutes earlier** with a deliberately wrong code: hub served the blob (572 B, `self_scope=true`), agent logged *the recovery code did not unwrap the identity escrow … exit status 1*, nothing written — which also proves links 6 and 7 ran independently of the success. **Scenario E proven live:** both retrievals raised `escrow_blob_served` (warning, operator-only); the first mailed the operator, the second was cooldown-suppressed AND that suppression is itself recorded; the customer leg reads `skipped/operator_only` on both. **R persisted nowhere, searched not claimed:** 0 lines in the agent journal, 0 in the controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover `felhom-idesc-*` staging dirs, and the staged-secret dir empty — with a **positive control** (a planted copy was found: 1, then 0 after removal) so the sweep is a measurement rather than an unfalsifiable absence. **THE §8.2 TRADE, made deliberately and recorded in the handler:** obtaining the blob used to require the operator to arm recovery mode; it now needs only the box's own credential. They still cannot open it (the hub never held R; a wrong code fails closed at age's scrypt KDF). `escrowSelfServiceRetrieval` is a single named constant — flipping it to false re-imposes recovery mode and changes nothing else, so the operator can overrule the trade for the cost of a boolean. **Red-proofs observed:** removing the ownership check served host B's blob to host A; removing the audit record made the retrieval silent; returning `PBSToken` instead of `ResticRepoPassword` yielded a plausible bundle with a non-matching key; commenting the `Options.EscrowRecovery` wiring failed the AST seam test. **Seam discipline:** the wiring is asserted by walking `main` → `runDaemon` → `buildLocalAPIServer` and checking the composite literal, not by `strings.Contains` — links 6 and 7 were two of this project's six built-but-never-wired instances and the fix must not become the seventh | CC | -| **R-200** | **The DR password-injection seam has a handler, a route and tests — and no form.** `POST /backup/offbox/inject-password` is routed (`controller/internal/web/server.go:510`) to `offboxInjectPasswordHandler` (`offbox_handlers.go:174-196`) → `InjectOffboxPassword` (`backup/offbox.go:541`). **No template in the repository contains that path or any form posting to it** (grep over `internal/web/templates/`: one unrelated hit, an XSS comment) | **PLUMBING SHIPPED** (controller **v0.195.0**); **the FORM is not built — still open** | — | **The tenth instance of this project's built-but-never-wired class, and the exact shape `CLAUDE.md` and `felhom.eu/CLAUDE.md`'s seam-wiring rule were written for:** handler tests that POST directly (`offbox_escrow_test.go:167,180`) prove nothing about reachability. To use the only implemented recovery seam today, a person must hand-craft an authenticated POST with a session cookie and CSRF token. **Note the layering while fixing it:** this form takes a **64-hex repo password**, not a recovery code (`offboxRepoPwPattern`, `offbox.go:543`) — they are different secrets at different layers, and the operator's 2026-08-04 ruling asks for a form that takes **R**. Build the R form and treat this one as the operator/DR fallback it was written as, but **ship it with a render test per branch of whatever gate it sits behind**. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 9 **THE DIAGNOSTIC HALF IS DONE AND IT ANSWERED THE QUESTION.** `--recover-offsite-check` is a `docker exec` escape hatch in the shape of `--print-reset-code`: R on **STDIN** (never argv, never `ps`, never shell history, never a transcript), fetch+unseal via the agent, and a verdict of **two sha256 hashes**. **It compares and never installs** — the recovered password is not written to `offbox/repo_password`; a test asserts the data dir is byte-unchanged and its red-proof (adding the install call) fails it. Confirmed live: `repo_password` mtime still `2026-08-03 07:18:02` after the successful check at `2026-08-04 11:49`. **Exit codes are load-bearing** — `0` match, `2` a clean MISMATCH, `1` a step failed; "it failed" and "it worked and disagreed" must never share a status because only one is a finding about the system. A box with no local password reports distinctly (the rebuilt-box shape, where the next step is to INSTALL rather than compare). **WHAT IS NOT BUILT, deliberately:** no card, no form, no preview, no customer-facing text — building an interface on top of a chain nobody had walked is how the preceding three weeks went wrong. **What remains for this row:** link 9 (the recovered password placed so `WriteOffboxSecrets` keeps it) and the customer-facing shape the operator ruled on 2026-08-04 (yell → R form → preview → proceed), which is now priced against a chain that exists rather than one that is assumed | CC | -| **R-201** | **Nothing in the offsite DR chain has ever been exercised past the ceremony — and the one live proof that exists predates the field it is cited for.** `slice10d-identity-restore-spike-findings.md` §1 (2026-06-10) proved a real wrap→unwrap of an identity bundle with a real R on a secret-less box, byte-identical, wrong-R failing closed. **That bundle was `{tunnel_token, pbs_token}`.** `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** — a month later — and is **unit-proven only** | **OPEN — the KEY is proven recoverable; the RESTORE is not** | — | **Never exercised, in these words:** a blob has never been served to a box; a fork-4 bundle has never been unsealed with a real R outside a unit test (the only production caller is `--selftest=identity-consume`, `cmd/felhom-agent/main.go:2845`, reading R from `FELHOM_RECOVERY_CODE`); a recovered repo password has never been injected; an existing offsite repository has never been reopened with one; **no restore of any kind has ever been performed from a recovered secret.** The one live consume ever prepared (S5 Part 4-B, 2026-07-04) is still marked *"DEFERRED, not run"*. **`_recovery-inventory-2026-07-28.md` already recorded this accurately** (A.2.7, C.1 row 1, C-2) — this row exists so the gap has an owner and a closing condition, not only a description. **Closing condition = the drill** designed in `audits/RECON-offsite-dr-chain-2026-08-04.md` §10, whose pass condition is a **byte-identical sha256 of a sentinel file restored after a wipe** — explicitly NOT "the repository opened". **Do not run it before R-198 is fixed**: the drill would walk a chain that is missing a link everyone believed was there **MATERIALLY ADVANCED 2026-08-04, AND THE DISTINCTION MATTERS.** What was proven on hardware is that the offsite repository **password** comes back out of the sealed bundle byte-identical (R-199). What has STILL never happened: a recovered password **installed**, an existing repository **reopened** under one, and a **file restored** from it. **The drill's pass condition is unchanged and is not this** — it is a byte-identical sha256 of a sentinel file after a wipe and reinstall, and "the store opened" was explicitly ruled insufficient. Design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10. **The drill is now cheaper and better-founded than when it was designed:** links 6–8 are walked, so a failure during the drill can be localised instead of being an undifferentiated "recovery did not work", and R-198 means the ceremony that creates the drill's recovery code no longer destroys the key it is meant to protect | CC + operator | +| **R-200** | **The DR password-injection seam has a handler, a route and tests — and no form.** `POST /backup/offbox/inject-password` is routed (`controller/internal/web/server.go:510`) to `offboxInjectPasswordHandler` (`offbox_handlers.go:174-196`) → `InjectOffboxPassword` (`backup/offbox.go:541`). **No template in the repository contains that path or any form posting to it** (grep over `internal/web/templates/`: one unrelated hit, an XSS comment) | **PLUMBING COMPLETE** (controller **v0.196.0**); **the FORM is not built — still open** | — | **The tenth instance of this project's built-but-never-wired class, and the exact shape `CLAUDE.md` and `felhom.eu/CLAUDE.md`'s seam-wiring rule were written for:** handler tests that POST directly (`offbox_escrow_test.go:167,180`) prove nothing about reachability. To use the only implemented recovery seam today, a person must hand-craft an authenticated POST with a session cookie and CSRF token. **Note the layering while fixing it:** this form takes a **64-hex repo password**, not a recovery code (`offboxRepoPwPattern`, `offbox.go:543`) — they are different secrets at different layers, and the operator's 2026-08-04 ruling asks for a form that takes **R**. Build the R form and treat this one as the operator/DR fallback it was written as, but **ship it with a render test per branch of whatever gate it sits behind**. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 9 **THE DIAGNOSTIC HALF IS DONE AND IT ANSWERED THE QUESTION.** `--recover-offsite-check` is a `docker exec` escape hatch in the shape of `--print-reset-code`: R on **STDIN** (never argv, never `ps`, never shell history, never a transcript), fetch+unseal via the agent, and a verdict of **two sha256 hashes**. **It compares and never installs** — the recovered password is not written to `offbox/repo_password`; a test asserts the data dir is byte-unchanged and its red-proof (adding the install call) fails it. Confirmed live: `repo_password` mtime still `2026-08-03 07:18:02` after the successful check at `2026-08-04 11:49`. **Exit codes are load-bearing** — `0` match, `2` a clean MISMATCH, `1` a step failed; "it failed" and "it worked and disagreed" must never share a status because only one is a finding about the system. A box with no local password reports distinctly (the rebuilt-box shape, where the next step is to INSTALL rather than compare). **WHAT IS NOT BUILT, deliberately:** no card, no form, no preview, no customer-facing text — building an interface on top of a chain nobody had walked is how the preceding three weeks went wrong. **What remains for this row:** link 9 (the recovered password placed so `WriteOffboxSecrets` keeps it) and the customer-facing shape the operator ruled on 2026-08-04 (yell → R form → preview → proceed), which is now priced against a chain that exists rather than one that is assumed **PART 0 SHIPPED 2026-08-04 (v0.196.0):** `--recover-offsite-install` is the sibling of the check — same fetch/unseal path, same STDIN discipline for R — and it **places** the recovered password via `InjectOffboxPassword`. The confirmation is a SECOND invocation (`--confirm-install`): without it, both hashes print and nothing is written, so the operator sees the comparison before a write is possible. Three outcomes named distinctly: **installed** (no local password — the rebuilt-box shape), **unchanged** (identical key, nothing written), **refused** (a DIFFERENT key present — installing would clobber the key the current repository is encrypted under; exit 2, no force offered). It re-reads the file after writing rather than trusting the call. **Red-proof observed:** removing the confirmation gate makes the dry run write the password. The R-persistence test carries a **positive control** (a planted copy found, then removed and not found). **NOT YET EXERCISED AGAINST A LIVE RECOVERY** — the R-201 drill halted before step 9, so this is unit-proven only. **What remains for this row:** the customer-facing shape the operator ruled on 2026-08-04 (yell → recovery-code form → preview → proceed) | CC | +| **R-201** | **Nothing in the offsite DR chain has ever been exercised past the ceremony — and the one live proof that exists predates the field it is cited for.** `slice10d-identity-restore-spike-findings.md` §1 (2026-06-10) proved a real wrap→unwrap of an identity bundle with a real R on a secret-less box, byte-identical, wrong-R failing closed. **That bundle was `{tunnel_token, pbs_token}`.** `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** — a month later — and is **unit-proven only** | **OPEN — the drill was PREPARED and HALTED BEFORE THE WIPE (2026-08-04)** | — | **Never exercised, in these words:** a blob has never been served to a box; a fork-4 bundle has never been unsealed with a real R outside a unit test (the only production caller is `--selftest=identity-consume`, `cmd/felhom-agent/main.go:2845`, reading R from `FELHOM_RECOVERY_CODE`); a recovered repo password has never been injected; an existing offsite repository has never been reopened with one; **no restore of any kind has ever been performed from a recovered secret.** The one live consume ever prepared (S5 Part 4-B, 2026-07-04) is still marked *"DEFERRED, not run"*. **`_recovery-inventory-2026-07-28.md` already recorded this accurately** (A.2.7, C.1 row 1, C-2) — this row exists so the gap has an owner and a closing condition, not only a description. **Closing condition = the drill** designed in `audits/RECON-offsite-dr-chain-2026-08-04.md` §10, whose pass condition is a **byte-identical sha256 of a sentinel file restored after a wipe** — explicitly NOT "the repository opened". **Do not run it before R-198 is fixed**: the drill would walk a chain that is missing a link everyone believed was there **MATERIALLY ADVANCED 2026-08-04, AND THE DISTINCTION MATTERS.** What was proven on hardware is that the offsite repository **password** comes back out of the sealed bundle byte-identical (R-199). What has STILL never happened: a recovered password **installed**, an existing repository **reopened** under one, and a **file restored** from it. **The drill's pass condition is unchanged and is not this** — it is a byte-identical sha256 of a sentinel file after a wipe and reinstall, and "the store opened" was explicitly ruled insufficient. Design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10. **The drill is now cheaper and better-founded than when it was designed:** links 6–8 are walked, so a failure during the drill can be localised instead of being an undifferentiated "recovery did not work", and R-198 means the ceremony that creates the drill's recovery code no longer destroys the key it is meant to protect **THE DRILL RAN AND DID NOT REACH ITS VERDICT, and stopping was the correct call** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). Steps 1–4 completed; the wipe never happened; **nothing irreversible was done**. It halted because **the sentinel file was not in the off-site snapshot** (R-203) — wiping would have destroyed the only copy and proven nothing. **Sentinel sha256 `643166269103a25c…`, still on the box.** **What the attempt established live, all of it new:** (1) a rebuilt box's off-site run **refuses** with the orphan card and pushes `offbox_repo_orphaned` — the spike's predicted third outcome, measured for the first time, and it does NOT silently start a fresh history (this closes R-193's Q3); (2) the **orphan reset works** — move-aside to `/home/felhom-repo.orphaned-20260804`, never delete, fresh repo initialised, `offbox_repo_reset` pushed (operator-authorised during the session); (3) demo-hp's pre-rebuild history is **permanently unrecoverable** — its key sits in superseded row id 3 with `identity_blob` NULL, superseded 07:15:36, **four hours before v0.93.0 fixed the retention**; (4) **a precondition the runbook did not contain:** neither pre-existing off-site-toggled app has a restorable file leg — both are named-volume-only, which the off-site tier tars but the customer restore never unpacks — so a file-leg app (`calibre-web`, mandatory `userdata: media/books`) had to be deployed, and it is now in place as the fixture. **TO RESUME:** fix or scope R-203, re-run steps 4–5 and confirm the sentinel IS in the snapshot **by listing it, not by a green status**, then P4 + the STOP + steps 6–11. Everything else is already staged: versions, recovery code, working repository, file-leg app, sentinel. **The pass condition is unchanged** — a byte-identical sentinel sha256, not "the store opened" | CC + operator | | **R-202** | **The orphan card promises the customer their old backups "may later be restorable with the matching recovery code" — and after v0.93.0 that is true for supersessions from now on and FALSE for anything already orphaned.** `controller/internal/web/templates/backups_remote.html:66,69` states it unconditionally, in Hungarian, on the one surface where being wrong costs most | **OPEN — Part 5 hit its gate 2026-08-04; the card is UNTOUCHED and the sentence is still live** | R-199/R-201 (which generation an orphaned repo belongs to is not knowable to the box today) | — | **THE GATE, and why it was hit rather than squeezed past.** The condition was: ship it iff the hub can tell a box what it needs with **one** additional boolean on the escrow ACK it already sends. The hub *can* cheaply compute *"≥1 retained blob for this host carries an identity blob"* — one correlated predicate in `GetEscrowStatusForCustomer`, and the controller even has the right seam already (`SetEscrowStale`/`StaleBlob` is exactly this shape). **But that boolean does not answer the card's question.** The card renders on `RepoState == "orphaned"`, and the promise is about *the key THIS orphaned repository was written under*. A box does not know which escrow generation the orphaned remote belongs to; a box with a pre-v0.93.0 orphan and a post-v0.93.0 supersession would read the boolean TRUE and the promise would still be false — a **conditional** falsehood that looks verified, which is strictly worse on a customer-facing card than today's hedged one. **What would actually make it truthful** is knowing the orphaned repo's generation, which is the same knowledge R-199/R-201's unassembled chain needs. **Interim exposure, stated rather than buried:** the sentence remains live and remains false for both demo boxes. **Cheapest honest interim** (not taken here — it is a customer-copy change and the gate said leave it alone): drop the recoverability clause and say only that the old history is set aside and not deleted, which is true unconditionally | CC + operator | +| **R-203** | **A customer-declared MANDATORY data directory was silently absent from the off-site snapshot while the run reported `ok`.** Measured live on demo-hp 2026-08-04: `calibre-web` declares `userdata: media/books class: mandatory`; its live bind is `/mnt/sys_drive/userdata/media/books` (the sentinel file was there), while the off-site capture set looked for `/mnt/sys_drive/felhom-data/userdata/media/books`, which does not exist. Result: `[WARN] mandatory data path missing on disk, skipped from offsite`, then `backed up calibre-web (…, 0 mandatory path(s))` and `backup OK: 3 app(s), 3 snapshot(s)` — `last_status: ok`, `last_success` stamped, nothing customer-visible, nothing hub-visible | **OPEN — HALTED THE R-201 DRILL** | — | **THE MECHANISM, from source.** `NamespaceRoot(drivePath, inGuestDrive)` (`appbackup/paths.go:28-33`) appends the `felhom-data` segment **when the drive IS the system data path** (`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`, `backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is `/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an app deployed at `HDD_PATH=/mnt/sys_drive`, the two produce different directories. **The same compose used BOTH roots**, from one deploy: `${IMPORT_PATH}` → `/mnt/sys_drive/felhom-data/userdata/import/calibre` (with the segment), `${USERDATA_PATH}` → `/mnt/sys_drive/userdata/media/books` (without). **WHAT IS MEASURED vs NOT, because it changes the fix.** MEASURED: the paths disagree, the mandatory directory is absent from the snapshot, the run says `ok`, and the only signal is a container-log WARN. **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a SUPPORTED choice — it was used because demo-hp's only registered drive (`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working as designed), while `/mnt/sys_drive` was **accepted** (HTTP 202). **EITHER BRANCH IS A DEFECT:** if the system drive is a supported app namespace, userdata resolution is wrong for every app on it and their mandatory directories are silently unprotected; if it is not supported, the deploy accepted a namespace it should have refused one call after refusing the NAS. **NOT a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths (everything of theirs is in named volumes), so they are unaffected and their snapshots are real. **Fix shape:** make the two roots one function, whichever is right — and make a skipped MANDATORY path a customer/hub-visible signal rather than a WARN, because `ok` with a missing mandatory directory is this project's own *a path the customer thinks is protected is not in the snapshot* shape. Source: `audits/DRILL-r201-offsite-recovery-2026-08-04.md` §2 | CC | | **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **HALF SHIPPED** (hub **v0.93.0**) — **the guard's SCOPING stays OPEN** | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. **SPIKE 2026-08-04 — BOTH HALVES CONFIRMED WITH NUMBERS, still OPEN, still not fixed here** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q7). The query is quoted at source (`store.go:987`): `... ORDER BY id LIMIT 500` = **the oldest 500**. Reproduced against the live hub DB with demo-hp's real consume anchor `2026-07-23 09:53:41`: the guard sees `total=500, withOffsite=500`, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** — the entire evidence set ends **six days before** the 2026-08-03 rebuild. The true window totals are `1174 / 1063` (⇒ 111 without, matching the 111 offsite-less reports). So the e-mail's *"500 report(s) since carry no offbox target"* interpolates `ReportsSinceConsume` while `OffsiteReportsSinceConsume` was **500** — the message states the precise negation of its own measurement. `offsite_credential_restaged` has **never fired for any customer** (zero rows of that type in the DB — checked, not assumed). **A NEW REASON NOT TO FIX THIS IN ISOLATION, from the same spike:** under R-193's Q2 finding a successful auto-restage would have restored demo-hp's TRANSPORT while the box minted a new repo password anyway — the heal can protect the plumbing and **cannot** protect the data, and had it fired on 2026-08-04 both boxes would have looked healthy while their snapshots were orphaned. **That is strictly worse than the current loud failure.** Whatever shape the fix takes must say so in its message. **RECON 2026-08-04 adds two inputs and changes no verdict** (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **(1) The guard's own conclusion — that a fix here protects the plumbing and not the data — is now stronger, not weaker:** even a perfect restage leaves the rebuilt box minting a fresh repo password, and per **R-198** the old one is destroyed by the re-ceremony the box is pushed into. Whatever shape the message takes must say that in the same breath, or it will read as an all-clear. **(2) The recency-bounded discriminator this row asks for has a ready anchor the hub already receives:** the report ACK's `escrow{identity_blob_present, restic_pw_sha256}` moves when a box re-keys, so "evidence since the newest re-key" is computable from data already stored — the same observation R-197 makes, aimed at this guard's time window. **BOTH HONESTY HALVES SHIPPED 2026-08-04 (hub v0.93.0); THE GUARD'S LOGIC IS DELIBERATELY UNTOUCHED.** **(a) The message now describes what was measured.** The one stuck state is reported as the two situations it actually covers — **burned** (`OffsiteReportsSinceConsume == 0`) and **regressed** (> 0, the demo-hp shape) — each stating its own measurement and carrying its own recommendation; the regressed text explicitly WITHDRAWS Re-issue and points at what removes an offbox target (a guest rebuild, R-193). `offsite_reports_since_consume` rides the details for the first time. **(b) Every refusal to self-heal leaves a record** — a `notification_log` row on the operator channel, status `refused`, with its reason (the R-182 suppressed-e-mail precedent), riding the stuck event's 24 h cadence so it sits beside the e-mail it explains rather than accumulating per tick. The two conditions were split into separate branches solely so each can name its own reason; **the set of situations in which the heal fires is byte-for-byte what it was**. **(c) Not in the spec and done anyway, narrowing only:** `offsite_delivery_stuck` and `offsite_credential_restaged` are added to `operatorOnlyEvents`. Neither was ever registered and neither has a `customerMessages` entry — which is not a block — so a customer with a configured recipient was in line for an English e-mail about one-time passwords being *"likely burned"*. Measured live: `notification_log` holds operator rows for demo-hp and no customer rows, which is NOT evidence the leg was blocked (equally consistent with no configured recipient), so the register makes it structural. **WHAT STAYS OPEN, and it is this row now:** `CountReportsOffsiteSince` reads `ORDER BY id LIMIT 500` — the OLDEST 500 reports after the consume — so the counts describe the start of the window, not the present. Its correct shape (recency-bounded, rebuild-aware) depends on the recovery chain that is not yet assembled (R-199/R-200/R-201), so it was NOT fixed here. **The window is named inside the alert text** so the limitation travels with the number instead of being laundered into a confident sentence. **Red-proofs observed:** restoring the single hardcoded sentence fails both message scenarios (the first mutation attempt left the default branch in place and only the burned scenario failed — recorded because a mutation that does not remove every guard is not a red-proof); replacing the regressed branch with a bare `return` fails the refusal record and its cadence test. | CC | | **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **OPEN** — the (c) decision is TAKEN (accept the risk); the chain is still unassembled | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTP access key…)"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the one a rebuild destroys, and **nothing automatic ever restages it**: `WriteOffboxSecrets` mints a fresh 256-bit password whenever `/offbox/repo_password` is absent (`offbox.go:392`), and the only recovery path, `InjectOffboxPassword`, has **exactly one caller in the whole repo** — a web form a human pastes into (`web/offbox_handlers.go:189`). **(2) MEASURED, WITHOUT TOUCHING A BOX:** the hub already stores `restic_pw_sha256` on both the live and the superseded escrow, so the question is a hash comparison. demo-hp `8e03eddf…` → `8a9e33aa…`; demo-felhom `48741892…` → `c60c8bc7…`. **Both boxes minted a new repository password.** **(3) THE CONTRAST IN THIS ROW IS FALSE FOR THE DATA.** demo-felhom's 76-second "lucky" recovery restored **delivery only** — its pre-rebuild object carried **36 snapshots / 1.14 GB** (`repo_size_bytes 1136685919`) and it has reported `snapshot_count: 0` in all 109 reports since, with a changed repo password and **nothing marking its escrow stale for 13 h**. Both boxes lost repository continuity; one loudly, one silently, and **the silent one is worse**. **(4) Q3 IS UNMEASURED AND THE BINARY WAS WRONG.** Neither box could run on 2026-08-04 02:15 UTC (demo-hp had no target at all until 07:15; demo-felhom's target was `escrow_state: pending`, which `OffboxRunnable` blocks) — **the decisive run is 2026-08-05 ~02:15 UTC**. Source predicts a **third** outcome, neither 15 nor 1: same account + same repo path (`u629488-sub3:/home/felhom-repo`, and `repoPath` is a compile-time constant) + new password + `claimed: 1` ⇒ `ensureOffboxRepo` classifies **orphaned** and returns `ErrOffboxOrphaned` — **the run refuses and shows the orphan card**, it does not start a silent fresh history. **Record which of the three actually occurs; a prediction from source is not a measurement.** **(5) CANDIDATE (b) IS NOT IMPLEMENTABLE AS STATED** — the escrow is R-wrapped/zero-knowledge and the hub has no recovery code, so "recoverable from escrow at re-bootstrap" describes a customer-present ceremony, i.e. the manual form that already exists. **(6) CANDIDATE (a) ALREADY EXISTS AND IS WIRED TO THE WRONG EVENT:** `reissueOnReenroll`'s **F3 leg** does exactly this (`api/handler.go:1051-1084`) but sits behind `handleHostEnroll`'s mint-once-reuse short-circuit (`if existing != nil { return }`), and a **guest** rebuild leaves the `hosts` row intact — so F3 is never reached. **(7) A NEW CANDIDATE (c), not previously named and recommended second:** the **agent survives a guest rebuild**, already receives the repo password over the pinned local API (`POST /escrow/stage-secret`) and already writes it to a fixed 0600 path — it merely **wipes** it after the ceremony. Retaining it and serving it back is the only candidate that addresses the irreplaceable secret, and every seam it needs exists. Its cost is one deliberate trade the operator must make (a copy of the data key at rest on the Proxmox host — see D6). **SPIKE RECOMMENDATION: ship the honesty pass (R-196 + R-197) now; then decide (c). Do NOT ship (a) first — it would have hidden this.** **NOT CLOSED — WAITING-ON-OPERATOR for the (c)-vs-accept-it decision, stated at the end of the spike.** **RECON 2026-08-04 — the chain was traced link by link, and the picture is worse than the spike's** (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **(A) THE SPIKE'S CANDIDATE (b) IS OVERTURNED IN PART.** "Recoverable from escrow" is not blocked by zero-knowledge — the *hub* cannot open the blob, the *customer* can, with R. What is genuinely impossible is an **unattended** recovery. A **customer-present** one is a real design, and the operator has now ruled on its shape (below). **(B) THE CHAIN IS NOT ASSEMBLED — eleven links, and the automation stops at four.** Mint → stage → seal → store on the hub are PROVEN-LIVE. Then: the hub's blob-serving endpoints have **no client anywhere** (**R-199**); unsealing's only production caller is a `--selftest` mode reading R from an env var; nothing extracts `restic_repo_password` from the recovered bundle; the injection seam **has no form** (**R-200**); and reopening an existing repo with a recovered password has **never happened** (**R-201**). **(C) THE LOAD-BEARING NEW FACT, and it retires this row's own "hub v0.60.0 retains superseded escrow" premise: the retention does NOT retain the restic repo password** — `host_escrow_superseded` has no `identity_blob` column (**R-198**). Both demo boxes' old passwords were destroyed by the 2026-08-04 re-ceremonies, so the orphaned snapshots are unrecoverable for a **second, independent** reason; keeping R would not have helped. **(D) A FAIL-CLOSED REFUSAL IS IMPLEMENTABLE — this is the most useful thing settled.** The hub already tells every box, on every report ACK, `escrow{identity_blob_present, restic_pw_sha256, created_at}` (`hub/internal/api/handler.go:504-510`) — and the controller **discards it** whenever no offbox target exists (`report/escrow_confirm.go:75-84`). Persisting it (the `ClaimSync` set-only pattern, `report/claim_sync.go:39-53`) and refusing to mint when a blob covers a password we do not have needs **no new hub API and no new secret**. **(E) OPERATOR RULING, 2026-08-04, recorded verbatim:** *"If a node is a fresh install AND the hub has a recovery blob, then the controller should yell that recovery is available, and provide a form for the customer to enter the recovery key. After unlocking the blob, the controller should show what will be recovered before proceeding."* Priced row by row in the recon §9: fresh-install signal **exists** (the mint branch's own `os.Stat`); hub-has-a-blob **exists on the wire, S to persist**; the yell **S**; an **R** form **does not exist** (the UI has only ever *emitted* R) **S**; unsealing must cross agent→controller because the controller image ships no `age` — **M**, one new agent local-API endpoint mirroring `/escrow/ceremony/claim`, plus a narrow hub `GET /hosts//escrow`; the **preview is cheap and read-only** — `restic snapshots --json` + `stats --mode raw-data --json` are already how the box counts snapshots (`offbox.go:1234-1265`), so count, dates, sizes, app tags and paths are all knowable before committing **S**. **Security question put to the operator, not answered:** the form sits behind the dashboard password (bcrypt + CSRF + 5/min lockout); the preview exposes backup cadence and app names; the form is an **oracle** for a stolen R and must fail as generically as `UnwrapIdentity` already does; and R transits the agent, which is the same trade as option (c) in a smaller, time-bounded form. **(F) THE DRILL IS DESIGNED AND NOT RUN** (recon §10): demo-hp, ~3–4 h, R kept deliberately, sentinel file sha256 before and after, **pass = byte-identical sha256, NOT "the repository opened"**, fail = a snapshot count of 1. **Run R-198's fix first.** **(G) Q3 STILL UNMEASURED:** neither box has run since (`last_run` absent on both, 2026-08-04 09:49/09:56 reports) — the decisive run remains **2026-08-05 ~02:15 UTC**. **OPERATOR RULINGS 2026-08-04, and one of them changes what the other items are for.** (1) **Candidate (c) is REFUSED — the risk is accepted:** no repository password is retained on the Proxmox host. **That makes the customer-present recovery path the ONLY way back from a rebuild**, which is why R-198 was shipped the same day as a load-bearing fix rather than a tidy-up: with no host-retained copy, everything runs through the retained identity blob, and until hub v0.93.0 the ceremony destroyed it. (2) **Run the drill, after R-198** — R-198 has shipped, so the drill is the next session (design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10; tracked as R-201). (3) **Delete the orphaned ciphertext** — ~1.2 GB across the two demo boxes; **STILL OWED**, deliberately not done in v0.93.0 (a destructive act on a protected endpoint does not ride a schema-change release). **What v0.93.0 delivers against this row:** the key now SURVIVES a supersession (R-198) and a changed key is now REPORTED on the day (R-197). **What it does NOT:** the chain that hands the key back is still unassembled at three links — R-199 (no client for the hub's blob-serving endpoints), R-200 (no form for the injection seam), R-201 (never exercised end to end). This row stays open until the drill returns a byte-identical sentinel file. | CC + operator | | — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 58b56c8..3a61f92 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -63,6 +63,8 @@ | R-202 | **The orphan card promises recoverability unconditionally**, which after R-198 is true going forward and false for anything already orphaned | S | **gate hit 2026-08-04 — card untouched, sentence still live** | Blocked on knowing which escrow generation an orphaned repo belongs to (R-199/R-201). A single ACK boolean can say a retained recoverable blob EXISTS but not that one COVERS this repo; a conditional promise that can still be false is worse on that surface than a hedged one | | R-199 | **The hub served recovery blobs on endpoints with no client anywhere** — the DR capstone's retrieval had never been called by the agent, the hub UI, a script or a runbook | M | **SHIPPED + PROVEN-LIVE (hub v0.94.0 + agent v0.125.0, 2026-08-04)** | Adds the capability-map row *"the offsite repository password can be RECOVERED from the sealed escrow"* as PROVEN-LIVE — for the KEY only, explicitly not for a restore. Chain links 6–8 walked on demo-felhom: recovered sha256 == on-disk sha256 == the hub's stored hash. Carries a deliberate security trade (a box's own credential now suffices where an operator-armed recovery mode was required), recorded in the handler and overridable by one constant | | R-200 | **The password-injection seam had a handler and no form** | M | **plumbing SHIPPED (controller v0.195.0); the form is NOT built** | `--recover-offsite-check` compares by hash and refuses to install. The customer-facing shape (yell → recovery-code form → preview → proceed) is priced against a chain that now exists rather than one that was assumed | +| R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN | +| R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 | | R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE | | R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) | | R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |