From 67eced8fbf16aa0a6a2873ebc2a8f8cf0934e584 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 10 Aug 2026 10:10:43 +0200 Subject: [PATCH] demo-felhom is protected again, and the authorised recovery could never have worked Checked before acting, and the check is the finding. The box's local key and the hub's sealed escrow key hash to the SAME value (c60c8bc737a6b7c6...), and that key answers "wrong password or no key found" against its own repository. Running the recovery would have returned a key the box already held and which was already proven not to open the store. The store was written under 48741892f0ef4d59... -- host_escrow_superseded id=4, superseded 2026-08-04 07:20:08, identity_blob NULL. The restic password lives only in the identity bundle (escrow/identity.go:39, read by recover.go:91), so it is unrecoverable by construction; the surviving K-escrow payload is 64 bytes, a wrapped key, far too small to carry it. Same shape the register already records for demo-hp, four hours the wrong side of the retention fix. Took the operator's stated fallback instead: the orphan reset through the customer's own card. Old store moved aside, never deleted, to /home/felhom-repo.orphaned-20260810 (1.2 GB); fresh repository under the current key; offbox_repo_reset audited hub-side. Then PROVEN rather than assumed -- last_status ok, 10s, and the snapshot's CONTENTS listed: opengist compose files, manifest.json and volume-dumps/opengist_opengist_data.tar. Not an empty backup calling itself successful. R-202 gains hard evidence: the orphan card promises those set-aside backups may be restorable later with their recovery code. For these 1.2 GB that is false and unfixable, and it is said to the customers most likely to read it. --- STATUS.md | 10 +++++----- documentation/backlog/OPEN-ITEMS.md | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/STATUS.md b/STATUS.md index a3ee1ce..25deff6 100644 --- a/STATUS.md +++ b/STATUS.md @@ -21,11 +21,11 @@ untouched and still blocked, as intended. first scheduled run since the rebuild is tonight at 04:15. Two apps it had before the rehearsal (opengist, privatebin) were never reinstalled; only Calibre-Web was, as the walk needed. -**demo-felhom has had no off-site backup for a week, and it will not fix itself.** Its repository is -flagged **orphaned**; the last attempt was 2026-08-05 and **no run has ever succeeded**. The hub -raised `offsite_stale` this morning and mailed you — that alarm is **true**. The remedy is the -recovery ceremony on that machine's own screen, using its recovery code, which is the one thing I -should not do without you saying so. *(R-278.)* +**demo-felhom is protected again — and the recovery you authorised turned out to be impossible.** Before running it I checked, and the sealed package holds *the same key the machine already had* — a key that provably does not open its own backup store. Recovering it would have handed back something useless. The store was written under an older key whose sealed copy was **not retained** (the retention fix landed hours too late for it), so **those 1.2 GB are permanently unreadable by anyone, including us**. + +So I took your stated fallback: the old store was **moved aside, not deleted** (`/home/felhom-repo.orphaned-20260810`), a fresh one was created under the current key, and a real backup ran — **succeeded in 10 seconds**, and I listed what is inside it rather than trusting the green tick: OpenGist's configuration, its manifest and its data volume. **The week without off-site protection is over.** *(R-278 closed.)* + +**One thing that needs your judgement, not mine.** The card that offered this told the customer their set-aside backups *may be restorable later with their recovery code*. For these ones that is simply untrue, and it is said to precisely the people who have just lost their history. *(R-202 — now evidenced.)* ## What works diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index df81b0e..315453b 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -270,7 +270,7 @@ unconditional promise, which CAMPAIGN-11 §7 step 7 measured the customer-facing | **R-199** | **The hub serves recovery blobs on two endpoints that have no client anywhere in the system.** `handleReEnroll` (`hub/internal/api/dr.go:101`) and `handleGetRestoreDirective` (`:155`) return `identity_escrow_b64` + `k_escrow_b64`, gated on operator-armed recovery mode. Census: **zero** callers in `felhom-agent` (no `ReEnroll` symbol at all; `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`), **zero** in the hub UI or any template, **zero** in `scripts/` or any runbook | **SHIPPED + PROVEN-LIVE 2026-08-04** (hub **v0.94.0**, agent **v0.125.0**) | — | **The documented retrieval path is a human with `sqlite3`:** `SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …` on a `kubectl cp`-ed `hub.db` — recorded in project memory as the 2026-07-04 S5 prep, where hub-side blob serving was explicitly deemed *"Part 3 NOT needed"*. **This is the built-but-never-wired class at the DR capstone**, and it is why the chain from a dead node to an open repository has no automatable middle. **What a recovery flow actually needs is smaller than what exists:** a narrow `GET /hosts//escrow` authed with the box's own per-host key, serving opaque bytes to the box that owns them — zero-knowledge untouched, and far lighter than `re-enroll`, which **rotates the host API key** and returns the new key in the response body (`dr.go:130,148`). **Decide before building:** whether `re-enroll`/`restore-directive` should get a client, be replaced by the narrow GET, or be retired. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 6 **LINKS 6, 7 AND 8 ARE ASSEMBLED AND WALKED.** **Link 6:** `GET /api/v1/hosts/{host_id}/escrow` — the box-authenticated MIRROR of the PUT that stores the blob, self-scoped by the per-host key (global may read any, the same asymmetry the PUT has). The operator-driven DR endpoints are UNTOUCHED and pinned by a test that exercises them with recovery mode off and on. **Link 7:** `POST /escrow/recover-offsite-password` on the agent's pinned local API gives `UnwrapIdentityBundle` its first production caller in two months. **Link 8:** it extracts and returns **only** the repository password (not the tunnel token, not the PBS token, not the WG key — the controller is a trust tier down). **PROVEN ON HARDWARE, demo-felhom, 2026-08-04 13:49 CEST:** on-disk `c60c8bc737a6…` vs recovered `c60c8bc737a6…` — **MATCH**, and the same hash the hub independently stores as `restic_pw_sha256`, so three sources agree. **Scenario B proven live 5 minutes earlier** with a deliberately wrong code: hub served the blob (572 B, `self_scope=true`), agent logged *the recovery code did not unwrap the identity escrow … exit status 1*, nothing written — which also proves links 6 and 7 ran independently of the success. **Scenario E proven live:** both retrievals raised `escrow_blob_served` (warning, operator-only); the first mailed the operator, the second was cooldown-suppressed AND that suppression is itself recorded; the customer leg reads `skipped/operator_only` on both. **R persisted nowhere, searched not claimed:** 0 lines in the agent journal, 0 in the controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover `felhom-idesc-*` staging dirs, and the staged-secret dir empty — with a **positive control** (a planted copy was found: 1, then 0 after removal) so the sweep is a measurement rather than an unfalsifiable absence. **THE §8.2 TRADE, made deliberately and recorded in the handler:** obtaining the blob used to require the operator to arm recovery mode; it now needs only the box's own credential. They still cannot open it (the hub never held R; a wrong code fails closed at age's scrypt KDF). `escrowSelfServiceRetrieval` is a single named constant — flipping it to false re-imposes recovery mode and changes nothing else, so the operator can overrule the trade for the cost of a boolean. **Red-proofs observed:** removing the ownership check served host B's blob to host A; removing the audit record made the retrieval silent; returning `PBSToken` instead of `ResticRepoPassword` yielded a plausible bundle with a non-matching key; commenting the `Options.EscrowRecovery` wiring failed the AST seam test. **Seam discipline:** the wiring is asserted by walking `main` → `runDaemon` → `buildLocalAPIServer` and checking the composite literal, not by `strings.Contains` — links 6 and 7 were two of this project's six built-but-never-wired instances and the fix must not become the seventh | CC | | **R-200** | **The DR password-injection seam has a handler, a route and tests — and no form.** `POST /backup/offbox/inject-password` is routed (`controller/internal/web/server.go:510`) to `offboxInjectPasswordHandler` (`offbox_handlers.go:174-196`) → `InjectOffboxPassword` (`backup/offbox.go:541`). **No template in the repository contains that path or any form posting to it** (grep over `internal/web/templates/`: one unrelated hit, an XSS comment) | **PLUMBING COMPLETE** (controller **v0.196.0**); **the FORM is not built — still open** | — | **The tenth instance of this project's built-but-never-wired class, and the exact shape `CLAUDE.md` and `felhom.eu/CLAUDE.md`'s seam-wiring rule were written for:** handler tests that POST directly (`offbox_escrow_test.go:167,180`) prove nothing about reachability. To use the only implemented recovery seam today, a person must hand-craft an authenticated POST with a session cookie and CSRF token. **Note the layering while fixing it:** this form takes a **64-hex repo password**, not a recovery code (`offboxRepoPwPattern`, `offbox.go:543`) — they are different secrets at different layers, and the operator's 2026-08-04 ruling asks for a form that takes **R**. Build the R form and treat this one as the operator/DR fallback it was written as, but **ship it with a render test per branch of whatever gate it sits behind**. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 9 **THE DIAGNOSTIC HALF IS DONE AND IT ANSWERED THE QUESTION.** `--recover-offsite-check` is a `docker exec` escape hatch in the shape of `--print-reset-code`: R on **STDIN** (never argv, never `ps`, never shell history, never a transcript), fetch+unseal via the agent, and a verdict of **two sha256 hashes**. **It compares and never installs** — the recovered password is not written to `offbox/repo_password`; a test asserts the data dir is byte-unchanged and its red-proof (adding the install call) fails it. Confirmed live: `repo_password` mtime still `2026-08-03 07:18:02` after the successful check at `2026-08-04 11:49`. **Exit codes are load-bearing** — `0` match, `2` a clean MISMATCH, `1` a step failed; "it failed" and "it worked and disagreed" must never share a status because only one is a finding about the system. A box with no local password reports distinctly (the rebuilt-box shape, where the next step is to INSTALL rather than compare). **WHAT IS NOT BUILT, deliberately:** no card, no form, no preview, no customer-facing text — building an interface on top of a chain nobody had walked is how the preceding three weeks went wrong. **What remains for this row:** link 9 (the recovered password placed so `WriteOffboxSecrets` keeps it) and the customer-facing shape the operator ruled on 2026-08-04 (yell → R form → preview → proceed), which is now priced against a chain that exists rather than one that is assumed **PART 0 SHIPPED 2026-08-04 (v0.196.0):** `--recover-offsite-install` is the sibling of the check — same fetch/unseal path, same STDIN discipline for R — and it **places** the recovered password via `InjectOffboxPassword`. The confirmation is a SECOND invocation (`--confirm-install`): without it, both hashes print and nothing is written, so the operator sees the comparison before a write is possible. Three outcomes named distinctly: **installed** (no local password — the rebuilt-box shape), **unchanged** (identical key, nothing written), **refused** (a DIFFERENT key present — installing would clobber the key the current repository is encrypted under; exit 2, no force offered). It re-reads the file after writing rather than trusting the call. **Red-proof observed:** removing the confirmation gate makes the dry run write the password. The R-persistence test carries a **positive control** (a planted copy found, then removed and not found). **NOT YET EXERCISED AGAINST A LIVE RECOVERY** — the R-201 drill halted before step 9, so this is unit-proven only. **What remains for this row:** the customer-facing shape the operator ruled on 2026-08-04 (yell → recovery-code form → preview → proceed) | CC | | **R-201** | **Nothing in the offsite DR chain has ever been exercised past the ceremony — and the one live proof that exists predates the field it is cited for.** `slice10d-identity-restore-spike-findings.md` §1 (2026-06-10) proved a real wrap→unwrap of an identity bundle with a real R on a secret-less box, byte-identical, wrong-R failing closed. **That bundle was `{tunnel_token, pbs_token}`.** `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** — a month later — and is **unit-proven only** | **BOTH HALVES PASSED — DATA 2026-08-04, JOURNEY 2026-08-07** — the customer file came back byte-identical (four times now), and on the fifth walk the customer's own journey completed with **zero guest command lines**. **It does NOT claim:** that the journey is smooth (R-252/R-253 are two unsignposted stops on it), nor that the fingerprint discriminator's POSITIVE half is proven (the mint guard fired, so there was no divergent key for shape (c) to catch — only its negative half was measured) | — | **Never exercised, in these words:** a blob has never been served to a box; a fork-4 bundle has never been unsealed with a real R outside a unit test (the only production caller is `--selftest=identity-consume`, `cmd/felhom-agent/main.go:2845`, reading R from `FELHOM_RECOVERY_CODE`); a recovered repo password has never been injected; an existing offsite repository has never been reopened with one; **no restore of any kind has ever been performed from a recovered secret.** The one live consume ever prepared (S5 Part 4-B, 2026-07-04) is still marked *"DEFERRED, not run"*. **`_recovery-inventory-2026-07-28.md` already recorded this accurately** (A.2.7, C.1 row 1, C-2) — this row exists so the gap has an owner and a closing condition, not only a description. **Closing condition = the drill** designed in `audits/RECON-offsite-dr-chain-2026-08-04.md` §10, whose pass condition is a **byte-identical sha256 of a sentinel file restored after a wipe** — explicitly NOT "the repository opened". **Do not run it before R-198 is fixed**: the drill would walk a chain that is missing a link everyone believed was there **MATERIALLY ADVANCED 2026-08-04, AND THE DISTINCTION MATTERS.** What was proven on hardware is that the offsite repository **password** comes back out of the sealed bundle byte-identical (R-199). What has STILL never happened: a recovered password **installed**, an existing repository **reopened** under one, and a **file restored** from it. **The drill's pass condition is unchanged and is not this** — it is a byte-identical sha256 of a sentinel file after a wipe and reinstall, and "the store opened" was explicitly ruled insufficient. Design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10. **The drill is now cheaper and better-founded than when it was designed:** links 6–8 are walked, so a failure during the drill can be localised instead of being an undifferentiated "recovery did not work", and R-198 means the ceremony that creates the drill's recovery code no longer destroys the key it is meant to protect **THE DRILL RAN AND DID NOT REACH ITS VERDICT, and stopping was the correct call** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). Steps 1–4 completed; the wipe never happened; **nothing irreversible was done**. It halted because **the sentinel file was not in the off-site snapshot** (R-203) — wiping would have destroyed the only copy and proven nothing. **Sentinel sha256 `643166269103a25c…`, still on the box.** **What the attempt established live, all of it new:** (1) a rebuilt box's off-site run **refuses** with the orphan card and pushes `offbox_repo_orphaned` — the spike's predicted third outcome, measured for the first time, and it does NOT silently start a fresh history (this closes R-193's Q3); (2) the **orphan reset works** — move-aside to `/home/felhom-repo.orphaned-20260804`, never delete, fresh repo initialised, `offbox_repo_reset` pushed (operator-authorised during the session); (3) demo-hp's pre-rebuild history is **permanently unrecoverable** — its key sits in superseded row id 3 with `identity_blob` NULL, superseded 07:15:36, **four hours before v0.93.0 fixed the retention**; (4) **a precondition the runbook did not contain:** neither pre-existing off-site-toggled app has a restorable file leg — both are named-volume-only, which the off-site tier tars but the customer restore never unpacks — so a file-leg app (`calibre-web`, mandatory `userdata: media/books`) had to be deployed, and it is now in place as the fixture. **TO RESUME:** fix or scope R-203, re-run steps 4–5 and confirm the sentinel IS in the snapshot **by listing it, not by a green status**, then P4 + the STOP + steps 6–11. Everything else is already staged: versions, recovery code, working repository, file-leg app, sentinel. **The pass condition is unchanged** — a byte-identical sentinel sha256, not "the store opened" **UNBLOCKED 2026-08-04 by R-203 (controller v0.197.0).** The sentinel now lands in the off-site snapshot and is listed there by name and size — the thing whose absence halted the drill. **Everything the resumed run needs is already in place on demo-hp:** agent v0.125.0, controller v0.197.0, the recovery code held by the operator (`R_DEMO-HP`), a working off-site repository (3 snapshots), `calibre-web` deployed with a mandatory userdata path, and the sentinel at sha256 `643166269103a25c…` — **verified byte-identical after the R-203 migration moved it to the corrected directory**. **What remains is exactly steps 4–11 of the drill**: re-verify the snapshot listing, take the deliberate rollback archive (P4), the §7 STOP, then wipe, reinstall, recover, install, restore and compare. **The pass condition is unchanged** — a byte-identical sentinel sha256, not "the store opened" **NIGHT RUN 2026-08-04 (`audits/DRILL-r201-night-run-2026-08-04.md`). THE HEADLINE: after a real rebuild — controller data volume destroyed, sentinel deleted from disk — the customer's recovery code produced `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`, BYTE-IDENTICAL to the pre-wipe on-disk key and to the hub's independent record. The off-site backup key is recoverable after a machine is rebuilt, and that had never been shown.** **Step 7's assertion PASSED:** `identity_blob` 572 B and `restic_pw_sha256` unchanged across the wipe (`updated_at` still 11:11:37) — nothing re-escrowed itself. **Step 9a passed:** the key installed cleanly on a bare box (the "installed" branch's first real run); **9b:** the apply kept it. **THE VERDICT WAS NOT REACHED** — step 10 never ran, so there is no post-restore sha256 and no snapshot count. **That is not a FAIL** (nothing came back wrong and no fresh history was started); it is a wall, and the wall is **R-204**. **The wipe was faithful to the incident, deliberately:** the 2026-08-03 rebuild R-193 is filed against was NOT a guest reprovision — the journal shows guest 9201 running continuously with no `pct destroy`/`pct restore`/`--selftest=provision` — so a controller-data-volume wipe reproduces it, and an unrehearsed provisioning chain improvised unattended is what §8.10 exists to prevent. **A precondition had drifted and was repaired, not worked around:** the staged snapshot had lost the sentinel because the afternoon's experiment produced a later same-day snapshot and `forget --keep-daily 7 --group-by host,tags` pruned the good one — **a good snapshot is not durable against a later bad run on the same day.** **TO FINISH (~5 min, operator present):** re-claim the box, confirm the escrow (**NOT a new ceremony** — it would supersede the identity blob and destroy the key under test), run a backup, restore the sentinel and compare to `643166269103a25c…`. Rollback available: the verified archive `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst` **THE DRILL PASSED (`audits/DRILL-r201-night-run-2026-08-04.md`).** demo-hp's controller data volume was destroyed and the sentinel deleted from disk; the customer's recovery code recovered the key (`8a9e33aa4da6…`, byte-identical to the pre-wipe on-disk key AND to the hub's independent record); it installed on the bare box; the **existing repository OPENED** (`repo_state: null`, **3 snapshots — NOT 1**, 42 026 B = the pre-wipe size exactly); and the customer restore flow returned **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` — byte-identical to the pre-wipe sentinel.** **The Felhom backup story is proved end to end for the first time.** **Step 7's assertion held:** `identity_blob` 572 B and `restic_pw_sha256` unchanged across the wipe — nothing re-escrowed itself, and no ceremony was run at any point (superseded rows still 2). **The wipe was faithful to the incident:** the 2026-08-03 rebuild R-193 is filed against was a controller-DATA-VOLUME loss, not a guest reprovision — the journal shows guest 9201 up throughout with no `pct destroy`/`pct restore`/`--selftest=provision`. **IT TOOK FOUR UNDOCUMENTED STEPS (→ R-204):** an operator Re-issue; a re-claim (whose escape hatch needs a controller restart to work at all); a manual escrow confirm; and `mode=full` on the restore, because the default `mode=unit` returns the app definition and NOT the customer's files. **A customer hitting this alone today would not get their data back.** Box left healthy, claimed, re-armed, sentinel restored to its live location. **⚠ WALKED TO COMPLETION 2026-08-06/07 — DATA PASS, JOURNEY FAIL, and the row stays open.** The full walk ran overnight on a new venue: built from the published ISO, fixtured, soaked through a real scheduled cycle, destroyed, rebuilt, and finished in the morning with the operator's emailed code. **DATA: PASS** — all three sentinels byte-identical out of snapshot `f5c53b03`, accented filename bytes included. **JOURNEY: FAIL** — the customer had **no route at all** to enter the code they hold: the recovery screen had retired itself, and the remote page offered to CREATE a new code instead. Root cause **R-241** (the credential self-heal writes a fresh repository key and moves the box out of the recovery-offer's pristine case, while orphan detection is unreachable behind `escrow_state: pending`). Recovery needed three guest command lines. **Also established:** the credential chain runs end to end unaided on an unclaimed box (first live sighting of its success line), and **R-239** — a fresh install lands on controller 0.203.0 while 0.205.0 is released, so R-234 and R-237 are undelivered. Full record: `tests/finalwalk-r201-2026-08-07/journal.md` **✅ CLOSED 2026-08-07 — BOTH HALVES PASS, on the fifth walk** (`documentation/tests/walk5-r201-2026-08-07/journal.md`). A fresh appliance was installed from the published ISO on `demo-hp` (VM 325), given three sentinels, escrowed, backed up off-site, then **destroyed on purpose** — guest and data volumes — and rebuilt through the documented day-0 path. **THE DATA: PASS** — all three sentinels byte-identical out of snapshot `5b0f20f7`, including the accented filename's *bytes*, read back with `os.listdir` on a bytes path so no decode round trip could launder a `U+FFFD`. **THE JOURNEY: PASS — zero guest command lines were needed to progress**, against three on the previous walk; the reset-code hatch was used once, in Phase A only, where §3 permits it. **RTO 71.7 s** from login to an open store (12.44 s of it the unseal itself; ~22 s a harness retry of mine). **The recovery screen appeared without being sought** (`/` → `/launcher` → `/recovery`) and answered all three questions, with a sealed-at timestamp matching `host_escrow.created_at` exactly. **What made the difference is R-241's mint guard, exercised live for the first time:** at 14:58:52Z the rebuilt box collected its re-staged credential, configured the transport and **refused to mint a repository password** over the sealed package — where the previous walk minted one and lost the journey silently. **Two obstacles remain and are filed rather than absorbed into this close — R-252 and R-253** — neither needing a shell, both cleared from the dashboard, and neither signposted; the journey succeeds and is not yet smooth. **This close does NOT claim** that shape (c) fired positively (it did not — with no local key the offer comes from shape (a); shape (c) was measured in its negative half in Phase A), nor that a customer would clear R-252/R-253 unaided. | **CLOSED 2026-08-07** | -| **R-202** | **The orphan card promises the customer their old backups "may later be restorable with the matching recovery code" — and after v0.93.0 that is true for supersessions from now on and FALSE for anything already orphaned.** `controller/internal/web/templates/backups_remote.html:66,69` states it unconditionally, in Hungarian, on the one surface where being wrong costs most | **OPEN — Part 5 hit its gate 2026-08-04; the card is UNTOUCHED and the sentence is still live** | R-199/R-201 (which generation an orphaned repo belongs to is not knowable to the box today) | — | **THE GATE, and why it was hit rather than squeezed past.** The condition was: ship it iff the hub can tell a box what it needs with **one** additional boolean on the escrow ACK it already sends. The hub *can* cheaply compute *"≥1 retained blob for this host carries an identity blob"* — one correlated predicate in `GetEscrowStatusForCustomer`, and the controller even has the right seam already (`SetEscrowStale`/`StaleBlob` is exactly this shape). **But that boolean does not answer the card's question.** The card renders on `RepoState == "orphaned"`, and the promise is about *the key THIS orphaned repository was written under*. A box does not know which escrow generation the orphaned remote belongs to; a box with a pre-v0.93.0 orphan and a post-v0.93.0 supersession would read the boolean TRUE and the promise would still be false — a **conditional** falsehood that looks verified, which is strictly worse on a customer-facing card than today's hedged one. **What would actually make it truthful** is knowing the orphaned repo's generation, which is the same knowledge R-199/R-201's unassembled chain needs. **Interim exposure, stated rather than buried:** the sentence remains live and remains false for both demo boxes. **Cheapest honest interim** (not taken here — it is a customer-copy change and the gate said leave it alone): drop the recoverability clause and say only that the old history is set aside and not deleted, which is true unconditionally | CC + operator | +| **R-202** | **NOW EVIDENCED, NOT ARGUED (2026-08-10).** demo-felhom’s orphan card told the customer that the set-aside backups may be restorable later *with their corresponding recovery code*. **For those 1.2 GB that is false and unfixable:** they were written under `48741892f0ef4d59…`, whose `identity_blob` is NULL, and the restic password exists nowhere else. The card promises a route that does not exist, to exactly the customers most likely to read it — the ones who have just lost their history. **The orphan card promises the customer their old backups "may later be restorable with the matching recovery code" — and after v0.93.0 that is true for supersessions from now on and FALSE for anything already orphaned.** `controller/internal/web/templates/backups_remote.html:66,69` states it unconditionally, in Hungarian, on the one surface where being wrong costs most | **OPEN — Part 5 hit its gate 2026-08-04; the card is UNTOUCHED and the sentence is still live** | R-199/R-201 (which generation an orphaned repo belongs to is not knowable to the box today) | — | **THE GATE, and why it was hit rather than squeezed past.** The condition was: ship it iff the hub can tell a box what it needs with **one** additional boolean on the escrow ACK it already sends. The hub *can* cheaply compute *"≥1 retained blob for this host carries an identity blob"* — one correlated predicate in `GetEscrowStatusForCustomer`, and the controller even has the right seam already (`SetEscrowStale`/`StaleBlob` is exactly this shape). **But that boolean does not answer the card's question.** The card renders on `RepoState == "orphaned"`, and the promise is about *the key THIS orphaned repository was written under*. A box does not know which escrow generation the orphaned remote belongs to; a box with a pre-v0.93.0 orphan and a post-v0.93.0 supersession would read the boolean TRUE and the promise would still be false — a **conditional** falsehood that looks verified, which is strictly worse on a customer-facing card than today's hedged one. **What would actually make it truthful** is knowing the orphaned repo's generation, which is the same knowledge R-199/R-201's unassembled chain needs. **Interim exposure, stated rather than buried:** the sentence remains live and remains false for both demo boxes. **Cheapest honest interim** (not taken here — it is a customer-copy change and the gate said leave it alone): drop the recoverability clause and say only that the old history is set aside and not deleted, which is true unconditionally | CC + operator | | **R-203** | **A customer-declared MANDATORY data directory was silently absent from the off-site snapshot while the run reported `ok`.** Measured live on demo-hp 2026-08-04: `calibre-web` declares `userdata: media/books class: mandatory`; its live bind is `/mnt/sys_drive/userdata/media/books` (the sentinel file was there), while the off-site capture set looked for `/mnt/sys_drive/felhom-data/userdata/media/books`, which does not exist. Result: `[WARN] mandatory data path missing on disk, skipped from offsite`, then `backed up calibre-web (…, 0 mandatory path(s))` and `backup OK: 3 app(s), 3 snapshot(s)` — `last_status: ok`, `last_success` stamped, nothing customer-visible, nothing hub-visible | **SHIPPED + PROVEN-LIVE 2026-08-04** (controller **v0.197.0**) | — | **THE MECHANISM, from source.** `NamespaceRoot(drivePath, inGuestDrive)` (`appbackup/paths.go:28-33`) appends the `felhom-data` segment **when the drive IS the system data path** (`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`, `backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is `/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an app deployed at `HDD_PATH=/mnt/sys_drive`, the two produce different directories. **The same compose used BOTH roots**, from one deploy: `${IMPORT_PATH}` → `/mnt/sys_drive/felhom-data/userdata/import/calibre` (with the segment), `${USERDATA_PATH}` → `/mnt/sys_drive/userdata/media/books` (without). **WHAT IS MEASURED vs NOT, because it changes the fix.** MEASURED: the paths disagree, the mandatory directory is absent from the snapshot, the run says `ok`, and the only signal is a container-log WARN. **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a SUPPORTED choice — it was used because demo-hp's only registered drive (`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working as designed), while `/mnt/sys_drive` was **accepted** (HTTP 202). **EITHER BRANCH IS A DEFECT:** if the system drive is a supported app namespace, userdata resolution is wrong for every app on it and their mandatory directories are silently unprotected; if it is not supported, the deploy accepted a namespace it should have refused one call after refusing the NAS. **NOT a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths (everything of theirs is in named volumes), so they are unaffected and their snapshots are real. **Fix shape:** make the two roots one function, whichever is right — and make a skipped MANDATORY path a customer/hub-visible signal rather than a WARN, because `ok` with a missing mandatory directory is this project's own *a path the customer thinks is protected is not in the snapshot* shape. Source: `audits/DRILL-r201-offsite-recovery-2026-08-04.md` §2 **BOTH HALVES SHIPPED.** **(1) The paths.** `appbackup`'s helpers take a NAMESPACE ROOT; the census found **FIVE** bare-drive-path callers, not the four the spec named — the fifth is the **FileBrowser mount builder** (`web/handlers.go`), i.e. the customer's own file browser would have shown the wrong directory on a non-enrolled path (latent: the system drive is deliberately never a registered `StoragePath`). The rule now has **ONE expression** (`appbackup.NamespaceRootFor` / `IsEnrolledDrive`); there were already **two** copies and **they differed** — `backup.Manager.namespaceRoot` compared without `filepath.Clean`, `stacks.Manager.inGuest` with it, so a trailing slash from config would have flipped the mode in one package and not the other. `ComputeFabBuckets` now receives the namespace root, which is what `ComputeCaptureSet` has always received, so the export and the backup describe the same directories by construction. **(2) The verdict.** `last_status` gains **`incomplete`** — minted, because `ok`|`error`|`running` had nothing meaning *"it ran, and this app is not fully protected"*. **Not `error`:** the rest of the run worked, so `SnapshotCount` and the `LastSuccess` anchor still record what WAS captured. It reaches the operator through the **existing** per-run digest (`backup_run_failures`) — a new event type would be a two-repo change and the hub drops anything outside `allowedEventTypes`. **§8.4's narrowing is a NO-OP and no customer warning disappears:** `TierOffsite`'s `tierKeeps()` already admits mandatory only, demonstrated by widening the tier filter alone and watching the class check hold the line. **THE SPEC'S §8.3 RISK DOES NOT EXIST, and this is the correction owed:** `ExportDataMounts` lives in `delete.go` but is **export-only** — its single production caller is the `.fab` adapter, nothing deletes on its result, and the delete path's own guard `ProtectedHDDPaths` is layout-agnostic by construction (it protects BOTH `/…` and `/felhom-data/…`). It shipped as its own commit anyway. **PROVEN LIVE on demo-hp:** the bind moved `/mnt/sys_drive/userdata/media/books` → `/mnt/sys_drive/felhom-data/userdata/media/books`, the capture log went `0 mandatory path(s)` → **`1 mandatory path(s)`**, and **the sentinel is in the snapshot's own file listing** — `-rw-r--r-- 1000 1000 181 … /mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt` — not a green status. **Red-proofs:** the bare-path call makes the two paths differ; inverting the drive-kind comparison breaks every enrolled row; leaving the export site bare emits the short path; and the verdict fails under both an unreachable gap-recording and an unconditional `ok`. **One red-proof PASSED and the test was wrong, not the code** — the first Scenario-C test only reached `offboxCaptureSet` while the mutation lives in `runOffboxInternal`; a run-level test replaced it | CC | | **R-204** | **A rebuilt box can recover its off-site key and still cannot use it: the remedy that reconfigures the tier is the thing that blocks the recovery.** Measured end to end on demo-hp during the 2026-08-04 night drill, after a real controller-data wipe | **ALL FOUR ITEMS CLOSED 2026-08-05** (items 1–3 controller v0.198.0 + hub v0.95.0; item 4 controller v0.199.0 + hub v0.96.0) | — | **THE CHAIN, each link measured.** (1) A rebuilt controller **cannot configure its off-site target at all**: `offsite-apply: consume one-time password: no unconsumed offsite password` — the previous controller consumed it (ledger: created `07:11:51`, consumed `07:12:06`). That is R-193, reconfirmed live. (2) The documented remedy is an operator **Re-issue**, which works — and **sets `stale_at` on the escrow** (measured: `2026-08-04 20:15:49`) **while `restic_pw_sha256` is unchanged**, i.e. R-196's false staleness. (3) A stale escrow makes the hub **withhold the hash from the report ACK**, so `EscrowAutoConfirmer` can never flip `pending → escrowed`, and `OffboxRunnable` (`configured && escrowed`) **refuses every off-site run**. (4) The only documented way to clear a stale escrow is **a fresh ceremony — which supersedes the identity blob and destroys the key being recovered.** **So the recovery and its precondition are mutually exclusive as built.** The key comes back (proven — `8a9e33aa4da6…` recovered byte-identical after the wipe, and installed) and then cannot be used to open the repository. **(5) A FOURTH link, undocumented anywhere:** a rebuilt box is **unclaimed**, and the claim gate correctly intercepts every non-claim route (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`), so **no controller endpoint responds at all** until the customer re-claims. That step appears in no design document and comes first. **Fix shape, not decided here:** either the Re-issue stops marking the escrow stale on evidence it does not have (R-196's own fix), or a recovered-and-verified key is allowed to confirm the escrow without a ceremony — the hash comparison that `EscrowAutoConfirmer` already performs is exactly the evidence needed, and it is being withheld precisely when it would be conclusive. **Do NOT fix by widening `OffboxRunnable`** — the atomicity guarantee it enforces (no un-recoverable ciphertext) is the reason the escrow exists. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3 **ALL FOUR WALKED AND MEASURED during the passing 2026-08-04 drill, so this row is now a known-good manual runbook AND the gap list.** (1) **R-193:** a rebuilt controller cannot configure its off-site tier — `no unconsumed offsite password` (ledger: created `07:11:51`, consumed `07:12:06` by its predecessor). Remedy: operator Re-issue. (2) **The claim gate:** a rebuilt box is unclaimed, so the gate intercepts EVERY controller endpoint (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`) — a first step of every recovery that appears in no design document. **And the local escape hatch does not work unaided:** `--print-reset-code` writes the new hash to `settings.json` while the RUNNING controller keeps its old copy in memory, so `effectiveClaimCode()` never sees it and the claim fails with *"Hibás vagy lejárt kód"*. **The controller must be restarted between minting and claiming** — two attempts failed before this was diagnosed. (3) **R-196:** the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged → the hub withholds the hash → auto-confirm can never fire → `OffboxRunnable` refuses every run. Cleared here with the **manual** confirm (`/backup/offbox/confirm-escrow`) — NOT a ceremony, which would have superseded the identity blob and destroyed the recovered key. (4) **`mode=unit` is the restore default and returns the recovery unit, NOT the userdata leg.** A customer told to "restore from off-site" gets their app definition back and not their documents, and nothing in that outcome says so. **Fix priorities, in the order they hurt:** (4) is a customer-facing trap on the last step; (3) is R-196's fix; (2) needs the escape hatch to reload settings (or the claim state to survive a rebuild); (1) is R-193's restage. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3–§4 | CC + operator **OUTCOME 2026-08-05 — controller v0.198.0 + hub v0.95.0.** **Item 1 (the reset code needs a restart) — CLOSED, proven live.** `effectiveClaimCode` reads through to the persisted claim state, so a code minted by the separate `--print-reset-code` process is seen without a restart; the precedence rule between settings and config is unchanged. Read-through, not a TTL: a TTL leaves a window in which a superseded code still works, and that is the mutation `TestClaimCode_SupersededByASecondMint_RefusedImmediately` kills. Fails closed on an unreadable state. **Live on demo-felhom 9201, nothing restarted (`restarts=0`, container older than both mints): the superseded code returned „Hibás vagy lejárt kód" and the current one was accepted first time.** **Item 2 (a re-issue marks a healthy escrow stale) — CLOSED, → R-196.** Test-proven; deliberately NOT fired live on demo-hp. **Item 3 (the restore's default silently returns the wrong thing) — CLOSED, proven live.** A `mode=unit` restore now names what came back, what did not and the step that gets it; the wizard's intent card states its scope BEFORE the choice; the full-restore size gate is untouched and pinned as unchanged. **The default stays `unit`** — all three wizard forms set `mode` explicitly, so changing it would alter nothing the customer sees while silently changing a mode-less POST. **ITEM 4 REMAINS AND IS THE WHOLE OF WHAT IS LEFT HERE: a rebuilt box cannot obtain an off-site credential unaided**, because the one-time password was spent by its predecessor, so an operator Re-issue is still required. **Its dependency is the one-shot credential design decision — it needs an operator ruling and belongs to → R-193.** Not begun in this session, deliberately. **ITEM 4 CLOSED 2026-08-05 — controller v0.199.0 + hub v0.96.0.** The box now DECLARES that it needs a credential (`offsite.state=needs_credential`) instead of reporting an absence the hub cannot interpret; the hub's new `internal/offsiteheal` answers it. **Operator ruling, recorded because a ruling that lives only in a conversation binds nobody (R-96): the trigger is a state the BOX DECLARES, not an inference.** An absent off-site object has FOUR meanings — never configured, mid-restart, a transient config read failure, rebuilt-and-stranded — and the hub cannot tell them apart; the box can, from two local facts (a fresh data area AND a hub-held recovery package). **Both halves are required:** freshness alone would make every un-configured box in the fleet ask for a credential, which is what `TestOffsiteDeclare_NeverHadOffsiteSaysNothing` exists to catch. The reconciler mirrors `pbsdrheal`: declared states only, a two-DISTINCT-REPORT debounce (derived from the ~15-min report cadence), **restage before mint**, an event per remediation, and a healthy box is a pure no-op. **PROVEN LIVE:** demo-felhom 9201 arranged (reversibly) into the stranded shape produced report id=16743 carrying `{enabled:false, state:needs_credential, quota_gb:0, repo_size_bytes:0}`, and the single declaration was **absorbed by the debounce** — no self-heal event fired — with the box restored the same minute. **What is deliberately NOT automated: the escrow ceremony.** A credential is replaceable; the recovery code is not. **Credential automatic, key customer-present.** **Second ruling recorded: the dashboard-password exposure on the recovery preview is METADATA (backup dates, app names), not content, and is ACCEPTED.**| | **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **CLOSED 2026-08-05 — the guard's scoping half closed BY REPLACEMENT** (hub v0.96.0) | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. **SPIKE 2026-08-04 — BOTH HALVES CONFIRMED WITH NUMBERS, still OPEN, still not fixed here** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q7). The query is quoted at source (`store.go:987`): `... ORDER BY id LIMIT 500` = **the oldest 500**. Reproduced against the live hub DB with demo-hp's real consume anchor `2026-07-23 09:53:41`: the guard sees `total=500, withOffsite=500`, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** — the entire evidence set ends **six days before** the 2026-08-03 rebuild. The true window totals are `1174 / 1063` (⇒ 111 without, matching the 111 offsite-less reports). So the e-mail's *"500 report(s) since carry no offbox target"* interpolates `ReportsSinceConsume` while `OffsiteReportsSinceConsume` was **500** — the message states the precise negation of its own measurement. `offsite_credential_restaged` has **never fired for any customer** (zero rows of that type in the DB — checked, not assumed). **A NEW REASON NOT TO FIX THIS IN ISOLATION, from the same spike:** under R-193's Q2 finding a successful auto-restage would have restored demo-hp's TRANSPORT while the box minted a new repo password anyway — the heal can protect the plumbing and **cannot** protect the data, and had it fired on 2026-08-04 both boxes would have looked healthy while their snapshots were orphaned. **That is strictly worse than the current loud failure.** Whatever shape the fix takes must say so in its message. **RECON 2026-08-04 adds two inputs and changes no verdict** (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **(1) The guard's own conclusion — that a fix here protects the plumbing and not the data — is now stronger, not weaker:** even a perfect restage leaves the rebuilt box minting a fresh repo password, and per **R-198** the old one is destroyed by the re-ceremony the box is pushed into. Whatever shape the message takes must say that in the same breath, or it will read as an all-clear. **(2) The recency-bounded discriminator this row asks for has a ready anchor the hub already receives:** the report ACK's `escrow{identity_blob_present, restic_pw_sha256}` moves when a box re-keys, so "evidence since the newest re-key" is computable from data already stored — the same observation R-197 makes, aimed at this guard's time window. **BOTH HONESTY HALVES SHIPPED 2026-08-04 (hub v0.93.0); THE GUARD'S LOGIC IS DELIBERATELY UNTOUCHED.** **(a) The message now describes what was measured.** The one stuck state is reported as the two situations it actually covers — **burned** (`OffsiteReportsSinceConsume == 0`) and **regressed** (> 0, the demo-hp shape) — each stating its own measurement and carrying its own recommendation; the regressed text explicitly WITHDRAWS Re-issue and points at what removes an offbox target (a guest rebuild, R-193). `offsite_reports_since_consume` rides the details for the first time. **(b) Every refusal to self-heal leaves a record** — a `notification_log` row on the operator channel, status `refused`, with its reason (the R-182 suppressed-e-mail precedent), riding the stuck event's 24 h cadence so it sits beside the e-mail it explains rather than accumulating per tick. The two conditions were split into separate branches solely so each can name its own reason; **the set of situations in which the heal fires is byte-for-byte what it was**. **(c) Not in the spec and done anyway, narrowing only:** `offsite_delivery_stuck` and `offsite_credential_restaged` are added to `operatorOnlyEvents`. Neither was ever registered and neither has a `customerMessages` entry — which is not a block — so a customer with a configured recipient was in line for an English e-mail about one-time passwords being *"likely burned"*. Measured live: `notification_log` holds operator rows for demo-hp and no customer rows, which is NOT evidence the leg was blocked (equally consistent with no configured recipient), so the register makes it structural. **WHAT STAYS OPEN, and it is this row now:** `CountReportsOffsiteSince` reads `ORDER BY id LIMIT 500` — the OLDEST 500 reports after the consume — so the counts describe the start of the window, not the present. Its correct shape (recency-bounded, rebuild-aware) depends on the recovery chain that is not yet assembled (R-199/R-200/R-201), so it was NOT fixed here. **The window is named inside the alert text** so the limitation travels with the number instead of being laundered into a confident sentence. **Red-proofs observed:** restoring the single hardcoded sentence fails both message scenarios (the first mutation attempt left the default branch in place and only the burned scenario failed — recorded because a mutation that does not remove every guard is not a red-proof); replacing the regressed branch with a bare `return` fails the refusal record and its cadence test. | CC **CLOSED 2026-08-05 (hub v0.96.0), by REPLACEMENT rather than repair.** The guard inferred the shape by counting how many of the **OLDEST 500** reports after a consume carried an offbox target — on demo-hp all 500 predated the rebuild, so the checker reported the *regressed* shape with confidence and declined to heal for 108 reports while the box sat stranded. A DECLARED state needs no window, no count and no inference, so `shapeDeclared` now **outranks both inferred shapes**, the stuck e-mail states what the box declared instead of prescribing Re-issue, and `maybeHeal` **stands down with a record** because `internal/offsiteheal` owns the remediation — two mechanisms healing one customer would double-issue, and this one can only mint, so it would also skip the cheap restage path. The counting fields remain for boxes on a pre-v0.199.0 controller, where there is still nothing better to go on.| @@ -506,7 +506,7 @@ applied.** The one that matters: Scenario A **fails against today's tree** with | **R-275** | **`--uninstall` leaves five 0600 `agent.json.*` credential backups, and the reinstall hands them to the new service account.** `/etc/felhom-agent/` survives with `agent.json.{campaign8-before,campaign9-before,campaign9-prev,pre-e-target-move,pre-prunegate.bak}`, each carrying a 64-char `hub.api_key` and a 59-char `proxmox.token`. The teardown claims to remove *"config (+ its .bak backups)"* and `scripts/CHANGELOG` F1 records *"uninstall now purges the agent config's `.bak*` siblings (one held a live hub api_key)"* — **that fix does not match the filenames in use, and it misses `agent.json.pre-prunegate.bak`, a file that literally ends in `.bak`.** **Exposure assessed, not assumed:** these are SUPERSEDED — the orphaned key hashes to `a5d2222a…`, the hub's current demo-hp key to `8c59d1b6…`, and the Proxmox token was deleted by the same uninstall. **But the reinstall recreates `felhom-agent` at uid 999, the same uid the deleted account had**, so three of the backups become the new account's files — verified readable as `felhom-agent`. A fresh install's service account inherits read access to the prior install's credentials; superseded today, live if the backups were recent (R-179's precedent). **Also left, undeclared:** `/etc/felhom/{.bootstrap-done,appliance-pairing-code}`, `felhom-bootstrap.service` + `/usr/local/sbin/felhom-bootstrap.sh`, the `vmbr9` stanza in `/etc/network/interfaces`, and `/etc/sudoers.d/felhom-agent.bak-pre-e2a` (21 KB — **INERT: sudo skips dotted filenames, verified with `sudo -l -U felhom-agent`; `visudo -c -f` parsing it OK is NOT evidence sudo loads it**) | **READY (S) — NEW 2026-08-09** | — | Purge by directory, not by glob; and do not let a new service account reuse a uid that owns old secrets | CC | | **R-276** | **RANK 2 — an uninstalled box keeps a live WireGuard tunnel into Felhom's off-site endpoint, and the teardown says nothing.** After `--uninstall` on demo-hp, `wg-quick@wg-felhom` is **enabled and active**, `/etc/wireguard/wg-felhom.conf` present, handshake to `167.233.158.164:443` **52 s old**, counters 5.86 GiB in / 2.48 GiB sent. It appears in **neither** the WIPED nor the KEPT list, though the BYO install disclosure names it prominently on the way in (*"an OUTBOUND WireGuard tunnel to the Felhom hub"*). A host told to leave Felhom retains a live network path into Felhom infrastructure, its hub-side peer registration intact, and nobody is told | **READY (S) — NEW 2026-08-09** | — | Tear the tunnel down and deregister the peer, or list it under KEPT with the reason and the removal command | CC | | **R-277** | **Three hub surfaces jointly present a HEALTHY off-site tier as an absent one — and it produced a wrong operator statement during this run.** For demo-hp on 2026-08-09 the box was pushing off-site daily without a gap (18 restic snapshots, `last_status: ok`), yet: (a) the customer page's Backup panel read `Snapshots 0 / Repo Size 0 MB / Integrity Unknown` — it renders the **local disk tier**, while the healthy `offsite` object sits **in the same report** unrendered on that panel; (b) the Offsite page read `0.0 GB` — true, but a 162 KB repo rounds to nothing; (c) a stale `offsite_delivery_stuck` event from **2026-08-07 10:19** (not recurring) reads as current state. **Three independent surfaces agreeing on a wrong picture is how a working backup gets "fixed".** It did exactly that here: the rehearsal reported a fleet-wide off-site outage to the operator and had to retract it. **Note the true half:** demo-felhom IS genuinely stuck (`offsite.state=needs_credential`, no run has ever succeeded) → **R-278** | **READY (S) — NEW 2026-08-09** | — | Render the offsite object on the offsite row; show bytes not rounded GB; distinguish a live alarm from event history | CC | -| **R-278** | **demo-felhom's off-site tier has never completed a run and has been stuck for six days.** `offsite.state=needs_credential` since the 2026-08-03 guest rebuild; the hub's own alarm reads *"enabled + escrowed but no run has EVER succeeded"*; the controller's `offsite-credential-retry` job runs every 5 minutes and completes in 0 s, doing nothing. R-193's fix (the recovery SCREEN, controller 0.200.0) is present on the box, so the remedy exists — it just needs the customer-present ceremony that nobody has run, which is R-243's shape (*"a machine waiting for its recovery code can stop backing up off-site without alarming us"*) landing on a real box. **Contrast that makes it a defect and not a chore:** demo-hp, same rebuild, same day, recovered and has 18 snapshots | **READY (S) — NEW 2026-08-09** | — | Either the self-heal reconciler owns this shape end-to-end, or the box must say plainly on the dashboard that it is unprotected pending the recovery code | CC | +| **R-278** | **demo-felhom's off-site tier has never completed a run and has been stuck for six days.** `offsite.state=needs_credential` since the 2026-08-03 guest rebuild; the hub's own alarm reads *"enabled + escrowed but no run has EVER succeeded"*; the controller's `offsite-credential-retry` job runs every 5 minutes and completes in 0 s, doing nothing. R-193's fix (the recovery SCREEN, controller 0.200.0) is present on the box, so the remedy exists — it just needs the customer-present ceremony that nobody has run, which is R-243's shape (*"a machine waiting for its recovery code can stop backing up off-site without alarming us"*) landing on a real box. **Contrast that makes it a defect and not a chore:** demo-hp, same rebuild, same day, recovered and has 18 snapshots | **CLOSED 2026-08-10 — protection RESTORED, and the recovery it waited for could never have worked** | — | Either the self-heal reconciler owns this shape end-to-end, or the box must say plainly on the dashboard that it is unprotected pending the recovery code **THE REMEDY THIS ROW ASSUMED WAS IMPOSSIBLE, and that is the finding.** The operator authorised the recovery ceremony on 2026-08-10; it was **not run**, because three measurements taken first showed it could not help. The box’s local key hashes to `c60c8bc737a6b7c6…`; the hub’s sealed escrow key hashes to **the same value**; and that key answers `Fatal: wrong password or no key found` against the repository. **Recovery would have returned a key the box already held and which was already proven not to open the store.** The repository was written under `48741892f0ef4d59…` (`host_escrow_superseded` id=4, superseded 2026-08-04 07:20:08) whose **`identity_blob` is NULL** — and the restic password lives ONLY in the identity bundle (`escrow/identity.go:39`, read by `escrow/recover.go:91` through `UnwrapIdentityBundle`), so it is unrecoverable by construction. Corroborated: the surviving K-escrow payload is **64 bytes**, a wrapped key, far too small to carry a bundle with a password. **The same shape the register already records for demo-hp** (*"its key sits in superseded row id 3 with identity_blob NULL … four hours before v0.93.0 fixed the retention"*). **What was done instead, on the operator’s stated fallback:** the orphan reset, through the customer’s own card — the old store **moved aside, never deleted**, to `/home/felhom-repo.orphaned-20260810` (1.2 GB); a fresh repository initialised under the current key; `offbox_repo_reset` audited hub-side at 08:06:31. **PROVEN rather than assumed:** `last_status: ok`, `last_success: 2026-08-10T08:07:33Z`, 10 s — and the snapshot’s CONTENTS listed, not just its count: `opengist/compose/{.felhom.yml,app.yaml,docker-compose.yml}`, `manifest.json`, and `volume-dumps/opengist_opengist_data.tar`. A week without off-site protection ends here | CC | | **R-279** | **There is no operator-triggerable off-site backup.** The only route to `POST /backup/offbox/run` is the customer's own dashboard session; `signed_jobs` carries opaque operator-SIGNED blobs and the hub holds no signing key. This cost the rehearsal a stop: preparing the run needed one off-site push and there was no operator path to it. Sibling of **R-177** (no operator-triggerable fill check) | **READY (XS) — NEW 2026-08-09** | — | Same shape as R-177; solve both together | CC | | **R-280** | **RANK 1 — after a reinstall the data drive cannot be re-attached through ANY dashboard route, and the restore page promises it is "two clicks".** Measured on the rebuilt demo-hp, 2026-08-09. The restore page diagnoses the situation perfectly and then sends the customer to an empty page: *„Előbb csatold vissza az adatmeghajtót. A mentéseid megvannak, és a meghajtók is megvannak — újratelepítés után viszont a gép még nem ismeri őket, ezért most nincs hová visszaállítani. **Ez két kattintás:** Tárhely → Meghajtók, »Meglévő meghajtó csatolása«."* **It is not two clicks; it is zero possible clicks.** `GET /api/disks/candidates` → `{"initialize":[],"attach":[]}`, so both wizards render an empty selector, and `Tárhely → Meghajtók` reads „Nincs regisztrált adattároló" with an empty unregistered list. **The agent is not at fault** — `GET /api/disks` returns the NVMe in full (1.0 TB, SMART PASSED, `mount_path:/mnt/nvme-1tb`, `guest_attached:false`), so the channel and enumeration work. **ROOT CAUSE:** `handleDiskCandidates` builds both lists from `ListCandidateDisks`, the UNCLAIMED-disk scan; demo-hp's NVMe is deliberately BOTH the user-data drive and the `felhom-backup` target (`operations/nodes.md`), so it is claimed and never offered. That filter is **correct for `initialize`** (never offer to format a disk in use — `/storage/init` even says so: *„Rendszer- és biztonsági-mentés meghajtók itt nem jelennek meg — azok védettek"*) and **over-broad for `attach`**, which is non-destructive by definition and whose own page says *„A meghajtón lévő adatok nem törlődnek — a csatolás csak elérhetővé teszi azokat."* **It cascades:** no store → Calibre-Web's install page degrades to *„Nincs regisztrált adattároló — adja meg kézzel az útvonalat"* and demands a hand-typed `E-könyvtár útvonal`; no app → the restore rows read „Nincs telepítve". **THE ESCAPE HATCH WORKS AND NO CUSTOMER COULD FIND IT:** `POST /settings/storage/add` with `storage_path=/mnt/sys_drive` succeeded first try (*„Adattároló sikeresen hozzáadva"*) — and `/mnt/sys_drive` is an internal path, the very one registered before the wipe. Once registered, everything unblocked and the deploy form became a proper picker (*„Tárhely (sys_drive) — 64.2 GB szabad"*). **This is R-220's successor:** R-220 was closed as "drives unenrollable after a rebuild — fixed"; enumeration is fixed, OFFERING is not | **READY (M) — NEW 2026-08-09** | — | Populate `attach` from mounted-but-unregistered filesystems rather than from the unclaimed-DISK scan; and never print "two clicks" without asserting the destination is non-empty | CC | | **R-281** | ~~**The hub said NOTHING through an entire reinstall — and the tripwire for a sealed-backup unseal did not fire on a real unseal.**~~ **WITHDRAWN 2026-08-09 — THE FINDING WAS AN ARTEFACT OF MY OWN MEASUREMENT, AND IT WAS WRONG IN BOTH DIRECTIONS.** The operator's mailbox settled it: the hub fired **twenty events** on 2026-08-09, and `escrow_blob_served` **DID** fire — 10:19:41 UTC / **12:19 CEST**, eight minutes before the verified restore. **Cause of the false reading, ESTABLISHED (not guessed):** the P7 query copied `/data/hub.db` **without `hub.db-wal`**. The hub runs SQLite in WAL mode (R-172), so every write since the last checkpoint was invisible. **The signature is an exact match:** P7 reported *"2 events all day, newest `db_dump_completed` 00:30:07"*, and the number of rows on 08-09 at or before 00:30:07 is **exactly 2**. **The two obvious alternatives were TESTED AND REFUTED**, not waved away: a **timezone offset** — all nine mailbox stamps equal the hub's UTC + 2 h exactly (`escrow_blob_served` 10:19→12:19, `host_down` 09:28→11:28, and seven more), so the window was right; and a **wrong customer key or wrong store** — the same table and key return the correct rows now. A live re-run cannot reproduce the fault because the WAL has since been checkpointed; the case rests on the command text plus the 2-of-2 count signature, and that is stated rather than dressed up as a reproduction. **This is a trap this project has already documented** — `operations/nodes.md` says copying `hub.db` alone is *"valid but stale … the worst failure shape"* — and I had avoided it correctly earlier in the same session before hitting it. **Split out: → R-285** (the real, opposite defect) and **→ R-286** (the measurement lesson) | **WITHDRAWN 2026-08-09** | — | Superseded by R-285/R-286 | CC |