RECON: trace the offsite DR chain link by link — it does not join up (R-198..R-201)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
Read-only recon of the escrow -> recovery chain, from a dead node to an open
repository. No production code, no build, no version bump.
Headline: the hub's superseded-escrow retention does NOT retain the offsite
repository password. host_escrow_superseded has no identity_blob column and
demoteCurrentEscrowTx copies only the K-escrow blob, so what survives a
supersession is the PBS datastore key, not the restic repo password. The next
escrow ceremony -- which the system tells a rebuilt box's customer to run --
destroys the last copy. Both demo boxes crossed that line on 2026-08-04.
Also established:
- the hub's blob-serving endpoints (re-enroll / restore-directive) have zero
callers anywhere: agent, hub UI, scripts, runbooks (R-199)
- POST /backup/offbox/inject-password is routed and handled but no template
contains the form (R-200)
- nothing in the recovery path has ever been exercised; the one live
round-trip proof (2026-06-10) predates the ResticRepoPassword field (R-201)
- a fail-closed mint refusal IS implementable: the report ACK already carries
escrow{identity_blob_present, restic_pw_sha256} and the controller discards
it whenever no offbox target exists
Corrections: yesterday's spike annotated (candidate (b) overturned in part --
unattended recovery is impossible, customer-present is not); capability-map
retention claim struck through and replaced with what the code does.
Deliverable: documentation/audits/RECON-offsite-dr-chain-2026-08-04.md
Register: new R-198..R-201; R-193 and R-192 updated; STATUS.md refreshed.
This commit is contained in:
1 parent
d26f49ad68
commit
3f2b7bc023
6 files changed
+726
-310
No files matched your search
@@ -36,7 +36,7 @@
|
||||
| Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`<iso>.rootpw.txt`) | scripts v1.24.0 (`scripts/iso/felhom-bootstrap.sh` `network_gate`/`sweep_nics`, `build-felhom-iso.sh` rootpw emission) | **PROVEN-LIVE (nested drill — nested ≠ metal: metal proof rides the next real multi-NIC install)** | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` — dead-NIC install from the virgin v1.24.0 ISO baked the 192.168.100.2 fallback (WITH a dead default gateway), the R-59 screen painted on the console (screendump captured), and after the cable move the box swept to the working NIC, re-leased and **self-registered at the hub unaided in under a minute**; the drill also caught + fixed the stale-fallback-route trap (flush before the bounded dhclient) and verified the emitted rootpw against the installed box's shadow hash | R-59 ships as a first-boot gate, not an install-time abort (recorded deviation — the fallback is the auto-installer's own, initrd hook out of scope); sweep is structurally first-boot-only (`state.json` gate + unit done-flag condition); a box past install-start gets the screen but its interfaces are never touched |
|
||||
| Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. **Deliverability (R-4), gmail half DONE 2026-07-18:** the rehearsal's claim email was the first sent under the tightened DMARC `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu remains Viktor's open half.** (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) |
|
||||
| Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends |
|
||||
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test |
|
||||
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention) — ⚠ CORRECTED 2026-08-04, the claim as written is FALSE for the offsite tier:** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`). ~~so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable")~~ **It retains the K-ESCROW BLOB ONLY. `host_escrow_superseded` has no `identity_blob` column and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`) does not copy one, so what stays recoverable with the old recovery code is the PBS datastore key — NOT the restic repository password, which lives in `identity_blob` and is OVERWRITTEN by the next ceremony.** Measured live 2026-08-04: both current rows hold `blob`=383 B **and** `identity_blob`=572 B; both superseded rows hold `blob`=383 B only. The reinstall-orphan incident this sentence names is therefore still "history destroyed" for Tier-3 — and the same false promise is made to the customer in Hungarian on the orphan card. → **R-198**, evidence `audits/RECON-offsite-dr-chain-2026-08-04.md` §7. **There is also no read path:** `ListSupersededEscrow` has zero production callers; retrieval is `sqlite3` on the hub pod, by hand. Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test |
|
||||
| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). |
|
||||
| **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** |
|
||||
| **Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard** | agent v0.104.0 → **v0.121.0**, hub v0.77.0 → **v0.91.0** | **PROVEN-LIVE (2026-08-03)** | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why this was NOT PROVEN-LIVE until now:** rotation had not been observed selecting both tiers across consecutive UNATTENDED cadences — at a 24h cadence a multi-day window — and a single passing run proves the code path, not the schedule. → **R-85**. **R-86 (agent v0.121.0 + hub v0.91.0, 2026-08-03) replaced the schedule and the observation became possible in one afternoon**, because what has to be observed is no longer a multi-day rotation but a RULE: a tier is due when its newest archive that has settled ~24 h has not been proven. **UPGRADE EVIDENCE — a real unattended run on demo-felhom (2026-08-03), triggered by DUE-NESS, not by a timer.** The scheduler's own log: `15:14:38 restore-test tier is DUE … target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z … reason="newest settled archive … has not been proven"` → `proxmox-backup-client restore --crypt-mode=encrypt` under the agent's own token → `15:25:08 gate decision class=guest_destroy guest=990000 allowed=true` → `15:25:14 scratch guest torn down` → **`15:25:14 backup: scheduled restore-test passed archive=felhom-pbs:… duration_s=635.1`**. A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored, booted, verified and destroyed in 635 s, unattended. **The three things a timer could not show**, all verified after it: the state names THAT archive (`{"felhom-pbs":{"archive":"…2026-07-28T04:49:43Z","proven_at":"2026-08-03T13:25:14Z"}}`); a second evaluation reports `due=false … is already proven` and runs nothing; and **an agent restart runs nothing**, which is the defect a person actually noticed — every deploy used to restart the timer. Teardown verified at all three layers: guest absent from `pct list`, **zero** `990000` volumes in `lvs`, and the hub-side `restore_tests[]` entry deliberately RETAINED (it IS the proof the staleness check reads). **R-189 (agent v0.122.0, 2026-08-03) closed the reporting half of this row, and it was a REAL gap in the evidence path:** the proof above reached the hub only because no restart intervened — `restore_tests[]` came solely from an in-memory store, so the 15:25:14 PASS was in fact LOST when the agent restarted 2 m 43 s later for a deploy (`0 restore-tests` on the next two host-reports). Under per-archive due-ness the box would not have repeated the work for a week. The persisted per-tier proof (with the archive, and now the tier) is merged into the report, so a proof survives a restart — one entry per tier, newest wins, and a record that cannot be described honestly is not emitted. **THE HOST TIER'S HALF OF THIS ROW WAS OPTIMISTIC UNTIL 2026-08-03, and it is worth saying plainly.** Every live restore-test cited here is on the OFFSITE tier. The HOST tier was not merely unproven on demo-felhom — it was **unprovable**, and on demo-hp equally: the agent's token had no ACL on `/storage/felhom-backup`, the storage both boxes configure as `local_backup_target`, so the content API answered `{"data":[]}` through the token while root listed three archives. The scheduler skipped it as *"no settled archive yet"* — which is exactly what a brand-new tier reports — so nothing ever said so (**R-185**). **CLOSED 2026-08-03 — agent v0.123.0 + installer 1.24.0.** The grant is applied on both demo boxes (the token now lists 3 and 4 archives), the installer's reuse arm grants on a pre-existing target, and the agent now **asks whether it may read each tier it depends on** instead of inferring it from an empty list: a critical degraded capability naming the storage and the missing role, which the hub alerted and emailed on **while the box was still blind**. **THE HOST TIER IS NOW PROVEN-LIVE, UNATTENDED, ON BOTH DEMO BOXES (2026-08-0Line truncated
|
||||
|
||||
@@ -0,0 +1,531 @@
|
||||
# RECON — can a customer actually get their backups back?
|
||||
|
||||
**Date:** 2026-08-04 · **Items:** R-193, R-192 (+ new: R-198…R-201) · **Class:** recon — **no production code**
|
||||
**Session scope:** read-only everywhere. Nothing built, deployed, wiped, rotated, re-issued or deleted.
|
||||
|
||||
> **Headline, before the detail.** The chain is **not** assembled, and the reason is worse than "two
|
||||
> links are manual". **The hub's superseded-escrow retention — the feature built for exactly this
|
||||
> incident — does not retain the offsite repository password.** `host_escrow_superseded` has no
|
||||
> `identity_blob` column, and `demoteCurrentEscrowTx` copies only the K-escrow blob. So the last copy
|
||||
> of a repo password is destroyed by the **next escrow ceremony** — the very act the system tells the
|
||||
> customer to perform when their escrow goes `pending` after a rebuild. The recovery window closes
|
||||
> because the remedy closes it.
|
||||
|
||||
---
|
||||
|
||||
## 1. Confirmed baselines (re-read on arrival)
|
||||
|
||||
| Repo | `main` @ commit | tree | Version |
|
||||
|------|-----------------|------|---------|
|
||||
| `felhom-agent` | `856a127cd60b0be797867c162f0ae49c7b64993e` | clean | v0.124.1 |
|
||||
| `felhom-controller` | `0887fd676d608e7c103fa13b3e87536ad2481811` | clean | v0.194.0 |
|
||||
| `felhom.eu` | `d26f49ad680e2be2ce2888fd58dff7dee886e4ad` | clean | hub v0.92.0 |
|
||||
|
||||
All three match the prompt's cited commits exactly. `git status --porcelain` empty in each.
|
||||
|
||||
**Highest register ID: R-197** (`OPEN-ITEMS.md` R-196/R-197 were minted by yesterday's spike;
|
||||
R-194/R-195 by the R-190 arc). Grepped across `OPEN-ITEMS.md`, `ROADMAP.md`, `STATUS.md`,
|
||||
`CONTEXT.md`. New items in this session therefore start at **R-198**.
|
||||
|
||||
---
|
||||
|
||||
## 2. Instruments — what each can and cannot answer
|
||||
|
||||
| # | Instrument | Exact read | Known limit |
|
||||
|---|---|---|---|
|
||||
| I1 | source | the three repos at the commits above | the authority; every claim below cites a file:line |
|
||||
| I2 | hub SQLite snapshot | `kubectl -n felhom-system exec pod/hub-6775889d95-gc8sk -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | the `-wal` is load-bearing (copied; `PRAGMA integrity_check` → `ok`) |
|
||||
| I2-fresh | freshness proof | `MAX(host_reports.received_at)` = `2026-08-04 10:00:40`, `datetime('now')` = `2026-08-04 10:03:48` | **positive observable**: newest row 3 m 08 s old, not "the query did not error" |
|
||||
| I3 | live filesystem, demo-felhom guest 9201 | `ssh felhom-pve "pct exec 9201 -- ls -la --time-style=full-iso …"` | read-only `ls`; mtimes only, no file contents read |
|
||||
| I4 | caller census | `grep -rn <symbol> --include=*.go` across all four repos | proves absence of a caller in Go; templates checked separately |
|
||||
|
||||
**Not used, deliberately:** no ceremony, no re-issue, no rotation, no `selftest=identity-consume`, no
|
||||
write of any kind, nothing touched on the storage endpoint.
|
||||
|
||||
---
|
||||
|
||||
## 3. Q1 — the chain, link by link
|
||||
|
||||
From a dead node to an open repository. **Status vocabulary as specified.**
|
||||
|
||||
| # | Link | Component / function | Who invokes it | Status |
|
||||
|---|---|---|---|---|
|
||||
| 1 | repo password is minted | controller `Manager.WriteOffboxSecrets` → mint at `internal/backup/offbox.go:392-401`; generator `:405` | `ApplyOffsiteTarget` (`offbox.go:474`) ← `ConfigureOffbox` ← the apply-bridge (`offsiteapply.go:221,256`) | **PROVEN-LIVE** — fires on every fresh data dir; measured twice this week (I2 hash change, I3 mtime) |
|
||||
| 2 | password staged to the agent | controller `PushOffboxPasswordForEscrow` (`offbox.go:530`) → agent `POST /escrow/stage-secret` (`localapi/server.go:518`) → `escrow.StagedResticPasswordPath()` = `/var/lib/felhom-agent/escrow-stage/restic_repo_password`, 0600 (`identity.go:44-47`) | the controller enable/ceremony path | **PROVEN-LIVE** |
|
||||
| 3 | sealed under R at the ceremony | `AttachResticPassword` (`identity.go:73`) → `WrapIdentityBundle` → `WrapIdentity` (`identity.go:114`, `age -p -a`, scrypt + ChaCha20-Poly1305) | agent `--selftest=escrow-create`, driven by the controller wizard (`/escrow/ceremony`, `localapi/server.go:525-527`) | **PROVEN-LIVE** — customer-facing wizard first live firing 2026-07-18 (`00-capability-map.md`, escrow row) |
|
||||
| 4 | stored by the hub | `PUT /api/v1/hosts/<id>/escrow` (`api/handler.go:239`, `handleHostEscrowPut :1104`) → `host_escrow.identity_blob` | agent `main.go:2937` | **PROVEN-LIVE** — measured: `identity_blob` present, 572 B, for **both** hosts (I2) |
|
||||
| 5 | **old blob retained on supersession** | `demoteCurrentEscrowTx` (`store/store.go:2547-2556`) → `host_escrow_superseded` | `SaveHostEscrow`, `DeleteHost` | **BROKEN FOR THIS PURPOSE.** The INSERT copies `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256` — **not `identity_blob`**, and the table has **no such column** (`PRAGMA table_info`, I2). The K-escrow survives; **the restic repo password does not** |
|
||||
| 6 | served back to a re-enrolling box | hub `handleReEnroll` (`api/dr.go:101`) and `handleGetRestoreDirective` (`:155`), returning `identity_escrow_b64`; recovery-mode gated, armed with the **global operator key** (`:34`) | **NOBODY.** I4: zero callers in `felhom-agent` (no `ReEnroll` symbol, no `/re-enroll` or `/restore-directive` string anywhere in the agent's hub client, `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`); zero in the hub UI (no template mentions them); zero in `scripts/` or any runbook | **implemented-never-exercised, and CLIENTLESS.** The only documented retrieval is by hand: `sqlite3 "SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …"` on a `kubectl cp`-ed `hub.db` (project memory, 2026-07-04 S5 prep) |
|
||||
| 7 | unsealed with R | `UnwrapIdentityBundle` (`identity.go:172`) → `UnwrapIdentity` (`:139`) | **`runSelftestIdentityConsume` only** (`cmd/felhom-agent/main.go:2845-2900`), reading R from `FELHOM_RECOVERY_CODE`. I4: no other production caller | **manual-only — the only caller is a `--selftest` mode** |
|
||||
| 8 | the repo password extracted | none. The selftest writes the **whole bundle JSON** to `-keydest` 0600 (`main.go:2872`) and prints *"identity recovered (tunnel_token + pbs_token)"* — it does not mention, extract or route `restic_repo_password` at all | a human reads the JSON | **missing** |
|
||||
| 9 | placed on the rebuilt controller | `InjectOffboxPassword` (`offbox.go:541`) | one caller: `offboxInjectPasswordHandler` (`web/offbox_handlers.go:189`) at `POST /backup/offbox/inject-password` (`web/server.go:510`). **I4: no template in the repo contains that path or any form posting to it** | **manual-only, and UI-unreachable** — the handler exists, the form does not. A person must POST it by hand with a valid session cookie + CSRF token |
|
||||
| 10 | the existing repo opens | `ensureOffboxRepo` (`offbox.go:653`): `restic cat config` → success clears `RepoState` | the scheduled/manual offbox run | **implemented-never-exercised** on a *recovered* password (exercised constantly on a live one) |
|
||||
| 11 | a backup or restore proves it | — | — | **never** |
|
||||
|
||||
### The chain in one sentence
|
||||
|
||||
Links 1–4 are real and proven. **Link 5 silently drops the payload.** Links 6–9 are each individually
|
||||
manual, and link 6 has no client at all. Link 11 has never happened.
|
||||
|
||||
---
|
||||
|
||||
## 4. Q2 — where the chain stops today
|
||||
|
||||
**The first link that is not automatic is #5, and it is not merely manual — it is lossy.**
|
||||
|
||||
Two distinct stopping points, and the order matters:
|
||||
|
||||
**(a) If the box has NOT re-escrowed since the rebuild** — the current `host_escrow.identity_blob`
|
||||
still seals the *old* password. Recovery is possible in principle, and a person must:
|
||||
|
||||
1. Arm recovery mode on the hub (`PUT /api/v1/admin/hosts/<id>/recovery-mode`, global operator key) —
|
||||
or skip it entirely, because nothing consumes the endpoint it gates;
|
||||
2. `kubectl cp` / `exec cat` the hub DB and `sqlite3 … writefile('/root/idblob', identity_blob)`;
|
||||
3. copy the blob to a machine with the agent binary and `age`;
|
||||
4. `FELHOM_RECOVERY_CODE=… felhom-agent --selftest=identity-consume -blob /root/idblob -keydest /root/bundle.json`;
|
||||
5. open `bundle.json` by hand and read `restic_repo_password` out of it;
|
||||
6. log in to the customer's dashboard and `POST /backup/offbox/inject-password` **by hand** (curl with
|
||||
session + CSRF) — there is no form;
|
||||
7. re-run the apply so `WriteOffboxSecrets` finds the file present and keeps it;
|
||||
8. trigger a run and hope `restic cat config` succeeds.
|
||||
|
||||
Eight manual steps, five of them requiring operator-tier access to the hub, one requiring the customer
|
||||
to be present with R, and one (step 6) with no user interface at all.
|
||||
|
||||
**(b) If the box HAS re-escrowed** — which is what the system pushes the customer to do, because a
|
||||
rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`,
|
||||
`offbox.go:570`), and the card says *„Helyreállítási kód szükséges"* — then the old `identity_blob`
|
||||
has been **overwritten** by `SaveHostEscrow` and **not carried** into the superseded row. At that point
|
||||
**no procedure exists**, with or without R, at any level of effort. The password is gone.
|
||||
|
||||
Both demo boxes are in state (b) as of 2026-08-04 07:15:36 (demo-hp) and 07:20:08 (demo-felhom) — I2,
|
||||
`host_escrow.updated_at`. This is an **independent, and much stronger, reason** the 51 orphaned
|
||||
snapshots are unrecoverable than "nobody kept the recovery codes". Keeping R would not have helped.
|
||||
|
||||
---
|
||||
|
||||
## 5. Q3 — can a rebuilt controller know it should recover?
|
||||
|
||||
### RULING: **YES — the hub already tells it, on every single report, and the controller throws the answer away.**
|
||||
|
||||
**What the box knows at the mint decision.** `WriteOffboxSecrets` is called from
|
||||
`ApplyOffsiteTarget`, driven by the apply-bridge with the hub's offsite **descriptor**, whose fields
|
||||
are `host, user, port, repo_path, host_fingerprint, quota_gb` (`offsiteapply.go:198-204`). **Nothing
|
||||
about escrow is in scope at that point.** Locally, the only signal is the one the mint branch already
|
||||
reads: `os.Stat(m.offboxPwPath())` returning `IsNotExist` — which *is* the fresh-data-dir signal, at
|
||||
exactly the right line, but cannot distinguish "never configured" from "rebuilt".
|
||||
|
||||
**What the box is nonetheless already told.** Every report ACK carries an `escrow` object
|
||||
(`api/handler.go:504-510`), consumed as `report.EscrowStatus`
|
||||
(`internal/report/escrow_confirm.go:19-23`):
|
||||
|
||||
```go
|
||||
type EscrowStatus struct {
|
||||
IdentityBlobPresent bool `json:"identity_blob_present"`
|
||||
ResticPwSHA256 string `json:"restic_pw_sha256"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
}
|
||||
```
|
||||
|
||||
That is precisely *"the hub has a blob, and here is the hash of the password it covers"* — arriving
|
||||
every report cycle, on a channel that already exists, with no new API and no new secret.
|
||||
|
||||
**And it is discarded.** `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:75-84`) returns
|
||||
immediately unless `Pending()` or `Escrowed()` — both of which require `OffboxConfigured()` **and** a
|
||||
non-nil target (`cmd/controller/main.go:580-590`). On a freshly rebuilt box, before the offsite apply,
|
||||
there is no target, so the status is dropped on the floor. It is never persisted anywhere: I4 finds
|
||||
exactly **one** reference to the ACK field in the whole controller
|
||||
(`cmd/controller/main.go:641: escrowConfirmer.Reconcile(resp.Escrow)`).
|
||||
|
||||
**Therefore a fail-closed refusal IS implementable**, with no new hub endpoint:
|
||||
|
||||
- persist the last-seen ACK escrow status alongside settings (the precedent exists — `ClaimSync`
|
||||
caches the hub's claim-code state into `settings.json` idempotently by generation,
|
||||
`internal/report/claim_sync.go:39-53`, and it does so *set-only, never cleared on hub silence*,
|
||||
which is the correct shape here too);
|
||||
- at the mint branch, refuse when the password file is absent **and** the cached status says
|
||||
`identity_blob_present && restic_pw_sha256 != ""`.
|
||||
|
||||
**One ordering caveat, stated rather than assumed.** On a truly fresh box the apply can race the first
|
||||
ACK. Measured on demo-hp: first post-rebuild report `2026-08-03 06:12:19`, offsite apply
|
||||
`2026-08-04 07:12` — the ACK preceded the apply by 25 hours, but that gap was an incident, not a
|
||||
design. A fail-closed rule handles the race correctly by construction: refuse to mint until the escrow
|
||||
status is known, and retry next cycle (the apply-bridge is already built to retry and to persist
|
||||
nothing on a partial failure, `offsiteapply.go:1-6`).
|
||||
|
||||
**Corroborating measurement that the data dir really is fresh on a rebuild** (I3, demo-felhom guest
|
||||
9201, `/var/lib/docker/volumes/felhom-controller-data/_data/data/`):
|
||||
|
||||
| file | mtime |
|
||||
|---|---|
|
||||
| the data directory's parent | `2026-08-03 07:17:25` |
|
||||
| `encryption.key` (32 B) | `2026-08-03 07:17:26` |
|
||||
| `offbox/` | `2026-08-03 07:18:02` |
|
||||
| `offbox/repo_password` (64 B) | `2026-08-03 07:18:02` |
|
||||
|
||||
The volume, the app-secret encryption key and the repo password are all newborn, 36 seconds apart.
|
||||
Corroborated hub-side (I2): `json_extract(report_json,'$.claimed')` reads **1 → 0 → 1** across the
|
||||
rebuild on both boxes — demo-hp `0` in 5 reports `2026-08-03 06:12:19`–`06:21:03`; demo-felhom `0` in 4
|
||||
reports `07:17:54`–`07:19:46`. `Claimed` is set only by `SetClaimed()`, whose sole caller is the claim
|
||||
submit handler (`web/claim.go:349`), so the return to `1` was a **human re-claim**, not automation.
|
||||
|
||||
> **Note, not acted on:** a fresh `encryption.key` means every pre-rebuild `app.yaml` `ENC:` secret is
|
||||
> undecryptable — the C-1 circular dependency in `_recovery-inventory-2026-07-28.md` fired on both
|
||||
> demo boxes this week. That is outside R-193's scope and is recorded as an observation (§13).
|
||||
|
||||
---
|
||||
|
||||
## 6. Q4 — what the stale-escrow detector actually covers
|
||||
|
||||
### RULING: **the prompt's premise conflates two different mechanisms. `escrow_confirm.go` DOES fire on a mint. But it fires into a log file, and the remedy it prescribes destroys the last copy of the old key.**
|
||||
|
||||
**The conflation, stated because it matters.** Yesterday's spike's sentence *"wired to the one path
|
||||
that doesn't change the key and absent from the one that does"* is about the **hub's `escrow_stale`
|
||||
event**, emitted by `ReissueCredentials` (`hub/internal/offsite/offsite.go:198-201`) — that is R-196,
|
||||
and it stands. It is **not** about `controller/internal/report/escrow_confirm.go`. Two different
|
||||
detectors; only one of them was ever claimed to be misplaced.
|
||||
|
||||
**What `escrow_confirm.go` does on a mint, from source.** After a rebuild the apply creates a target
|
||||
with `EscrowState: "pending"` (the `cur == nil` fallthrough, `offbox.go:482-494`). So `Pending()` is
|
||||
true, the pending branch runs (`escrow_confirm.go:86-107`), the hub's hash (sealing the OLD password)
|
||||
is compared against `LocalHash()` (the NEW minted one), they differ, and it logs:
|
||||
|
||||
```
|
||||
[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password
|
||||
(hub hash %.12s… != local %.12s…) — run the escrow ceremony … staying pending
|
||||
```
|
||||
|
||||
**So: confirmed, it fires.** The detector is not absent from the mint path.
|
||||
|
||||
**Three limits, each of which matters more than the fact that it fires.**
|
||||
|
||||
1. **It is a controller-local log line.** No event, no e-mail, no hub-side signal. `StaleBlob()` — the
|
||||
flag the web card renders — is set **only** by the escrowed branch (`reconcileEscrowed`,
|
||||
`:130-165`); the pending branch sets no display flag. The operator learns nothing.
|
||||
2. **Its prescription is the destructive act.** *"run the escrow ceremony"* → a new ceremony →
|
||||
`SaveHostEscrow` overwrites `host_escrow.identity_blob` → `demoteCurrentEscrowTx` does not carry it
|
||||
→ **the old repo password ceases to exist**. The detector correctly identifies the situation and
|
||||
then tells the customer to close the door.
|
||||
3. **It cannot distinguish the two situations that need different answers** — "you re-keyed
|
||||
deliberately" and "your machine was rebuilt and your history is about to be orphaned" produce the
|
||||
identical warning.
|
||||
|
||||
So: **less needs building than the prompt feared for the *detection*, and more needs building for the
|
||||
*consequence*.** The comparison exists; what does not exist is a path from that comparison to either a
|
||||
signal or a recovery.
|
||||
|
||||
---
|
||||
|
||||
## 7. Q5 — is the superseded-escrow retention real, and what does it retain?
|
||||
|
||||
### RULING: **the retention is real, it retains the wrong blob, and nothing can read it back.**
|
||||
|
||||
**What it retains — source and live, agreeing.** `demoteCurrentEscrowTx` (`store/store.go:2547-2556`):
|
||||
|
||||
```sql
|
||||
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
|
||||
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
|
||||
FROM host_escrow WHERE host_id = ?
|
||||
```
|
||||
|
||||
`identity_blob` is not in the column list. Live schema (I2, `PRAGMA table_info`):
|
||||
|
||||
| table | columns |
|
||||
|---|---|
|
||||
| `host_escrow` | `host_id, blob, key_fingerprint, posture, created_at, updated_at, **identity_blob**, directive_json, restic_pw_sha256, stale_at` |
|
||||
| `host_escrow_superseded` | `id, host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at` |
|
||||
|
||||
**`blob` is the K-escrow — the PBS datastore encryption key**, wrapped by the PBS-native scrypt path.
|
||||
**`identity_blob` is the age-wrapped `IdentityBundle`** — the one carrying `ResticRepoPassword`
|
||||
(`felhom-agent/internal/escrow/identity.go:34-39`), plus `TunnelToken`, `PBSToken` and
|
||||
`WGPrivateKey`. Live sizes confirm they are different objects: both hosts' current rows hold
|
||||
`blob` = 383 B **and** `identity_blob` = 572 B; both superseded rows hold `blob` = 383 B and nothing
|
||||
else.
|
||||
|
||||
**So, exactly what a customer with the matching OLD recovery code could recover from a retained blob:**
|
||||
|
||||
- ✅ the PBS datastore encryption key **K** — i.e. the Plane-2 whole-guest offsite ciphertext;
|
||||
- ❌ **not** the restic repository password — the Tier-3 offsite app-data history;
|
||||
- ❌ not the tunnel token, not the PBS access token, not the WireGuard private key.
|
||||
|
||||
**And there is no read path.** I4: `ListSupersededEscrow` (`store/store.go:2617`) has **zero
|
||||
production callers** — only two tests. The only production surface is `CountSupersededEscrow`, used
|
||||
for a number on the host page (`web/hosts.go:573`) and a log line (`api/handler.go:1151`). The schema
|
||||
comment says so itself: *"The ACK and restore-serving read `host_escrow` (the CURRENT row) — never
|
||||
this table."* Retrieval is `sqlite3` on the hub pod, by hand, by the operator.
|
||||
|
||||
**The claim this contradicts.** `00-capability-map.md` (escrow-ceremony row) records hub v0.60.0 as:
|
||||
|
||||
> *"a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS
|
||||
> it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code
|
||||
> (turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable')."*
|
||||
|
||||
The same claim appears in the schema comment (`store.go:370-375`) and, in Hungarian, on the customer's
|
||||
own orphan card (`controller/internal/web/templates/backups_remote.html:66,69`):
|
||||
|
||||
> *„A meglévő mentések nem sérültek — a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók
|
||||
> lehetnek."* · *„A régi előzmény félretéve marad (nem törlődik), és a hozzá tartozó helyreállítási
|
||||
> kóddal később visszaállítható lehet."*
|
||||
|
||||
**For the offsite restic repository — the incident it names — that is false.** The retained blob does
|
||||
not contain the key that opens it. This is the **ninth** entry in `CLAUDE.md`'s table of comments
|
||||
asserting an invariant the code does not provide, and the first that is also **customer-facing copy**.
|
||||
→ **R-198**, and the capability-map correction in §9.
|
||||
|
||||
---
|
||||
|
||||
## 8. Q6 — has any part of this ever been exercised?
|
||||
|
||||
**Method:** read every audit, validation doc, findings file and REPORT that mentions escrow consume,
|
||||
identity-consume, re-enroll or inject-password; then check what the cited evidence actually covers.
|
||||
|
||||
### Proven
|
||||
|
||||
| what | evidence | scope of the proof |
|
||||
|---|---|---|
|
||||
| ceremony (create + seal + upload) | `00-capability-map.md` escrow row; `tests/VALIDATION-n100-rehearsal-2026-07-18.md` S6 — customer drove the wizard, R claimed one-shot 16:56:39, auto-confirmed 16:56:41 | the **create** side only |
|
||||
| identity-bundle round-trip with a real R on a secret-less box | `tests/slice10d-identity-restore-spike-findings.md` §1, 2026-06-10 — wrap→unwrap byte-identical (`sha256` match); wrong-R **fails closed**, no plaintext emitted | **PRE-FORK-4.** The bundle then was `{tunnel_token, pbs_token}`. `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** (`felhom-agent/CHANGELOG.md:2344`) — a month later. **The proof does not cover the field this whole question is about** |
|
||||
| `ResticRepoPassword` survives the wrap/unwrap | agent v0.77.0 CHANGELOG: *"`IdentityBundle` round-trip carries `ResticRepoPassword` byte-exact + not-in-blob + wrong-R fails closed"* | **unit test only** |
|
||||
| `InjectOffboxPassword` places and refuses correctly | `controller/internal/backup/offbox_test.go:952-990`, incl. the companion showing a fresh mint without inject | **unit test only** |
|
||||
|
||||
### Never exercised — named in those words
|
||||
|
||||
- **A blob has never been served to a box.** Link 6 has no client (§3).
|
||||
- **A fork-4 bundle has never been unsealed with a real R outside a unit test.** The only production
|
||||
caller is a `--selftest` mode; the one live consume ever prepared (S5 Part 4-B, 2026-07-04) was
|
||||
explicitly **deferred and never run** — its own resume note still reads *"DEFERRED 2026-07-04, not
|
||||
run"* (project memory; `06-offsite-connectivity.md:327` marks the destructive drill operator-gated).
|
||||
- **A recovered repo password has never been injected into a controller.**
|
||||
- **An existing offsite repository has never been reopened with a recovered password.**
|
||||
- **No restore of any kind has ever been performed from a recovered secret.**
|
||||
|
||||
`_recovery-inventory-2026-07-28.md` already said most of this and said it accurately — A.2.7 records
|
||||
*"the **consume** side is proven at spike level … the destructive in-place drill is explicitly
|
||||
operator-gated and unrun"*, and C.1 row 1 says *"escrow **creation** proven live; **consume never
|
||||
exercised destructively**"*. **That document is right and this session confirms it.** What it did not
|
||||
catch — because it was written from `host_escrow`, not from the supersession path — is Q5.
|
||||
|
||||
**Capability-map correction owed:** not for a "recovery proven" claim (no row makes one), but for the
|
||||
**retention** claim quoted in §7. See §9.
|
||||
|
||||
---
|
||||
|
||||
## 9. Q7 — the operator's ruled shape, assessed row by row
|
||||
|
||||
> **Operator ruling, 2026-08-04, verbatim:** *If a node is a fresh install AND the hub has a recovery
|
||||
> blob, then the controller should yell that recovery is available, and provide a form for the customer
|
||||
> to enter the recovery key. After unlocking the blob, the controller should show what will be
|
||||
> recovered before proceeding.*
|
||||
|
||||
| The ruling requires | Exists? | What it needs, and what it costs |
|
||||
|---|---|---|
|
||||
| the controller knows it is a **fresh install** | **YES, at the right line** | `os.Stat(m.offboxPwPath())` → `IsNotExist` is already the mint branch (`offbox.go:392`). It is trustworthy as *"this controller has no repo password"*, which is the only fact the decision needs; it does **not** by itself distinguish first-ever install from rebuild — that is what the next row supplies. Corroborators exist but are weaker: `Claimed` is set-only and was measured re-set by a human within minutes (§5); a fresh `encryption.key` is a stronger signal but is not currently read by anything. **Cost: zero — the branch already exists.** |
|
||||
| the controller knows **the hub has a blob** | **YES on the wire, NO in the code** | `EscrowStatus{IdentityBlobPresent, ResticPwSHA256, CreatedAt}` arrives on every report ACK and is discarded when no offbox target exists (§5). **Needs:** persist it (the `ClaimSync` set-only-by-generation pattern is the precedent, `claim_sync.go:39-53`). **Cost: small — controller only, one settings field, one reconciler, no hub change.** |
|
||||
| it can **yell** rather than silently mint | **NO** | The refusal belongs **inside `WriteOffboxSecrets`, at the mint branch** — the one place that decides. Returning a new sentinel (`ErrOffboxRecoveryAvailable`, sibling of `ErrOffboxOrphaned`) propagates through `ApplyOffsiteTarget` → `ConfigureOffbox` → the apply-bridge, which already treats a failed configure as "persist nothing, retry next cycle" (`offsiteapply.go:5-6, 256`) — so a refusal is **fail-safe by construction**, not a new failure mode. Meanwhile the customer sees a card on `/backups/remote` (the orphan card at `backups_remote.html:60-70` is the exact precedent) and offsite runs stay blocked, which they already are (`OffboxRunnable`). **Cost: small-medium — controller only.** **Design hazard to name:** a first-ever install of a *re-used customer record* would also trip this; the refusal must be dismissible ("start fresh anyway"), and dismissal must be the act that mints. |
|
||||
| a **form takes R** | **NO — and R is not a password** | The controller UI **emits** R (`backups_escrow.html:235-236`, `/backup/escrow/…/claim` returns it once) and has **never accepted** one. The only thing resembling an entry point is `POST /backup/offbox/inject-password`, which takes a **64-hex repo password** (`offboxRepoPwPattern`, `offbox.go:543`) — a different secret at a different layer — and **has no form in any template** (§3 link 9). **Cost: small for the form; the hard part is the next row.** |
|
||||
| the blob is **unsealed** with R | **NO — and there is a real boundary** | Unsealing is `age -d`. `ageBinary = "/usr/bin/age"` (`identity.go:24`) and the code live in the **agent**. The **controller image does not ship `age`** (`controller/Dockerfile:56-70` — restic, openssh-client, sshpass, sqlite3, docker-cli; no age) and must not: the agent is the privileged tier by design. **The crossing already has a transport and a precedent:** the pinned local API carries `POST /escrow/ceremony` + `GET /escrow/ceremony/status` + `POST /escrow/ceremony/claim` (`localapi/server.go:524-527`) — i.e. the agent **already** handles R server-side for the create direction and hands the result back exactly once. **Needs: one new agent local-API endpoint** (`POST /escrow/unseal`: blob + R in, the single `restic_repo_password` field out, R never logged, R never persisted) — the exact mirror of the claim endpoint. **Plus** a way for the blob to reach the agent: the agent's hub client has no escrow **GET** (only the `PUT` at `main.go:2937`), and `dr.go`'s re-enroll is the wrong tool (it rotates the host API key and needs operator-armed recovery mode). **Needs: a second small hub endpoint** — `GET /hosts/<id>/escrow`, per-host-key authed, serving opaque bytes to the box that owns them. Zero-knowledge is untouched: the hub still never decrypts, and the bytes are useless without R. **Cost: medium — three repos, two new endpoints, both narrow.** |
|
||||
| **"show what will be recovered" before proceeding** | **the primitive EXISTS; the flow does not** | `restic snapshots --json` + `restic stats --mode raw-data --json` are already how the box counts snapshots and sizes the repo (`offboxRecordStats`, `offbox.go:1234-1265`). Both are **read-only against the repository** — nothing is written, no lock is taken, no `init`. So a preview can show: **snapshot count, per-snapshot time, host, tags (= app names), paths, and the deduplicated repo size** — all before committing. **Committing requires only that the recovered password be written to `offboxPwPath()`**, and even that is reversible in principle. **The clean shape:** unseal → hold the recovered password **in memory** → run the two read-only probes against the existing repo → render → and only on the customer's confirmation call `InjectOffboxPassword`. **Cost: small — one handler, one template, no new restic capability.** **Caveat, measured not assumed:** `offboxRecordStats` today also persists `RepoSizeBytes` into settings as a side effect; a preview path must not reuse it verbatim. |
|
||||
| **proceeding** actually recovers | **NO end-to-end** | `InjectOffboxPassword` → next `WriteOffboxSecrets` keeps the file → `ensureOffboxRepo` probes and clears `RepoState`. Every piece is implemented; **none has ever run against a recovered password** (§8). **Cost: zero new code, but it is precisely what the drill in §10 has to prove.** |
|
||||
|
||||
### Things the ruling implies that do not exist, priced
|
||||
|
||||
| missing piece | cost | note |
|
||||
|---|---|---|
|
||||
| persisted ACK escrow status + fail-closed mint refusal | **S** | controller only; the `ClaimSync` pattern applies directly |
|
||||
| the "recovery is available" card + dismissal | **S** | one template, one flag; the orphan card is the template precedent |
|
||||
| a customer-facing **recovery-code** form (distinct from the password form) | **S** | the password form's handler exists but has no UI; both need building |
|
||||
| agent local-API `POST /escrow/unseal` | **M** | new privileged surface; mirror of `/escrow/ceremony/claim` |
|
||||
| hub `GET /hosts/<id>/escrow` (opaque, per-host key) | **S** | new read surface on the hub; zero-knowledge preserved |
|
||||
| preview handler + template driven by read-only restic probes | **S** | primitive already exists |
|
||||
| **carrying `identity_blob` into `host_escrow_superseded`** | **S** | one column, one `SELECT` list — **and without it the whole feature protects only boxes that have not yet re-escrowed** (→ R-198). This is a prerequisite, not a nice-to-have |
|
||||
|
||||
### The security question the ruling raises — stated, not answered
|
||||
|
||||
**What gates the surface today.** The controller dashboard is behind `RequireAuth`
|
||||
(`internal/web/auth.go:52`): a bcrypt password (`effectivePasswordHash()` — customer-set in
|
||||
`settings.json`, else operator-provisioned in `controller.yaml`), a 7-day session cookie, 5 failed
|
||||
logins per minute then lockout, and `CsrfProtect` on every mutating route. The claim gate serves only
|
||||
the claim page on an unclaimed box. So a recovery form would sit **behind the customer's dashboard
|
||||
password**, on a controller reachable at whatever address that box's dashboard is reachable at.
|
||||
|
||||
**What a preview would expose to someone who guesses or steals R.** Not file contents — but the
|
||||
snapshot list is not nothing: **backup dates and cadence, the app names (restic tags), the paths, and
|
||||
the repository size.** That is a map of the household's digital life, and it is precisely the metadata
|
||||
the zero-knowledge posture otherwise keeps out of everyone's hands including the operator's.
|
||||
|
||||
**Three specifics the operator should weigh, without this session choosing:**
|
||||
|
||||
1. **Guessing R is not the threat; holding the dashboard password is.** R is ten EFF words
|
||||
(~129 bits, `00-capability-map.md` agent v0.93.0 note) and `age`'s scrypt is deliberately slow —
|
||||
online guessing is not a route. But **anyone with the dashboard password can already restore live
|
||||
offsite data** (`_recovery-inventory-2026-07-28.md` A.1.3–A.1.6). The marginal new exposure is
|
||||
confined to the *orphaned/previous* repository, which today nobody can read at all.
|
||||
2. **The form is an oracle.** A correct/incorrect R answer distinguishable in the UI turns the
|
||||
dashboard into a verification service for a stolen recovery code. `UnwrapIdentity` already
|
||||
fails closed with a single generic error (`identity.go:157`) — the UI must not do better than that,
|
||||
and should rate-limit like the login path does.
|
||||
3. **R will pass through the agent.** Whatever the form does, R crosses the local API to a
|
||||
root-adjacent daemon. The ceremony already does this in the other direction, so the precedent is
|
||||
set — but it means R is momentarily present in agent process memory on the Proxmox host, which is
|
||||
the same host D6 says the operator has root on. **This is the same trade as spike option (c), in a
|
||||
smaller and time-bounded form**, and it should be decided alongside it rather than separately.
|
||||
|
||||
---
|
||||
|
||||
## 10. Q8 — the drill design
|
||||
|
||||
**Why a deliberate cycle.** No recovery code exists for either demo box, and — per §7 — even one would
|
||||
no longer open the old repositories. The proof therefore requires creating R, keeping it, and walking
|
||||
the full circle. **This section is the design; the drill is not run in this session.**
|
||||
|
||||
### Target and preconditions
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Box** | **`demo-hp`** (HP t740, node `felhom-host`, host `demo-hp-bb76ea`). Tier 0 — disposable, and the designated drill host by the 2026-07-25 operator ruling. **Not demo-felhom** — it is the N100 that also carries the ISO/rehearsal lineage; leave one demo box in a known-good state throughout |
|
||||
| **Guest** | the customer guest on that host (rebuild target) |
|
||||
| **Data** | one deliberately identifiable app + a sentinel file whose content is a known string and whose sha256 is recorded before the wipe |
|
||||
| **Prereq** | the box is healthy, offsite target configured and `escrow_state: escrowed`, and its **current** `host_escrow.identity_blob` seals the password of the repo that will be tested |
|
||||
| **Duration** | ~3–4 h wall clock, of which ~45 min is operator-attended (steps 2, 5, 8, 9) |
|
||||
| **Blocking dependency** | **steps 8–9 cannot be executed as designed today** — links 6, 8 and 9 of the chain do not exist as automation. The drill as written uses the manual substitutes (§4a) and **that is the point**: it measures the *existing* chain honestly before anything is built |
|
||||
|
||||
### The cycle
|
||||
|
||||
| # | Step | Who | Observable that must be recorded |
|
||||
|---|---|---|---|
|
||||
| 1 | Record the starting state: `host_escrow.restic_pw_sha256`, `identity_blob` length, `$.offsite.snapshot_count`, `repo_size_bytes` | CC | a hash and two counters, from the hub DB |
|
||||
| 2 | **Run the escrow ceremony through the customer wizard and KEEP R** | **HUMAN** | R written to the operator's password manager under a named entry; **its storage location recorded in the drill report — the code itself never** |
|
||||
| 3 | Deploy an app, write the sentinel file, record its sha256 | CC | sha256 + path |
|
||||
| 4 | Run an off-site backup through the real UI endpoint; wait for completion | CC | `$.offsite.last_status = ok`, `snapshot_count` increased by ≥1, `last_success` stamped |
|
||||
| 5 | **Verify the snapshot from outside**: `restic snapshots --json` against the repo | CC | snapshot id + tag + timestamp — the pre-wipe truth to compare against |
|
||||
| 6 | **Wipe**: destroy the guest and reinstall it (the same procedure that produced the 2026-08-03 rebuild) | **HUMAN** decision, CC executes | new `controller_started`; `claimed` drops to 0; a new `encryption.key` mtime |
|
||||
| 7 | Let the box come back and reach the mint. **Do NOT run a new ceremony** | CC | the controller log carries the `escrow-confirm` mismatch WARN (§6); `escrow_state: pending`; `host_escrow.identity_blob` **unchanged** (this is the assertion that keeps recovery possible) |
|
||||
| 8 | **Recover**: export `identity_blob` from the hub, unseal with the kept R, extract `restic_repo_password` | **HUMAN** (R is theirs) | `identity-consume` exits 0; the bundle JSON contains a non-empty `restic_repo_password`; **the value is never printed or logged** |
|
||||
| 9 | **Inject** the recovered password (`POST /backup/offbox/inject-password`, by hand — there is no form), re-apply, run an off-site backup | **HUMAN**-assisted | `restic cat config` succeeds; **`snapshot_count` ≥ the step-5 count + 1**, not 1 |
|
||||
| 10 | **Restore the sentinel** through the customer restore flow and compare sha256 | CC | **byte-identical sha256 to step 3** |
|
||||
| 11 | Record everything, tear down nothing that would hide the evidence | CC | drill report |
|
||||
|
||||
### The single observation that makes this a FAILURE rather than a pass
|
||||
|
||||
> **Step 10's sha256 does not match step 3's — or step 9 reports a snapshot count of 1.**
|
||||
|
||||
A snapshot count of 1 means a **new, empty history** was started over the old one: the repository did
|
||||
not reopen, and every earlier snapshot is orphaned. **Explicitly: "the repository opened" is not the
|
||||
pass condition.** `restic cat config` succeeding proves the password is right; it does not prove the
|
||||
customer's data came back. Only the sha256 comparison does.
|
||||
|
||||
### Secondary failures worth recording separately (each is a finding, not a pass/fail)
|
||||
|
||||
- Step 7 finds `identity_blob` **changed** — something re-escrowed automatically → the recovery window
|
||||
closes without a human act (would be new and serious).
|
||||
- Step 8's bundle has an **empty** `restic_repo_password` — a pre-fork-4 or hash-less blob shape.
|
||||
- Step 9's `ensureOffboxRepo` classifies `orphaned` despite the correct password → the classifier or
|
||||
the transport, not the key.
|
||||
|
||||
### What could make it fail for uninteresting reasons
|
||||
|
||||
The `age` binary absent on the machine running step 8; the hub DB export producing a truncated blob
|
||||
(use `writefile`, verify the byte length against `length(identity_blob)` = 572); the CSRF token in
|
||||
step 9 (`felhom_session` is dropped by curl's cookie jar — project memory
|
||||
`controller-api-headless-access`); and the R-71a settle gate deferring the apply in step 9 (wait for
|
||||
its GO line rather than retrying).
|
||||
|
||||
### Rollback, and whether the loss is acceptable
|
||||
|
||||
**If the recovery does not work, what has been lost is:** demo-hp's guest, its app data since the last
|
||||
whole-guest archive, and its off-site history. **On that box that is acceptable** — it is Tier 0,
|
||||
explicitly disposable, and it is the box the target-selection runbook names for drills. **The
|
||||
rollback** is a whole-guest restore from the local/PBS archive taken immediately before step 6, which
|
||||
is a proven path on this fleet (restore-test passed on demo-hp's host tier in 109.3 s on 2026-08-04).
|
||||
**Take that archive deliberately as part of step 6 and verify it exists before destroying anything** —
|
||||
do not rely on the nightly.
|
||||
|
||||
**Not acceptable, and therefore not proposed:** running this on demo-felhom (leaves no healthy demo
|
||||
box), on DooPlex (Tier 2, it *is* the recovery chain), or on any real customer's box.
|
||||
|
||||
---
|
||||
|
||||
## 11. Every query used
|
||||
|
||||
| # | Source | Query / read | Value read |
|
||||
|---|---|---|---|
|
||||
| 1 | git | `git -C <repo> rev-parse HEAD`; `git status --porcelain` | the three commits in §1; all empty |
|
||||
| 2 | hub DB | `PRAGMA integrity_check` | `ok` |
|
||||
| 3 | hub DB | `SELECT MAX(received_at) FROM host_reports; SELECT datetime('now')` | `2026-08-04 10:00:40` / `2026-08-04 10:03:48` |
|
||||
| 4 | hub DB | `SELECT host_id, length(blob), length(identity_blob), substr(restic_pw_sha256,1,12), created_at, updated_at FROM host_escrow` | `demo-felhom-8363b5, 383, 572, c60c8bc737a6, 2026-08-04T07:20:07Z, 2026-08-04 07:20:08` · `demo-hp-bb76ea, 383, 572, 8a9e33aa4da6, 2026-08-04T07:15:36Z, 2026-08-04 07:15:36` |
|
||||
| 5 | hub DB | `SELECT id, host_id, length(blob), substr(restic_pw_sha256,1,12), created_at, superseded_at FROM host_escrow_superseded` | `3, demo-hp-bb76ea, 383, 8e03eddf9ff7, 2026-07-23T10:01:17Z, 2026-08-04 07:15:36` · `4, demo-felhom-8363b5, 383, 48741892f0ef, 2026-07-21T08:38:33Z, 2026-08-04 07:20:08` |
|
||||
| 6 | hub DB | `PRAGMA table_info(host_escrow)` / `(host_escrow_superseded)` | the two column lists in §7 — `identity_blob` present in the first, **absent from the second** |
|
||||
| 7 | hub DB | latest report per customer: `json_extract(report_json,'$.claimed' / '$.offsite.escrow_state' / '.snapshot_count' / '.repo_state' / '.last_status' / '.last_run')` | demo-felhom `2026-08-04 09:49:42`: `1, escrowed, 0, NULL, NULL, NULL` · demo-hp `2026-08-04 09:56:57`: `1, escrowed, 0, NULL, NULL, NULL` |
|
||||
| 8 | hub DB | `claimed` grouped over reports since `2026-08-03 05:00` | demo-hp: `1` ×124, **`0` ×5 (06:12:19→06:21:03)** · demo-felhom: `1` ×120, **`0` ×4 (07:17:54→07:19:46)** |
|
||||
| 9 | hub DB | `SELECT customer_id, event_type, severity, created_at FROM events WHERE created_at > '2026-08-03 00:00'` | no `escrow_stale`, no `offsite_reissued`, no `offbox_repo_orphaned`, no claim event in the window; `controller_started` ×4 demo-hp / ×4 demo-felhom around the rebuilds |
|
||||
| 10 | guest 9201 (I3) | `ls -la --time-style=full-iso …/felhom-controller-data/_data/data{,/offbox}` | the mtime table in §5 |
|
||||
| 11 | source (I4) | `grep -rn "InjectOffboxPassword" --include=*.go .` | 1 definition, 1 handler caller, 5 test uses, 1 comment — **no production caller besides the handler** |
|
||||
| 12 | source (I4) | `grep -rn "inject" internal/web/templates/` | one unrelated hit (an XSS comment) — **no form** |
|
||||
| 13 | source (I4) | `grep -rni "reenroll\|re_enroll" --include=*.go .` in `felhom-agent` | **zero hits** |
|
||||
| 14 | source (I4) | `grep -rn "re-enroll\|recovery-mode\|restore-directive" --include=*.sh --include=*.md --include=*.py` workspace-wide | only design docs + project memory; no script, no runbook step |
|
||||
| 15 | source (I4) | `grep -rn "ListSupersededEscrow"` in `felhom.eu` | 2 test callers, 0 production |
|
||||
| 16 | source (I4) | `grep -rn "\.Escrow\b" --include=*.go .` in `felhom-controller` | one hit: `cmd/controller/main.go:641` |
|
||||
| 17 | source | `felhom-agent/CHANGELOG.md` | `v0.77.0 — fork-4: escrow the offsite restic repo password under R (2026-07-09)` |
|
||||
| 18 | source | `controller/Dockerfile:36-70` | runtime package list — **no `age`** |
|
||||
|
||||
---
|
||||
|
||||
## 12. What could not be established
|
||||
|
||||
Named, not guessed.
|
||||
|
||||
1. **The 2026-08-05 ~02:15 UTC scheduled run's outcome.** Still unmeasured — both boxes became
|
||||
runnable only on 2026-08-04 morning and neither has run since (query 7: `last_run` absent on both).
|
||||
Yesterday's spike predicts `ErrOffboxOrphaned`; that prediction is **unchanged and still unproven**.
|
||||
2. **Whether demo-hp's data dir was fresh.** Established for demo-felhom by direct mtime measurement
|
||||
(I3). demo-hp was not measured — it has no baked SSH key and reaching it needs the break-glass
|
||||
credential, which this read-only session did not use. The hub-side `claimed 1→0→1` signature is
|
||||
identical on both boxes, which is suggestive, not decisive.
|
||||
3. **Whether the orphaned ciphertext still exists at the provider.** Unchanged from yesterday: reading
|
||||
the endpoint needs the SFTP credential, which was not touched.
|
||||
4. **Whether any pre-2026-07-21 escrow generation ever existed for demo-felhom.** The superseded table
|
||||
holds two rows in the entire database; earlier generations are not retained (and, per §7, would not
|
||||
have carried the identity blob anyway).
|
||||
5. **Whether `age` is present on any machine that would run step 8 of the drill.** Not checked — the
|
||||
agent declares `/usr/bin/age` as a runtime dependency but this session did not verify it on either
|
||||
demo host.
|
||||
|
||||
---
|
||||
|
||||
## 13. Observations — noticed, not acted on
|
||||
|
||||
1. **A fresh `encryption.key` on every rebuild** (I3: 32 B, mtime `2026-08-03 07:17:26`) means every
|
||||
pre-rebuild `app.yaml` `ENC:` secret on that box is undecryptable. This is exactly circular
|
||||
dependency **C-1** in `_recovery-inventory-2026-07-28.md`, and it fired live on both demo boxes this
|
||||
week without anyone naming it. It is a strictly larger blast radius than the offsite repo and it is
|
||||
**not** covered by R-193. Worth its own item if the operator agrees it is not already implied by C-1.
|
||||
2. **`Claimed` returning to `1` within minutes** of a rebuild, with no event of any kind recording it,
|
||||
means "the customer claimed this box" has no audit trail. There is no `customer_claimed` event type.
|
||||
3. **`offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting** — harmless today,
|
||||
a trap for anyone reusing it as a read-only preview primitive (§9).
|
||||
4. **`handleReEnroll` returns the new API key in the response body** (`dr.go:148`) while also logging
|
||||
the rotation. Not a defect (the box needs it), but it is a secret in an HTTP response on an endpoint
|
||||
nothing currently calls — worth a second look before anything starts calling it.
|
||||
|
||||
---
|
||||
|
||||
## 14. The operator's decisions — stated, and NOT answered here
|
||||
|
||||
1. **Run the drill (§10) or not?** It is the only thing that converts *"the pieces look right"* into
|
||||
*"a customer got their data back"*. It costs demo-hp for an afternoon and, if it fails, that box's
|
||||
history. **Recommendation: yes — but fix `identity_blob` retention (R-198) first**, because a drill
|
||||
that walks the current chain will pass or fail on a chain that is missing a link everyone believed
|
||||
was there.
|
||||
2. **The orphaned snapshots at the endpoint** (40.9 MB + 1.14 GB, presumed present, §12.3). They are
|
||||
**unrecoverable by two independent mechanisms** now: no recovery code was kept, *and* the sealed
|
||||
passwords were destroyed by the 2026-08-04 re-ceremonies. They consume the 50 GB soft quota
|
||||
indefinitely; nothing prunes a moved-aside restic repo. **Delete, or leave?** This session did not
|
||||
act on it and states it as the operator's.
|
||||
3. **The unchanged R-193 question** — agent-retained repo password (spike option c) versus accepting
|
||||
that every rebuild starts a new repository. **This session adds one input to that decision:** the
|
||||
"accept it, provided the system says so loudly" branch currently rests on a retention feature that
|
||||
does not retain the relevant secret. Under option (d)-alone as it stands today, *"the old history
|
||||
stays recoverable with your recovery code"* is not true. Either R-198 is fixed, or that branch has
|
||||
to be re-described honestly to the customer.
|
||||
</content>
|
||||
</invoke>
|
||||
@@ -10,6 +10,39 @@ what it measured) · **Class:** spike — **no production code shipped for R-193
|
||||
|
||||
---
|
||||
|
||||
> ### ANNOTATION 2026-08-04 (later the same day) — read this before acting on §Q8
|
||||
>
|
||||
> A follow-up recon traced the escrow→recovery chain link by link
|
||||
> (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **Two corrections to this document, and they pull
|
||||
> in opposite directions.**
|
||||
>
|
||||
> **(1) Q8 candidate (b) — "not implementable as specified" — is TOO STRONG and is overturned in
|
||||
> part.** Zero-knowledge means the *hub* cannot open the blob; the *customer* can, with R, which is
|
||||
> what the ceremony exists for. The mechanism to hand a rebuilt box its old repo password back —
|
||||
> customer-present, not unattended — is a real design, and the operator has since ruled on its shape.
|
||||
> What (b) got right is narrower than what it said: **an UNATTENDED rebuild cannot recover from a
|
||||
> zero-knowledge escrow.** A customer-present one can. The recon prices that shape row by row.
|
||||
>
|
||||
> **(2) A LOAD-BEARING FACT THIS SPIKE DID NOT CHECK, and it makes (d)-alone weaker than ranked
|
||||
> here.** `host_escrow_superseded` **has no `identity_blob` column**, and `demoteCurrentEscrowTx`
|
||||
> (`hub/internal/store/store.go:2547-2556`) copies only the K-escrow `blob`. So the retained
|
||||
> "superseded" blob holds the **PBS datastore key**, not the **restic repository password**. The
|
||||
> retention feature does not retain the secret it was built to retain, and the **next escrow ceremony
|
||||
> — which the system tells the customer to run whenever a rebuild leaves the escrow `pending` — is
|
||||
> what destroys the last copy.** Both demo boxes crossed that line on 2026-08-04 at 07:15:36 and
|
||||
> 07:20:08. → **R-198**.
|
||||
>
|
||||
> **Consequence for this document's recommendation.** "Ship (d), then decide (c)" still holds, but
|
||||
> (d)'s implicit promise — *the old history stays recoverable with the recovery code* — is **false as
|
||||
> the code stands**, and is repeated verbatim to the customer in Hungarian on the orphan card. (d) is
|
||||
> not complete without R-198.
|
||||
>
|
||||
> **Unchanged and still correct:** Q1, Q2, Q4 (R-196), Q5, Q6, Q7 (R-192) and the Q8 (a)/(c)/(d)
|
||||
> analyses. **Q3 remains UNMEASURED** — neither box has run since; the 2026-08-05 ~02:15 UTC run is
|
||||
> still the decisive one.
|
||||
|
||||
---
|
||||
|
||||
## 0. Why this exists
|
||||
|
||||
Two candidate fixes were named when R-193 was filed, and neither could be weighed, because two prior
|
||||
@@ -412,6 +445,12 @@ re-keyed, `host_leaf_changed` 2026-08-03 06:09:40), so re-enroll returns the exi
|
||||
|
||||
### (b) The credential becomes recoverable from escrow at re-bootstrap
|
||||
|
||||
> **ANNOTATED 2026-08-04 (recon) — this ruling is overturned in part. See the annotation at the top.**
|
||||
> The constraint is real but narrower than written: an **unattended** rebuild cannot recover from a
|
||||
> zero-knowledge escrow; a **customer-present** one can, and the operator has ruled on that shape.
|
||||
> The recon also found that the chain this option would use is clientless at three links and lossy at
|
||||
> a fourth (`RECON-offsite-dr-chain-2026-08-04.md` §3–§4).
|
||||
|
||||
- **BLOCKED AS STATED, and this is a hard constraint, not an estimate.** The escrow blob is
|
||||
**R-wrapped and zero-knowledge** — the hub holds opaque bytes and has no recovery code
|
||||
(`api/handler.go`, and D6 in `CONTEXT.md` S-3). **A rebuilt box cannot unseal it without the
|
||||
|
||||
@@ -39,8 +39,12 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-195** | ~~**A customer with no machine ever bound e-mailed an `expected_dbdump_missed` ERROR every morning.** `david` — a real prospective customer whose record was created 2026-08-01 16:51:49 with **hosts=0, host_deletions=0, host_reports=0, reports=0** — raised the alarm at 03:00 UTC on 08-02, 08-03 and 08-04~~ | **SHIPPED** (hub **v0.92.0**, 2026-08-04) | — | **The mechanism, and it is the interesting half: the skip that protects every other silent customer is keyed off having reported at least once.** `CheckBackupDeadlines`' down-skip reads `StalenessChecker.GetState()`, whose map is seeded from `store.GetCustomers()` — a query **over the `reports` table**. A customer with zero reports appears in no row, gets no state, and `GetState()` returns `""` rather than `"down"`. The skip therefore misses exactly the customer it would most obviously cover. (Corroborated live: `peti-felhom` is active with a host deleted 2026-07-15 and **does not** alarm — it has 482 old reports, so it is `down` and skipped.) The backup half was already safe (`reportJSON == ""` → skip); the DB-dump half had no guard at all. **Fix:** `store.HasEverBoundHost(customerID)` = a live `hosts` row **OR** a `host_deletions` tombstone, consulted once per customer at the top of the deadline loop. **The discriminator is deliberately "was a host EVER bound", NOT "has a report arrived"** — a box that was installed and went silent is a real fault and must keep alarming; that is the case the change could break and it has its own test. Fail-**open** on a read error (an unreadable binding must never SUPPRESS a real alarm), and the deferral is LOGGED with its own counter (the v0.73.0 Part-7 precedent: a quiet check must not look like a check that did not run). The anchored-verdict structure is untouched. **Red-proof observed:** deleting the guard fails `TestCheckBackupDeadlines_NeverBoundHost_Silent` with `got [expected_dbdump_missed]` — verbatim the event `david` sent three mornings running. `david`'s record was NOT modified; the record was correct and the alarm was what was wrong | — |
|
||||
| **R-196** | **`escrow_stale` is wired to the ONE path that does not change the repo password, and absent from the path that does.** `ReissueCredentials` (`hub/internal/offsite/offsite.go:150-228`) resets **only** the Hetzner sub-account/box password and stages a fresh one-time secret — it contains no reference to a restic password and *cannot*, since that password is generated on the box and never leaves it except into the R-wrapped escrow. Yet it calls `MarkEscrowStale` on the stated grounds that *"the restic repo password just changed"* (`offsite.go:198-201`), and the same false premise is repeated at `api/handler.go:1067-1069` and in R-39's record | **OPEN** | — | **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting a guarantee the code does not provide** — and the first where the comment factually describes a *different function*. It survived because its EFFECT (a stale escrow) is real, so nobody checked its CAUSE. **The live defect, not a documentation nit:** on the ordinary Re-issue shape (a consumed-but-failed install on a box that still holds its `repo_password` file) the box re-applies, `WriteOffboxSecrets` finds the file present and **keeps it**, the repo password is unchanged — and the hub has told the customer in Hungarian that their recovery escrow is stale and asked them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.** **The inverse is the worse half and is R-193's:** demo-felhom's repo password *did* change on 2026-08-03 with **no Re-issue anywhere in its history** (its only `escrow_stale`/`offsite_reissued` pair is dated 2026-07-21 08:29:29) and therefore **nothing marked its escrow stale for 13 h**. **Fix shape:** mark the escrow stale on the evidence that it IS stale — a changed `restic_pw_sha256` (→ R-197) — not on a Re-issue; and correct all three comments in the same commit. **Not established, so not asserted:** whether the 2026-07-21 Re-issue on demo-felhom re-sealed an unchanged password (the predicted false-staleness shape) — `host_escrow_superseded` holds only two rows in the whole DB, both from 2026-08-04, so the prior generation is not retained. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q4 | CC |
|
||||
| **R-197** | **The hub holds both halves of the evidence that a box's offsite DATA key changed, and reads neither.** `restic_pw_sha256` is stored on `host_escrow` and carried to `host_escrow_superseded` on every re-escrow. Comparing the two is what let this spike answer its hardest question in one query — and **nothing in the hub does it** | **OPEN** | — | **Why this is the cheapest real fix on the table.** A changed repo password means the previous offsite repository is now unopenable by the box, i.e. the customer's off-site history is orphaned. That is the single most consequential state change in the backup system, it is **already fully observable from data the hub owns**, and today it produces **no event, no e-mail, no card and no log line** — demo-felhom's went unremarked for 13 hours and would have gone unremarked indefinitely had this spike not run. **Two-line verdict:** on `SaveHostEscrow`, if the incoming `restic_pw_sha256` differs from the row being superseded, emit a distinct operator event naming the orphaned generation. **Deliberately an EVENT, not a heal** — nothing should act on this automatically until R-193's (c)-vs-accept decision is taken; the point is that the operator learns on the day. **Pair with R-196**, which is the same signal aimed at the right trigger. **Generalises past this row:** *a comparison the system could be making from data it already stores, and is not, is a silence with no cost of entry* — cf. R-190's store-grant probe, where the state was read and the TRANSITION was not. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q8 option (d) | CC |
|
||||
| **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **OPEN** | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. **SPIKE 2026-08-04 — BOTH HALVES CONFIRMED WITH NUMBERS, still OPEN, still not fixed here** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q7). The query is quoted at source (`store.go:987`): `... ORDER BY id LIMIT 500` = **the oldest 500**. Reproduced against the live hub DB with demo-hp's real consume anchor `2026-07-23 09:53:41`: the guard sees `total=500, withOffsite=500`, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** — the entire evidence set ends **six days before** the 2026-08-03 rebuild. The true window totals are `1174 / 1063` (⇒ 111 without, matching the 111 offsite-less reports). So the e-mail's *"500 report(s) since carry no offbox target"* interpolates `ReportsSinceConsume` while `OffsiteReportsSinceConsume` was **500** — the message states the precise negation of its own measurement. `offsite_credential_restaged` has **never fired for any customer** (zero rows of that type in the DB — checked, not assumed). **A NEW REASON NOT TO FIX THIS IN ISOLATION, from the same spike:** under R-193's Q2 finding a successful auto-restage would have restoLine truncated
|
||||
| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **OPEN** | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTP access key…)"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the one a rebuild destroys, and **nothing automaLine truncated
|
||||
| **R-198** | **The hub's superseded-escrow retention does NOT retain the offsite repository password — and the ceremony the system tells the customer to run is what destroys the last copy.** `host_escrow_superseded` has **no `identity_blob` column**, and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`) copies only `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256`. `blob` is the **K-escrow** (the PBS datastore key, PBS-native scrypt); the **restic repo password lives in `identity_blob`** (`felhom-agent/internal/escrow/identity.go:34-39`, age-wrapped `IdentityBundle`). Measured live: both hosts' current rows hold `blob`=383 B **and** `identity_blob`=572 B; both superseded rows hold `blob`=383 B and nothing else | **OPEN** | — | **This is the NINTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does not provide, and the first that is ALSO customer-facing copy.** The claim appears three times: the schema comment (`store.go:370-375`, *"so the old passphrase stays customer-R-recoverable … turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable'"*), the capability map's escrow row, and — in Hungarian, to the customer, on the orphan card — `controller/internal/web/templates/backups_remote.html:66,69` (*„a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók lehetnek"*). **For the offsite restic repository, the incident it names, all three are false.** **Why it is worse than a missing column:** a rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`, `offbox.go:570`), the card says „Helyreállítási kód szükséges", and the controller's own detector logs *"run the escrow ceremony"* (`report/escrow_confirm.go:100`) — so **the prescribed remedy is the act that overwrites `host_escrow.identity_blob` and loses the old password forever**. Both demo boxes crossed that line on **2026-08-04 at 07:15:36 (demo-hp) and 07:20:08 (demo-felhom)**. **This is an independent, stronger reason the 51 orphaned demo snapshots are unrecoverable than "nobody kept the recovery codes" — keeping R would not have helped.** **Fix shape (small):** add `identity_blob` to the superseded table and to `demoteCurrentEscrowTx`'s SELECT list; pin it with a test that asserts the CONSEQUENCE (a superseded row can still yield a repo password) rather than the mechanism. **Then correct all three claims in the same commit** — including the Hungarian card, which must not promise what the system cannot do. **Prerequisite for R-193's (d)-alone branch** and for the drill. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §7 | CC |
|
||||
| **R-199** | **The hub serves recovery blobs on two endpoints that have no client anywhere in the system.** `handleReEnroll` (`hub/internal/api/dr.go:101`) and `handleGetRestoreDirective` (`:155`) return `identity_escrow_b64` + `k_escrow_b64`, gated on operator-armed recovery mode. Census: **zero** callers in `felhom-agent` (no `ReEnroll` symbol at all; `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`), **zero** in the hub UI or any template, **zero** in `scripts/` or any runbook | **OPEN** | — | **The documented retrieval path is a human with `sqlite3`:** `SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …` on a `kubectl cp`-ed `hub.db` — recorded in project memory as the 2026-07-04 S5 prep, where hub-side blob serving was explicitly deemed *"Part 3 NOT needed"*. **This is the built-but-never-wired class at the DR capstone**, and it is why the chain from a dead node to an open repository has no automatable middle. **What a recovery flow actually needs is smaller than what exists:** a narrow `GET /hosts/<id>/escrow` authed with the box's own per-host key, serving opaque bytes to the box that owns them — zero-knowledge untouched, and far lighter than `re-enroll`, which **rotates the host API key** and returns the new key in the response body (`dr.go:130,148`). **Decide before building:** whether `re-enroll`/`restore-directive` should get a client, be replaced by the narrow GET, or be retired. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 6 | CC |
|
||||
| **R-200** | **The DR password-injection seam has a handler, a route and tests — and no form.** `POST /backup/offbox/inject-password` is routed (`controller/internal/web/server.go:510`) to `offboxInjectPasswordHandler` (`offbox_handlers.go:174-196`) → `InjectOffboxPassword` (`backup/offbox.go:541`). **No template in the repository contains that path or any form posting to it** (grep over `internal/web/templates/`: one unrelated hit, an XSS comment) | **OPEN** | — | **The tenth instance of this project's built-but-never-wired class, and the exact shape `CLAUDE.md` and `felhom.eu/CLAUDE.md`'s seam-wiring rule were written for:** handler tests that POST directly (`offbox_escrow_test.go:167,180`) prove nothing about reachability. To use the only implemented recovery seam today, a person must hand-craft an authenticated POST with a session cookie and CSRF token. **Note the layering while fixing it:** this form takes a **64-hex repo password**, not a recovery code (`offboxRepoPwPattern`, `offbox.go:543`) — they are different secrets at different layers, and the operator's 2026-08-04 ruling asks for a form that takes **R**. Build the R form and treat this one as the operator/DR fallback it was written as, but **ship it with a render test per branch of whatever gate it sits behind**. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 9 | CC |
|
||||
| **R-201** | **Nothing in the offsite DR chain has ever been exercised past the ceremony — and the one live proof that exists predates the field it is cited for.** `slice10d-identity-restore-spike-findings.md` §1 (2026-06-10) proved a real wrap→unwrap of an identity bundle with a real R on a secret-less box, byte-identical, wrong-R failing closed. **That bundle was `{tunnel_token, pbs_token}`.** `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** — a month later — and is **unit-proven only** | **OPEN** | — | **Never exercised, in these words:** a blob has never been served to a box; a fork-4 bundle has never been unsealed with a real R outside a unit test (the only production caller is `--selftest=identity-consume`, `cmd/felhom-agent/main.go:2845`, reading R from `FELHOM_RECOVERY_CODE`); a recovered repo password has never been injected; an existing offsite repository has never been reopened with one; **no restore of any kind has ever been performed from a recovered secret.** The one live consume ever prepared (S5 Part 4-B, 2026-07-04) is still marked *"DEFERRED, not run"*. **`_recovery-inventory-2026-07-28.md` already recorded this accurately** (A.2.7, C.1 row 1, C-2) — this row exists so the gap has an owner and a closing condition, not only a description. **Closing condition = the drill** designed in `audits/RECON-offsite-dr-chain-2026-08-04.md` §10, whose pass condition is a **byte-identical sha256 of a sentinel file restored after a wipe** — explicitly NOT "the repository opened". **Do not run it before R-198 is fixed**: the drill would walk a chain that is missing a link everyone believed was there | CC + operator |
|
||||
| **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **OPEN** | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. **SPIKE 2026-08-04 — BOTH HALVES CONFIRMED WITH NUMBERS, still OPEN, still not fixed here** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q7). The query is quoted at source (`store.go:987`): `... ORDER BY id LIMIT 500` = **the oldest 500**. Reproduced against the live hub DB with demo-hp's real consume anchor `2026-07-23 09:53:41`: the guard sees `total=500, withOffsite=500`, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** — the entire evidence set ends **six days before** the 2026-08-03 rebuild. The true window totals are `1174 / 1063` (⇒ 111 without, matching the 111 offsite-less reports). So the e-mail's *"500 report(s) since carry no offbox target"* interpolates `ReportsSinceConsume` while `OffsiteReportsSinceConsume` was **500** — the message states the precise negation of its own measurement. `offsite_credential_restaged` has **never fired for any customer** (zero rows of that type in the DB — checked, not assumed). **A NEW REASON NOT TO FIX THIS IN ISOLATION, from the same spike:** under R-193's Q2 finding a successful auto-restage would have restoLine truncated
|
||||
| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **OPEN** | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTP access key…)"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the one a rebuild destroys, and **nothing automaLine truncated
|
||||
| — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC |
|
||||
| — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator |
|
||||
| **R-90** | ~~ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged~~ | **CLOSED — the operator rescaled ep0 to a CX33 on 2026-08-03** | — | **MEASURED ON THE BOX, not read from an invoice:** `felhom-hetzner` reports `Mem: 7757` MB total (**8 GB**, was 3.8) and `nproc` **4**. **The interim lever survived and was checked rather than assumed** — a resize is a stop/start, so "the swapfile is still there" was an assumption until measured: `/swapfile`, 4 GiB, dated `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`), 0 B in use on an idle box. **THE 40 GB LOCAL DISK DID NOT CHANGE** and must not be "corrected" alongside the RAM: `/` is 38 G, 58% used. This was a CPU/RAM resize only, so every disk figure in the runbooks still stands — the separate 98 G volume at `/mnt/pbs-datastore` (R-82 P0.3) is unaffected. **Why this was BLOCKED and no longer is:** the row recorded CX33 as *"confirmed unavailable even powered OFF"* — the Cost-Optimized line's limited availability, not a power-state problem. It became available and the operator took it. **Documentation corrected** (`RUNBOOK-ep0-datastore-volume`, `RUNBOOK-pbs-prune-serverside` ×2, `runbooks/offsite-endpoint.md` ×2, `runbooks/target-selection.md`) and **audit/evidence documents ANNOTATED, not revised** (`SPIKE-connectivity-wireguard-2026-07-03`, campaign-10 `phaseA-journal`) — they record what was true when written and that is their value. **Still open and still the operator's, deliberately untouched:** `target-selection.md`'s *"D-d did not name ep0 either way. Confirm it explicitly."* | — |
|
||||
|
||||
Reference in new issue
Block a user