From 6362bb6cb6ec86bbb737264b75c0833bf1dfee4e Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 12 Aug 2026 18:43:07 +0200 Subject: [PATCH] =?UTF-8?q?hub=20v0.103.0=20=E2=80=94=20a=20host=20can=20r?= =?UTF-8?q?ead=20the=20packages=20we=20kept=20for=20it=20(R-311)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ListSupersededEscrow had zero production callers for nineteen days. It is the only reader of a retained identity_blob, so the retention shipped in v0.93.0 was material the product could not reach - proven on the fixture 2026-08-12, where a code that opens a retained package was answered as a code that opened nothing. New GET /api/v1/hosts//escrow/retained: self-scoped exactly as the current-row GET, same recovery-mode gate, same audit event written BEFORE the bytes leave, capped at 16. Rows with a NULL identity_blob are WITHHELD and returned as unopenable_count. They retain the PBS key, not the repository password, so they can never open what the caller is asking about; serving them would have the agent try packages that cannot succeed and would let the screen claim an earlier package is openable on exactly the boxes the original defect hurt. The count is returned because their existence is load-bearing and underivable. The trade, stated rather than waved through: the hub still cannot read any of it - sealed bytes in, sealed bytes out, no decrypt path, no recovery code ever held. What widens is volume, bounded by self-scope, the recovery-mode gate and the cap. The response is a NAMED TYPE, not a map, so the wire-contract gate can resolve it; the wire is declared as a fourth ROOT and the gate now checks 182 tags rather than 174. A positive control shows that check is name-presence, not decodability - filed as R-315 rather than reported as coverage. Six tests through the real endpoint; four red-proofs asserted applied. --- STATUS.md | 35 +-- .../architecture/00-capability-map.md | 2 +- documentation/backlog/OPEN-ITEMS.md | 7 +- hub/CHANGELOG.md | 32 +++ hub/internal/api/escrow_retained_get_test.go | 218 ++++++++++++++++++ hub/internal/api/handler.go | 159 +++++++++++++ scripts/wire_contract_gate.py | 6 + 7 files changed, 443 insertions(+), 16 deletions(-) create mode 100644 hub/internal/api/escrow_retained_get_test.go diff --git a/STATUS.md b/STATUS.md index a3871c2..56d4b70 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,6 +1,6 @@ # STATUS — what works, what's broken, what's next -**Updated 2026-08-12 (late evening — retained-key drill).** +**Updated 2026-08-12 (night — the door, part one).** > **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates > part of it in plain words, and **nothing may exist only here**. **Items, not paragraphs. One screen.** @@ -11,12 +11,13 @@ *(The golden-vouch and floor-raise asks that stood here are **done** — the hub reads golden **0.213.0**, floor **0.213.0**, agent **0.128.0**. Checked live, not assumed.)* -- **R-307 — a countdown is already running on `demo-felhom`**, started 10 Aug, **firing 24 August**, - after which that set-aside off-site copy is deleted for good. Today's drill did not start it and - deliberately did not cancel it. Cancel it (copy kept, costs storage) or let it run (copy deleted). - **Doing nothing chooses deletion.** -- **R-304 — decide what we promise about old backups.** The retention works, but only an operator can - use it; the customer is told their correct code is wrong. Either build the door, or stop promising. +- **R-312 — one decision, and it is the interesting one.** The customer is now told the truth about an + older code, but there is still no button: restoring from a set-aside copy means either threading an + alternative location through the restore code, or adopting that copy as the machine's current one. + The second is a different product decision. **Nothing is broken while this waits.** +- **R-313 — the copy you told me to keep cannot be opened by anyone.** `demo-felhom`'s set-aside store + holds 36 snapshots and one key, and that key was destroyed by the bug we fixed on 4 August. Keeping + it is still the right call; it should be a decision, not an accumulation. - **R-303 — one coherence decision**, ranked low: a box can still raise the "we cannot open your new backups" card while counting down to deleting the old ones. The two no longer contradict each other, but the state is odd and the wrong fix would hide a real second fault. @@ -39,6 +40,12 @@ Both demo machines are home, healthy and reporting on the approved pair. Off-sit fifth deliberately left, because it is true where it renders. - **One name per secret, box side** (R-295): the dashboard code is „Beállító kód" everywhere; „Visszaállító kód" is retired. It collided with the escrow „Helyreállítási kód" and cost a real code. +- **A correct recovery code is no longer called wrong** (R-311, three components). If a customer types + the code for an older set of backups, the machine now checks the packages we kept, recognises it, and + says so: *your code is correct, it belongs to an earlier package, we kept it, your current backups are + fine, write to us*. It deliberately promises no restore, because there is no button yet. +- **The countdown on `demo-felhom` is cancelled** on your ruling (R-307). Nothing was deleted; the + 24 August deadline is gone. See R-313 for what that copy turns out to be. - **Both installer fixes are now PUBLISHED** as `installer-v1.27.0` (R-297 + R-300). Each fault was watched happening first, on a machine reset to factory state: the old installer really did build a machine on a base image from July, and our own uninstall really did block our own next install. @@ -50,11 +57,10 @@ Both demo machines are home, healthy and reporting on the approved pair. Off-sit everything on it is lost.** First act of the visit: copy the ~3.6 GB off before anything is reinstalled — it is currently the only copy in existence. Whether it stays parked is your call and is deliberately left open. -- **Kept backups can be opened — but not by the customer** (R-304). We now know for certain the old key - is kept and **does** open the old backups: three planted files came back byte-for-byte from a store the - machine itself could no longer read. But nothing in the product reads a kept key, so it takes an - operator with a shell. Worse, a customer who types their **correct** old code is told it *"did not open - the sealed bundle"*. The keeping is real; the promise that they can use it is not. +- **Kept backups can be opened — but still only by us** (R-304 partly closed, R-312 open). The machine + now recognises an older code and says so plainly instead of hedging. What it still cannot do is hand + the customer their old files: that needs the restore code to accept a second location, which is real + work rather than wiring. Today the honest answer is "your code is right, write to us" — and we can. - **The dnsmasq fix helps a machine once** (R-305). On a machine that never had Felhom it works. On the second reinstall the leftover comes back, because the package is never removed — so the machine looks, to our own installer, as if the household had installed it. Watched happening the same afternoon. @@ -64,5 +70,6 @@ Both demo machines are home, healthy and reporting on the approved pair. Off-sit ## Working on next -Your two decisions above (R-307's deadline is the hard one); then R-304's shape; then the hub naming; -then the 2026-08-09 batch (R-279 … R-292), still untriaged against everything since. +R-312's shape (the button, or deliberately no button); then R-305, because the tester's second +reinstall still hits the dnsmasq wall; then the hub naming; then the 2026-08-09 batch +(R-279 … R-292), still untriaged against everything since. diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index f3b4eb7..922eb53 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -36,7 +36,7 @@ | Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`.rootpw.txt`) | scripts v1.24.0 (`scripts/iso/felhom-bootstrap.sh` `network_gate`/`sweep_nics`, `build-felhom-iso.sh` rootpw emission) | **PROVEN-LIVE (nested drill — nested ≠ metal: metal proof rides the next real multi-NIC install)** | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` — dead-NIC install from the virgin v1.24.0 ISO baked the 192.168.100.2 fallback (WITH a dead default gateway), the R-59 screen painted on the console (screendump captured), and after the cable move the box swept to the working NIC, re-leased and **self-registered at the hub unaided in under a minute**; the drill also caught + fixed the stale-fallback-route trap (flush before the bounded dhclient) and verified the emitted rootpw against the installed box's shadow hash | R-59 ships as a first-boot gate, not an install-time abort (recorded deviation — the fallback is the auto-installer's own, initrd hook out of scope); sweep is structurally first-boot-only (`state.json` gate + unit done-flag condition); a box past install-start gets the screen but its interfaces are never touched | | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. **Deliverability (R-4), gmail half DONE 2026-07-18:** the rehearsal's claim email was the first sent under the tightened DMARC `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu remains Viktor's open half.** (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) | | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends | -| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention) — ⚠ the claim as written was FALSE for the offsite tier for two months; **FIXED in hub v0.93.0 (2026-08-04), and the row below states what ships TODAY:** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`) — and **since v0.93.0 the retained row carries `identity_blob` as well as the K-escrow blob**, so a previous passphrase does now stay recoverable with the recovery code that sealed it — **OPERATOR-ONLY, and the customer-facing half of that sentence is FALSE (R-304, drill 2026-08-12).** The retention was exercised end-to-end for the first time that day and **it works**: the retained row carried the identity blob byte-identically (`sha256 a10032341c8584ed…`), the old recovery code unsealed it, and three planted files — including a Hungarian accented filename verified as raw bytes — restored **byte-identical** from a store the box itself could no longer open (negative control first: `Fatal: wrong password or no key found`). **What does not exist is the door.** `ListSupersededEscrow` (`hub/internal/store/store.go:2841`) is the only reader of a retained `identity_blob` and has **zero production callers**; the product's recovery path (`POST /escrow/recover-offsite-password` → `GetHostDRBundle`, `store.go:3152`) selects `FROM host_escrow` — the CURRENT row only. Asked with the code that demonstrably opens the retained row, the product answers **"the recovery code did not open the sealed bundle"**. So: recoverable by an operator with SQLite, `age` and a shell; **not recoverable by the customer, who is told their correct code is wrong.** **What was wrong until v0.93.0, recorded because it is the ninth entry in `CLAUDE.md`'s comment-vs-code table and the first that was also customer-facing copy:** `host_escrow_superseded` had no `identity_blob` column and `demoteCurrentEscrowTx` did not copy one, so what survived a supersession was the **PBS datastore key only** — never the restic repository password, which lives in `identity_blob`. The destroying act was the **escrow ceremony** a rebuilt box asks its customer to run. Measured live 2026-08-04, before the fix: both current rows held `blob`=383 B **and** `identity_blob`=572 B; both retained rows held `blob`=383 B only. → **R-198** (SHIPPED), evidence `audits/RECON-offsite-dr-chain-2026-08-04.md` §7. **THREE SCOPE LIMITS THIS ROW MUST NOT BE READ PAST.** (1) **Nothing was backfilled and nothing could be** — rows superseded before v0.93.0 were written without the blob and their source rows are already overwritten; both demo boxes' pre-2026-08-04 repository passwords are gone permanently. (2) **A retained key is not a restore — but as of 2026-08-04 evening it IS a recovered key.** See the row below. (3) The customer-facing orphan card still promises recoverability unconditionally (R-202, gate hit, card untouched). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test | +| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention) — ⚠ the claim as written was FALSE for the offsite tier for two months; **FIXED in hub v0.93.0 (2026-08-04), and the row below states what ships TODAY:** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`) — and **since v0.93.0 the retained row carries `identity_blob` as well as the K-escrow blob**, so a previous passphrase does now stay recoverable with the recovery code that sealed it — **OPERATOR-ONLY, and the customer-facing half of that sentence is FALSE (R-304, drill 2026-08-12).** The retention was exercised end-to-end for the first time that day and **it works**: the retained row carried the identity blob byte-identically (`sha256 a10032341c8584ed…`), the old recovery code unsealed it, and three planted files — including a Hungarian accented filename verified as raw bytes — restored **byte-identical** from a store the box itself could no longer open (negative control first: `Fatal: wrong password or no key found`). **What does not exist is the door.** `ListSupersededEscrow` (`hub/internal/store/store.go:2841`) is the only reader of a retained `identity_blob` and has **zero production callers**; the product's recovery path (`POST /escrow/recover-offsite-password` → `GetHostDRBundle`, `store.go:3152`) selects `FROM host_escrow` — the CURRENT row only. Asked with the code that demonstrably opens the retained row, the product answers **"the recovery code did not open the sealed bundle"**. So: recoverable by an operator with SQLite, `age` and a shell; **not recoverable by the customer, who is told their correct code is wrong.** — **AMENDED 2026-08-12 evening (R-311, shipped hub v0.103.0 + agent v0.129.0 + controller v0.214.0):** the customer is **no longer told their code is wrong.** The agent now tries the retained packages when the current one refuses (`GET /hosts//escrow/retained` → `ErrCodeOpensRetained` → HTTP 422), and the screen says the code is CORRECT, names the supersession date, says the earlier package is kept and the current backups are unaffected, and routes to support. **What is still true and must not be read away:** there is no in-product ROUTE to the set-aside data (R-312 — every restore entry point resolves its repository from settings and its password from one file; adding an alternative is new surface, not wiring), so the recovery itself remains operator-performed. The status of this capability is therefore **"the customer is told the truth and handed to a human"**, not "the customer can recover their old history". **What was wrong until v0.93.0, recorded because it is the ninth entry in `CLAUDE.md`'s comment-vs-code table and the first that was also customer-facing copy:** `host_escrow_superseded` had no `identity_blob` column and `demoteCurrentEscrowTx` did not copy one, so what survived a supersession was the **PBS datastore key only** — never the restic repository password, which lives in `identity_blob`. The destroying act was the **escrow ceremony** a rebuilt box asks its customer to run. Measured live 2026-08-04, before the fix: both current rows held `blob`=383 B **and** `identity_blob`=572 B; both retained rows held `blob`=383 B only. → **R-198** (SHIPPED), evidence `audits/RECON-offsite-dr-chain-2026-08-04.md` §7. **THREE SCOPE LIMITS THIS ROW MUST NOT BE READ PAST.** (1) **Nothing was backfilled and nothing could be** — rows superseded before v0.93.0 were written without the blob and their source rows are already overwritten; both demo boxes' pre-2026-08-04 repository passwords are gone permanently. (2) **A retained key is not a restore — but as of 2026-08-04 evening it IS a recovered key.** See the row below. (3) The customer-facing orphan card still promises recoverability unconditionally (R-202, gate hit, card untouched). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test | | **The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code** | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE (2026-08-04)** | On demo-felhom, through the real endpoints end to end: the box fetched its own sealed blob from the hub with its own per-host credential (hub log: *escrow blob SERVED … 572 opaque bytes, self_scope=true*), the agent unsealed it with the customer's recovery code, and the extracted repository password's sha256 was **byte-identical** to the one on disk — `c60c8bc737a6…`, which is ALSO the hash the hub had independently stored, so three sources agree. Five minutes earlier the same path with a WRONG code failed closed at age's KDF with nothing written, which proves links 6 and 7 ran independently of the success. R was searched for afterwards and found in 0 log lines and 0 files, with a positive control confirming the search would have found it. Evidence: per-repo CHANGELOGs; `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 links 6–8 | **WHAT THIS ROW DOES NOT CLAIM, stated because the previous over-claim here was struck out four hours earlier.** It covers the KEY, not the DATA. **R-199 BACK-POINTER (omitted when this row was written): the recovery chain's link inventory and the per-link status live in `audits/RECON-offsite-dr-chain-2026-08-04.md` §3; links 1–8 are walked, 9–11 are not.** **Re-confirmed 2026-08-04 evening by an attempt to prove the DATA half:** the R-201 drill was prepared on demo-hp and **halted before the wipe** — the sentinel file was not in the off-site snapshot (R-203), so the wipe would have destroyed it and proven nothing. **No file has still ever been restored from an off-site backup after a wipe** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). The install half of the chain (controller v0.196.0 `--recover-offsite-install`, R-200) is likewise unit-proven only — it has never run against a live recovery. A recovered password has never been **installed** (the diagnostic compares and refuses to write, by design), no existing repository has ever been **reopened** under one, and **no file has ever been restored** from an off-site history via a recovered key. Links 9–11 of the chain are open (R-200's remaining half, R-201). The proof also used a box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is exactly what the drill covers and it has not run || DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 90e311e..6f526c2 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -607,7 +607,12 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-304** | **The retained escrow key works, and the customer is told their correct code is wrong.** DRILL 2026-08-12 answered the three questions separately, on `demo-felhom`, with planted data. **(a) retention: WORKS** — the first retained row in fleet history to carry material (`host_escrow_superseded` id 11, `identity_blob` 572 B), byte-identical (`sha256 a10032341c8584ed…`) to the pre-supersession `host_escrow` row. **(b) the material opens the old store: YES** — unsealed with the OLD recovery code it yielded a password byte-identical to the pre-change one (`sha c60c8bc737a6b7c6…`), and restored three planted files **byte-identical** from a store the box itself could no longer open (negative control first: `Fatal: wrong password or no key found`), **including a Hungarian accented filename verified as raw bytes**. **(c) the customer's route: DOES NOT EXIST, and misinforms.** `ListSupersededEscrow` (`store.go:2841`) is the only reader of a retained `identity_blob` and has **zero production callers** — five call sites, all `_test.go`; the product path (`POST /escrow/recover-offsite-password` → `FetchIdentityEscrow` → `GetHostDRBundle`, `store.go:3152`) selects `FROM host_escrow` — the CURRENT row only. Asked for the old password with the code that demonstrably opens the retained row, the product answered **"the recovery code did not open the sealed bundle — nothing was written"**. **This is the R-224 class again**: there an unreachable hub was reported as a bad code; here a VALID code for retained history is reported as a bad code, and the customer's attempt ends there. **Consequence:** the census answer stands (it was about retention); the countdown banner's promise is true in substance and false in practice; **any capability-map claim that the customer can recover the old history with their recovery code is false today and must move** | **READY (L) — NEW 2026-08-12, RANK 1** | R-198, R-199, R-224, R-241 | Decide the shape: serve retained rows on the recovery path (needs a "which package?" choice — a customer may have several), or stop promising retrieval anywhere the customer cannot perform it. **Until one of those, the honest position is that retention is an operator-only capability.** At minimum, the refusal must stop asserting the code is wrong when the hub simply never looked | operator + CC | | **R-305** | **The R-300 cleanup fires exactly once per machine, and the second reinstall hits the original wall.** PROVEN on `drill-r50` 2026-08-12, not deduced. The uninstall stops+disables dnsmasq but deliberately never **purges** the package, so on every later install `dpkg-query` reports it installed, preflight records `dnsmasq_preexisting=yes`, the agent's `EnsureDnsmasq` re-enables it, and the uninstall then only *restarts* it — leaving it `enabled`/`active` on `0.0.0.0:53` exactly as before the fix. Cycle 1: `no` → disabled → `:53 FREE` → next install PASSES. Cycle 2: `yes` → left running → **cycle-3 byo preflight FAILED, exit 1**. The improved refusal does fire and names the command, so the customer is not stranded — but they must run by hand a cleanup Felhom could have run itself, on a box Felhom does own | **READY (M) — NEW 2026-08-12, RANK 2** | R-300 | Distinguish "the package pre-dates Felhom" from "a previous **Felhom** install left it" — e.g. keep the ownership record outside `state.json` (which the uninstall deletes), or record Felhom-installed-ness where the agent installs it. **Do not fix by purging on hosts we may not own** | CC | | **R-306** | **`--preflight-only` says "no state written" and writes state — with an answer that can be wrong.** `_state_put` short-circuits on `DRY_RUN` only (`felhom-host-install.sh:418`), so a preflight-only run creates `/var/lib/felhom-install/state.json`. Observed live: after a run whose banner read `PRE-FLIGHT PASS (mode=byo) — no state written, no install step executed`, the file existed containing `{"completed": [], "dnsmasq_preexisting": "yes"}`. Both the banner and the flag's own comment at line 226 assert the opposite. **The harm is not the file, it is the value**: the runbook recommends preflight-only first, then the same command without the flag, so on a box carrying a Felhom leftover the wrong ownership answer is baked in before the real install begins | **READY (S) — NEW 2026-08-12, RANK 3** | R-300, R-305 | Either make `_state_put` a no-op under `PREFLIGHT_ONLY` (and record ownership at install instead), or correct both claims. A comment asserting an invariant needs a test pinning it | CC | -| **R-307** | **`demo-felhom` carries a LIVE abandon countdown that this drill did not start — and the end state says there should be none.** `settings.json`: `abandon_started_at 2026-08-10T08:06:31Z`, `abandon_at 2026-08-24T08:06:31Z`, `abandon_repo_path /home/felhom-repo.orphaned-20260810`. The drill's fence forbade starting, shortening or triggering a countdown, and none was; but its required end state was *"no abandon countdown anywhere"*, and one exists. **The two ways to satisfy that are not equivalent and the choice is the operator's:** cancel it (the orphaned repository is kept indefinitely — storage cost, no data risk) or let it run (on **2026-08-24** that repository is deleted, **irreversibly**). **Doing nothing selects deletion by default.** Deliberately not resolved by CC | **BLOCKED ON OPERATOR — NEW 2026-08-12** | R-301, R-302, R-303 | Decide before 2026-08-24 | operator | +| **R-307** | **`demo-felhom` carries a LIVE abandon countdown that this drill did not start — and the end state says there should be none.** `settings.json`: `abandon_started_at 2026-08-10T08:06:31Z`, `abandon_at 2026-08-24T08:06:31Z`, `abandon_repo_path /home/felhom-repo.orphaned-20260810`. The drill's fence forbade starting, shortening or triggering a countdown, and none was; but its required end state was *"no abandon countdown anywhere"*, and one exists. **The two ways to satisfy that are not equivalent and the choice is the operator's:** cancel it (the orphaned repository is kept indefinitely — storage cost, no data risk) or let it run (on **2026-08-24** that repository is deleted, **irreversibly**). **Doing nothing selects deletion by default.** Deliberately not resolved by CC. **RULED 2026-08-12 (operator): KEEP the set-aside history — cancel the countdown.** Done the same evening through the product's own operator path (`--abandon-stop`, which refuses rather than silently no-opping), with the container stopped first so the running controller could not overwrite `settings.json` from memory. **Proved, not trusted to the exit code:** `abandon_started_at` and `abandon_at` are GONE (`AbandonStatus` returns `Active=false` when `AbandonAt` is empty — `offbox_abandon.go:111-113`, so no countdown renders); `abandon_repo_path` deliberately REMAINS as the pointer to the preserved store; the store still holds **36 snapshot objects** and its full `config/data/index/keys/locks/snapshots` structure; both repositories still present on the endpoint; **nothing deleted anywhere**. See R-313 — what was preserved cannot currently be opened by anyone | **CLOSED — countdown cancelled 2026-08-12 on the operator's ruling** | R-301, R-302, R-303, R-313 | — | CC | | **R-308** | **The stored controller password no longer opens `demo-felhom`.** `PASSWORD` in `~/.config/credentials` on DooPlex returns `Hibás jelszó` against `https://127.0.0.1:443/login` with `Host: felhom.demo-felhom.eu`. **NOT the known quoting trap** — the surrounding quotes were stripped before use and the stored value is unquoted (16 bytes for 15 chars), so this is a genuinely stale credential, not a mis-read. Cost during the drill: the dashboard was unreachable headlessly, so the offsite push had to be made with `restic` directly instead of the page's own button, which left the app-backup→unit→offsite chain unexercised. `R_DEMO-FELHOM` **was** correct and has since been rotated by this session's final ceremony (updated in place; prior file backed up alongside) | **READY (S) — NEW 2026-08-12, RANK 2** | — | Recover or reset the demo-felhom dashboard password and re-store it; note which boxes share one and which do not | operator | | **R-309** | **The day-0 runbook says pushing the installer publishes it. It has not since R-110.** `documentation/runbooks/day0-install.md` §C.1: *"it is always the current `main` script. There is no release tag, no staging copy and no version selector; pushing `scripts/felhom-host-install.sh` publishes it."* R-110 moved `/scripts/` to the **installer tag's** tree — `manifests/webpage.yaml` carries **two** `--ref=installer-v…` pins (lines 327, 372) and `hostinstall_gates.py` gate 6 enforces that they name a tag. Confirmed live 2026-08-12: `https://felhom.eu/scripts/felhom-host-install.sh` served `SCRIPT_VERSION="1.25.0"` while `main` had `1.27.0`, three and a half hours after the push. **The runbook's version is the dangerous one to believe in both directions** — it would have an operator think a pushed fix is live when it is not, and think a pushed mistake is live when it is not | **READY (S) — NEW 2026-08-12, RANK 3** | R-110 | Correct §C.1 to name the tag-and-manifest publication step (as performed today) | CC | | **R-310** | **Two small edges on the installer, neither costing more than a moment.** (1) The R-297 operator-named refusal states the vouched version twice in consecutive sentences (*"…but the vouched golden is 0.213.0. The vouched golden is 0.213.0."*). (2) `--uninstall` reads its typed vmid confirmation from `/dev/tty` and `--force` deliberately does **not** bypass it, so teardown cannot be scripted without a pty — correct for an irreversible destroy, but undocumented; it surfaces as `line 891: /dev/tty: No such device or address` and an rc=1 that looks like a failure rather than a refusal to proceed unattended | **READY (S) — NEW 2026-08-12, RANK 4** | R-297 | Drop the duplicated sentence; add one runbook line naming the pty requirement | CC | +| **R-311** | **A correct recovery code for a retained package stopped being reported as wrong.** The drill of 2026-08-12 proved the retained key WORKS (it opened a set-aside store and restored planted files byte-identical) while the product answered *"the recovery code did not open the sealed bundle"*. **What was already right:** the customer-facing screen did not bluntly accuse — R-222/R-226 hedged, naming both causes and the kept package. **What was wrong was the hedge's reason:** *„innen nem tudjuk megkülönböztetni őket"* was true only because NOTHING EVER LOOKED. **Shipped, three repos.** Hub v0.103.0: `GET /hosts//escrow/retained` — the first production caller `ListSupersededEscrow` has ever had — self-scoped, same recovery-mode gate, same audit event before the bytes leave, rows with a NULL `identity_blob` WITHHELD and returned as `unopenable_count` (they can never open anything, and serving them would let the screen promise recovery on exactly the boxes the original defect hurt), capped at 16. Agent v0.129.0: `FetchRetained` consulted ONLY after the current package refuses, `ErrCodeOpensRetained` + the supersession date, local API answers **422**, bounded at 6 attempts (~1 s of scrypt each), and fail-safe in every direction — a nil fetcher, a 404 from an older hub, a transport failure or a malformed package all leave the original refusal standing. Controller v0.214.0: class `RecoveryCodeOpensRetained` gated on `FeatureRetainedRecoveryClass` (MinAgent 0.129.0, SEPARATE from the R-224 gate because a box can sit between 0.126.0 and 0.129.0), and a message that says the code is correct, names the date, says the earlier package is kept, says the CURRENT backups are unaffected, and **deliberately promises no restore** — it routes to support, which can do it by hand. **The lie's true home was the agent:** removing the CONTROLLER case yields the neutral message (R-224's safe default catches it); removing the AGENT's retained lookup brings back the fail-closed wrong-code error in those exact words — both demonstrated, mutations asserted applied | **CLOSED — shipped hub v0.103.0 + agent v0.129.0 + controller v0.214.0** | R-199, R-222, R-224, R-226, R-304 | — | CC | +| **R-312** | **There is no in-product route from the recovery screen to a set-aside store, and building one is not wiring — it is new surface.** Established read-only before any code was written (the session's §4 spike). Every restore entry point resolves the repository from `m.settings.GetOffboxTarget()` and the password from the single `offboxPwPath()` file: `offboxLatestSnapshot` (`offbox_restore.go:85-86`), `offboxSnapshotSize` (`:139-140`), `RestoreOffboxScratch` (`:206+`). **There is no repo-path parameter anywhere in the chain** — a grep for one returns nothing. The only existing seam that installs a recovered password, `InjectOffboxPassword` (`offbox.go:667`), writes that same one file, i.e. **ADOPTS** the set-aside store as the machine's current target. So the two options are (a) thread an alternative (repo, password) through three functions plus the UI, or (b) adopt — and adoption is a different product decision. **The session HALTED here by its own rule** and shipped R-311 alone. What the drill did to read the set-aside store was `restic` by hand with `-r ` and an overridden `RESTIC_PASSWORD_FILE`; **that distance is exactly what (b)-to-(c) costs** | **READY (L) — NEW 2026-08-12** | R-304, R-311 | Decide (a) vs (b). If (a): the parameter is small and contained; the UI question ("which package?" when a customer has several) is the real design work. **Do not choose (b) without deciding what happens to the machine's current off-site history** | operator + CC | +| **R-313** | **`demo-felhom`'s set-aside store is UNRECOVERABLE — 36 snapshots whose key we destroyed ourselves.** `/home/felhom-repo.orphaned-20260810` holds **36 snapshot objects and exactly one key slot**, and it does NOT open with the box's current password (`Fatal: wrong password or no key found`, exit 1 — measured). Its password is the one hashed `48741892f0ef4d59…`, which is retained row id 4 — **`identity_blob` NULL**, a pre-v0.93.0 row. So the material was dropped by the R-198 defect during its two-month window, and no recovery code in existence opens that store. **This is the concrete, still-present cost of R-198**, sitting on the endpoint rather than in a post-mortem. It also means the operator's ruling to KEEP it (R-307, countdown cancelled — see below) preserves bytes nobody can read: correct as a decision, and worth knowing as a fact | **READY (S) — NEW 2026-08-12** | R-198, R-307 | Decide whether to keep paying storage for 36 unreadable snapshots, or delete them deliberately and record why. **Not urgent and not automatic — but it should be a decision rather than an accumulation** | operator | +| **R-314** | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route | **READY (S) — NEW 2026-08-12, RANK 3** | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC | +| **R-315** | **The wire-contract gate's positive control FAILS on the new wire: it checks name-presence, not decodability.** R-311 declared `hub -> agent (GET /escrow/retained)` as a fourth ROOT, and the gate's tag count rose 174 → 182, so the fields ARE inspected. But renaming the agent-side `superseded_at` json tag to `superseded_at_RENAMED` **still passed** — because the string `superseded_at` also occurs as a map key in the agent's local-API response, and the check is a repo-wide name search. The gate documents this ("name-reachability is not use"), so it is a known limit rather than a regression — but it means **declaring this wire bought documentation, not enforcement**, and a report that claimed coverage would have been wrong. The mutation was asserted to have applied before the run | **READY (M) — NEW 2026-08-12, RANK 3** | R-311 | Make the check resolve the RECEIVER'S mirror type and compare field-by-field, or state per-root which kind of check it got. **A gate whose positive control fails is an instrument nobody has calibrated** | CC | diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 35a2fa1..14a6464 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,35 @@ +## v0.103.0 — a host can read the packages we kept for it (2026-08-12, R-311) + +**`ListSupersededEscrow` had zero production callers for nineteen days.** It is the only reader of a +retained `identity_blob`, and nothing called it — so the retention shipped in v0.93.0 was material the +product could not reach. Proven on the fixture 2026-08-12: a recovery code that opens a retained +package (unsealed by hand, and it restored planted files byte-identical from a store the box itself +could no longer open) was answered by the product as a code that opened nothing. + +**New: `GET /api/v1/hosts//escrow/retained`** — the retained siblings of the existing +`/escrow` row, self-scoped to the asking host exactly as that one is, behind the same §8.3 +recovery-mode gate, and audited with the same `escrow_blob_served` event BEFORE the bytes leave. + +**What it will not serve.** Rows whose `identity_blob` is NULL — every pre-v0.93.0 row — are withheld +and returned as `unopenable_count` instead. They retain the PBS datastore key and not the repository +password, so they can never open what the caller is asking about; serving them would have the agent +try packages that cannot succeed and would let the recovery screen claim an earlier package is +openable on exactly the boxes the original defect hurt. The count is returned because their EXISTENCE +is load-bearing and underivable: on such a box a correct old code opens nothing, and the honest reason +is a defect of ours rather than the customer's typing. + +**The trade, stated rather than waved through.** The hub still cannot read any of it — sealed bytes in, +sealed bytes out, no decrypt path, no recovery code ever held. What widens is VOLUME: a host key that +could fetch one opaque package can now fetch N. Bounded by the same self-scope, the same recovery-mode +gate, and an explicit `retainedEscrowServeCap` (16) so a long supersession history cannot become an +unbounded read. + +Pinned by six tests driven through the real endpoint, with four red-proofs each asserted to have +applied: serving the CURRENT row instead (count 2→1), dropping the unopenable guard (count 1→2, +unopenable 1→0), dropping self-scope (403→200), and collapsing the route suffix (count 1→0). + +--- + ## v0.102.0 — the hub refuses to vouch a version that cannot be installed (2026-08-09, R-273) **The guard owed since 2026-08-09 morning.** Agent v0.128.0 had been published as a package and never diff --git a/hub/internal/api/escrow_retained_get_test.go b/hub/internal/api/escrow_retained_get_test.go new file mode 100644 index 0000000..c615dad --- /dev/null +++ b/hub/internal/api/escrow_retained_get_test.go @@ -0,0 +1,218 @@ +package api + +import ( + "encoding/base64" + "encoding/json" + "net/http" + "testing" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// R-311 (v0.103.0) — GET /hosts//escrow/retained. +// +// WHY THESE TESTS EXIST. `ListSupersededEscrow` had zero production callers for nineteen days; the +// retention it reads was proven on 2026-08-12 to hold the right material and to open a store the box +// itself could not, while the customer was told their (correct) code opened nothing. These tests pin +// the endpoint that ends that, and — more importantly — they pin the two things it must NOT do: +// serve a package that cannot be opened, and let one host read another's. + +type retainedResp struct { + HostID string `json:"host_id"` + Count int `json:"count"` + UnopenableCount int `json:"unopenable_count"` + TruncatedCount int `json:"truncated_count"` + Packages []struct { + Index int `json:"index"` + SupersededAt string `json:"superseded_at"` + KeyFingerprint string `json:"key_fingerprint"` + IdentityEscrowB64 string `json:"identity_escrow_b64"` + } `json:"packages"` +} + +func getRetained(t *testing.T, h *Handler, hostID, key string) (int, retainedResp) { + t.Helper() + rr := do(h, http.MethodGet, "/hosts/"+hostID+"/escrow/retained", key, "") + var out retainedResp + if rr.Code == http.StatusOK { + if err := json.Unmarshal(rr.Body.Bytes(), &out); err != nil { + t.Fatalf("decode retained response: %v (body %s)", err, rr.Body.String()) + } + } + return rr.Code, out +} + +// seedSuperseded drives the REAL supersession path (a PUT sealing a different password demotes the +// current row) rather than inserting into host_escrow_superseded directly — so what the test proves +// includes the demote wiring, not just this handler's SELECT. +func seedSuperseded(t *testing.T, h *Handler, hostID, key string, gens []struct{ sha, identity string }) { + t.Helper() + for i, g := range gens { + if rr := do(h, http.MethodPut, "/hosts/"+hostID+"/escrow", key, escrowBodyWithHash([]byte("k"), g.sha, g.identity)); rr.Code != http.StatusOK { + t.Fatalf("seed PUT %d = %d (%s)", i, rr.Code, rr.Body.String()) + } + } +} + +// THE ONE THAT MATTERS: a retained package that carries key material is served, and the caller can +// tell it apart from the current row. +// +// RED-PROOF: change the handler's SELECT source from ListSupersededEscrow to GetHostDRBundle (i.e. +// serve the CURRENT row) → the served blob becomes "age-gen3" → this FAILS on the body comparison, +// which is exactly the confusion the endpoint exists to end. +func TestRetainedEscrowGet_ServesRetainedPackages(t *testing.T) { + h, st, _ := newTestHandler(t) + st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"}) + seedSuperseded(t, h, "h1", "HKEY", []struct{ sha, identity string }{ + {"SHA_GEN1", "age-gen1"}, + {"SHA_GEN2", "age-gen2"}, + {"SHA_GEN3", "age-gen3"}, // current after this + }) + + code, out := getRetained(t, h, "h1", "HKEY") + if code != http.StatusOK { + t.Fatalf("GET retained = %d, want 200", code) + } + if out.Count != 2 || len(out.Packages) != 2 { + t.Fatalf("want 2 retained packages, got count=%d len=%d", out.Count, len(out.Packages)) + } + if out.UnopenableCount != 0 { + t.Errorf("unopenable_count = %d, want 0 (every seeded row carried an identity blob)", out.UnopenableCount) + } + // The retained packages must be the SUPERSEDED generations, never the current one. If this ever + // serves gen3 the endpoint is answering the wrong question and the customer is misled again. + got := map[string]bool{} + for _, p := range out.Packages { + b, err := base64.StdEncoding.DecodeString(p.IdentityEscrowB64) + if err != nil { + t.Fatalf("package %d: identity blob is not base64: %v", p.Index, err) + } + got[string(b)] = true + if p.SupersededAt == "" { + t.Errorf("package %d has no superseded_at — the screen needs a date to name the package", p.Index) + } + } + if !got["age-gen1"] || !got["age-gen2"] { + t.Errorf("served packages = %v, want the two SUPERSEDED generations (age-gen1, age-gen2)", got) + } + if got["age-gen3"] { + t.Error("the CURRENT package was served as retained — that is the wrong-package confusion this endpoint exists to end") + } +} + +// A retained row with NO identity blob (every pre-v0.93.0 row) must be WITHHELD and COUNTED. Serving +// it would have the agent try a package that cannot open anything, and would let the screen claim an +// earlier package is recoverable on exactly the boxes hurt by the original defect. +// +// RED-PROOF: delete the `if len(e.IdentityBlob) == 0 { unopenable++; continue }` guard → the empty +// row is served as a package → Count becomes 2 and UnopenableCount 0 → this FAILS twice. +func TestRetainedEscrowGet_WithholdsAndCountsUnopenableRows(t *testing.T) { + h, st, _ := newTestHandler(t) + st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"}) + // gen1 carries NO identity blob — the pre-v0.93.0 shape. + seedSuperseded(t, h, "h1", "HKEY", []struct{ sha, identity string }{ + {"SHA_GEN1", ""}, + {"SHA_GEN2", "age-gen2"}, + {"SHA_GEN3", "age-gen3"}, + }) + + code, out := getRetained(t, h, "h1", "HKEY") + if code != http.StatusOK { + t.Fatalf("GET retained = %d, want 200", code) + } + if out.Count != 1 { + t.Errorf("count = %d, want 1 (only gen2 carries material)", out.Count) + } + if out.UnopenableCount != 1 { + t.Errorf("unopenable_count = %d, want 1 — the caller cannot derive this and needs it to explain "+ + "why a correct old code opens nothing on such a box", out.UnopenableCount) + } + for _, p := range out.Packages { + if p.IdentityEscrowB64 == "" { + t.Error("a package with an empty identity blob was served — it can never open anything") + } + } +} + +// SELF-SCOPE. The current-row GET has this and a retained read is strictly more material, so the same +// asymmetry must hold: a host key reads only its own. +// +// RED-PROOF: remove the `!isGlobal && authHostID != pathHostID` branch → h2's key reads h1's packages +// → this FAILS with 200. +func TestRetainedEscrowGet_SelfScoped(t *testing.T) { + h, st, _ := newTestHandler(t) + st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY1"}) + st.UpsertHost(&store.Host{HostID: "h2", CustomerID: "c2", APIKey: "HKEY2"}) + seedSuperseded(t, h, "h1", "HKEY1", []struct{ sha, identity string }{ + {"SHA_GEN1", "age-gen1"}, + {"SHA_GEN2", "age-gen2"}, + }) + + rr := do(h, http.MethodGet, "/hosts/h1/escrow/retained", "HKEY2", "") + if rr.Code != http.StatusForbidden { + t.Fatalf("h2 reading h1's retained packages = %d, want 403 — a host key must never be a fleet-wide reader", rr.Code) + } +} + +// A host that has never re-sealed gets a clean, empty answer — not a 404 and not a fault. "No retained +// package" is an ordinary situation and must be distinguishable from "something is broken", or the +// agent's try-loop cannot tell them apart either. +// +// RED-PROOF: make the handler 404 on an empty list → this FAILS with 404. +func TestRetainedEscrowGet_NoneIsACleanEmptyAnswer(t *testing.T) { + h, st, _ := newTestHandler(t) + st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"}) + if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k"), "SHA1", "age-1")); rr.Code != http.StatusOK { + t.Fatalf("seed PUT = %d", rr.Code) + } + + code, out := getRetained(t, h, "h1", "HKEY") + if code != http.StatusOK { + t.Fatalf("GET retained on a never-superseded host = %d, want 200", code) + } + if out.Count != 0 || len(out.Packages) != 0 || out.UnopenableCount != 0 { + t.Errorf("want an empty answer, got count=%d packages=%d unopenable=%d", out.Count, len(out.Packages), out.UnopenableCount) + } +} + +// The retained route must not be reachable without a key at all. +// +// RED-PROOF: drop the checkAuthHost block → this FAILS with 200. +func TestRetainedEscrowGet_RequiresAuth(t *testing.T) { + h, st, _ := newTestHandler(t) + st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"}) + if rr := do(h, http.MethodGet, "/hosts/h1/escrow/retained", "", ""); rr.Code != http.StatusUnauthorized { + t.Fatalf("unauthenticated retained GET = %d, want 401", rr.Code) + } +} + +// ROUTING: /escrow and /escrow/retained must stay two different answers. A prefix "tidy-up" that +// collapsed them would route retained reads to the current row — silently the wrong package. +// +// RED-PROOF: change the retained case's suffix to "/escrow" (so the earlier case wins) → the retained +// route returns the CURRENT-row shape, which has no "packages" key → this FAILS. +func TestRetainedEscrowGet_IsADistinctRouteFromTheCurrentRow(t *testing.T) { + h, st, _ := newTestHandler(t) + st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"}) + seedSuperseded(t, h, "h1", "HKEY", []struct{ sha, identity string }{ + {"SHA_GEN1", "age-gen1"}, + {"SHA_GEN2", "age-gen2"}, + }) + + cur := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", "") + var curBody map[string]any + if err := json.Unmarshal(cur.Body.Bytes(), &curBody); err != nil { + t.Fatalf("decode current-row body: %v", err) + } + if _, hasPackages := curBody["packages"]; hasPackages { + t.Error("the CURRENT-row route grew a packages key — the two routes have been merged") + } + if _, hasPresent := curBody["present"]; !hasPresent { + t.Error("the current-row route lost its present flag") + } + + _, ret := getRetained(t, h, "h1", "HKEY") + if ret.Count != 1 { + t.Fatalf("retained route count = %d, want 1 — it is answering the current row's question", ret.Count) + } +} diff --git a/hub/internal/api/handler.go b/hub/internal/api/handler.go index 42d25ae..b10266d 100644 --- a/hub/internal/api/handler.go +++ b/hub/internal/api/handler.go @@ -245,6 +245,13 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { case r.Method == http.MethodGet && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"): hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow") h.handleHostEscrowGet(w, r, hostID) + // R-311 (v0.103.0): the RETAINED siblings of the row above. The two cases cannot collide — one + // ends `/escrow`, the other `/escrow/retained` — but do NOT "tidy" them into a single prefix + // match: a prefix match would route retained reads to the CURRENT row, which is silently the + // wrong package and is exactly the confusion this endpoint exists to end. + case r.Method == http.MethodGet && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow/retained"): + hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow/retained") + h.handleHostEscrowRetainedGet(w, r, hostID) // G1 break-glass: day-0 vaults the root@pam console credential (self-scoped host key); the // operator retrieves it via the /admin/ path (global key only). case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/recovery-credential"): @@ -1371,6 +1378,158 @@ func (h *Handler) handleHostEscrowGet(w http.ResponseWriter, r *http.Request, pa }) } +// handleHostEscrowRetainedGet serves a host its own RETAINED (superseded) sealed identity packages. +// +// ── WHY THIS EXISTS (R-311) ───────────────────────────────────────────────────────────────────── +// +// `ListSupersededEscrow` has been the only reader of a retained identity blob since v0.93.0 and had +// ZERO production callers — every call site was a test. The consequence, measured on the fixture +// 2026-08-12: a recovery code that demonstrably opens a retained package (proven by hand: unsealed, +// and it restored planted files byte-identical from a store the box could no longer open) was +// reported to the customer as a code that opened nothing. The screen already hedged that with two +// possible causes (R-222/R-226) and said it could not tell them apart. **It could not tell them +// apart because nothing ever looked.** This endpoint is what makes looking possible. +// +// ── WHAT THIS DOES NOT CHANGE ─────────────────────────────────────────────────────────────────── +// +// The hub still cannot read any of it. Sealed bytes in, sealed bytes out; there is no decrypt path +// here and the hub has never held a recovery code. What DOES widen is volume: a host key that could +// previously fetch one opaque package can now fetch N. The trade is stated rather than waved through +// — see the audit — and it is bounded three ways: the same self-scope as the current row, the same +// recovery-mode gate, and an explicit cap so a host with a long supersession history cannot turn one +// request into an unbounded read. +// +// Rows whose `identity_blob` is NULL are NOT served and are counted separately. They are the +// pre-v0.93.0 rows; they retain the PBS key and not the repository password, so they can never open +// anything the caller is asking about. Serving them would make the agent try packages that cannot +// succeed and would let the screen claim an earlier package is openable when it is not — the same +// false-explanation trap `SupersededPresent` already avoids at store.go. They are counted because +// their EXISTENCE is a true and load-bearing fact: on those boxes a correct old code opens nothing, +// and the honest reason is a defect of ours, not the customer's typing. +const retainedEscrowServeCap = 16 + +// RetainedEscrowPackage / RetainedEscrowResponse are the WIRE for GET /hosts//escrow/retained. +// +// They are named types rather than a `map[string]any` on purpose: the wire-contract gate resolves a +// declared ROOT by TYPE, so an untyped map is a cross-repo contract the gate cannot see. This wire is +// declared in `scripts/wire_contract_gate.py` ROOTS — hub → agent — and the agent's mirror is +// `felhom-agent/internal/hub.RetainedEscrowResponse`. Change a tag here and the gate fails there, +// which is the entire point. +type RetainedEscrowPackage struct { + // Index is a label WITHIN ONE RESPONSE. Not durable, never persisted, never a lookup key. + Index int `json:"index"` + // SupersededAt is when this package stopped being the current one. It is the only field here a + // customer ever sees — it is how they recognise which recovery code they are holding. + SupersededAt string `json:"superseded_at"` + KeyFingerprint string `json:"key_fingerprint"` + // IdentityEscrowB64 is OPAQUE ciphertext. The hub cannot open it and never could. + IdentityEscrowB64 string `json:"identity_escrow_b64"` +} + +type RetainedEscrowResponse struct { + HostID string `json:"host_id"` + Count int `json:"count"` + // UnopenableCount is the number of retained rows withheld because they carry no key material + // (every pre-v0.93.0 row). Underivable by the caller and load-bearing: on a box with only those, + // a perfectly correct old code opens nothing and the reason is a defect of ours. + UnopenableCount int `json:"unopenable_count"` + TruncatedCount int `json:"truncated_count"` + Packages []RetainedEscrowPackage `json:"packages"` +} + +func (h *Handler) handleHostEscrowRetainedGet(w http.ResponseWriter, r *http.Request, pathHostID string) { + authHostID, _, isGlobal, ok := h.checkAuthHost(r) + if !ok { + http.Error(w, "Unauthorized", http.StatusUnauthorized) + return + } + if pathHostID == "" { + http.Error(w, "Missing host_id", http.StatusBadRequest) + return + } + // SELF-SCOPED, identically to the current-row GET. Without this line any host key is a fleet-wide + // reader of every retained package, which is strictly worse than the same hole on one row. + if !isGlobal && authHostID != pathHostID { + h.logger.Printf("[WARN] retained escrow GET REFUSED: host %s asked for %s's packages (self-scope)", authHostID, pathHostID) + http.Error(w, "Forbidden: host_id mismatch", http.StatusForbidden) + return + } + host, err := h.store.GetHost(pathHostID) + if err != nil { + http.Error(w, "Internal error", http.StatusInternalServerError) + return + } + if host == nil { + http.Error(w, "Unknown host_id", http.StatusNotFound) + return + } + // The SAME §8.3 gate as the current row. A retained package is not less sensitive than the + // current one; if self-service retrieval is ever switched off, it must go dark with it. + if !escrowSelfServiceRetrieval && !host.InRecoveryMode(time.Now().UTC()) { + h.logger.Printf("[WARN] retained escrow GET REFUSED for %s — self-service retrieval is disabled and recovery mode is not armed", pathHostID) + http.Error(w, "Forbidden: host not in recovery mode (operator must arm it)", http.StatusForbidden) + return + } + + retained, rerr := h.store.ListSupersededEscrow(pathHostID) + if rerr != nil { + h.logger.Printf("[ERROR] retained escrow GET for %s: %v", pathHostID, rerr) + http.Error(w, "Internal error", http.StatusInternalServerError) + return + } + packages := make([]RetainedEscrowPackage, 0, len(retained)) + unopenable := 0 + for _, e := range retained { + if len(e.IdentityBlob) == 0 { + unopenable++ + continue + } + if len(packages) >= retainedEscrowServeCap { + continue + } + packages = append(packages, RetainedEscrowPackage{ + Index: len(packages), + SupersededAt: e.UpdatedAt, + KeyFingerprint: e.KeyFingerprint, + IdentityEscrowB64: base64.StdEncoding.EncodeToString(e.IdentityBlob), + }) + } + truncated := 0 + if n := len(retained) - unopenable; n > len(packages) { + truncated = n - len(packages) + } + + // The audit row is written BEFORE the bytes leave, exactly as the current-row GET does, so a + // retrieval cannot be served without its record. Severity matches that path deliberately: this is + // a recovery in progress and nothing else, and it must reach the operator by e-mail. + if host.CustomerID != "" && len(packages) > 0 { + msg := fmt.Sprintf("Retained recovery packages served: host %s retrieved %d retained sealed package(s) (%d unopenable pre-v0.93.0 row(s) withheld). "+ + "This is the recovery path in use — the packages cannot be opened without the customer's recovery code, which the hub never holds. "+ + "If no recovery is in progress on that box, investigate.", pathHostID, len(packages), unopenable) + details, _ := json.Marshal(map[string]any{ + "host_id": pathHostID, + "served": len(packages), + "unopenable": unopenable, + "truncated": truncated, + "self_scope": !isGlobal, + }) + if _, eerr := h.store.SaveEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub"); eerr != nil { + h.logger.Printf("[WARN] %s event save FAILED for %s (serving anyway): %v", eventEscrowBlobServed, pathHostID, eerr) + } else if h.dispatcher != nil { + go h.dispatcher.ProcessEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub") + } + } + h.logger.Printf("[WARN] retained escrow SERVED to host %s (%d package(s), %d unopenable withheld, %d over cap, self_scope=%v)", + pathHostID, len(packages), unopenable, truncated, !isGlobal) + writeJSON(w, http.StatusOK, RetainedEscrowResponse{ + HostID: pathHostID, + Count: len(packages), + UnopenableCount: unopenable, + TruncatedCount: truncated, + Packages: packages, + }) +} + // maybeEmitRepoKeyChanged raises ONE operator signal per supersession when the sealed offsite repo // password demonstrably changed. Both hashes have been stored since SLICE 3 (host_escrow and, since // v0.60.0, host_escrow_superseded) and NOTHING compared them: demo-felhom's repository password diff --git a/scripts/wire_contract_gate.py b/scripts/wire_contract_gate.py index 1ebf096..3f8cc98 100644 --- a/scripts/wire_contract_gate.py +++ b/scripts/wire_contract_gate.py @@ -81,6 +81,12 @@ ROOTS = [ "controller", "internal/report", "Report", "hub"), ("hub -> controller (report ACK, `escrow` object)", "hub", "internal/store", "EscrowStatus", "controller"), + # R-311. Declared the moment the wire was created, because the gate covers only what is DECLARED + # and a silent pass is indistinguishable from coverage. The hub response is a named type rather + # than a map[string]any precisely so this root can resolve — an untyped map is a cross-repo + # contract nothing can check. + ("hub -> agent (GET /hosts//escrow/retained)", + "hub", "internal/api", "RetainedEscrowResponse", "agent"), ] # Tag names whose literal string carries no information in a repo-wide search. NOT CHECKED.