hub v0.103.0 — a host can read the packages we kept for it (R-311)
ListSupersededEscrow had zero production callers for nineteen days. It is the only reader of a retained identity_blob, so the retention shipped in v0.93.0 was material the product could not reach - proven on the fixture 2026-08-12, where a code that opens a retained package was answered as a code that opened nothing. New GET /api/v1/hosts/<id>/escrow/retained: self-scoped exactly as the current-row GET, same recovery-mode gate, same audit event written BEFORE the bytes leave, capped at 16. Rows with a NULL identity_blob are WITHHELD and returned as unopenable_count. They retain the PBS key, not the repository password, so they can never open what the caller is asking about; serving them would have the agent try packages that cannot succeed and would let the screen claim an earlier package is openable on exactly the boxes the original defect hurt. The count is returned because their existence is load-bearing and underivable. The trade, stated rather than waved through: the hub still cannot read any of it - sealed bytes in, sealed bytes out, no decrypt path, no recovery code ever held. What widens is volume, bounded by self-scope, the recovery-mode gate and the cap. The response is a NAMED TYPE, not a map, so the wire-contract gate can resolve it; the wire is declared as a fourth ROOT and the gate now checks 182 tags rather than 174. A positive control shows that check is name-presence, not decodability - filed as R-315 rather than reported as coverage. Six tests through the real endpoint; four red-proofs asserted applied.
This commit is contained in:
@@ -1,6 +1,6 @@
|
|||||||
# STATUS — what works, what's broken, what's next
|
# STATUS — what works, what's broken, what's next
|
||||||
|
|
||||||
**Updated 2026-08-12 (late evening — retained-key drill).**
|
**Updated 2026-08-12 (night — the door, part one).**
|
||||||
|
|
||||||
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
|
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
|
||||||
> part of it in plain words, and **nothing may exist only here**. **Items, not paragraphs. One screen.**
|
> part of it in plain words, and **nothing may exist only here**. **Items, not paragraphs. One screen.**
|
||||||
@@ -11,12 +11,13 @@
|
|||||||
*(The golden-vouch and floor-raise asks that stood here are **done** — the hub reads golden **0.213.0**,
|
*(The golden-vouch and floor-raise asks that stood here are **done** — the hub reads golden **0.213.0**,
|
||||||
floor **0.213.0**, agent **0.128.0**. Checked live, not assumed.)*
|
floor **0.213.0**, agent **0.128.0**. Checked live, not assumed.)*
|
||||||
|
|
||||||
- **R-307 — a countdown is already running on `demo-felhom`**, started 10 Aug, **firing 24 August**,
|
- **R-312 — one decision, and it is the interesting one.** The customer is now told the truth about an
|
||||||
after which that set-aside off-site copy is deleted for good. Today's drill did not start it and
|
older code, but there is still no button: restoring from a set-aside copy means either threading an
|
||||||
deliberately did not cancel it. Cancel it (copy kept, costs storage) or let it run (copy deleted).
|
alternative location through the restore code, or adopting that copy as the machine's current one.
|
||||||
**Doing nothing chooses deletion.**
|
The second is a different product decision. **Nothing is broken while this waits.**
|
||||||
- **R-304 — decide what we promise about old backups.** The retention works, but only an operator can
|
- **R-313 — the copy you told me to keep cannot be opened by anyone.** `demo-felhom`'s set-aside store
|
||||||
use it; the customer is told their correct code is wrong. Either build the door, or stop promising.
|
holds 36 snapshots and one key, and that key was destroyed by the bug we fixed on 4 August. Keeping
|
||||||
|
it is still the right call; it should be a decision, not an accumulation.
|
||||||
- **R-303 — one coherence decision**, ranked low: a box can still raise the "we cannot open your new
|
- **R-303 — one coherence decision**, ranked low: a box can still raise the "we cannot open your new
|
||||||
backups" card while counting down to deleting the old ones. The two no longer contradict each other,
|
backups" card while counting down to deleting the old ones. The two no longer contradict each other,
|
||||||
but the state is odd and the wrong fix would hide a real second fault.
|
but the state is odd and the wrong fix would hide a real second fault.
|
||||||
@@ -39,6 +40,12 @@ Both demo machines are home, healthy and reporting on the approved pair. Off-sit
|
|||||||
fifth deliberately left, because it is true where it renders.
|
fifth deliberately left, because it is true where it renders.
|
||||||
- **One name per secret, box side** (R-295): the dashboard code is „Beállító kód" everywhere;
|
- **One name per secret, box side** (R-295): the dashboard code is „Beállító kód" everywhere;
|
||||||
„Visszaállító kód" is retired. It collided with the escrow „Helyreállítási kód" and cost a real code.
|
„Visszaállító kód" is retired. It collided with the escrow „Helyreállítási kód" and cost a real code.
|
||||||
|
- **A correct recovery code is no longer called wrong** (R-311, three components). If a customer types
|
||||||
|
the code for an older set of backups, the machine now checks the packages we kept, recognises it, and
|
||||||
|
says so: *your code is correct, it belongs to an earlier package, we kept it, your current backups are
|
||||||
|
fine, write to us*. It deliberately promises no restore, because there is no button yet.
|
||||||
|
- **The countdown on `demo-felhom` is cancelled** on your ruling (R-307). Nothing was deleted; the
|
||||||
|
24 August deadline is gone. See R-313 for what that copy turns out to be.
|
||||||
- **Both installer fixes are now PUBLISHED** as `installer-v1.27.0` (R-297 + R-300). Each fault was
|
- **Both installer fixes are now PUBLISHED** as `installer-v1.27.0` (R-297 + R-300). Each fault was
|
||||||
watched happening first, on a machine reset to factory state: the old installer really did build a
|
watched happening first, on a machine reset to factory state: the old installer really did build a
|
||||||
machine on a base image from July, and our own uninstall really did block our own next install.
|
machine on a base image from July, and our own uninstall really did block our own next install.
|
||||||
@@ -50,11 +57,10 @@ Both demo machines are home, healthy and reporting on the approved pair. Off-sit
|
|||||||
everything on it is lost.** First act of the visit: copy the ~3.6 GB off before anything is
|
everything on it is lost.** First act of the visit: copy the ~3.6 GB off before anything is
|
||||||
reinstalled — it is currently the only copy in existence. Whether it stays parked is your call and is
|
reinstalled — it is currently the only copy in existence. Whether it stays parked is your call and is
|
||||||
deliberately left open.
|
deliberately left open.
|
||||||
- **Kept backups can be opened — but not by the customer** (R-304). We now know for certain the old key
|
- **Kept backups can be opened — but still only by us** (R-304 partly closed, R-312 open). The machine
|
||||||
is kept and **does** open the old backups: three planted files came back byte-for-byte from a store the
|
now recognises an older code and says so plainly instead of hedging. What it still cannot do is hand
|
||||||
machine itself could no longer read. But nothing in the product reads a kept key, so it takes an
|
the customer their old files: that needs the restore code to accept a second location, which is real
|
||||||
operator with a shell. Worse, a customer who types their **correct** old code is told it *"did not open
|
work rather than wiring. Today the honest answer is "your code is right, write to us" — and we can.
|
||||||
the sealed bundle"*. The keeping is real; the promise that they can use it is not.
|
|
||||||
- **The dnsmasq fix helps a machine once** (R-305). On a machine that never had Felhom it works. On the
|
- **The dnsmasq fix helps a machine once** (R-305). On a machine that never had Felhom it works. On the
|
||||||
second reinstall the leftover comes back, because the package is never removed — so the machine looks,
|
second reinstall the leftover comes back, because the package is never removed — so the machine looks,
|
||||||
to our own installer, as if the household had installed it. Watched happening the same afternoon.
|
to our own installer, as if the household had installed it. Watched happening the same afternoon.
|
||||||
@@ -64,5 +70,6 @@ Both demo machines are home, healthy and reporting on the approved pair. Off-sit
|
|||||||
|
|
||||||
## Working on next
|
## Working on next
|
||||||
|
|
||||||
Your two decisions above (R-307's deadline is the hard one); then R-304's shape; then the hub naming;
|
R-312's shape (the button, or deliberately no button); then R-305, because the tester's second
|
||||||
then the 2026-08-09 batch (R-279 … R-292), still untriaged against everything since.
|
reinstall still hits the dnsmasq wall; then the hub naming; then the 2026-08-09 batch
|
||||||
|
(R-279 … R-292), still untriaged against everything since.
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
@@ -607,7 +607,12 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
|
|||||||
| **R-304** | **The retained escrow key works, and the customer is told their correct code is wrong.** DRILL 2026-08-12 answered the three questions separately, on `demo-felhom`, with planted data. **(a) retention: WORKS** — the first retained row in fleet history to carry material (`host_escrow_superseded` id 11, `identity_blob` 572 B), byte-identical (`sha256 a10032341c8584ed…`) to the pre-supersession `host_escrow` row. **(b) the material opens the old store: YES** — unsealed with the OLD recovery code it yielded a password byte-identical to the pre-change one (`sha c60c8bc737a6b7c6…`), and restored three planted files **byte-identical** from a store the box itself could no longer open (negative control first: `Fatal: wrong password or no key found`), **including a Hungarian accented filename verified as raw bytes**. **(c) the customer's route: DOES NOT EXIST, and misinforms.** `ListSupersededEscrow` (`store.go:2841`) is the only reader of a retained `identity_blob` and has **zero production callers** — five call sites, all `_test.go`; the product path (`POST /escrow/recover-offsite-password` → `FetchIdentityEscrow` → `GetHostDRBundle`, `store.go:3152`) selects `FROM host_escrow` — the CURRENT row only. Asked for the old password with the code that demonstrably opens the retained row, the product answered **"the recovery code did not open the sealed bundle — nothing was written"**. **This is the R-224 class again**: there an unreachable hub was reported as a bad code; here a VALID code for retained history is reported as a bad code, and the customer's attempt ends there. **Consequence:** the census answer stands (it was about retention); the countdown banner's promise is true in substance and false in practice; **any capability-map claim that the customer can recover the old history with their recovery code is false today and must move** | **READY (L) — NEW 2026-08-12, RANK 1** | R-198, R-199, R-224, R-241 | Decide the shape: serve retained rows on the recovery path (needs a "which package?" choice — a customer may have several), or stop promising retrieval anywhere the customer cannot perform it. **Until one of those, the honest position is that retention is an operator-only capability.** At minimum, the refusal must stop asserting the code is wrong when the hub simply never looked | operator + CC |
|
| **R-304** | **The retained escrow key works, and the customer is told their correct code is wrong.** DRILL 2026-08-12 answered the three questions separately, on `demo-felhom`, with planted data. **(a) retention: WORKS** — the first retained row in fleet history to carry material (`host_escrow_superseded` id 11, `identity_blob` 572 B), byte-identical (`sha256 a10032341c8584ed…`) to the pre-supersession `host_escrow` row. **(b) the material opens the old store: YES** — unsealed with the OLD recovery code it yielded a password byte-identical to the pre-change one (`sha c60c8bc737a6b7c6…`), and restored three planted files **byte-identical** from a store the box itself could no longer open (negative control first: `Fatal: wrong password or no key found`), **including a Hungarian accented filename verified as raw bytes**. **(c) the customer's route: DOES NOT EXIST, and misinforms.** `ListSupersededEscrow` (`store.go:2841`) is the only reader of a retained `identity_blob` and has **zero production callers** — five call sites, all `_test.go`; the product path (`POST /escrow/recover-offsite-password` → `FetchIdentityEscrow` → `GetHostDRBundle`, `store.go:3152`) selects `FROM host_escrow` — the CURRENT row only. Asked for the old password with the code that demonstrably opens the retained row, the product answered **"the recovery code did not open the sealed bundle — nothing was written"**. **This is the R-224 class again**: there an unreachable hub was reported as a bad code; here a VALID code for retained history is reported as a bad code, and the customer's attempt ends there. **Consequence:** the census answer stands (it was about retention); the countdown banner's promise is true in substance and false in practice; **any capability-map claim that the customer can recover the old history with their recovery code is false today and must move** | **READY (L) — NEW 2026-08-12, RANK 1** | R-198, R-199, R-224, R-241 | Decide the shape: serve retained rows on the recovery path (needs a "which package?" choice — a customer may have several), or stop promising retrieval anywhere the customer cannot perform it. **Until one of those, the honest position is that retention is an operator-only capability.** At minimum, the refusal must stop asserting the code is wrong when the hub simply never looked | operator + CC |
|
||||||
| **R-305** | **The R-300 cleanup fires exactly once per machine, and the second reinstall hits the original wall.** PROVEN on `drill-r50` 2026-08-12, not deduced. The uninstall stops+disables dnsmasq but deliberately never **purges** the package, so on every later install `dpkg-query` reports it installed, preflight records `dnsmasq_preexisting=yes`, the agent's `EnsureDnsmasq` re-enables it, and the uninstall then only *restarts* it — leaving it `enabled`/`active` on `0.0.0.0:53` exactly as before the fix. Cycle 1: `no` → disabled → `:53 FREE` → next install PASSES. Cycle 2: `yes` → left running → **cycle-3 byo preflight FAILED, exit 1**. The improved refusal does fire and names the command, so the customer is not stranded — but they must run by hand a cleanup Felhom could have run itself, on a box Felhom does own | **READY (M) — NEW 2026-08-12, RANK 2** | R-300 | Distinguish "the package pre-dates Felhom" from "a previous **Felhom** install left it" — e.g. keep the ownership record outside `state.json` (which the uninstall deletes), or record Felhom-installed-ness where the agent installs it. **Do not fix by purging on hosts we may not own** | CC |
|
| **R-305** | **The R-300 cleanup fires exactly once per machine, and the second reinstall hits the original wall.** PROVEN on `drill-r50` 2026-08-12, not deduced. The uninstall stops+disables dnsmasq but deliberately never **purges** the package, so on every later install `dpkg-query` reports it installed, preflight records `dnsmasq_preexisting=yes`, the agent's `EnsureDnsmasq` re-enables it, and the uninstall then only *restarts* it — leaving it `enabled`/`active` on `0.0.0.0:53` exactly as before the fix. Cycle 1: `no` → disabled → `:53 FREE` → next install PASSES. Cycle 2: `yes` → left running → **cycle-3 byo preflight FAILED, exit 1**. The improved refusal does fire and names the command, so the customer is not stranded — but they must run by hand a cleanup Felhom could have run itself, on a box Felhom does own | **READY (M) — NEW 2026-08-12, RANK 2** | R-300 | Distinguish "the package pre-dates Felhom" from "a previous **Felhom** install left it" — e.g. keep the ownership record outside `state.json` (which the uninstall deletes), or record Felhom-installed-ness where the agent installs it. **Do not fix by purging on hosts we may not own** | CC |
|
||||||
| **R-306** | **`--preflight-only` says "no state written" and writes state — with an answer that can be wrong.** `_state_put` short-circuits on `DRY_RUN` only (`felhom-host-install.sh:418`), so a preflight-only run creates `/var/lib/felhom-install/state.json`. Observed live: after a run whose banner read `PRE-FLIGHT PASS (mode=byo) — no state written, no install step executed`, the file existed containing `{"completed": [], "dnsmasq_preexisting": "yes"}`. Both the banner and the flag's own comment at line 226 assert the opposite. **The harm is not the file, it is the value**: the runbook recommends preflight-only first, then the same command without the flag, so on a box carrying a Felhom leftover the wrong ownership answer is baked in before the real install begins | **READY (S) — NEW 2026-08-12, RANK 3** | R-300, R-305 | Either make `_state_put` a no-op under `PREFLIGHT_ONLY` (and record ownership at install instead), or correct both claims. A comment asserting an invariant needs a test pinning it | CC |
|
| **R-306** | **`--preflight-only` says "no state written" and writes state — with an answer that can be wrong.** `_state_put` short-circuits on `DRY_RUN` only (`felhom-host-install.sh:418`), so a preflight-only run creates `/var/lib/felhom-install/state.json`. Observed live: after a run whose banner read `PRE-FLIGHT PASS (mode=byo) — no state written, no install step executed`, the file existed containing `{"completed": [], "dnsmasq_preexisting": "yes"}`. Both the banner and the flag's own comment at line 226 assert the opposite. **The harm is not the file, it is the value**: the runbook recommends preflight-only first, then the same command without the flag, so on a box carrying a Felhom leftover the wrong ownership answer is baked in before the real install begins | **READY (S) — NEW 2026-08-12, RANK 3** | R-300, R-305 | Either make `_state_put` a no-op under `PREFLIGHT_ONLY` (and record ownership at install instead), or correct both claims. A comment asserting an invariant needs a test pinning it | CC |
|
||||||
| **R-307** | **`demo-felhom` carries a LIVE abandon countdown that this drill did not start — and the end state says there should be none.** `settings.json`: `abandon_started_at 2026-08-10T08:06:31Z`, `abandon_at 2026-08-24T08:06:31Z`, `abandon_repo_path /home/felhom-repo.orphaned-20260810`. The drill's fence forbade starting, shortening or triggering a countdown, and none was; but its required end state was *"no abandon countdown anywhere"*, and one exists. **The two ways to satisfy that are not equivalent and the choice is the operator's:** cancel it (the orphaned repository is kept indefinitely — storage cost, no data risk) or let it run (on **2026-08-24** that repository is deleted, **irreversibly**). **Doing nothing selects deletion by default.** Deliberately not resolved by CC | **BLOCKED ON OPERATOR — NEW 2026-08-12** | R-301, R-302, R-303 | Decide before 2026-08-24 | operator |
|
| **R-307** | **`demo-felhom` carries a LIVE abandon countdown that this drill did not start — and the end state says there should be none.** `settings.json`: `abandon_started_at 2026-08-10T08:06:31Z`, `abandon_at 2026-08-24T08:06:31Z`, `abandon_repo_path /home/felhom-repo.orphaned-20260810`. The drill's fence forbade starting, shortening or triggering a countdown, and none was; but its required end state was *"no abandon countdown anywhere"*, and one exists. **The two ways to satisfy that are not equivalent and the choice is the operator's:** cancel it (the orphaned repository is kept indefinitely — storage cost, no data risk) or let it run (on **2026-08-24** that repository is deleted, **irreversibly**). **Doing nothing selects deletion by default.** Deliberately not resolved by CC. **RULED 2026-08-12 (operator): KEEP the set-aside history — cancel the countdown.** Done the same evening through the product's own operator path (`--abandon-stop`, which refuses rather than silently no-opping), with the container stopped first so the running controller could not overwrite `settings.json` from memory. **Proved, not trusted to the exit code:** `abandon_started_at` and `abandon_at` are GONE (`AbandonStatus` returns `Active=false` when `AbandonAt` is empty — `offbox_abandon.go:111-113`, so no countdown renders); `abandon_repo_path` deliberately REMAINS as the pointer to the preserved store; the store still holds **36 snapshot objects** and its full `config/data/index/keys/locks/snapshots` structure; both repositories still present on the endpoint; **nothing deleted anywhere**. See R-313 — what was preserved cannot currently be opened by anyone | **CLOSED — countdown cancelled 2026-08-12 on the operator's ruling** | R-301, R-302, R-303, R-313 | — | CC |
|
||||||
| **R-308** | **The stored controller password no longer opens `demo-felhom`.** `PASSWORD` in `~/.config/credentials` on DooPlex returns `Hibás jelszó` against `https://127.0.0.1:443/login` with `Host: felhom.demo-felhom.eu`. **NOT the known quoting trap** — the surrounding quotes were stripped before use and the stored value is unquoted (16 bytes for 15 chars), so this is a genuinely stale credential, not a mis-read. Cost during the drill: the dashboard was unreachable headlessly, so the offsite push had to be made with `restic` directly instead of the page's own button, which left the app-backup→unit→offsite chain unexercised. `R_DEMO-FELHOM` **was** correct and has since been rotated by this session's final ceremony (updated in place; prior file backed up alongside) | **READY (S) — NEW 2026-08-12, RANK 2** | — | Recover or reset the demo-felhom dashboard password and re-store it; note which boxes share one and which do not | operator |
|
| **R-308** | **The stored controller password no longer opens `demo-felhom`.** `PASSWORD` in `~/.config/credentials` on DooPlex returns `Hibás jelszó` against `https://127.0.0.1:443/login` with `Host: felhom.demo-felhom.eu`. **NOT the known quoting trap** — the surrounding quotes were stripped before use and the stored value is unquoted (16 bytes for 15 chars), so this is a genuinely stale credential, not a mis-read. Cost during the drill: the dashboard was unreachable headlessly, so the offsite push had to be made with `restic` directly instead of the page's own button, which left the app-backup→unit→offsite chain unexercised. `R_DEMO-FELHOM` **was** correct and has since been rotated by this session's final ceremony (updated in place; prior file backed up alongside) | **READY (S) — NEW 2026-08-12, RANK 2** | — | Recover or reset the demo-felhom dashboard password and re-store it; note which boxes share one and which do not | operator |
|
||||||
| **R-309** | **The day-0 runbook says pushing the installer publishes it. It has not since R-110.** `documentation/runbooks/day0-install.md` §C.1: *"it is always the current `main` script. There is no release tag, no staging copy and no version selector; pushing `scripts/felhom-host-install.sh` publishes it."* R-110 moved `/scripts/` to the **installer tag's** tree — `manifests/webpage.yaml` carries **two** `--ref=installer-v…` pins (lines 327, 372) and `hostinstall_gates.py` gate 6 enforces that they name a tag. Confirmed live 2026-08-12: `https://felhom.eu/scripts/felhom-host-install.sh` served `SCRIPT_VERSION="1.25.0"` while `main` had `1.27.0`, three and a half hours after the push. **The runbook's version is the dangerous one to believe in both directions** — it would have an operator think a pushed fix is live when it is not, and think a pushed mistake is live when it is not | **READY (S) — NEW 2026-08-12, RANK 3** | R-110 | Correct §C.1 to name the tag-and-manifest publication step (as performed today) | CC |
|
| **R-309** | **The day-0 runbook says pushing the installer publishes it. It has not since R-110.** `documentation/runbooks/day0-install.md` §C.1: *"it is always the current `main` script. There is no release tag, no staging copy and no version selector; pushing `scripts/felhom-host-install.sh` publishes it."* R-110 moved `/scripts/` to the **installer tag's** tree — `manifests/webpage.yaml` carries **two** `--ref=installer-v…` pins (lines 327, 372) and `hostinstall_gates.py` gate 6 enforces that they name a tag. Confirmed live 2026-08-12: `https://felhom.eu/scripts/felhom-host-install.sh` served `SCRIPT_VERSION="1.25.0"` while `main` had `1.27.0`, three and a half hours after the push. **The runbook's version is the dangerous one to believe in both directions** — it would have an operator think a pushed fix is live when it is not, and think a pushed mistake is live when it is not | **READY (S) — NEW 2026-08-12, RANK 3** | R-110 | Correct §C.1 to name the tag-and-manifest publication step (as performed today) | CC |
|
||||||
| **R-310** | **Two small edges on the installer, neither costing more than a moment.** (1) The R-297 operator-named refusal states the vouched version twice in consecutive sentences (*"…but the vouched golden is 0.213.0. The vouched golden is 0.213.0."*). (2) `--uninstall` reads its typed vmid confirmation from `/dev/tty` and `--force` deliberately does **not** bypass it, so teardown cannot be scripted without a pty — correct for an irreversible destroy, but undocumented; it surfaces as `line 891: /dev/tty: No such device or address` and an rc=1 that looks like a failure rather than a refusal to proceed unattended | **READY (S) — NEW 2026-08-12, RANK 4** | R-297 | Drop the duplicated sentence; add one runbook line naming the pty requirement | CC |
|
| **R-310** | **Two small edges on the installer, neither costing more than a moment.** (1) The R-297 operator-named refusal states the vouched version twice in consecutive sentences (*"…but the vouched golden is 0.213.0. The vouched golden is 0.213.0."*). (2) `--uninstall` reads its typed vmid confirmation from `/dev/tty` and `--force` deliberately does **not** bypass it, so teardown cannot be scripted without a pty — correct for an irreversible destroy, but undocumented; it surfaces as `line 891: /dev/tty: No such device or address` and an rc=1 that looks like a failure rather than a refusal to proceed unattended | **READY (S) — NEW 2026-08-12, RANK 4** | R-297 | Drop the duplicated sentence; add one runbook line naming the pty requirement | CC |
|
||||||
|
| **R-311** | **A correct recovery code for a retained package stopped being reported as wrong.** The drill of 2026-08-12 proved the retained key WORKS (it opened a set-aside store and restored planted files byte-identical) while the product answered *"the recovery code did not open the sealed bundle"*. **What was already right:** the customer-facing screen did not bluntly accuse — R-222/R-226 hedged, naming both causes and the kept package. **What was wrong was the hedge's reason:** *„innen nem tudjuk megkülönböztetni őket"* was true only because NOTHING EVER LOOKED. **Shipped, three repos.** Hub v0.103.0: `GET /hosts/<id>/escrow/retained` — the first production caller `ListSupersededEscrow` has ever had — self-scoped, same recovery-mode gate, same audit event before the bytes leave, rows with a NULL `identity_blob` WITHHELD and returned as `unopenable_count` (they can never open anything, and serving them would let the screen promise recovery on exactly the boxes the original defect hurt), capped at 16. Agent v0.129.0: `FetchRetained` consulted ONLY after the current package refuses, `ErrCodeOpensRetained` + the supersession date, local API answers **422**, bounded at 6 attempts (~1 s of scrypt each), and fail-safe in every direction — a nil fetcher, a 404 from an older hub, a transport failure or a malformed package all leave the original refusal standing. Controller v0.214.0: class `RecoveryCodeOpensRetained` gated on `FeatureRetainedRecoveryClass` (MinAgent 0.129.0, SEPARATE from the R-224 gate because a box can sit between 0.126.0 and 0.129.0), and a message that says the code is correct, names the date, says the earlier package is kept, says the CURRENT backups are unaffected, and **deliberately promises no restore** — it routes to support, which can do it by hand. **The lie's true home was the agent:** removing the CONTROLLER case yields the neutral message (R-224's safe default catches it); removing the AGENT's retained lookup brings back the fail-closed wrong-code error in those exact words — both demonstrated, mutations asserted applied | **CLOSED — shipped hub v0.103.0 + agent v0.129.0 + controller v0.214.0** | R-199, R-222, R-224, R-226, R-304 | — | CC |
|
||||||
|
| **R-312** | **There is no in-product route from the recovery screen to a set-aside store, and building one is not wiring — it is new surface.** Established read-only before any code was written (the session's §4 spike). Every restore entry point resolves the repository from `m.settings.GetOffboxTarget()` and the password from the single `offboxPwPath()` file: `offboxLatestSnapshot` (`offbox_restore.go:85-86`), `offboxSnapshotSize` (`:139-140`), `RestoreOffboxScratch` (`:206+`). **There is no repo-path parameter anywhere in the chain** — a grep for one returns nothing. The only existing seam that installs a recovered password, `InjectOffboxPassword` (`offbox.go:667`), writes that same one file, i.e. **ADOPTS** the set-aside store as the machine's current target. So the two options are (a) thread an alternative (repo, password) through three functions plus the UI, or (b) adopt — and adoption is a different product decision. **The session HALTED here by its own rule** and shipped R-311 alone. What the drill did to read the set-aside store was `restic` by hand with `-r <alt repo>` and an overridden `RESTIC_PASSWORD_FILE`; **that distance is exactly what (b)-to-(c) costs** | **READY (L) — NEW 2026-08-12** | R-304, R-311 | Decide (a) vs (b). If (a): the parameter is small and contained; the UI question ("which package?" when a customer has several) is the real design work. **Do not choose (b) without deciding what happens to the machine's current off-site history** | operator + CC |
|
||||||
|
| **R-313** | **`demo-felhom`'s set-aside store is UNRECOVERABLE — 36 snapshots whose key we destroyed ourselves.** `/home/felhom-repo.orphaned-20260810` holds **36 snapshot objects and exactly one key slot**, and it does NOT open with the box's current password (`Fatal: wrong password or no key found`, exit 1 — measured). Its password is the one hashed `48741892f0ef4d59…`, which is retained row id 4 — **`identity_blob` NULL**, a pre-v0.93.0 row. So the material was dropped by the R-198 defect during its two-month window, and no recovery code in existence opens that store. **This is the concrete, still-present cost of R-198**, sitting on the endpoint rather than in a post-mortem. It also means the operator's ruling to KEEP it (R-307, countdown cancelled — see below) preserves bytes nobody can read: correct as a decision, and worth knowing as a fact | **READY (S) — NEW 2026-08-12** | R-198, R-307 | Decide whether to keep paying storage for 36 unreadable snapshots, or delete them deliberately and record why. **Not urgent and not automatic — but it should be a decision rather than an accumulation** | operator |
|
||||||
|
| **R-314** | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route | **READY (S) — NEW 2026-08-12, RANK 3** | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC |
|
||||||
|
| **R-315** | **The wire-contract gate's positive control FAILS on the new wire: it checks name-presence, not decodability.** R-311 declared `hub -> agent (GET /escrow/retained)` as a fourth ROOT, and the gate's tag count rose 174 → 182, so the fields ARE inspected. But renaming the agent-side `superseded_at` json tag to `superseded_at_RENAMED` **still passed** — because the string `superseded_at` also occurs as a map key in the agent's local-API response, and the check is a repo-wide name search. The gate documents this ("name-reachability is not use"), so it is a known limit rather than a regression — but it means **declaring this wire bought documentation, not enforcement**, and a report that claimed coverage would have been wrong. The mutation was asserted to have applied before the run | **READY (M) — NEW 2026-08-12, RANK 3** | R-311 | Make the check resolve the RECEIVER'S mirror type and compare field-by-field, or state per-root which kind of check it got. **A gate whose positive control fails is an instrument nobody has calibrated** | CC |
|
||||||
|
|||||||
@@ -1,3 +1,35 @@
|
|||||||
|
## v0.103.0 — a host can read the packages we kept for it (2026-08-12, R-311)
|
||||||
|
|
||||||
|
**`ListSupersededEscrow` had zero production callers for nineteen days.** It is the only reader of a
|
||||||
|
retained `identity_blob`, and nothing called it — so the retention shipped in v0.93.0 was material the
|
||||||
|
product could not reach. Proven on the fixture 2026-08-12: a recovery code that opens a retained
|
||||||
|
package (unsealed by hand, and it restored planted files byte-identical from a store the box itself
|
||||||
|
could no longer open) was answered by the product as a code that opened nothing.
|
||||||
|
|
||||||
|
**New: `GET /api/v1/hosts/<id>/escrow/retained`** — the retained siblings of the existing
|
||||||
|
`/escrow` row, self-scoped to the asking host exactly as that one is, behind the same §8.3
|
||||||
|
recovery-mode gate, and audited with the same `escrow_blob_served` event BEFORE the bytes leave.
|
||||||
|
|
||||||
|
**What it will not serve.** Rows whose `identity_blob` is NULL — every pre-v0.93.0 row — are withheld
|
||||||
|
and returned as `unopenable_count` instead. They retain the PBS datastore key and not the repository
|
||||||
|
password, so they can never open what the caller is asking about; serving them would have the agent
|
||||||
|
try packages that cannot succeed and would let the recovery screen claim an earlier package is
|
||||||
|
openable on exactly the boxes the original defect hurt. The count is returned because their EXISTENCE
|
||||||
|
is load-bearing and underivable: on such a box a correct old code opens nothing, and the honest reason
|
||||||
|
is a defect of ours rather than the customer's typing.
|
||||||
|
|
||||||
|
**The trade, stated rather than waved through.** The hub still cannot read any of it — sealed bytes in,
|
||||||
|
sealed bytes out, no decrypt path, no recovery code ever held. What widens is VOLUME: a host key that
|
||||||
|
could fetch one opaque package can now fetch N. Bounded by the same self-scope, the same recovery-mode
|
||||||
|
gate, and an explicit `retainedEscrowServeCap` (16) so a long supersession history cannot become an
|
||||||
|
unbounded read.
|
||||||
|
|
||||||
|
Pinned by six tests driven through the real endpoint, with four red-proofs each asserted to have
|
||||||
|
applied: serving the CURRENT row instead (count 2→1), dropping the unopenable guard (count 1→2,
|
||||||
|
unopenable 1→0), dropping self-scope (403→200), and collapsing the route suffix (count 1→0).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## v0.102.0 — the hub refuses to vouch a version that cannot be installed (2026-08-09, R-273)
|
## v0.102.0 — the hub refuses to vouch a version that cannot be installed (2026-08-09, R-273)
|
||||||
|
|
||||||
**The guard owed since 2026-08-09 morning.** Agent v0.128.0 had been published as a package and never
|
**The guard owed since 2026-08-09 morning.** Agent v0.128.0 had been published as a package and never
|
||||||
|
|||||||
@@ -0,0 +1,218 @@
|
|||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-311 (v0.103.0) — GET /hosts/<id>/escrow/retained.
|
||||||
|
//
|
||||||
|
// WHY THESE TESTS EXIST. `ListSupersededEscrow` had zero production callers for nineteen days; the
|
||||||
|
// retention it reads was proven on 2026-08-12 to hold the right material and to open a store the box
|
||||||
|
// itself could not, while the customer was told their (correct) code opened nothing. These tests pin
|
||||||
|
// the endpoint that ends that, and — more importantly — they pin the two things it must NOT do:
|
||||||
|
// serve a package that cannot be opened, and let one host read another's.
|
||||||
|
|
||||||
|
type retainedResp struct {
|
||||||
|
HostID string `json:"host_id"`
|
||||||
|
Count int `json:"count"`
|
||||||
|
UnopenableCount int `json:"unopenable_count"`
|
||||||
|
TruncatedCount int `json:"truncated_count"`
|
||||||
|
Packages []struct {
|
||||||
|
Index int `json:"index"`
|
||||||
|
SupersededAt string `json:"superseded_at"`
|
||||||
|
KeyFingerprint string `json:"key_fingerprint"`
|
||||||
|
IdentityEscrowB64 string `json:"identity_escrow_b64"`
|
||||||
|
} `json:"packages"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func getRetained(t *testing.T, h *Handler, hostID, key string) (int, retainedResp) {
|
||||||
|
t.Helper()
|
||||||
|
rr := do(h, http.MethodGet, "/hosts/"+hostID+"/escrow/retained", key, "")
|
||||||
|
var out retainedResp
|
||||||
|
if rr.Code == http.StatusOK {
|
||||||
|
if err := json.Unmarshal(rr.Body.Bytes(), &out); err != nil {
|
||||||
|
t.Fatalf("decode retained response: %v (body %s)", err, rr.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return rr.Code, out
|
||||||
|
}
|
||||||
|
|
||||||
|
// seedSuperseded drives the REAL supersession path (a PUT sealing a different password demotes the
|
||||||
|
// current row) rather than inserting into host_escrow_superseded directly — so what the test proves
|
||||||
|
// includes the demote wiring, not just this handler's SELECT.
|
||||||
|
func seedSuperseded(t *testing.T, h *Handler, hostID, key string, gens []struct{ sha, identity string }) {
|
||||||
|
t.Helper()
|
||||||
|
for i, g := range gens {
|
||||||
|
if rr := do(h, http.MethodPut, "/hosts/"+hostID+"/escrow", key, escrowBodyWithHash([]byte("k"), g.sha, g.identity)); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("seed PUT %d = %d (%s)", i, rr.Code, rr.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// THE ONE THAT MATTERS: a retained package that carries key material is served, and the caller can
|
||||||
|
// tell it apart from the current row.
|
||||||
|
//
|
||||||
|
// RED-PROOF: change the handler's SELECT source from ListSupersededEscrow to GetHostDRBundle (i.e.
|
||||||
|
// serve the CURRENT row) → the served blob becomes "age-gen3" → this FAILS on the body comparison,
|
||||||
|
// which is exactly the confusion the endpoint exists to end.
|
||||||
|
func TestRetainedEscrowGet_ServesRetainedPackages(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||||
|
seedSuperseded(t, h, "h1", "HKEY", []struct{ sha, identity string }{
|
||||||
|
{"SHA_GEN1", "age-gen1"},
|
||||||
|
{"SHA_GEN2", "age-gen2"},
|
||||||
|
{"SHA_GEN3", "age-gen3"}, // current after this
|
||||||
|
})
|
||||||
|
|
||||||
|
code, out := getRetained(t, h, "h1", "HKEY")
|
||||||
|
if code != http.StatusOK {
|
||||||
|
t.Fatalf("GET retained = %d, want 200", code)
|
||||||
|
}
|
||||||
|
if out.Count != 2 || len(out.Packages) != 2 {
|
||||||
|
t.Fatalf("want 2 retained packages, got count=%d len=%d", out.Count, len(out.Packages))
|
||||||
|
}
|
||||||
|
if out.UnopenableCount != 0 {
|
||||||
|
t.Errorf("unopenable_count = %d, want 0 (every seeded row carried an identity blob)", out.UnopenableCount)
|
||||||
|
}
|
||||||
|
// The retained packages must be the SUPERSEDED generations, never the current one. If this ever
|
||||||
|
// serves gen3 the endpoint is answering the wrong question and the customer is misled again.
|
||||||
|
got := map[string]bool{}
|
||||||
|
for _, p := range out.Packages {
|
||||||
|
b, err := base64.StdEncoding.DecodeString(p.IdentityEscrowB64)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("package %d: identity blob is not base64: %v", p.Index, err)
|
||||||
|
}
|
||||||
|
got[string(b)] = true
|
||||||
|
if p.SupersededAt == "" {
|
||||||
|
t.Errorf("package %d has no superseded_at — the screen needs a date to name the package", p.Index)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !got["age-gen1"] || !got["age-gen2"] {
|
||||||
|
t.Errorf("served packages = %v, want the two SUPERSEDED generations (age-gen1, age-gen2)", got)
|
||||||
|
}
|
||||||
|
if got["age-gen3"] {
|
||||||
|
t.Error("the CURRENT package was served as retained — that is the wrong-package confusion this endpoint exists to end")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A retained row with NO identity blob (every pre-v0.93.0 row) must be WITHHELD and COUNTED. Serving
|
||||||
|
// it would have the agent try a package that cannot open anything, and would let the screen claim an
|
||||||
|
// earlier package is recoverable on exactly the boxes hurt by the original defect.
|
||||||
|
//
|
||||||
|
// RED-PROOF: delete the `if len(e.IdentityBlob) == 0 { unopenable++; continue }` guard → the empty
|
||||||
|
// row is served as a package → Count becomes 2 and UnopenableCount 0 → this FAILS twice.
|
||||||
|
func TestRetainedEscrowGet_WithholdsAndCountsUnopenableRows(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||||
|
// gen1 carries NO identity blob — the pre-v0.93.0 shape.
|
||||||
|
seedSuperseded(t, h, "h1", "HKEY", []struct{ sha, identity string }{
|
||||||
|
{"SHA_GEN1", ""},
|
||||||
|
{"SHA_GEN2", "age-gen2"},
|
||||||
|
{"SHA_GEN3", "age-gen3"},
|
||||||
|
})
|
||||||
|
|
||||||
|
code, out := getRetained(t, h, "h1", "HKEY")
|
||||||
|
if code != http.StatusOK {
|
||||||
|
t.Fatalf("GET retained = %d, want 200", code)
|
||||||
|
}
|
||||||
|
if out.Count != 1 {
|
||||||
|
t.Errorf("count = %d, want 1 (only gen2 carries material)", out.Count)
|
||||||
|
}
|
||||||
|
if out.UnopenableCount != 1 {
|
||||||
|
t.Errorf("unopenable_count = %d, want 1 — the caller cannot derive this and needs it to explain "+
|
||||||
|
"why a correct old code opens nothing on such a box", out.UnopenableCount)
|
||||||
|
}
|
||||||
|
for _, p := range out.Packages {
|
||||||
|
if p.IdentityEscrowB64 == "" {
|
||||||
|
t.Error("a package with an empty identity blob was served — it can never open anything")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// SELF-SCOPE. The current-row GET has this and a retained read is strictly more material, so the same
|
||||||
|
// asymmetry must hold: a host key reads only its own.
|
||||||
|
//
|
||||||
|
// RED-PROOF: remove the `!isGlobal && authHostID != pathHostID` branch → h2's key reads h1's packages
|
||||||
|
// → this FAILS with 200.
|
||||||
|
func TestRetainedEscrowGet_SelfScoped(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY1"})
|
||||||
|
st.UpsertHost(&store.Host{HostID: "h2", CustomerID: "c2", APIKey: "HKEY2"})
|
||||||
|
seedSuperseded(t, h, "h1", "HKEY1", []struct{ sha, identity string }{
|
||||||
|
{"SHA_GEN1", "age-gen1"},
|
||||||
|
{"SHA_GEN2", "age-gen2"},
|
||||||
|
})
|
||||||
|
|
||||||
|
rr := do(h, http.MethodGet, "/hosts/h1/escrow/retained", "HKEY2", "")
|
||||||
|
if rr.Code != http.StatusForbidden {
|
||||||
|
t.Fatalf("h2 reading h1's retained packages = %d, want 403 — a host key must never be a fleet-wide reader", rr.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A host that has never re-sealed gets a clean, empty answer — not a 404 and not a fault. "No retained
|
||||||
|
// package" is an ordinary situation and must be distinguishable from "something is broken", or the
|
||||||
|
// agent's try-loop cannot tell them apart either.
|
||||||
|
//
|
||||||
|
// RED-PROOF: make the handler 404 on an empty list → this FAILS with 404.
|
||||||
|
func TestRetainedEscrowGet_NoneIsACleanEmptyAnswer(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||||
|
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k"), "SHA1", "age-1")); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("seed PUT = %d", rr.Code)
|
||||||
|
}
|
||||||
|
|
||||||
|
code, out := getRetained(t, h, "h1", "HKEY")
|
||||||
|
if code != http.StatusOK {
|
||||||
|
t.Fatalf("GET retained on a never-superseded host = %d, want 200", code)
|
||||||
|
}
|
||||||
|
if out.Count != 0 || len(out.Packages) != 0 || out.UnopenableCount != 0 {
|
||||||
|
t.Errorf("want an empty answer, got count=%d packages=%d unopenable=%d", out.Count, len(out.Packages), out.UnopenableCount)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The retained route must not be reachable without a key at all.
|
||||||
|
//
|
||||||
|
// RED-PROOF: drop the checkAuthHost block → this FAILS with 200.
|
||||||
|
func TestRetainedEscrowGet_RequiresAuth(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/h1/escrow/retained", "", ""); rr.Code != http.StatusUnauthorized {
|
||||||
|
t.Fatalf("unauthenticated retained GET = %d, want 401", rr.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ROUTING: /escrow and /escrow/retained must stay two different answers. A prefix "tidy-up" that
|
||||||
|
// collapsed them would route retained reads to the current row — silently the wrong package.
|
||||||
|
//
|
||||||
|
// RED-PROOF: change the retained case's suffix to "/escrow" (so the earlier case wins) → the retained
|
||||||
|
// route returns the CURRENT-row shape, which has no "packages" key → this FAILS.
|
||||||
|
func TestRetainedEscrowGet_IsADistinctRouteFromTheCurrentRow(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||||
|
seedSuperseded(t, h, "h1", "HKEY", []struct{ sha, identity string }{
|
||||||
|
{"SHA_GEN1", "age-gen1"},
|
||||||
|
{"SHA_GEN2", "age-gen2"},
|
||||||
|
})
|
||||||
|
|
||||||
|
cur := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", "")
|
||||||
|
var curBody map[string]any
|
||||||
|
if err := json.Unmarshal(cur.Body.Bytes(), &curBody); err != nil {
|
||||||
|
t.Fatalf("decode current-row body: %v", err)
|
||||||
|
}
|
||||||
|
if _, hasPackages := curBody["packages"]; hasPackages {
|
||||||
|
t.Error("the CURRENT-row route grew a packages key — the two routes have been merged")
|
||||||
|
}
|
||||||
|
if _, hasPresent := curBody["present"]; !hasPresent {
|
||||||
|
t.Error("the current-row route lost its present flag")
|
||||||
|
}
|
||||||
|
|
||||||
|
_, ret := getRetained(t, h, "h1", "HKEY")
|
||||||
|
if ret.Count != 1 {
|
||||||
|
t.Fatalf("retained route count = %d, want 1 — it is answering the current row's question", ret.Count)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -245,6 +245,13 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
case r.Method == http.MethodGet && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
|
case r.Method == http.MethodGet && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
|
||||||
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
|
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
|
||||||
h.handleHostEscrowGet(w, r, hostID)
|
h.handleHostEscrowGet(w, r, hostID)
|
||||||
|
// R-311 (v0.103.0): the RETAINED siblings of the row above. The two cases cannot collide — one
|
||||||
|
// ends `/escrow`, the other `/escrow/retained` — but do NOT "tidy" them into a single prefix
|
||||||
|
// match: a prefix match would route retained reads to the CURRENT row, which is silently the
|
||||||
|
// wrong package and is exactly the confusion this endpoint exists to end.
|
||||||
|
case r.Method == http.MethodGet && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow/retained"):
|
||||||
|
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow/retained")
|
||||||
|
h.handleHostEscrowRetainedGet(w, r, hostID)
|
||||||
// G1 break-glass: day-0 vaults the root@pam console credential (self-scoped host key); the
|
// G1 break-glass: day-0 vaults the root@pam console credential (self-scoped host key); the
|
||||||
// operator retrieves it via the /admin/ path (global key only).
|
// operator retrieves it via the /admin/ path (global key only).
|
||||||
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/recovery-credential"):
|
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/recovery-credential"):
|
||||||
@@ -1371,6 +1378,158 @@ func (h *Handler) handleHostEscrowGet(w http.ResponseWriter, r *http.Request, pa
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// handleHostEscrowRetainedGet serves a host its own RETAINED (superseded) sealed identity packages.
|
||||||
|
//
|
||||||
|
// ── WHY THIS EXISTS (R-311) ─────────────────────────────────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// `ListSupersededEscrow` has been the only reader of a retained identity blob since v0.93.0 and had
|
||||||
|
// ZERO production callers — every call site was a test. The consequence, measured on the fixture
|
||||||
|
// 2026-08-12: a recovery code that demonstrably opens a retained package (proven by hand: unsealed,
|
||||||
|
// and it restored planted files byte-identical from a store the box could no longer open) was
|
||||||
|
// reported to the customer as a code that opened nothing. The screen already hedged that with two
|
||||||
|
// possible causes (R-222/R-226) and said it could not tell them apart. **It could not tell them
|
||||||
|
// apart because nothing ever looked.** This endpoint is what makes looking possible.
|
||||||
|
//
|
||||||
|
// ── WHAT THIS DOES NOT CHANGE ───────────────────────────────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// The hub still cannot read any of it. Sealed bytes in, sealed bytes out; there is no decrypt path
|
||||||
|
// here and the hub has never held a recovery code. What DOES widen is volume: a host key that could
|
||||||
|
// previously fetch one opaque package can now fetch N. The trade is stated rather than waved through
|
||||||
|
// — see the audit — and it is bounded three ways: the same self-scope as the current row, the same
|
||||||
|
// recovery-mode gate, and an explicit cap so a host with a long supersession history cannot turn one
|
||||||
|
// request into an unbounded read.
|
||||||
|
//
|
||||||
|
// Rows whose `identity_blob` is NULL are NOT served and are counted separately. They are the
|
||||||
|
// pre-v0.93.0 rows; they retain the PBS key and not the repository password, so they can never open
|
||||||
|
// anything the caller is asking about. Serving them would make the agent try packages that cannot
|
||||||
|
// succeed and would let the screen claim an earlier package is openable when it is not — the same
|
||||||
|
// false-explanation trap `SupersededPresent` already avoids at store.go. They are counted because
|
||||||
|
// their EXISTENCE is a true and load-bearing fact: on those boxes a correct old code opens nothing,
|
||||||
|
// and the honest reason is a defect of ours, not the customer's typing.
|
||||||
|
const retainedEscrowServeCap = 16
|
||||||
|
|
||||||
|
// RetainedEscrowPackage / RetainedEscrowResponse are the WIRE for GET /hosts/<id>/escrow/retained.
|
||||||
|
//
|
||||||
|
// They are named types rather than a `map[string]any` on purpose: the wire-contract gate resolves a
|
||||||
|
// declared ROOT by TYPE, so an untyped map is a cross-repo contract the gate cannot see. This wire is
|
||||||
|
// declared in `scripts/wire_contract_gate.py` ROOTS — hub → agent — and the agent's mirror is
|
||||||
|
// `felhom-agent/internal/hub.RetainedEscrowResponse`. Change a tag here and the gate fails there,
|
||||||
|
// which is the entire point.
|
||||||
|
type RetainedEscrowPackage struct {
|
||||||
|
// Index is a label WITHIN ONE RESPONSE. Not durable, never persisted, never a lookup key.
|
||||||
|
Index int `json:"index"`
|
||||||
|
// SupersededAt is when this package stopped being the current one. It is the only field here a
|
||||||
|
// customer ever sees — it is how they recognise which recovery code they are holding.
|
||||||
|
SupersededAt string `json:"superseded_at"`
|
||||||
|
KeyFingerprint string `json:"key_fingerprint"`
|
||||||
|
// IdentityEscrowB64 is OPAQUE ciphertext. The hub cannot open it and never could.
|
||||||
|
IdentityEscrowB64 string `json:"identity_escrow_b64"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type RetainedEscrowResponse struct {
|
||||||
|
HostID string `json:"host_id"`
|
||||||
|
Count int `json:"count"`
|
||||||
|
// UnopenableCount is the number of retained rows withheld because they carry no key material
|
||||||
|
// (every pre-v0.93.0 row). Underivable by the caller and load-bearing: on a box with only those,
|
||||||
|
// a perfectly correct old code opens nothing and the reason is a defect of ours.
|
||||||
|
UnopenableCount int `json:"unopenable_count"`
|
||||||
|
TruncatedCount int `json:"truncated_count"`
|
||||||
|
Packages []RetainedEscrowPackage `json:"packages"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *Handler) handleHostEscrowRetainedGet(w http.ResponseWriter, r *http.Request, pathHostID string) {
|
||||||
|
authHostID, _, isGlobal, ok := h.checkAuthHost(r)
|
||||||
|
if !ok {
|
||||||
|
http.Error(w, "Unauthorized", http.StatusUnauthorized)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if pathHostID == "" {
|
||||||
|
http.Error(w, "Missing host_id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// SELF-SCOPED, identically to the current-row GET. Without this line any host key is a fleet-wide
|
||||||
|
// reader of every retained package, which is strictly worse than the same hole on one row.
|
||||||
|
if !isGlobal && authHostID != pathHostID {
|
||||||
|
h.logger.Printf("[WARN] retained escrow GET REFUSED: host %s asked for %s's packages (self-scope)", authHostID, pathHostID)
|
||||||
|
http.Error(w, "Forbidden: host_id mismatch", http.StatusForbidden)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
host, err := h.store.GetHost(pathHostID)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if host == nil {
|
||||||
|
http.Error(w, "Unknown host_id", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// The SAME §8.3 gate as the current row. A retained package is not less sensitive than the
|
||||||
|
// current one; if self-service retrieval is ever switched off, it must go dark with it.
|
||||||
|
if !escrowSelfServiceRetrieval && !host.InRecoveryMode(time.Now().UTC()) {
|
||||||
|
h.logger.Printf("[WARN] retained escrow GET REFUSED for %s — self-service retrieval is disabled and recovery mode is not armed", pathHostID)
|
||||||
|
http.Error(w, "Forbidden: host not in recovery mode (operator must arm it)", http.StatusForbidden)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
retained, rerr := h.store.ListSupersededEscrow(pathHostID)
|
||||||
|
if rerr != nil {
|
||||||
|
h.logger.Printf("[ERROR] retained escrow GET for %s: %v", pathHostID, rerr)
|
||||||
|
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
packages := make([]RetainedEscrowPackage, 0, len(retained))
|
||||||
|
unopenable := 0
|
||||||
|
for _, e := range retained {
|
||||||
|
if len(e.IdentityBlob) == 0 {
|
||||||
|
unopenable++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if len(packages) >= retainedEscrowServeCap {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
packages = append(packages, RetainedEscrowPackage{
|
||||||
|
Index: len(packages),
|
||||||
|
SupersededAt: e.UpdatedAt,
|
||||||
|
KeyFingerprint: e.KeyFingerprint,
|
||||||
|
IdentityEscrowB64: base64.StdEncoding.EncodeToString(e.IdentityBlob),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
truncated := 0
|
||||||
|
if n := len(retained) - unopenable; n > len(packages) {
|
||||||
|
truncated = n - len(packages)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The audit row is written BEFORE the bytes leave, exactly as the current-row GET does, so a
|
||||||
|
// retrieval cannot be served without its record. Severity matches that path deliberately: this is
|
||||||
|
// a recovery in progress and nothing else, and it must reach the operator by e-mail.
|
||||||
|
if host.CustomerID != "" && len(packages) > 0 {
|
||||||
|
msg := fmt.Sprintf("Retained recovery packages served: host %s retrieved %d retained sealed package(s) (%d unopenable pre-v0.93.0 row(s) withheld). "+
|
||||||
|
"This is the recovery path in use — the packages cannot be opened without the customer's recovery code, which the hub never holds. "+
|
||||||
|
"If no recovery is in progress on that box, investigate.", pathHostID, len(packages), unopenable)
|
||||||
|
details, _ := json.Marshal(map[string]any{
|
||||||
|
"host_id": pathHostID,
|
||||||
|
"served": len(packages),
|
||||||
|
"unopenable": unopenable,
|
||||||
|
"truncated": truncated,
|
||||||
|
"self_scope": !isGlobal,
|
||||||
|
})
|
||||||
|
if _, eerr := h.store.SaveEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub"); eerr != nil {
|
||||||
|
h.logger.Printf("[WARN] %s event save FAILED for %s (serving anyway): %v", eventEscrowBlobServed, pathHostID, eerr)
|
||||||
|
} else if h.dispatcher != nil {
|
||||||
|
go h.dispatcher.ProcessEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
h.logger.Printf("[WARN] retained escrow SERVED to host %s (%d package(s), %d unopenable withheld, %d over cap, self_scope=%v)",
|
||||||
|
pathHostID, len(packages), unopenable, truncated, !isGlobal)
|
||||||
|
writeJSON(w, http.StatusOK, RetainedEscrowResponse{
|
||||||
|
HostID: pathHostID,
|
||||||
|
Count: len(packages),
|
||||||
|
UnopenableCount: unopenable,
|
||||||
|
TruncatedCount: truncated,
|
||||||
|
Packages: packages,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
// maybeEmitRepoKeyChanged raises ONE operator signal per supersession when the sealed offsite repo
|
// maybeEmitRepoKeyChanged raises ONE operator signal per supersession when the sealed offsite repo
|
||||||
// password demonstrably changed. Both hashes have been stored since SLICE 3 (host_escrow and, since
|
// password demonstrably changed. Both hashes have been stored since SLICE 3 (host_escrow and, since
|
||||||
// v0.60.0, host_escrow_superseded) and NOTHING compared them: demo-felhom's repository password
|
// v0.60.0, host_escrow_superseded) and NOTHING compared them: demo-felhom's repository password
|
||||||
|
|||||||
@@ -81,6 +81,12 @@ ROOTS = [
|
|||||||
"controller", "internal/report", "Report", "hub"),
|
"controller", "internal/report", "Report", "hub"),
|
||||||
("hub -> controller (report ACK, `escrow` object)",
|
("hub -> controller (report ACK, `escrow` object)",
|
||||||
"hub", "internal/store", "EscrowStatus", "controller"),
|
"hub", "internal/store", "EscrowStatus", "controller"),
|
||||||
|
# R-311. Declared the moment the wire was created, because the gate covers only what is DECLARED
|
||||||
|
# and a silent pass is indistinguishable from coverage. The hub response is a named type rather
|
||||||
|
# than a map[string]any precisely so this root can resolve — an untyped map is a cross-repo
|
||||||
|
# contract nothing can check.
|
||||||
|
("hub -> agent (GET /hosts/<id>/escrow/retained)",
|
||||||
|
"hub", "internal/api", "RetainedEscrowResponse", "agent"),
|
||||||
]
|
]
|
||||||
|
|
||||||
# Tag names whose literal string carries no information in a repo-wide search. NOT CHECKED.
|
# Tag names whose literal string carries no information in a repo-wide search. NOT CHECKED.
|
||||||
|
|||||||
Reference in New Issue
Block a user