Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| d102ca5767 | |||
| 684cd2eb11 | |||
| 4906aeb3f9 | |||
| 8b188bea68 | |||
| 6362bb6cb6 | |||
| 1d4985d87c | |||
| c1319a91a8 |
+23
@@ -15,6 +15,29 @@
|
||||
> would make one of the two audiences stop reading. `STATUS.md` is also a **view of `OPEN-ITEMS.md`**
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
## Material retained is not history recoverable — ask the three questions separately (2026-08-12)
|
||||
|
||||
**"We keep the old key" and "the customer can get their old backups back" are three questions, and
|
||||
collapsing them is how a false belief survived a week.** Ask them one at a time, and answer each with
|
||||
its own evidence:
|
||||
|
||||
**(a) Is the material retained?** **(b) Does it, once unsealed, still open the old store?**
|
||||
**(c) Does the customer have a ROUTE to (b) through the product?**
|
||||
|
||||
Measured 2026-08-12, and the answers were **yes, yes, no** — a shape nobody had predicted. Our records
|
||||
predicted (a) yes and (b) no ("a box we fill and cannot open"); the drill found (b) works and **(c) was
|
||||
the missing one.** `ListSupersededEscrow` had held the right bytes since v0.93.0 with **zero production
|
||||
callers**, so the capability existed and was unreachable. The same afternoon's follow-up shipped a
|
||||
partial (c) — the product now RECOGNISES a correct code for an earlier package — and established that
|
||||
the rest of (c) is new surface, not wiring, because every restore entry point resolves its repository
|
||||
from settings and its password from one file (R-312).
|
||||
|
||||
**The generalisation, which is the part worth keeping:** a stored artifact proves storage, never
|
||||
usability. Between "the bytes are kept" and "the customer is helped" sit an unseal, a consumer and a
|
||||
route, and each has been separately absent in this project. **Before claiming a capability, name which
|
||||
of the three you have evidence for** — and if a comment or a card asserts the third, find the code path
|
||||
that performs it or move the claim.
|
||||
|
||||
## Two rules earned twice each (2026-08-12)
|
||||
|
||||
**A fact must be recorded at the moment it is a fact, not inferred later from something adjacent.**
|
||||
|
||||
@@ -1,87 +1,169 @@
|
||||
# REPORT — DRILL: the retained key, and the two fixes nobody had watched work (2026-08-12)
|
||||
# REPORT — The door, part one: a correct code stops being called wrong (2026-08-12 night)
|
||||
|
||||
**Class:** drill (unattended, destructive on Tier 0) + spike for Phase C's first step
|
||||
**Venues:** `drill-r50` (nested PVE on DooPlex), `demo-felhom` (guest 9201) — both Tier 0.
|
||||
**`demo-hp` was never touched. `peti-felhom` was never contacted. No abandon countdown was started,
|
||||
shortened or triggered.**
|
||||
**Full record:** `documentation/audits/DRILL-retained-key-2026-08-12.md`
|
||||
**Three repos.** hub **v0.103.0** · agent **v0.129.0** · controller **v0.214.0** · golden **0.214.0**.
|
||||
Register: **R-311 CLOSED**, R-307 CLOSED, **R-312 / R-313 / R-314 / R-315 opened**. Ceiling
|
||||
R-310 → **R-315**.
|
||||
|
||||
---
|
||||
|
||||
## The answer to the question this drill existed to answer
|
||||
## 1. The spike's answer, first and in plain language
|
||||
|
||||
**(a) Is the old key kept? YES** — proven for the first time in the fleet's history.
|
||||
**(b) Does the kept key open the old backups? YES** — three planted files, including a Hungarian
|
||||
accented filename verified as raw bytes, restored **byte-identical** from a store the machine itself
|
||||
could no longer open.
|
||||
**(c) Can the customer get there through the product? NO — and they are told their correct code is
|
||||
wrong.**
|
||||
**Can a customer restore from a set-aside store with the machinery that already exists? NO — and
|
||||
building it is new surface, not wiring.** Established read-only, at `file:line`, before a line was
|
||||
written.
|
||||
|
||||
The brief said to be ready for the answer to be no, and our own records predicted retention would be
|
||||
*"a box we fill and cannot open"*. **That was half right, and the wrong half was the one nobody had
|
||||
checked.** The box opens. What does not exist is the door: `ListSupersededEscrow`
|
||||
(`hub/internal/store/store.go:2841`) is the only reader of a retained key and has **zero production
|
||||
callers**; the recovery path selects `FROM host_escrow` — the current row only. Asked with the very
|
||||
code that had just opened the retained row by hand, the product answered *"the recovery code did not
|
||||
open the sealed bundle — nothing was written"*. → **R-304, rank 1**
|
||||
Every restore entry point resolves the repository from `settings.GetOffboxTarget()` and the password
|
||||
from the single `offboxPwPath()` file: `offboxLatestSnapshot` (`offbox_restore.go:85-86`),
|
||||
`offboxSnapshotSize` (`:139-140`), `RestoreOffboxScratch` (`:206+`). **A grep for a repo-path
|
||||
parameter anywhere in the restore chain returns nothing.** The only seam that installs a recovered
|
||||
password, `InjectOffboxPassword` (`offbox.go:667`), writes that same one file — i.e. **adoption**.
|
||||
|
||||
Consequences: the census answer **stands**; the countdown banner's promise is **true in substance,
|
||||
false in practice**; the capability map's recovery claim **has been moved** with today's evidence.
|
||||
What the drill did to read the set-aside store was `restic` **by hand**, with `-r <alt repo>` and an
|
||||
overridden `RESTIC_PASSWORD_FILE`. **That distance is exactly what (b)-to-(c) costs.**
|
||||
|
||||
## What shipped
|
||||
**So the session halted at Part 3 by its own rule, shipped Part 2, and hands back options → R-312.**
|
||||
Cost to find out: ~35 minutes, read-only.
|
||||
|
||||
**`installer-v1.27.0` published** — tag cut and **both** `--ref`s in `manifests/webpage.yaml` bumped
|
||||
(sidecar line 327, init container line 372). Publication was earned: both faults were watched
|
||||
happening first, from a machine reset to factory state.
|
||||
## 2. Part 0 — the countdown, cancelled on your ruling
|
||||
|
||||
- **R-300 CLOSED** — pre-fix uninstall left dnsmasq `enabled`/`active` on `0.0.0.0:53`; the next byo
|
||||
install refused, exit 1. Fixed path: recorded `not present before Felhom` → `stopping + disabling
|
||||
it` → `:53 FREE` → preflight PASS. The owner's side proven too (record `yes` → left running).
|
||||
- **R-297 CLOSED** — a stale `golden-0.98.3.tar.zst` planted as newest-by-filename; v1.25.0 took it
|
||||
with no comparison and **the box came up on controller 0.98.3** against a vouched 0.213.0 — below
|
||||
the floor and below v0.206.0 where the off-site recovery screen exists. Fixed path re-fetched and
|
||||
sha-verified the vouched golden (landed 0.213.0); an operator-named stale archive was **refused**.
|
||||
Through the product's own operator path (`--abandon-stop`, which refuses rather than silently
|
||||
no-opping), with the container **stopped first** so the running controller could not overwrite
|
||||
`settings.json` from memory. **Proved, not trusted to the exit code:**
|
||||
|
||||
## Findings opened — ceiling R-303 → R-310
|
||||
- `abandon_started_at` and `abandon_at` — **gone**. `AbandonStatus` returns `Active=false` when
|
||||
`AbandonAt` is empty (`offbox_abandon.go:111-113`), so no countdown renders.
|
||||
- `abandon_repo_path` — **deliberately kept**, as the pointer to the preserved store.
|
||||
- The set-aside store — **still there**: 36 snapshot objects, full `config/data/index/keys/locks/
|
||||
snapshots` structure. Both repositories still on the endpoint. **Nothing deleted anywhere.**
|
||||
|
||||
| # | Rank | What |
|
||||
**And the thing you should know about what was preserved (R-313):** it holds **36 snapshots and one
|
||||
key slot**, and it does **not** open with the box's current password (`Fatal: wrong password or no key
|
||||
found`, exit 1 — measured). Its key is the one hashed `48741892f0ef…` — retained row id 4,
|
||||
`identity_blob` **NULL**, a pre-v0.93.0 row. **The material was dropped by the R-198 defect during its
|
||||
two-month window, so no recovery code in existence opens that store.** Keeping it is still the right
|
||||
call — deleting is irreversible and a decision, not an accumulation — but it is 36 unreadable
|
||||
snapshots, and that is the concrete, still-present cost of R-198 sitting on the endpoint.
|
||||
|
||||
## 3. Part 2 — what shipped, and a correction to the premise
|
||||
|
||||
**The premise needed correcting first.** The task described the customer being told *"the recovery code
|
||||
did not open the sealed bundle"*. That is the **agent's local-API** reply. The **customer-facing
|
||||
screen already hedged** (R-222/R-226) — it named both causes, named the kept package and its date, and
|
||||
said it could not tell them apart. That was **honest**; it could not tell them apart **because nothing
|
||||
ever looked**. So what shipped is smaller and more precise than "stop the lie": **the hedge becomes an
|
||||
answer.**
|
||||
|
||||
- **hub v0.103.0** — `GET /hosts/<id>/escrow/retained`, the **first production caller
|
||||
`ListSupersededEscrow` has ever had**. Self-scoped identically, same recovery-mode gate, same audit
|
||||
event written before the bytes leave, capped at 16. Rows with a NULL `identity_blob` are **withheld
|
||||
and counted** (`unopenable_count`): they can never open what the caller is asking about, and serving
|
||||
them would let the screen promise recovery on exactly the boxes the original defect hurt.
|
||||
- **agent v0.129.0** — retained packages tried **only after** the current one refuses; `422` with
|
||||
`superseded_at`; bounded at 6 attempts (~1 s of scrypt each); fail-safe in every direction.
|
||||
- **controller v0.214.0** — class `RecoveryCodeOpensRetained`, gated on MinAgent **0.129.0** via a
|
||||
**second, separate** trust flag (a box can sit between 0.126.0 and 0.129.0). The message says the
|
||||
code is correct, names the date, says the package is kept, says the **current** backups are
|
||||
unaffected, and **promises no restore** — it routes to support, which can do it.
|
||||
|
||||
**The trade you should see stated:** the hub still cannot read any of it — sealed bytes in, sealed
|
||||
bytes out, no decrypt path, no recovery code ever held. What widens is **volume**: a host key that
|
||||
could fetch one opaque package can now fetch N, bounded by self-scope, the recovery-mode gate and the
|
||||
cap.
|
||||
|
||||
## 4. Red-proofs — and where the lie actually lives
|
||||
|
||||
Every mutation asserted to have applied before its run.
|
||||
|
||||
| Repo | Mutation | Outcome |
|
||||
|---|---|---|
|
||||
| **R-304** | **1** | Retained key has no product route; the correct old code is reported as wrong |
|
||||
| **R-305** | 2 | The R-300 cleanup fires **once per machine** — the leftover returns on the second reinstall (proven, cycles 2/3) |
|
||||
| **R-308** | 2 | Stored controller `PASSWORD` no longer opens demo-felhom (`Hibás jelszó`) — not the quoting trap |
|
||||
| **R-306** | 3 | `--preflight-only` says *"no state written"* and writes `state.json` — with an ownership answer that can be wrong |
|
||||
| **R-309** | 3 | The day-0 runbook says pushing publishes the installer; false since R-110 (measured: public URL served 1.25.0 while `main` had 1.27.0) |
|
||||
| **R-310** | 4 | Duplicated sentence in the golden refusal; `--uninstall` needs a pty and `--force` does not bypass it |
|
||||
| **R-307** | — | **Operator decision, deadline 2026-08-24** — see below |
|
||||
| hub | serve the CURRENT row instead of retained | FAILS (count 2→1) |
|
||||
| hub | drop the unopenable guard | FAILS (count 1→2, unopenable 1→0) |
|
||||
| hub | drop self-scope | FAILS (403→200) |
|
||||
| hub | collapse the route suffix | FAILS (count 1→0) |
|
||||
| **agent** | **remove the retained lookup** | **FAILS — the fail-closed wrong-code error returns. THE LIE COMES BACK.** |
|
||||
| agent | + 6 more (nil fetcher, wrong code, fetch failure, bounded attempts, predates-field, success path) | all pinned |
|
||||
| controller | delete the new case | FAILS — but the customer gets the **neutral** message, because R-224's safe default catches it |
|
||||
| controller | make 422 unconditional | FAILS — an agent that never looked is read as having looked |
|
||||
| controller | route 400 to the new class | FAILS — a mistype is congratulated |
|
||||
|
||||
## What needs you
|
||||
**Answering the question directly:** the lie returns when the **agent's** retained lookup is removed,
|
||||
not when the controller's case is. R-224's safe default is doing its job one layer up.
|
||||
|
||||
**`demo-felhom` carries a live abandon countdown** — started 2026-08-10, **firing 2026-08-24**, for
|
||||
`/home/felhom-repo.orphaned-20260810`. This drill did **not** start it and deliberately did **not**
|
||||
cancel it. The brief's end state asked for no countdown anywhere; satisfying that means choosing:
|
||||
**cancel it** (copy kept indefinitely, storage cost, no data risk) or **let it run** (copy deleted,
|
||||
irreversibly). **Doing nothing selects deletion.** → **R-307**
|
||||
## 5. The claim guard, and a gate whose positive control failed
|
||||
|
||||
## End state
|
||||
**The claim guard had a blind spot the size of the recovery screen** — it scanned templates only,
|
||||
while every recovery message is a Go string in a handler. It now scans `recovery_handlers.go` too, and
|
||||
**on its first run convicted a pre-existing unregistered claim**. 8 → 10 registered claims.
|
||||
|
||||
- **`demo-felhom`** — up, reporting, healthy, on the vouched pair; `repo_password` restored to the
|
||||
original (`sha c60c8bc737a6b7c6…`), escrow re-sealed and uploaded, off-site repo reachable
|
||||
(`restic snapshots` exit 0). Its recovery code was rotated by the final ceremony and
|
||||
`R_DEMO-FELHOM` updated in place (prior file backed up alongside). Planted data removed; eight
|
||||
secret-bearing files **shredded**.
|
||||
- **`demo-hp`** — untouched, reporting.
|
||||
- **`drill-r50`** — **reverted to snapshot `virgin`, powered off.**
|
||||
- **Hub** — two new retained rows (the P1 and P2 blobs), deliberately kept as the fixture proving the
|
||||
retention works. `drill-r50-0a4f9a` re-used, not duplicated: no new scratch customer.
|
||||
- **Off-site** — only demo-felhom's own repository path touched, `backup` the only mutating verb used.
|
||||
**No prune, no forget, no delete, no rename anywhere.** One snapshot added and deliberately left:
|
||||
`6ea85413`, 66 KiB, tagged `drill-retained-key-20260812` — removable by ID if you want it gone.
|
||||
**The wire-contract gate: declared, and honestly weaker than it looks (R-315).** The hub response was
|
||||
made a **named type** so the gate could resolve it; the wire is declared as a fourth ROOT and the tag
|
||||
count rose **174 → 182**, so the fields are inspected. But a positive control — renaming the
|
||||
agent-side `superseded_at` tag — **still passed**, because the check is repo-wide name-presence and
|
||||
the string also occurs as a map key elsewhere. The gate documents this ("name-reachability is not
|
||||
use"), so it is a known limit, not a regression — **but declaring this wire bought documentation, not
|
||||
enforcement**, and saying otherwise would have been false.
|
||||
|
||||
## Honest gaps
|
||||
## 6. Live state
|
||||
|
||||
- **The Phase A logs did not survive** the intermediate revert to `virgin`. Every quotation in the
|
||||
audit is verbatim from the live run, but the raw files are gone. Procedural lesson, recorded.
|
||||
- The planted data reached the store via `restic` directly, not the dashboard button, because of
|
||||
R-308 — so the app-backup→unit→offsite chain went unexercised. Not what this drill measured.
|
||||
- Wall clock **≈ 1 h 13 min** against a 4–5 h envelope. Nothing was dropped; Phase C ran concurrently
|
||||
with Phase B on a different machine.
|
||||
| | |
|
||||
|---|---|
|
||||
| controller | **0.214.0** on guest 9201, `Up … (healthy)` |
|
||||
| agent | **0.129.0** on `felhom-pve`, unit active, journal clean |
|
||||
| hub | **0.103.0** — see §7 |
|
||||
| golden | **0.214.0** baked + published |
|
||||
|
||||
### Live proof on hardware — the 422, end to end
|
||||
|
||||
```
|
||||
OLD code (opens retained row 11) HTTP 422 opens_retained: True
|
||||
superseded_at: 2026-08-12T15:18:55Z
|
||||
retained_has_restic_pw: True
|
||||
"the recovery code is correct, but it belongs to an
|
||||
EARLIER sealed package (superseded …), not the one
|
||||
currently held"
|
||||
WRONG code (negative control) HTTP 400 "the recovery code did not open the sealed bundle"
|
||||
```
|
||||
|
||||
The hub half measured directly too: `GET …/escrow/retained` → **200**, `count=2`,
|
||||
**`unopenable_count=1`** — that one being retained row id 4, the pre-v0.93.0 row whose material R-198
|
||||
destroyed. The withholding rule is doing exactly what it was written for, on real data.
|
||||
|
||||
**What was NOT walked, and why.** The customer's rendered sentence was **not** produced end-to-end.
|
||||
`recoveryUnlockHandler` redirects to `/backups/remote` when `!recoveryOffer()`, and `demo-felhom`
|
||||
holds its own repository password again (restored yesterday), so it is correctly **not** in the
|
||||
offered state. Walking it would mean removing that password to fake a rebuilt box — destabilising a
|
||||
healthy machine to render a sentence whose logic is pinned by six handler tests and whose upstream 422
|
||||
is proven live. I did not. **Method stated: endpoint-level for the agent and hub, handler-level for the
|
||||
message.** What the customer DOES see on this box today is the orphan card, and it is honest:
|
||||
*„Megnyitni innen egyelőre nem lehet, és ez nem a kódodon múlik."*
|
||||
|
||||
### A correction I have to make about my own last report — R-308 was wrong
|
||||
|
||||
I reported that the stored controller password no longer opens `demo-felhom`. **It does.** I had
|
||||
stripped only DOUBLE quotes from the `~/.config/credentials` value; the values are wrapped in
|
||||
**SINGLE** quotes, so I was sending a literal `'` as part of the password. Unquoted correctly it is 13
|
||||
characters and logs in first try — **HTTP 302 with a session cookie**.
|
||||
|
||||
The same bug then made this session's first live R-311 test read as a **failure** (HTTP 400) for
|
||||
twenty minutes, and I nearly filed the fix as broken. It is the **third** wrong "the credential is
|
||||
stale" verdict this project has produced from that one trap. R-308 is **withdrawn**; the real lesson
|
||||
is filed with it — never let a shell decide what a secret is.
|
||||
|
||||
## 7. What was dropped, named plainly
|
||||
|
||||
- **Part 3 (the route) — HALTED at the spike, by the task's own rule.** → R-312.
|
||||
- **§7's fixture walk was not re-run end-to-end.** Yesterday's drill already proved the byte-identical
|
||||
restore from a set-aside store; today's change is upstream of it (which sentence is shown), the
|
||||
dashboard is unreachable headlessly (R-308), and the restore route does not exist (R-312). What was
|
||||
proved live is the 422 itself.
|
||||
- Explicitly out of scope and still open, so it does not read as forgotten: **R-305** (the removal fix
|
||||
helps a machine once — the tester's second reinstall still hits it), the hub emails naming the
|
||||
retired secret, **R-309** (the runbook's publication claim), the CI runs that fail with no log, the
|
||||
twenty unread facts, the nine grey claims, **R-303**.
|
||||
|
||||
## 8. Bypass, stated as required
|
||||
|
||||
`git push --no-verify` was used **once**, on `felhom-agent`. The `release-complete` gate refuses a
|
||||
CHANGELOG entry whose tag and package do not exist; `release-agent.sh` refuses a tree that is not
|
||||
pushed. Circular by construction. The bypass was immediately followed by the real release
|
||||
(`release-agent.sh 0.129.0`), and the gates were re-run afterwards: **green**.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# STATUS — what works, what's broken, what's next
|
||||
|
||||
**Updated 2026-08-12 (late evening — retained-key drill).**
|
||||
**Updated 2026-08-12 (night — the door, part one).**
|
||||
|
||||
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
|
||||
> part of it in plain words, and **nothing may exist only here**. **Items, not paragraphs. One screen.**
|
||||
@@ -8,23 +8,30 @@
|
||||
|
||||
## Waiting on you
|
||||
|
||||
*(The golden-vouch and floor-raise asks that stood here are **done** — the hub reads golden **0.213.0**,
|
||||
floor **0.213.0**, agent **0.128.0**. Checked live, not assumed.)*
|
||||
*(Vouched and delivered 2026-08-12 night: golden **0.214.0**, agent **0.129.0**, min agent **0.129.0**,
|
||||
floor **0.214.0**. **Both demo machines are now on controller 0.214.0 + agent 0.129.0** — checked on the
|
||||
boxes, not assumed. The safety guard was watched doing its job on the way: `demo-hp` was **held** at
|
||||
0.213.0 while its agent was still 0.128.0 (*"agent 0.128.0 < MinAgent 0.129.0 — controller floor
|
||||
withheld"*), and took the update 6 seconds after the agent caught up. That guard exists because a box
|
||||
once landed on a controller newer than its agent and a customer was told a correct recovery code was
|
||||
wrong; this is the first time it has been seen firing.)*
|
||||
|
||||
- **R-307 — a countdown is already running on `demo-felhom`**, started 10 Aug, **firing 24 August**,
|
||||
after which that set-aside off-site copy is deleted for good. Today's drill did not start it and
|
||||
deliberately did not cancel it. Cancel it (copy kept, costs storage) or let it run (copy deleted).
|
||||
**Doing nothing chooses deletion.**
|
||||
- **R-304 — decide what we promise about old backups.** The retention works, but only an operator can
|
||||
use it; the customer is told their correct code is wrong. Either build the door, or stop promising.
|
||||
- **R-312 — one decision, and it is the interesting one.** The customer is now told the truth about an
|
||||
older code, but there is still no button: restoring from a set-aside copy means either threading an
|
||||
alternative location through the restore code, or adopting that copy as the machine's current one.
|
||||
The second is a different product decision. **Nothing is broken while this waits.**
|
||||
- **R-313 — the copy you told me to keep cannot be opened by anyone.** `demo-felhom`'s set-aside store
|
||||
holds 36 snapshots and one key, and that key was destroyed by the bug we fixed on 4 August. Keeping
|
||||
it is still the right call; it should be a decision, not an accumulation.
|
||||
- **R-303 — one coherence decision**, ranked low: a box can still raise the "we cannot open your new
|
||||
backups" card while counting down to deleting the old ones. The two no longer contradict each other,
|
||||
but the state is odd and the wrong fix would hide a real second fault.
|
||||
|
||||
## What works
|
||||
|
||||
Both demo machines are home, healthy and reporting on the approved pair. Off-site is credentialed on
|
||||
`demo-hp` and its repository still opens with the machine's own key. `drill-r50` is blocked, as intended.
|
||||
Both demo machines are home, healthy and reporting on the approved pair — **controller 0.214.0, agent
|
||||
0.129.0**, delivered by the floor rather than by hand. Off-site is credentialed on `demo-hp` and its
|
||||
repository still opens with the machine's own key. `drill-r50` is reverted to `virgin`, powered off.
|
||||
|
||||
## Shipped
|
||||
|
||||
@@ -39,6 +46,12 @@ Both demo machines are home, healthy and reporting on the approved pair. Off-sit
|
||||
fifth deliberately left, because it is true where it renders.
|
||||
- **One name per secret, box side** (R-295): the dashboard code is „Beállító kód" everywhere;
|
||||
„Visszaállító kód" is retired. It collided with the escrow „Helyreállítási kód" and cost a real code.
|
||||
- **A correct recovery code is no longer called wrong** (R-311, three components). If a customer types
|
||||
the code for an older set of backups, the machine now checks the packages we kept, recognises it, and
|
||||
says so: *your code is correct, it belongs to an earlier package, we kept it, your current backups are
|
||||
fine, write to us*. It deliberately promises no restore, because there is no button yet.
|
||||
- **The countdown on `demo-felhom` is cancelled** on your ruling (R-307). Nothing was deleted; the
|
||||
24 August deadline is gone. See R-313 for what that copy turns out to be.
|
||||
- **Both installer fixes are now PUBLISHED** as `installer-v1.27.0` (R-297 + R-300). Each fault was
|
||||
watched happening first, on a machine reset to factory state: the old installer really did build a
|
||||
machine on a base image from July, and our own uninstall really did block our own next install.
|
||||
@@ -50,11 +63,10 @@ Both demo machines are home, healthy and reporting on the approved pair. Off-sit
|
||||
everything on it is lost.** First act of the visit: copy the ~3.6 GB off before anything is
|
||||
reinstalled — it is currently the only copy in existence. Whether it stays parked is your call and is
|
||||
deliberately left open.
|
||||
- **Kept backups can be opened — but not by the customer** (R-304). We now know for certain the old key
|
||||
is kept and **does** open the old backups: three planted files came back byte-for-byte from a store the
|
||||
machine itself could no longer read. But nothing in the product reads a kept key, so it takes an
|
||||
operator with a shell. Worse, a customer who types their **correct** old code is told it *"did not open
|
||||
the sealed bundle"*. The keeping is real; the promise that they can use it is not.
|
||||
- **Kept backups can be opened — but still only by us** (R-304 partly closed, R-312 open). The machine
|
||||
now recognises an older code and says so plainly instead of hedging. What it still cannot do is hand
|
||||
the customer their old files: that needs the restore code to accept a second location, which is real
|
||||
work rather than wiring. Today the honest answer is "your code is right, write to us" — and we can.
|
||||
- **The dnsmasq fix helps a machine once** (R-305). On a machine that never had Felhom it works. On the
|
||||
second reinstall the leftover comes back, because the package is never removed — so the machine looks,
|
||||
to our own installer, as if the household had installed it. Watched happening the same afternoon.
|
||||
@@ -64,5 +76,6 @@ Both demo machines are home, healthy and reporting on the approved pair. Off-sit
|
||||
|
||||
## Working on next
|
||||
|
||||
Your two decisions above (R-307's deadline is the hard one); then R-304's shape; then the hub naming;
|
||||
then the 2026-08-09 batch (R-279 … R-292), still untriaged against everything since.
|
||||
R-312's shape (the button, or deliberately no button); then R-305, because the tester's second
|
||||
reinstall still hits the dnsmasq wall; then the hub naming; then the 2026-08-09 batch
|
||||
(R-279 … R-292), still untriaged against everything since.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -450,8 +450,8 @@ whose credentials reach nothing else — and the only mutating verb used was `ba
|
||||
| Phase A — removal leftover | 16:50 → 17:12 | **~22 min** (4 installs, 4 uninstalls, 3 preflights) |
|
||||
| Phase B — stale golden | 17:13 → 17:34 | **~21 min** (2 full provisions) |
|
||||
| Phase C — retained key | 17:14 → 17:23 | **~25 min** (overlapped Phase B; different machine) |
|
||||
| Record, register, publication, teardown | 17:35 → 17:55 | **~20 min** |
|
||||
| **Total** | **16:42 → 17:55** | **≈ 1 h 13 min** |
|
||||
| Record, register, publication, teardown | 17:34 → 17:45 | **~11 min** |
|
||||
| **Total** | **16:42 → 17:45** | **≈ 1 h 03 min** |
|
||||
|
||||
Phase C ran concurrently with Phase B deliberately — different machines, no shared state. The
|
||||
4–5 hour envelope was not needed; nothing was hurried and nothing was dropped for time.
|
||||
|
||||
@@ -441,7 +441,7 @@ builds the receiving struct by hand cannot see a field that never decodes, which
|
||||
|---|---|---|
|
||||
| **R-264** | **Twenty-one facts the boxes report that the hub can now decode nowhere, each allowlisted with a reason rather than silently skipped — and for these the reason is "no consumer today, and one is arguably owed".** Split out of R-260 on 2026-08-08 so that closing the CLASS (gated) and fixing its sharpest instance (`operator_key_configured`) could not be mistaken for having decided what the hub should do with the rest. **The list, grouped by what a consumer would be for.** **(a) Guest-network health — `guest_net` and its seven children** (`checked_at`, `has_route`, `dhclient_alive`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`). The R-54 watchdog reports per-guest network state and self-heal counts every cycle and the hub — the component that emails the operator — models none of it. There is a live incident in this project's own record where a killed `dhclient` took a tunnel down for 1 h 15 m (`audits/INCIDENT-guest-dhclient-killed-2026-07-20.md`); a recurring-heal signal is exactly what would have surfaced it. **This is the strongest candidate of the twenty-one.** **(b) `selfupdate_pending` + `selfupdate_pending_version`** — an agent that has flipped its binary and never committed reports pending on every heartbeat so that "the operator sees WHY the version isn't advancing", and no operator can see it. **(c) `mgmt_plane.healed_recently`** — bounded: the hub DOES alarm on the `privsep_healed_at` timestamp beside it, so the recurring-clobber signal is not lost, only this flag. **(d) `restore_tests.mount_parity` + `mount_inventory`** — R-262's subject; the verdict is not lost (a mismatch fails before `Pass` is set) but the hub cannot tell a full-fidelity pass from a boot-only one. **(e) `pbs_dr.applied_at`.** **(f) Controller-side: `config_hash`, `reporting_disabled`, `stacks`, `storage.migrated_to`, `backup.last_db_dump`, `backup.last_integrity_check`** — the last two are backup-integrity timestamps, which is the "presence is not success" neighbourhood. **For each the question is the same and is NOT answered here: is it wanted? If the hub should act on it, model it and name what consults it. If it should not, the honest end is that the emitter stops sending it** — a fact emitted forever and consumed nowhere is a future false green waiting for someone to write a check against it. **Deliberately not decided in the G-1 session**, whose scope was the gate plus the operator-access instance; unilaterally removing emitters would also break the byte-identical cross-repo host-report golden and is a coordinated two-repo change | **READY** — owner Viktor |
|
||||
|
||||
| **R-265** | **A CI run can fail with NO LOG PERSISTED, and the alarm mail then points the operator at a log that does not exist.** Observed 2026-08-08 as run **264** (`650cc8a`, a **documentation-only** commit) sat between two green runs of identical gate code. **Measured rather than assumed — the shape is unmistakable:** every other run in the session took **18–34 s and has a log (HTTP 200)**; 264 took **834 s** (07:12:40 → 07:26:34 UTC) and `GET /actions/jobs/264/logs` returns **HTTP 500 — `actions_log/…/264.log.zst: file does not exist`**. The runner pod never restarted (`act-runner`, 0 restarts, 5 d 17 h uptime), so the runner did not die; the JOB hung and was reaped. **It is NOT a gate finding, and four independent facts say so:** the diff from the green run before it is Markdown only; the same content is green two commits later (run 265, `dd55a3f`, 33 s); the gate code is byte-identical across 263/264/265; and 260–262, which WERE real gate failures, all failed in under 35 s **with** logs. **THE CAUSE OF THE HANG IS UNDETERMINED and is deliberately not guessed at.** DooPlex was doing heavy work in that window (a 139 MB `kubectl cp`, and a `go run` compiling the whole hub module for the live-validation harness), which is a plausible contention story — but the box has 40 cores and sat at load ~5, so it is **not established** and is recorded as a hypothesis, not a cause. **THE FINDING THAT MATTERS IS THE SECOND-ORDER ONE, and it is this workflow's own stated purpose turned against it.** `gates.yml` exists because "a detector nobody hears is the defect R-29 filed, rebuilt one layer up", and its alarm mail says *"The failing gate names itself in the run log."* **Here there is no run log**, so an operator following that sentence finds nothing and cannot tell an infrastructure reap from a real conviction. Worse and **unverified**: the alarm step is `if: failure()`, and whether it even ran for a reaped job is unknown — if it did not, this was a red CI that alarmed nobody, which is exactly the shape the workflow was built to prevent. **Fix shape, not a decision:** (a) make the alarm mail state the run's DURATION and whether a log exists, so a log-less reap is self-identifying; (b) give the job an explicit `timeout-minutes` well under the reap so it fails fast, loudly and with a log; (c) establish whether the alarm fires at all on a reaped job — that is one deliberate test, and until it is run, "CI alarms on failure" is an assumption **SECOND SIGHTING, 2026-08-10 — and I nearly missed it.** felhom.eu CI **run 293** (`67eced8`, the demo-felhom recovery commit) **FAILED**, and its log is **unretrievable**: `/actions/jobs/293/logs` returns **HTTP 500**, while the runs either side of it (292 and 294) both return HTTP 200 with 12 290 bytes. So the failure is real and its reason is **not establishable** — exactly the shape this row was filed for. **What can be said:** the same tree passes the FULL gate set locally (`GITEA_BASE=… python3 scripts/repo_gates.py`, all eight OK), and run **294** on the very next commit passed in CI, so nothing persisted. **What cannot:** why 293 failed. **And the process failure is mine** — the end-of-session checklist says confirm your own push's CI run by ID, and I pushed 67eced8 without doing so; it was found only because the NEXT session happened to list recent runs. A pull check that is skipped is not a check| **CLOSED 2026-08-08 — `timeout-minutes: 5` on the gates job, and the alarm mail now states elapsed seconds and qualifies its own "names itself in the run log" sentence.** ⚠ **The unknown is NOT closed and must not be read as closed:** whether the `if: failure()` alarm fires for a REAPED job is still unverified. The timeout makes the reap unreachable in practice; it does not answer what happens inside one |
|
||||
| **R-265** | **A CI run can fail with NO LOG PERSISTED, and the alarm mail then points the operator at a log that does not exist.** Observed 2026-08-08 as run **264** (`650cc8a`, a **documentation-only** commit) sat between two green runs of identical gate code. **Measured rather than assumed — the shape is unmistakable:** every other run in the session took **18–34 s and has a log (HTTP 200)**; 264 took **834 s** (07:12:40 → 07:26:34 UTC) and `GET /actions/jobs/264/logs` returns **HTTP 500 — `actions_log/…/264.log.zst: file does not exist`**. The runner pod never restarted (`act-runner`, 0 restarts, 5 d 17 h uptime), so the runner did not die; the JOB hung and was reaped. **It is NOT a gate finding, and four independent facts say so:** the diff from the green run before it is Markdown only; the same content is green two commits later (run 265, `dd55a3f`, 33 s); the gate code is byte-identical across 263/264/265; and 260–262, which WERE real gate failures, all failed in under 35 s **with** logs. **THE CAUSE OF THE HANG IS UNDETERMINED and is deliberately not guessed at.** DooPlex was doing heavy work in that window (a 139 MB `kubectl cp`, and a `go run` compiling the whole hub module for the live-validation harness), which is a plausible contention story — but the box has 40 cores and sat at load ~5, so it is **not established** and is recorded as a hypothesis, not a cause. **THE FINDING THAT MATTERS IS THE SECOND-ORDER ONE, and it is this workflow's own stated purpose turned against it.** `gates.yml` exists because "a detector nobody hears is the defect R-29 filed, rebuilt one layer up", and its alarm mail says *"The failing gate names itself in the run log."* **Here there is no run log**, so an operator following that sentence finds nothing and cannot tell an infrastructure reap from a real conviction. Worse and **unverified**: the alarm step is `if: failure()`, and whether it even ran for a reaped job is unknown — if it did not, this was a red CI that alarmed nobody, which is exactly the shape the workflow was built to prevent. **Fix shape, not a decision:** (a) make the alarm mail state the run's DURATION and whether a log exists, so a log-less reap is self-identifying; (b) give the job an explicit `timeout-minutes` well under the reap so it fails fast, loudly and with a log; (c) establish whether the alarm fires at all on a reaped job — that is one deliberate test, and until it is run, "CI alarms on failure" is an assumption **SECOND SIGHTING, 2026-08-10 — and I nearly missed it.** felhom.eu CI **run 293** (`67eced8`, the demo-felhom recovery commit) **FAILED**, and its log is **unretrievable**: `/actions/jobs/293/logs` returns **HTTP 500**, while the runs either side of it (292 and 294) both return HTTP 200 with 12 290 bytes. So the failure is real and its reason is **not establishable** — exactly the shape this row was filed for. **What can be said:** the same tree passes the FULL gate set locally (`GITEA_BASE=… python3 scripts/repo_gates.py`, all eight OK), and run **294** on the very next commit passed in CI, so nothing persisted. **What cannot:** why 293 failed. **And the process failure is mine** — the end-of-session checklist says confirm your own push's CI run by ID, and I pushed 67eced8 without doing so; it was found only because the NEXT session happened to list recent runs. A pull check that is skipped is not a check| **CLOSED 2026-08-08 — `timeout-minutes: 5` on the gates job, and the alarm mail now states elapsed seconds and qualifies its own "names itself in the run log" sentence.** ⚠ **The unknown is NOT closed and must not be read as closed:** whether the `if: failure()` alarm fires for a REAPED job is still unverified. The timeout makes the reap unreachable in practice; it does not answer what happens inside one | **THIRD SIGHTING 2026-08-12 (controller run 324).** Failed after **11 minutes** where the previous run took 18 s; `/actions/runs/324/logs` and the web log endpoint both **404**. The jobs API *does* answer, and it names the failing step: **"Fetch the pushed commit and the sibling clone it needs"** — i.e. it died at the clone, before any gate ran, while a `felhom.eu` push was in flight. **Diagnosed rather than assumed:** the same sha passed all 11 gates locally on a clean tree, and an API **re-run of the same sha (326) succeeded** with no code change. So the jobs endpoint is a usable substitute for the missing log — it gives the failing STEP, which is most of what the operator needs — and `POST /actions/runs/<id>/rerun` (HTTP 201) is the cheap disambiguator between environmental and real.
|
||||
|
||||
**Explicitly still open, untouched by this session:** R-246 (the wrong stale flag on `demo-hp` —
|
||||
clearing it is an operator act hub-side), R-255, R-256, R-257, R-258, R-259, R-261, R-262, R-263, and
|
||||
@@ -607,7 +607,12 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
|
||||
| **R-304** | **The retained escrow key works, and the customer is told their correct code is wrong.** DRILL 2026-08-12 answered the three questions separately, on `demo-felhom`, with planted data. **(a) retention: WORKS** — the first retained row in fleet history to carry material (`host_escrow_superseded` id 11, `identity_blob` 572 B), byte-identical (`sha256 a10032341c8584ed…`) to the pre-supersession `host_escrow` row. **(b) the material opens the old store: YES** — unsealed with the OLD recovery code it yielded a password byte-identical to the pre-change one (`sha c60c8bc737a6b7c6…`), and restored three planted files **byte-identical** from a store the box itself could no longer open (negative control first: `Fatal: wrong password or no key found`), **including a Hungarian accented filename verified as raw bytes**. **(c) the customer's route: DOES NOT EXIST, and misinforms.** `ListSupersededEscrow` (`store.go:2841`) is the only reader of a retained `identity_blob` and has **zero production callers** — five call sites, all `_test.go`; the product path (`POST /escrow/recover-offsite-password` → `FetchIdentityEscrow` → `GetHostDRBundle`, `store.go:3152`) selects `FROM host_escrow` — the CURRENT row only. Asked for the old password with the code that demonstrably opens the retained row, the product answered **"the recovery code did not open the sealed bundle — nothing was written"**. **This is the R-224 class again**: there an unreachable hub was reported as a bad code; here a VALID code for retained history is reported as a bad code, and the customer's attempt ends there. **Consequence:** the census answer stands (it was about retention); the countdown banner's promise is true in substance and false in practice; **any capability-map claim that the customer can recover the old history with their recovery code is false today and must move** | **READY (L) — NEW 2026-08-12, RANK 1** | R-198, R-199, R-224, R-241 | Decide the shape: serve retained rows on the recovery path (needs a "which package?" choice — a customer may have several), or stop promising retrieval anywhere the customer cannot perform it. **Until one of those, the honest position is that retention is an operator-only capability.** At minimum, the refusal must stop asserting the code is wrong when the hub simply never looked | operator + CC |
|
||||
| **R-305** | **The R-300 cleanup fires exactly once per machine, and the second reinstall hits the original wall.** PROVEN on `drill-r50` 2026-08-12, not deduced. The uninstall stops+disables dnsmasq but deliberately never **purges** the package, so on every later install `dpkg-query` reports it installed, preflight records `dnsmasq_preexisting=yes`, the agent's `EnsureDnsmasq` re-enables it, and the uninstall then only *restarts* it — leaving it `enabled`/`active` on `0.0.0.0:53` exactly as before the fix. Cycle 1: `no` → disabled → `:53 FREE` → next install PASSES. Cycle 2: `yes` → left running → **cycle-3 byo preflight FAILED, exit 1**. The improved refusal does fire and names the command, so the customer is not stranded — but they must run by hand a cleanup Felhom could have run itself, on a box Felhom does own | **READY (M) — NEW 2026-08-12, RANK 2** | R-300 | Distinguish "the package pre-dates Felhom" from "a previous **Felhom** install left it" — e.g. keep the ownership record outside `state.json` (which the uninstall deletes), or record Felhom-installed-ness where the agent installs it. **Do not fix by purging on hosts we may not own** | CC |
|
||||
| **R-306** | **`--preflight-only` says "no state written" and writes state — with an answer that can be wrong.** `_state_put` short-circuits on `DRY_RUN` only (`felhom-host-install.sh:418`), so a preflight-only run creates `/var/lib/felhom-install/state.json`. Observed live: after a run whose banner read `PRE-FLIGHT PASS (mode=byo) — no state written, no install step executed`, the file existed containing `{"completed": [], "dnsmasq_preexisting": "yes"}`. Both the banner and the flag's own comment at line 226 assert the opposite. **The harm is not the file, it is the value**: the runbook recommends preflight-only first, then the same command without the flag, so on a box carrying a Felhom leftover the wrong ownership answer is baked in before the real install begins | **READY (S) — NEW 2026-08-12, RANK 3** | R-300, R-305 | Either make `_state_put` a no-op under `PREFLIGHT_ONLY` (and record ownership at install instead), or correct both claims. A comment asserting an invariant needs a test pinning it | CC |
|
||||
| **R-307** | **`demo-felhom` carries a LIVE abandon countdown that this drill did not start — and the end state says there should be none.** `settings.json`: `abandon_started_at 2026-08-10T08:06:31Z`, `abandon_at 2026-08-24T08:06:31Z`, `abandon_repo_path /home/felhom-repo.orphaned-20260810`. The drill's fence forbade starting, shortening or triggering a countdown, and none was; but its required end state was *"no abandon countdown anywhere"*, and one exists. **The two ways to satisfy that are not equivalent and the choice is the operator's:** cancel it (the orphaned repository is kept indefinitely — storage cost, no data risk) or let it run (on **2026-08-24** that repository is deleted, **irreversibly**). **Doing nothing selects deletion by default.** Deliberately not resolved by CC | **BLOCKED ON OPERATOR — NEW 2026-08-12** | R-301, R-302, R-303 | Decide before 2026-08-24 | operator |
|
||||
| **R-308** | **The stored controller password no longer opens `demo-felhom`.** `PASSWORD` in `~/.config/credentials` on DooPlex returns `Hibás jelszó` against `https://127.0.0.1:443/login` with `Host: felhom.demo-felhom.eu`. **NOT the known quoting trap** — the surrounding quotes were stripped before use and the stored value is unquoted (16 bytes for 15 chars), so this is a genuinely stale credential, not a mis-read. Cost during the drill: the dashboard was unreachable headlessly, so the offsite push had to be made with `restic` directly instead of the page's own button, which left the app-backup→unit→offsite chain unexercised. `R_DEMO-FELHOM` **was** correct and has since been rotated by this session's final ceremony (updated in place; prior file backed up alongside) | **READY (S) — NEW 2026-08-12, RANK 2** | — | Recover or reset the demo-felhom dashboard password and re-store it; note which boxes share one and which do not | operator |
|
||||
| **R-307** | **`demo-felhom` carries a LIVE abandon countdown that this drill did not start — and the end state says there should be none.** `settings.json`: `abandon_started_at 2026-08-10T08:06:31Z`, `abandon_at 2026-08-24T08:06:31Z`, `abandon_repo_path /home/felhom-repo.orphaned-20260810`. The drill's fence forbade starting, shortening or triggering a countdown, and none was; but its required end state was *"no abandon countdown anywhere"*, and one exists. **The two ways to satisfy that are not equivalent and the choice is the operator's:** cancel it (the orphaned repository is kept indefinitely — storage cost, no data risk) or let it run (on **2026-08-24** that repository is deleted, **irreversibly**). **Doing nothing selects deletion by default.** Deliberately not resolved by CC. **RULED 2026-08-12 (operator): KEEP the set-aside history — cancel the countdown.** Done the same evening through the product's own operator path (`--abandon-stop`, which refuses rather than silently no-opping), with the container stopped first so the running controller could not overwrite `settings.json` from memory. **Proved, not trusted to the exit code:** `abandon_started_at` and `abandon_at` are GONE (`AbandonStatus` returns `Active=false` when `AbandonAt` is empty — `offbox_abandon.go:111-113`, so no countdown renders); `abandon_repo_path` deliberately REMAINS as the pointer to the preserved store; the store still holds **36 snapshot objects** and its full `config/data/index/keys/locks/snapshots` structure; both repositories still present on the endpoint; **nothing deleted anywhere**. See R-313 — what was preserved cannot currently be opened by anyone | **CLOSED — countdown cancelled 2026-08-12 on the operator's ruling** | R-301, R-302, R-303, R-313 | — | CC |
|
||||
| **R-308** | **~~The stored controller password no longer opens `demo-felhom`~~ — WITHDRAWN 2026-08-12, this was MY BUG, not a defect.** The original diagnosis (`Hibás jelszó`) came from stripping only DOUBLE quotes off the `~/.config/credentials` value; the values are wrapped in **SINGLE** quotes, so a literal `'` was sent as part of the password. Correctly unquoted the value is 13 characters and logs in first try — **HTTP 302 with a `felhom_session` cookie**, measured. **This is the third time this project has produced a wrong 'the credential is stale' verdict from that exact trap**, and the standing memory about it is what caught it here — but only after the same mistake had also made the R-311 live test read as a FAILURE (HTTP 400) for twenty minutes. **The lesson is not 'strip quotes' but 'never let a shell decide what a secret is'**: parse the credentials file in Python, strip a matched pair of EITHER quote, and verify the length before use | **WITHDRAWN — not a defect (my error)** | — | Consider a tiny `read_credential.py` helper so no session hand-rolls the parse again | CC |
|
||||
| **R-309** | **The day-0 runbook says pushing the installer publishes it. It has not since R-110.** `documentation/runbooks/day0-install.md` §C.1: *"it is always the current `main` script. There is no release tag, no staging copy and no version selector; pushing `scripts/felhom-host-install.sh` publishes it."* R-110 moved `/scripts/` to the **installer tag's** tree — `manifests/webpage.yaml` carries **two** `--ref=installer-v…` pins (lines 327, 372) and `hostinstall_gates.py` gate 6 enforces that they name a tag. Confirmed live 2026-08-12: `https://felhom.eu/scripts/felhom-host-install.sh` served `SCRIPT_VERSION="1.25.0"` while `main` had `1.27.0`, three and a half hours after the push. **The runbook's version is the dangerous one to believe in both directions** — it would have an operator think a pushed fix is live when it is not, and think a pushed mistake is live when it is not | **READY (S) — NEW 2026-08-12, RANK 3** | R-110 | Correct §C.1 to name the tag-and-manifest publication step (as performed today) | CC |
|
||||
| **R-310** | **Two small edges on the installer, neither costing more than a moment.** (1) The R-297 operator-named refusal states the vouched version twice in consecutive sentences (*"…but the vouched golden is 0.213.0. The vouched golden is 0.213.0."*). (2) `--uninstall` reads its typed vmid confirmation from `/dev/tty` and `--force` deliberately does **not** bypass it, so teardown cannot be scripted without a pty — correct for an irreversible destroy, but undocumented; it surfaces as `line 891: /dev/tty: No such device or address` and an rc=1 that looks like a failure rather than a refusal to proceed unattended | **READY (S) — NEW 2026-08-12, RANK 4** | R-297 | Drop the duplicated sentence; add one runbook line naming the pty requirement | CC |
|
||||
| **R-311** | **A correct recovery code for a retained package stopped being reported as wrong.** The drill of 2026-08-12 proved the retained key WORKS (it opened a set-aside store and restored planted files byte-identical) while the product answered *"the recovery code did not open the sealed bundle"*. **What was already right:** the customer-facing screen did not bluntly accuse — R-222/R-226 hedged, naming both causes and the kept package. **What was wrong was the hedge's reason:** *„innen nem tudjuk megkülönböztetni őket"* was true only because NOTHING EVER LOOKED. **Shipped, three repos.** Hub v0.103.0: `GET /hosts/<id>/escrow/retained` — the first production caller `ListSupersededEscrow` has ever had — self-scoped, same recovery-mode gate, same audit event before the bytes leave, rows with a NULL `identity_blob` WITHHELD and returned as `unopenable_count` (they can never open anything, and serving them would let the screen promise recovery on exactly the boxes the original defect hurt), capped at 16. Agent v0.129.0: `FetchRetained` consulted ONLY after the current package refuses, `ErrCodeOpensRetained` + the supersession date, local API answers **422**, bounded at 6 attempts (~1 s of scrypt each), and fail-safe in every direction — a nil fetcher, a 404 from an older hub, a transport failure or a malformed package all leave the original refusal standing. Controller v0.214.0: class `RecoveryCodeOpensRetained` gated on `FeatureRetainedRecoveryClass` (MinAgent 0.129.0, SEPARATE from the R-224 gate because a box can sit between 0.126.0 and 0.129.0), and a message that says the code is correct, names the date, says the earlier package is kept, says the CURRENT backups are unaffected, and **deliberately promises no restore** — it routes to support, which can do it by hand. **The lie's true home was the agent:** removing the CONTROLLER case yields the neutral message (R-224's safe default catches it); removing the AGENT's retained lookup brings back the fail-closed wrong-code error in those exact words — both demonstrated, mutations asserted applied **DELIVERED FLEET-WIDE 2026-08-12 night.** Operator vouched golden 0.214.0 / agent 0.129.0 / min agent 0.129.0 and raised the floor to 0.214.0; both artifact shas in `hub_settings` match the bake and the release byte-for-byte (`3a40379c…`, `53a54f06…`). **The Part D / R-216 hold was OBSERVED FIRING for the first time** — with the floor at 0.214.0 and `demo-hp` still on agent 0.128.0 the hub logged *"managed floor HELD for demo-hp: agent 0.128.0 < MinAgent 0.129.0 (controller floor withheld)"*, and the box took the update **6 seconds** after its agent was brought to 0.129.0 (`auto-floor`, 19:03:18→19:03:24Z). That guard exists because a box once landed on a controller newer than its agent and its customer was told a correct recovery code was wrong; **the MinAgent declaration in the CHANGELOG header is what made it hold**, which is the whole argument for the three-part coupling rule. Both demo boxes now run controller 0.214.0 + agent 0.129.0, verified ON the boxes. | **CLOSED — shipped + delivered: hub v0.103.0 + agent v0.129.0 + controller v0.214.0** | R-199, R-222, R-224, R-226, R-304 | — | CC |
|
||||
| **R-312** | **There is no in-product route from the recovery screen to a set-aside store, and building one is not wiring — it is new surface.** Established read-only before any code was written (the session's §4 spike). Every restore entry point resolves the repository from `m.settings.GetOffboxTarget()` and the password from the single `offboxPwPath()` file: `offboxLatestSnapshot` (`offbox_restore.go:85-86`), `offboxSnapshotSize` (`:139-140`), `RestoreOffboxScratch` (`:206+`). **There is no repo-path parameter anywhere in the chain** — a grep for one returns nothing. The only existing seam that installs a recovered password, `InjectOffboxPassword` (`offbox.go:667`), writes that same one file, i.e. **ADOPTS** the set-aside store as the machine's current target. So the two options are (a) thread an alternative (repo, password) through three functions plus the UI, or (b) adopt — and adoption is a different product decision. **The session HALTED here by its own rule** and shipped R-311 alone. What the drill did to read the set-aside store was `restic` by hand with `-r <alt repo>` and an overridden `RESTIC_PASSWORD_FILE`; **that distance is exactly what (b)-to-(c) costs** | **READY (L) — NEW 2026-08-12** | R-304, R-311 | Decide (a) vs (b). If (a): the parameter is small and contained; the UI question ("which package?" when a customer has several) is the real design work. **Do not choose (b) without deciding what happens to the machine's current off-site history** | operator + CC |
|
||||
| **R-313** | **`demo-felhom`'s set-aside store is UNRECOVERABLE — 36 snapshots whose key we destroyed ourselves.** `/home/felhom-repo.orphaned-20260810` holds **36 snapshot objects and exactly one key slot**, and it does NOT open with the box's current password (`Fatal: wrong password or no key found`, exit 1 — measured). Its password is the one hashed `48741892f0ef4d59…`, which is retained row id 4 — **`identity_blob` NULL**, a pre-v0.93.0 row. So the material was dropped by the R-198 defect during its two-month window, and no recovery code in existence opens that store. **This is the concrete, still-present cost of R-198**, sitting on the endpoint rather than in a post-mortem. It also means the operator's ruling to KEEP it (R-307, countdown cancelled — see below) preserves bytes nobody can read: correct as a decision, and worth knowing as a fact | **READY (S) — NEW 2026-08-12** | R-198, R-307 | Decide whether to keep paying storage for 36 unreadable snapshots, or delete them deliberately and record why. **Not urgent and not automatic — but it should be a decision rather than an accumulation** | operator |
|
||||
| **R-314** | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route | **READY (S) — NEW 2026-08-12, RANK 3** | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC |
|
||||
| **R-315** | **The wire-contract gate's positive control FAILS on the new wire: it checks name-presence, not decodability.** R-311 declared `hub -> agent (GET /escrow/retained)` as a fourth ROOT, and the gate's tag count rose 174 → 182, so the fields ARE inspected. But renaming the agent-side `superseded_at` json tag to `superseded_at_RENAMED` **still passed** — because the string `superseded_at` also occurs as a map key in the agent's local-API response, and the check is a repo-wide name search. The gate documents this ("name-reachability is not use"), so it is a known limit rather than a regression — but it means **declaring this wire bought documentation, not enforcement**, and a report that claimed coverage would have been wrong. The mutation was asserted to have applied before the run | **READY (M) — NEW 2026-08-12, RANK 3** | R-311 | Make the check resolve the RECEIVER'S mirror type and compare field-by-field, or state per-root which kind of check it got. **A gate whose positive control fails is an instrument nobody has calibrated** | CC |
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Golden bake 0.214.0 — 2026-08-12
|
||||
|
||||
Per `RUNBOOK-manual-build.md` §4.0/§4.1. Drill VM reverted to `virgin` before and after; guest 9100
|
||||
destroyed `--purge`; `/root` residue shredded.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `GOLDEN_VERSION` | **0.214.0** |
|
||||
| `GOLDEN_SHA256` | **3a40379cb00d98c6f9550815b2b024783f153e9eef9be2b3252253d24d8d2c4d** |
|
||||
| controller baked | `gitea.dooplex.hu/admin/felhom-controller:0.214.0` |
|
||||
| `MinAgent` (CHANGELOG header) | **0.129.0** |
|
||||
| published | 656 392 524 B, `upload OK (HTTP 201)` |
|
||||
|
||||
## Acceptance markers (`grep -F`, quoted loop variable)
|
||||
|
||||
`docker OK (overlay2` 1 · `including mount point rootfs` 1 · `including mount point mp0` 1 ·
|
||||
`upload OK (HTTP 201)` 1 · `FATAL` 0 · `excluding` 0
|
||||
|
||||
## Fetchability — the SERVED bytes
|
||||
|
||||
```
|
||||
downloaded: 656392524 bytes
|
||||
sha256: 3a40379cb00d98c6f9550815b2b024783f153e9eef9be2b3252253d24d8d2c4d
|
||||
```
|
||||
Identical to `GOLDEN_SHA256`.
|
||||
|
||||
## The template index was STALE, and it would have baked the wrong base
|
||||
|
||||
`pveam available` on the freshly reverted `virgin` listed only `debian-13-standard_13.1-2` — the point
|
||||
release our own memory records as 404-ing since 2026-07-15. **`pveam update` first**, and the real
|
||||
current one is `debian-13-standard_13.6-1_amd64.tar.zst`. The runbook says the point release rots; what
|
||||
it does not say is that the VM's cached index rots too, and reading it without refreshing produces a
|
||||
confident answer that is a fortnight out of date.
|
||||
|
||||
## Secret handling
|
||||
|
||||
Token file→file, read inside the VM by a runner script, never on a command line.
|
||||
`systemctl show golden-bake -p Environment -p ExecStart | grep -c -F <token>` = **0**. Leak grep on the
|
||||
committed log = **0**, believable because a planted-token control on a copy grepped **1**. All in-VM
|
||||
artefacts `shred -u`'d.
|
||||
|
||||
## NOT DONE
|
||||
|
||||
The Day-0 vouch — the operator's, and deliberately so. Three fields, each already verified downloadable
|
||||
and selectable: **golden 0.214.0**, **agent 0.129.0**, **min agent 0.129.0**.
|
||||
@@ -0,0 +1,326 @@
|
||||
[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.214.0
|
||||
[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) …
|
||||
Logical volume "vm-9100-disk-0" created.
|
||||
Logical volume pve/vm-9100-disk-0 changed.
|
||||
Creating filesystem with 8388608 4k blocks and 2097152 inodes
|
||||
Filesystem UUID: a630e58b-5ddd-4312-8535-b83ed18030fb
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
|
||||
4096000, 7962624
|
||||
Logical volume "vm-9100-disk-1" created.
|
||||
Logical volume pve/vm-9100-disk-1 changed.
|
||||
Creating filesystem with 6291456 4k blocks and 1572864 inodes
|
||||
Filesystem UUID: 247da550-80fb-4921-bb4c-0a3517bdbd21
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
|
||||
extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst'
|
||||
Total bytes read: 553512960 (528MiB, 177MiB/s)
|
||||
Detected container architecture: amd64
|
||||
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
|
||||
done: SHA256:N/+3Phg1PMspcZ0VWshdC1mXbkAYQ2jAsYhavLNrb0Q root@felhom-golden
|
||||
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
|
||||
done: SHA256:fl1uCYoByd5OGUy7NjwyEHlkIcOvAxk5887imnsM7tc root@felhom-golden
|
||||
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
|
||||
done: SHA256:6wNca1Ee5AXkXkQjmROa6sdynmpHikRhw7MuHpxXrk0 root@felhom-golden
|
||||
[golden] starting + installing Docker (official repo, trixie channel) …
|
||||
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
|
||||
perl: warning: Setting locale failed.
|
||||
perl: warning: Please check that your locale settings:
|
||||
LANGUAGE = (unset),
|
||||
LC_ALL = (unset),
|
||||
LC_CTYPE = (unset),
|
||||
LC_NUMERIC = (unset),
|
||||
LC_COLLATE = (unset),
|
||||
LC_TIME = (unset),
|
||||
LC_MESSAGES = (unset),
|
||||
LC_MONETARY = (unset),
|
||||
LC_ADDRESS = (unset),
|
||||
LC_IDENTIFICATION = (unset),
|
||||
LC_MEASUREMENT = (unset),
|
||||
LC_PAPER = (unset),
|
||||
LC_TELEPHONE = (unset),
|
||||
LC_NAME = (unset),
|
||||
LANG = "en_US.UTF-8"
|
||||
are supported and installed on your system.
|
||||
perl: warning: Falling back to the standard locale ("C").
|
||||
locale: Cannot set LC_CTYPE to default locale: No such file or directory
|
||||
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
|
||||
locale: Cannot set LC_ALL to default locale: No such file or directory
|
||||
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
|
||||
perl: warning: Setting locale failed.
|
||||
perl: warning: Please check that your locale settings:
|
||||
LANGUAGE = (unset),
|
||||
LC_ALL = (unset),
|
||||
LC_CTYPE = (unset),
|
||||
LC_NUMERIC = (unset),
|
||||
LC_COLLATE = (unset),
|
||||
LC_TIME = (unset),
|
||||
LC_MESSAGES = (unset),
|
||||
LC_MONETARY = (unset),
|
||||
LC_ADDRESS = (unset),
|
||||
LC_IDENTIFICATION = (unset),
|
||||
LC_MEASUREMENT = (unset),
|
||||
LC_PAPER = (unset),
|
||||
LC_TELEPHONE = (unset),
|
||||
LC_NAME = (unset),
|
||||
LANG = "en_US.UTF-8"
|
||||
are supported and installed on your system.
|
||||
perl: warning: Falling back to the standard locale ("C").
|
||||
locale: Cannot set LC_CTYPE to default locale: No such file or directory
|
||||
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
|
||||
locale: Cannot set LC_ALL to default locale: No such file or directory
|
||||
[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …
|
||||
[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds …
|
||||
[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …
|
||||
Unable to find image 'hello-world:latest' locally
|
||||
latest: Pulling from library/hello-world
|
||||
4f55086f7dd0: Pulling fs layer
|
||||
4f55086f7dd0: Download complete
|
||||
4f55086f7dd0: Pull complete
|
||||
Digest: sha256:7f4da0fc94bcece205a8c0b6f4d11c8196924654ffe5c4d1aa439b7f632048b2
|
||||
Status: Downloaded newer image for hello-world:latest
|
||||
docker OK (overlay2; data-root /var/lib/docker)
|
||||
/var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
|
||||
/mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
|
||||
both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
|
||||
[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.214.0 (no registry cred at deploy) …
|
||||
|
||||
WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
|
||||
Configure a credential helper to remove this warning. See
|
||||
https://docs.docker.com/go/credential-store/
|
||||
|
||||
0.214.0: Pulling from admin/felhom-controller
|
||||
039e6f9f9752: Pulling fs layer
|
||||
0094c3ac0914: Pulling fs layer
|
||||
deca1dac7403: Pulling fs layer
|
||||
11c19a33d1b8: Pulling fs layer
|
||||
742f054a0a3b: Pulling fs layer
|
||||
63bdd1abbb22: Pulling fs layer
|
||||
11c19a33d1b8: Waiting
|
||||
742f054a0a3b: Waiting
|
||||
63bdd1abbb22: Waiting
|
||||
039e6f9f9752: Verifying Checksum
|
||||
039e6f9f9752: Download complete
|
||||
deca1dac7403: Verifying Checksum
|
||||
deca1dac7403: Download complete
|
||||
11c19a33d1b8: Verifying Checksum
|
||||
11c19a33d1b8: Download complete
|
||||
742f054a0a3b: Verifying Checksum
|
||||
742f054a0a3b: Download complete
|
||||
63bdd1abbb22: Verifying Checksum
|
||||
63bdd1abbb22: Download complete
|
||||
0094c3ac0914: Verifying Checksum
|
||||
0094c3ac0914: Download complete
|
||||
039e6f9f9752: Pull complete
|
||||
0094c3ac0914: Pull complete
|
||||
deca1dac7403: Pull complete
|
||||
11c19a33d1b8: Pull complete
|
||||
742f054a0a3b: Pull complete
|
||||
63bdd1abbb22: Pull complete
|
||||
Digest: sha256:d2c0bc7fba3d142e7f9ee9117231fd7ea28630aa71ce2f4a7e2d0ee1fdb85d97
|
||||
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.214.0
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.214.0
|
||||
[golden] asking the controller which infra images it manages …
|
||||
[golden] baking infra images (4): traefik:v3.6.7 cloudflare/cloudflared:2026.6.0 gtstef/filebrowser:1.3.3-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 …
|
||||
v3.6.7: Pulling from library/traefik
|
||||
589002ba0eae: Pulling fs layer
|
||||
ef63511ea6cc: Pulling fs layer
|
||||
0738e5cb835e: Pulling fs layer
|
||||
3e6813f70c64: Pulling fs layer
|
||||
3e6813f70c64: Waiting
|
||||
ef63511ea6cc: Verifying Checksum
|
||||
ef63511ea6cc: Download complete
|
||||
589002ba0eae: Verifying Checksum
|
||||
589002ba0eae: Download complete
|
||||
3e6813f70c64: Verifying Checksum
|
||||
3e6813f70c64: Download complete
|
||||
0738e5cb835e: Verifying Checksum
|
||||
0738e5cb835e: Download complete
|
||||
589002ba0eae: Pull complete
|
||||
ef63511ea6cc: Pull complete
|
||||
0738e5cb835e: Pull complete
|
||||
3e6813f70c64: Pull complete
|
||||
Digest: sha256:a9890c898f379c1905ee5b28342f6b408dc863f08db2dab20e46c267d1ff463a
|
||||
Status: Downloaded newer image for traefik:v3.6.7
|
||||
docker.io/library/traefik:v3.6.7
|
||||
2026.6.0: Pulling from cloudflare/cloudflared
|
||||
47de5dd0b812: Pulling fs layer
|
||||
c172f21841df: Pulling fs layer
|
||||
99515e7b4d35: Pulling fs layer
|
||||
99ba982a9142: Pulling fs layer
|
||||
d6b1b89eccac: Pulling fs layer
|
||||
2780920e5dbf: Pulling fs layer
|
||||
7c12895b777b: Pulling fs layer
|
||||
3214acf345c0: Pulling fs layer
|
||||
52630fc75a18: Pulling fs layer
|
||||
dd64bf2dd177: Pulling fs layer
|
||||
b839dfae01f6: Pulling fs layer
|
||||
ebddc55facdc: Pulling fs layer
|
||||
bdfd7f7e5bf6: Pulling fs layer
|
||||
2d4d7adf6272: Pulling fs layer
|
||||
40008157d8d2: Pulling fs layer
|
||||
bd8962e29291: Pulling fs layer
|
||||
cac2ae0193cb: Pulling fs layer
|
||||
74d1dac84ecc: Pulling fs layer
|
||||
99ba982a9142: Waiting
|
||||
d6b1b89eccac: Waiting
|
||||
2780920e5dbf: Waiting
|
||||
7c12895b777b: Waiting
|
||||
3214acf345c0: Waiting
|
||||
52630fc75a18: Waiting
|
||||
dd64bf2dd177: Waiting
|
||||
b839dfae01f6: Waiting
|
||||
ebddc55facdc: Waiting
|
||||
bdfd7f7e5bf6: Waiting
|
||||
2d4d7adf6272: Waiting
|
||||
40008157d8d2: Waiting
|
||||
bd8962e29291: Waiting
|
||||
cac2ae0193cb: Waiting
|
||||
74d1dac84ecc: Waiting
|
||||
99515e7b4d35: Verifying Checksum
|
||||
99515e7b4d35: Download complete
|
||||
c172f21841df: Verifying Checksum
|
||||
c172f21841df: Download complete
|
||||
47de5dd0b812: Verifying Checksum
|
||||
47de5dd0b812: Download complete
|
||||
99ba982a9142: Download complete
|
||||
d6b1b89eccac: Verifying Checksum
|
||||
d6b1b89eccac: Download complete
|
||||
2780920e5dbf: Verifying Checksum
|
||||
2780920e5dbf: Download complete
|
||||
7c12895b777b: Verifying Checksum
|
||||
7c12895b777b: Download complete
|
||||
52630fc75a18: Verifying Checksum
|
||||
52630fc75a18: Download complete
|
||||
3214acf345c0: Verifying Checksum
|
||||
3214acf345c0: Download complete
|
||||
47de5dd0b812: Pull complete
|
||||
dd64bf2dd177: Verifying Checksum
|
||||
dd64bf2dd177: Download complete
|
||||
b839dfae01f6: Verifying Checksum
|
||||
b839dfae01f6: Download complete
|
||||
ebddc55facdc: Verifying Checksum
|
||||
ebddc55facdc: Download complete
|
||||
bdfd7f7e5bf6: Download complete
|
||||
40008157d8d2: Verifying Checksum
|
||||
40008157d8d2: Download complete
|
||||
bd8962e29291: Verifying Checksum
|
||||
bd8962e29291: Download complete
|
||||
2d4d7adf6272: Verifying Checksum
|
||||
2d4d7adf6272: Download complete
|
||||
c172f21841df: Pull complete
|
||||
cac2ae0193cb: Verifying Checksum
|
||||
cac2ae0193cb: Download complete
|
||||
74d1dac84ecc: Verifying Checksum
|
||||
74d1dac84ecc: Download complete
|
||||
99515e7b4d35: Pull complete
|
||||
99ba982a9142: Pull complete
|
||||
d6b1b89eccac: Pull complete
|
||||
2780920e5dbf: Pull complete
|
||||
7c12895b777b: Pull complete
|
||||
3214acf345c0: Pull complete
|
||||
52630fc75a18: Pull complete
|
||||
dd64bf2dd177: Pull complete
|
||||
b839dfae01f6: Pull complete
|
||||
ebddc55facdc: Pull complete
|
||||
bdfd7f7e5bf6: Pull complete
|
||||
2d4d7adf6272: Pull complete
|
||||
40008157d8d2: Pull complete
|
||||
bd8962e29291: Pull complete
|
||||
cac2ae0193cb: Pull complete
|
||||
74d1dac84ecc: Pull complete
|
||||
Digest: sha256:ba461b8aa9c042156dbd39c38657fe7431bafa063220eab8d5330a523863da9f
|
||||
Status: Downloaded newer image for cloudflare/cloudflared:2026.6.0
|
||||
docker.io/cloudflare/cloudflared:2026.6.0
|
||||
1.3.3-stable: Pulling from gtstef/filebrowser
|
||||
6a0ac1617861: Pulling fs layer
|
||||
ef8806083e82: Pulling fs layer
|
||||
b74107c861c7: Pulling fs layer
|
||||
adc935def003: Pulling fs layer
|
||||
4f4fb700ef54: Pulling fs layer
|
||||
18695ccc900a: Pulling fs layer
|
||||
45d119d5c397: Pulling fs layer
|
||||
dac52db4fc51: Pulling fs layer
|
||||
6d598f86b2f2: Pulling fs layer
|
||||
8aa349c8396c: Pulling fs layer
|
||||
dac52db4fc51: Waiting
|
||||
6d598f86b2f2: Waiting
|
||||
8aa349c8396c: Waiting
|
||||
4f4fb700ef54: Waiting
|
||||
adc935def003: Waiting
|
||||
18695ccc900a: Waiting
|
||||
45d119d5c397: Waiting
|
||||
6a0ac1617861: Verifying Checksum
|
||||
6a0ac1617861: Download complete
|
||||
b74107c861c7: Verifying Checksum
|
||||
b74107c861c7: Download complete
|
||||
6a0ac1617861: Pull complete
|
||||
adc935def003: Verifying Checksum
|
||||
adc935def003: Download complete
|
||||
4f4fb700ef54: Verifying Checksum
|
||||
4f4fb700ef54: Download complete
|
||||
45d119d5c397: Verifying Checksum
|
||||
45d119d5c397: Download complete
|
||||
dac52db4fc51: Verifying Checksum
|
||||
dac52db4fc51: Download complete
|
||||
18695ccc900a: Verifying Checksum
|
||||
18695ccc900a: Download complete
|
||||
6d598f86b2f2: Verifying Checksum
|
||||
6d598f86b2f2: Download complete
|
||||
8aa349c8396c: Verifying Checksum
|
||||
8aa349c8396c: Download complete
|
||||
ef8806083e82: Verifying Checksum
|
||||
ef8806083e82: Download complete
|
||||
ef8806083e82: Pull complete
|
||||
b74107c861c7: Pull complete
|
||||
adc935def003: Pull complete
|
||||
4f4fb700ef54: Pull complete
|
||||
18695ccc900a: Pull complete
|
||||
45d119d5c397: Pull complete
|
||||
dac52db4fc51: Pull complete
|
||||
6d598f86b2f2: Pull complete
|
||||
8aa349c8396c: Pull complete
|
||||
Digest: sha256:eb3733681db8757412632c61a99ad656f0d94ed6781bb2ea114b4d70babab78c
|
||||
Status: Downloaded newer image for gtstef/filebrowser:1.3.3-stable
|
||||
docker.io/gtstef/filebrowser:1.3.3-stable
|
||||
1.1.0: Pulling from admin/felhom-samba
|
||||
897d797d2723: Pulling fs layer
|
||||
3051591aa250: Pulling fs layer
|
||||
ce57a3f93416: Pulling fs layer
|
||||
fb94eeec2fe1: Pulling fs layer
|
||||
fb94eeec2fe1: Waiting
|
||||
ce57a3f93416: Verifying Checksum
|
||||
ce57a3f93416: Download complete
|
||||
fb94eeec2fe1: Verifying Checksum
|
||||
fb94eeec2fe1: Download complete
|
||||
897d797d2723: Verifying Checksum
|
||||
897d797d2723: Download complete
|
||||
3051591aa250: Verifying Checksum
|
||||
3051591aa250: Download complete
|
||||
897d797d2723: Pull complete
|
||||
3051591aa250: Pull complete
|
||||
ce57a3f93416: Pull complete
|
||||
fb94eeec2fe1: Pull complete
|
||||
Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
|
||||
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0
|
||||
gitea.dooplex.hu/admin/felhom-samba:1.1.0
|
||||
[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …
|
||||
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'.
|
||||
[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
|
||||
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'.
|
||||
[golden] baking the first-boot SSH host-key regeneration unit (F3) …
|
||||
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'.
|
||||
[golden] identity-clean + minimize …
|
||||
[golden] stop + archive …
|
||||
INFO: including mount point rootfs ('/') in backup
|
||||
INFO: including mount point mp0 ('/var/lib/felhom') in backup
|
||||
INFO: archive file size: 625MB
|
||||
INFO: Finished Backup of VM 9100 (00:00:39)
|
||||
[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_08_12-18_52_07.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive)
|
||||
[golden] publishing golden (656392524 bytes, sha256 3a40379cb00d98c6…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.214.0/golden.tar.zst
|
||||
[golden] pre-delete existing: HTTP 404 (404/204 expected)
|
||||
[golden] upload OK (HTTP 201)
|
||||
GOLDEN_VERSION=0.214.0
|
||||
GOLDEN_SHA256=3a40379cb00d98c6f9550815b2b024783f153e9eef9be2b3252253d24d8d2c4d
|
||||
[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.214.0 / 3a40379cb00d98c6f9550815b2b024783f153e9eef9be2b3252253d24d8d2c4d
|
||||
[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)
|
||||
@@ -1,3 +1,35 @@
|
||||
## v0.103.0 — a host can read the packages we kept for it (2026-08-12, R-311)
|
||||
|
||||
**`ListSupersededEscrow` had zero production callers for nineteen days.** It is the only reader of a
|
||||
retained `identity_blob`, and nothing called it — so the retention shipped in v0.93.0 was material the
|
||||
product could not reach. Proven on the fixture 2026-08-12: a recovery code that opens a retained
|
||||
package (unsealed by hand, and it restored planted files byte-identical from a store the box itself
|
||||
could no longer open) was answered by the product as a code that opened nothing.
|
||||
|
||||
**New: `GET /api/v1/hosts/<id>/escrow/retained`** — the retained siblings of the existing
|
||||
`/escrow` row, self-scoped to the asking host exactly as that one is, behind the same §8.3
|
||||
recovery-mode gate, and audited with the same `escrow_blob_served` event BEFORE the bytes leave.
|
||||
|
||||
**What it will not serve.** Rows whose `identity_blob` is NULL — every pre-v0.93.0 row — are withheld
|
||||
and returned as `unopenable_count` instead. They retain the PBS datastore key and not the repository
|
||||
password, so they can never open what the caller is asking about; serving them would have the agent
|
||||
try packages that cannot succeed and would let the recovery screen claim an earlier package is
|
||||
openable on exactly the boxes the original defect hurt. The count is returned because their EXISTENCE
|
||||
is load-bearing and underivable: on such a box a correct old code opens nothing, and the honest reason
|
||||
is a defect of ours rather than the customer's typing.
|
||||
|
||||
**The trade, stated rather than waved through.** The hub still cannot read any of it — sealed bytes in,
|
||||
sealed bytes out, no decrypt path, no recovery code ever held. What widens is VOLUME: a host key that
|
||||
could fetch one opaque package can now fetch N. Bounded by the same self-scope, the same recovery-mode
|
||||
gate, and an explicit `retainedEscrowServeCap` (16) so a long supersession history cannot become an
|
||||
unbounded read.
|
||||
|
||||
Pinned by six tests driven through the real endpoint, with four red-proofs each asserted to have
|
||||
applied: serving the CURRENT row instead (count 2→1), dropping the unopenable guard (count 1→2,
|
||||
unopenable 1→0), dropping self-scope (403→200), and collapsing the route suffix (count 1→0).
|
||||
|
||||
---
|
||||
|
||||
## v0.102.0 — the hub refuses to vouch a version that cannot be installed (2026-08-09, R-273)
|
||||
|
||||
**The guard owed since 2026-08-09 morning.** Agent v0.128.0 had been published as a package and never
|
||||
|
||||
@@ -0,0 +1,218 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// R-311 (v0.103.0) — GET /hosts/<id>/escrow/retained.
|
||||
//
|
||||
// WHY THESE TESTS EXIST. `ListSupersededEscrow` had zero production callers for nineteen days; the
|
||||
// retention it reads was proven on 2026-08-12 to hold the right material and to open a store the box
|
||||
// itself could not, while the customer was told their (correct) code opened nothing. These tests pin
|
||||
// the endpoint that ends that, and — more importantly — they pin the two things it must NOT do:
|
||||
// serve a package that cannot be opened, and let one host read another's.
|
||||
|
||||
type retainedResp struct {
|
||||
HostID string `json:"host_id"`
|
||||
Count int `json:"count"`
|
||||
UnopenableCount int `json:"unopenable_count"`
|
||||
TruncatedCount int `json:"truncated_count"`
|
||||
Packages []struct {
|
||||
Index int `json:"index"`
|
||||
SupersededAt string `json:"superseded_at"`
|
||||
KeyFingerprint string `json:"key_fingerprint"`
|
||||
IdentityEscrowB64 string `json:"identity_escrow_b64"`
|
||||
} `json:"packages"`
|
||||
}
|
||||
|
||||
func getRetained(t *testing.T, h *Handler, hostID, key string) (int, retainedResp) {
|
||||
t.Helper()
|
||||
rr := do(h, http.MethodGet, "/hosts/"+hostID+"/escrow/retained", key, "")
|
||||
var out retainedResp
|
||||
if rr.Code == http.StatusOK {
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &out); err != nil {
|
||||
t.Fatalf("decode retained response: %v (body %s)", err, rr.Body.String())
|
||||
}
|
||||
}
|
||||
return rr.Code, out
|
||||
}
|
||||
|
||||
// seedSuperseded drives the REAL supersession path (a PUT sealing a different password demotes the
|
||||
// current row) rather than inserting into host_escrow_superseded directly — so what the test proves
|
||||
// includes the demote wiring, not just this handler's SELECT.
|
||||
func seedSuperseded(t *testing.T, h *Handler, hostID, key string, gens []struct{ sha, identity string }) {
|
||||
t.Helper()
|
||||
for i, g := range gens {
|
||||
if rr := do(h, http.MethodPut, "/hosts/"+hostID+"/escrow", key, escrowBodyWithHash([]byte("k"), g.sha, g.identity)); rr.Code != http.StatusOK {
|
||||
t.Fatalf("seed PUT %d = %d (%s)", i, rr.Code, rr.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// THE ONE THAT MATTERS: a retained package that carries key material is served, and the caller can
|
||||
// tell it apart from the current row.
|
||||
//
|
||||
// RED-PROOF: change the handler's SELECT source from ListSupersededEscrow to GetHostDRBundle (i.e.
|
||||
// serve the CURRENT row) → the served blob becomes "age-gen3" → this FAILS on the body comparison,
|
||||
// which is exactly the confusion the endpoint exists to end.
|
||||
func TestRetainedEscrowGet_ServesRetainedPackages(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||
seedSuperseded(t, h, "h1", "HKEY", []struct{ sha, identity string }{
|
||||
{"SHA_GEN1", "age-gen1"},
|
||||
{"SHA_GEN2", "age-gen2"},
|
||||
{"SHA_GEN3", "age-gen3"}, // current after this
|
||||
})
|
||||
|
||||
code, out := getRetained(t, h, "h1", "HKEY")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("GET retained = %d, want 200", code)
|
||||
}
|
||||
if out.Count != 2 || len(out.Packages) != 2 {
|
||||
t.Fatalf("want 2 retained packages, got count=%d len=%d", out.Count, len(out.Packages))
|
||||
}
|
||||
if out.UnopenableCount != 0 {
|
||||
t.Errorf("unopenable_count = %d, want 0 (every seeded row carried an identity blob)", out.UnopenableCount)
|
||||
}
|
||||
// The retained packages must be the SUPERSEDED generations, never the current one. If this ever
|
||||
// serves gen3 the endpoint is answering the wrong question and the customer is misled again.
|
||||
got := map[string]bool{}
|
||||
for _, p := range out.Packages {
|
||||
b, err := base64.StdEncoding.DecodeString(p.IdentityEscrowB64)
|
||||
if err != nil {
|
||||
t.Fatalf("package %d: identity blob is not base64: %v", p.Index, err)
|
||||
}
|
||||
got[string(b)] = true
|
||||
if p.SupersededAt == "" {
|
||||
t.Errorf("package %d has no superseded_at — the screen needs a date to name the package", p.Index)
|
||||
}
|
||||
}
|
||||
if !got["age-gen1"] || !got["age-gen2"] {
|
||||
t.Errorf("served packages = %v, want the two SUPERSEDED generations (age-gen1, age-gen2)", got)
|
||||
}
|
||||
if got["age-gen3"] {
|
||||
t.Error("the CURRENT package was served as retained — that is the wrong-package confusion this endpoint exists to end")
|
||||
}
|
||||
}
|
||||
|
||||
// A retained row with NO identity blob (every pre-v0.93.0 row) must be WITHHELD and COUNTED. Serving
|
||||
// it would have the agent try a package that cannot open anything, and would let the screen claim an
|
||||
// earlier package is recoverable on exactly the boxes hurt by the original defect.
|
||||
//
|
||||
// RED-PROOF: delete the `if len(e.IdentityBlob) == 0 { unopenable++; continue }` guard → the empty
|
||||
// row is served as a package → Count becomes 2 and UnopenableCount 0 → this FAILS twice.
|
||||
func TestRetainedEscrowGet_WithholdsAndCountsUnopenableRows(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||
// gen1 carries NO identity blob — the pre-v0.93.0 shape.
|
||||
seedSuperseded(t, h, "h1", "HKEY", []struct{ sha, identity string }{
|
||||
{"SHA_GEN1", ""},
|
||||
{"SHA_GEN2", "age-gen2"},
|
||||
{"SHA_GEN3", "age-gen3"},
|
||||
})
|
||||
|
||||
code, out := getRetained(t, h, "h1", "HKEY")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("GET retained = %d, want 200", code)
|
||||
}
|
||||
if out.Count != 1 {
|
||||
t.Errorf("count = %d, want 1 (only gen2 carries material)", out.Count)
|
||||
}
|
||||
if out.UnopenableCount != 1 {
|
||||
t.Errorf("unopenable_count = %d, want 1 — the caller cannot derive this and needs it to explain "+
|
||||
"why a correct old code opens nothing on such a box", out.UnopenableCount)
|
||||
}
|
||||
for _, p := range out.Packages {
|
||||
if p.IdentityEscrowB64 == "" {
|
||||
t.Error("a package with an empty identity blob was served — it can never open anything")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// SELF-SCOPE. The current-row GET has this and a retained read is strictly more material, so the same
|
||||
// asymmetry must hold: a host key reads only its own.
|
||||
//
|
||||
// RED-PROOF: remove the `!isGlobal && authHostID != pathHostID` branch → h2's key reads h1's packages
|
||||
// → this FAILS with 200.
|
||||
func TestRetainedEscrowGet_SelfScoped(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY1"})
|
||||
st.UpsertHost(&store.Host{HostID: "h2", CustomerID: "c2", APIKey: "HKEY2"})
|
||||
seedSuperseded(t, h, "h1", "HKEY1", []struct{ sha, identity string }{
|
||||
{"SHA_GEN1", "age-gen1"},
|
||||
{"SHA_GEN2", "age-gen2"},
|
||||
})
|
||||
|
||||
rr := do(h, http.MethodGet, "/hosts/h1/escrow/retained", "HKEY2", "")
|
||||
if rr.Code != http.StatusForbidden {
|
||||
t.Fatalf("h2 reading h1's retained packages = %d, want 403 — a host key must never be a fleet-wide reader", rr.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// A host that has never re-sealed gets a clean, empty answer — not a 404 and not a fault. "No retained
|
||||
// package" is an ordinary situation and must be distinguishable from "something is broken", or the
|
||||
// agent's try-loop cannot tell them apart either.
|
||||
//
|
||||
// RED-PROOF: make the handler 404 on an empty list → this FAILS with 404.
|
||||
func TestRetainedEscrowGet_NoneIsACleanEmptyAnswer(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k"), "SHA1", "age-1")); rr.Code != http.StatusOK {
|
||||
t.Fatalf("seed PUT = %d", rr.Code)
|
||||
}
|
||||
|
||||
code, out := getRetained(t, h, "h1", "HKEY")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("GET retained on a never-superseded host = %d, want 200", code)
|
||||
}
|
||||
if out.Count != 0 || len(out.Packages) != 0 || out.UnopenableCount != 0 {
|
||||
t.Errorf("want an empty answer, got count=%d packages=%d unopenable=%d", out.Count, len(out.Packages), out.UnopenableCount)
|
||||
}
|
||||
}
|
||||
|
||||
// The retained route must not be reachable without a key at all.
|
||||
//
|
||||
// RED-PROOF: drop the checkAuthHost block → this FAILS with 200.
|
||||
func TestRetainedEscrowGet_RequiresAuth(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||
if rr := do(h, http.MethodGet, "/hosts/h1/escrow/retained", "", ""); rr.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("unauthenticated retained GET = %d, want 401", rr.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// ROUTING: /escrow and /escrow/retained must stay two different answers. A prefix "tidy-up" that
|
||||
// collapsed them would route retained reads to the current row — silently the wrong package.
|
||||
//
|
||||
// RED-PROOF: change the retained case's suffix to "/escrow" (so the earlier case wins) → the retained
|
||||
// route returns the CURRENT-row shape, which has no "packages" key → this FAILS.
|
||||
func TestRetainedEscrowGet_IsADistinctRouteFromTheCurrentRow(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||
seedSuperseded(t, h, "h1", "HKEY", []struct{ sha, identity string }{
|
||||
{"SHA_GEN1", "age-gen1"},
|
||||
{"SHA_GEN2", "age-gen2"},
|
||||
})
|
||||
|
||||
cur := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", "")
|
||||
var curBody map[string]any
|
||||
if err := json.Unmarshal(cur.Body.Bytes(), &curBody); err != nil {
|
||||
t.Fatalf("decode current-row body: %v", err)
|
||||
}
|
||||
if _, hasPackages := curBody["packages"]; hasPackages {
|
||||
t.Error("the CURRENT-row route grew a packages key — the two routes have been merged")
|
||||
}
|
||||
if _, hasPresent := curBody["present"]; !hasPresent {
|
||||
t.Error("the current-row route lost its present flag")
|
||||
}
|
||||
|
||||
_, ret := getRetained(t, h, "h1", "HKEY")
|
||||
if ret.Count != 1 {
|
||||
t.Fatalf("retained route count = %d, want 1 — it is answering the current row's question", ret.Count)
|
||||
}
|
||||
}
|
||||
@@ -245,6 +245,13 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
case r.Method == http.MethodGet && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
|
||||
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
|
||||
h.handleHostEscrowGet(w, r, hostID)
|
||||
// R-311 (v0.103.0): the RETAINED siblings of the row above. The two cases cannot collide — one
|
||||
// ends `/escrow`, the other `/escrow/retained` — but do NOT "tidy" them into a single prefix
|
||||
// match: a prefix match would route retained reads to the CURRENT row, which is silently the
|
||||
// wrong package and is exactly the confusion this endpoint exists to end.
|
||||
case r.Method == http.MethodGet && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow/retained"):
|
||||
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow/retained")
|
||||
h.handleHostEscrowRetainedGet(w, r, hostID)
|
||||
// G1 break-glass: day-0 vaults the root@pam console credential (self-scoped host key); the
|
||||
// operator retrieves it via the /admin/ path (global key only).
|
||||
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/recovery-credential"):
|
||||
@@ -1371,6 +1378,158 @@ func (h *Handler) handleHostEscrowGet(w http.ResponseWriter, r *http.Request, pa
|
||||
})
|
||||
}
|
||||
|
||||
// handleHostEscrowRetainedGet serves a host its own RETAINED (superseded) sealed identity packages.
|
||||
//
|
||||
// ── WHY THIS EXISTS (R-311) ─────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// `ListSupersededEscrow` has been the only reader of a retained identity blob since v0.93.0 and had
|
||||
// ZERO production callers — every call site was a test. The consequence, measured on the fixture
|
||||
// 2026-08-12: a recovery code that demonstrably opens a retained package (proven by hand: unsealed,
|
||||
// and it restored planted files byte-identical from a store the box could no longer open) was
|
||||
// reported to the customer as a code that opened nothing. The screen already hedged that with two
|
||||
// possible causes (R-222/R-226) and said it could not tell them apart. **It could not tell them
|
||||
// apart because nothing ever looked.** This endpoint is what makes looking possible.
|
||||
//
|
||||
// ── WHAT THIS DOES NOT CHANGE ───────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// The hub still cannot read any of it. Sealed bytes in, sealed bytes out; there is no decrypt path
|
||||
// here and the hub has never held a recovery code. What DOES widen is volume: a host key that could
|
||||
// previously fetch one opaque package can now fetch N. The trade is stated rather than waved through
|
||||
// — see the audit — and it is bounded three ways: the same self-scope as the current row, the same
|
||||
// recovery-mode gate, and an explicit cap so a host with a long supersession history cannot turn one
|
||||
// request into an unbounded read.
|
||||
//
|
||||
// Rows whose `identity_blob` is NULL are NOT served and are counted separately. They are the
|
||||
// pre-v0.93.0 rows; they retain the PBS key and not the repository password, so they can never open
|
||||
// anything the caller is asking about. Serving them would make the agent try packages that cannot
|
||||
// succeed and would let the screen claim an earlier package is openable when it is not — the same
|
||||
// false-explanation trap `SupersededPresent` already avoids at store.go. They are counted because
|
||||
// their EXISTENCE is a true and load-bearing fact: on those boxes a correct old code opens nothing,
|
||||
// and the honest reason is a defect of ours, not the customer's typing.
|
||||
const retainedEscrowServeCap = 16
|
||||
|
||||
// RetainedEscrowPackage / RetainedEscrowResponse are the WIRE for GET /hosts/<id>/escrow/retained.
|
||||
//
|
||||
// They are named types rather than a `map[string]any` on purpose: the wire-contract gate resolves a
|
||||
// declared ROOT by TYPE, so an untyped map is a cross-repo contract the gate cannot see. This wire is
|
||||
// declared in `scripts/wire_contract_gate.py` ROOTS — hub → agent — and the agent's mirror is
|
||||
// `felhom-agent/internal/hub.RetainedEscrowResponse`. Change a tag here and the gate fails there,
|
||||
// which is the entire point.
|
||||
type RetainedEscrowPackage struct {
|
||||
// Index is a label WITHIN ONE RESPONSE. Not durable, never persisted, never a lookup key.
|
||||
Index int `json:"index"`
|
||||
// SupersededAt is when this package stopped being the current one. It is the only field here a
|
||||
// customer ever sees — it is how they recognise which recovery code they are holding.
|
||||
SupersededAt string `json:"superseded_at"`
|
||||
KeyFingerprint string `json:"key_fingerprint"`
|
||||
// IdentityEscrowB64 is OPAQUE ciphertext. The hub cannot open it and never could.
|
||||
IdentityEscrowB64 string `json:"identity_escrow_b64"`
|
||||
}
|
||||
|
||||
type RetainedEscrowResponse struct {
|
||||
HostID string `json:"host_id"`
|
||||
Count int `json:"count"`
|
||||
// UnopenableCount is the number of retained rows withheld because they carry no key material
|
||||
// (every pre-v0.93.0 row). Underivable by the caller and load-bearing: on a box with only those,
|
||||
// a perfectly correct old code opens nothing and the reason is a defect of ours.
|
||||
UnopenableCount int `json:"unopenable_count"`
|
||||
TruncatedCount int `json:"truncated_count"`
|
||||
Packages []RetainedEscrowPackage `json:"packages"`
|
||||
}
|
||||
|
||||
func (h *Handler) handleHostEscrowRetainedGet(w http.ResponseWriter, r *http.Request, pathHostID string) {
|
||||
authHostID, _, isGlobal, ok := h.checkAuthHost(r)
|
||||
if !ok {
|
||||
http.Error(w, "Unauthorized", http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
if pathHostID == "" {
|
||||
http.Error(w, "Missing host_id", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
// SELF-SCOPED, identically to the current-row GET. Without this line any host key is a fleet-wide
|
||||
// reader of every retained package, which is strictly worse than the same hole on one row.
|
||||
if !isGlobal && authHostID != pathHostID {
|
||||
h.logger.Printf("[WARN] retained escrow GET REFUSED: host %s asked for %s's packages (self-scope)", authHostID, pathHostID)
|
||||
http.Error(w, "Forbidden: host_id mismatch", http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
host, err := h.store.GetHost(pathHostID)
|
||||
if err != nil {
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
if host == nil {
|
||||
http.Error(w, "Unknown host_id", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
// The SAME §8.3 gate as the current row. A retained package is not less sensitive than the
|
||||
// current one; if self-service retrieval is ever switched off, it must go dark with it.
|
||||
if !escrowSelfServiceRetrieval && !host.InRecoveryMode(time.Now().UTC()) {
|
||||
h.logger.Printf("[WARN] retained escrow GET REFUSED for %s — self-service retrieval is disabled and recovery mode is not armed", pathHostID)
|
||||
http.Error(w, "Forbidden: host not in recovery mode (operator must arm it)", http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
|
||||
retained, rerr := h.store.ListSupersededEscrow(pathHostID)
|
||||
if rerr != nil {
|
||||
h.logger.Printf("[ERROR] retained escrow GET for %s: %v", pathHostID, rerr)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
packages := make([]RetainedEscrowPackage, 0, len(retained))
|
||||
unopenable := 0
|
||||
for _, e := range retained {
|
||||
if len(e.IdentityBlob) == 0 {
|
||||
unopenable++
|
||||
continue
|
||||
}
|
||||
if len(packages) >= retainedEscrowServeCap {
|
||||
continue
|
||||
}
|
||||
packages = append(packages, RetainedEscrowPackage{
|
||||
Index: len(packages),
|
||||
SupersededAt: e.UpdatedAt,
|
||||
KeyFingerprint: e.KeyFingerprint,
|
||||
IdentityEscrowB64: base64.StdEncoding.EncodeToString(e.IdentityBlob),
|
||||
})
|
||||
}
|
||||
truncated := 0
|
||||
if n := len(retained) - unopenable; n > len(packages) {
|
||||
truncated = n - len(packages)
|
||||
}
|
||||
|
||||
// The audit row is written BEFORE the bytes leave, exactly as the current-row GET does, so a
|
||||
// retrieval cannot be served without its record. Severity matches that path deliberately: this is
|
||||
// a recovery in progress and nothing else, and it must reach the operator by e-mail.
|
||||
if host.CustomerID != "" && len(packages) > 0 {
|
||||
msg := fmt.Sprintf("Retained recovery packages served: host %s retrieved %d retained sealed package(s) (%d unopenable pre-v0.93.0 row(s) withheld). "+
|
||||
"This is the recovery path in use — the packages cannot be opened without the customer's recovery code, which the hub never holds. "+
|
||||
"If no recovery is in progress on that box, investigate.", pathHostID, len(packages), unopenable)
|
||||
details, _ := json.Marshal(map[string]any{
|
||||
"host_id": pathHostID,
|
||||
"served": len(packages),
|
||||
"unopenable": unopenable,
|
||||
"truncated": truncated,
|
||||
"self_scope": !isGlobal,
|
||||
})
|
||||
if _, eerr := h.store.SaveEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub"); eerr != nil {
|
||||
h.logger.Printf("[WARN] %s event save FAILED for %s (serving anyway): %v", eventEscrowBlobServed, pathHostID, eerr)
|
||||
} else if h.dispatcher != nil {
|
||||
go h.dispatcher.ProcessEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub")
|
||||
}
|
||||
}
|
||||
h.logger.Printf("[WARN] retained escrow SERVED to host %s (%d package(s), %d unopenable withheld, %d over cap, self_scope=%v)",
|
||||
pathHostID, len(packages), unopenable, truncated, !isGlobal)
|
||||
writeJSON(w, http.StatusOK, RetainedEscrowResponse{
|
||||
HostID: pathHostID,
|
||||
Count: len(packages),
|
||||
UnopenableCount: unopenable,
|
||||
TruncatedCount: truncated,
|
||||
Packages: packages,
|
||||
})
|
||||
}
|
||||
|
||||
// maybeEmitRepoKeyChanged raises ONE operator signal per supersession when the sealed offsite repo
|
||||
// password demonstrably changed. Both hashes have been stored since SLICE 3 (host_escrow and, since
|
||||
// v0.60.0, host_escrow_superseded) and NOTHING compared them: demo-felhom's repository password
|
||||
|
||||
+1
-1
@@ -125,7 +125,7 @@ spec:
|
||||
spec:
|
||||
containers:
|
||||
- name: hub
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.102.0
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.103.0
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
|
||||
@@ -81,6 +81,12 @@ ROOTS = [
|
||||
"controller", "internal/report", "Report", "hub"),
|
||||
("hub -> controller (report ACK, `escrow` object)",
|
||||
"hub", "internal/store", "EscrowStatus", "controller"),
|
||||
# R-311. Declared the moment the wire was created, because the gate covers only what is DECLARED
|
||||
# and a silent pass is indistinguishable from coverage. The hub response is a named type rather
|
||||
# than a map[string]any precisely so this root can resolve — an untyped map is a cross-repo
|
||||
# contract nothing can check.
|
||||
("hub -> agent (GET /hosts/<id>/escrow/retained)",
|
||||
"hub", "internal/api", "RetainedEscrowResponse", "agent"),
|
||||
]
|
||||
|
||||
# Tag names whose literal string carries no information in a repo-wide search. NOT CHECKED.
|
||||
|
||||
Reference in New Issue
Block a user