R-311 proven live (HTTP 422 on hardware); R-308 WITHDRAWN — my quoting bug, not a stale credential

The live test read as a FAILURE for twenty minutes because I stripped only double
quotes from a credentials value wrapped in SINGLE ones, sending a literal ' as part
of the recovery code. Correctly unquoted: the old code returns 422 with
opens_retained=true and the supersession date; a wrong code still returns 400.

The same bug produced the R-308 finding in the previous report. The dashboard
password is fine - HTTP 302 with a session cookie on the first try. Third time this
project has produced a wrong 'the credential is stale' verdict from that one trap.

REPORT.md

@@ -112,8 +112,42 @@ enforcement**, and saying otherwise would have been false.
 | hub | **0.103.0** — see §7 |
 | golden | **0.214.0** baked + published |
 
-**Live validation was endpoint/handler-level, not a click-through**, and the reason is a finding:
-**the dashboard password of record no longer opens `demo-felhom` (R-308)**.
+### Live proof on hardware — the 422, end to end
+
+```
+OLD code (opens retained row 11)   HTTP 422  opens_retained: True
+                                             superseded_at: 2026-08-12T15:18:55Z
+                                             retained_has_restic_pw: True
+                                             "the recovery code is correct, but it belongs to an
+                                              EARLIER sealed package (superseded …), not the one
+                                              currently held"
+WRONG code (negative control)      HTTP 400  "the recovery code did not open the sealed bundle"
+```
+
+The hub half measured directly too: `GET …/escrow/retained` → **200**, `count=2`,
+**`unopenable_count=1`** — that one being retained row id 4, the pre-v0.93.0 row whose material R-198
+destroyed. The withholding rule is doing exactly what it was written for, on real data.
+
+**What was NOT walked, and why.** The customer's rendered sentence was **not** produced end-to-end.
+`recoveryUnlockHandler` redirects to `/backups/remote` when `!recoveryOffer()`, and `demo-felhom`
+holds its own repository password again (restored yesterday), so it is correctly **not** in the
+offered state. Walking it would mean removing that password to fake a rebuilt box — destabilising a
+healthy machine to render a sentence whose logic is pinned by six handler tests and whose upstream 422
+is proven live. I did not. **Method stated: endpoint-level for the agent and hub, handler-level for the
+message.** What the customer DOES see on this box today is the orphan card, and it is honest:
+*„Megnyitni innen egyelőre nem lehet, és ez nem a kódodon múlik."*
+
+### A correction I have to make about my own last report — R-308 was wrong
+
+I reported that the stored controller password no longer opens `demo-felhom`. **It does.** I had
+stripped only DOUBLE quotes from the `~/.config/credentials` value; the values are wrapped in
+**SINGLE** quotes, so I was sending a literal `'` as part of the password. Unquoted correctly it is 13
+characters and logs in first try — **HTTP 302 with a session cookie**.
+
+The same bug then made this session's first live R-311 test read as a **failure** (HTTP 400) for
+twenty minutes, and I nearly filed the fix as broken. It is the **third** wrong "the credential is
+stale" verdict this project has produced from that one trap. R-308 is **withdrawn**; the real lesson
+is filed with it — never let a shell decide what a secret is.
 
 ## 7. What was dropped, named plainly
 

documentation/backlog/OPEN-ITEMS.md

@@ -608,7 +608,7 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
 | **R-305** | **The R-300 cleanup fires exactly once per machine, and the second reinstall hits the original wall.** PROVEN on `drill-r50` 2026-08-12, not deduced. The uninstall stops+disables dnsmasq but deliberately never **purges** the package, so on every later install `dpkg-query` reports it installed, preflight records `dnsmasq_preexisting=yes`, the agent's `EnsureDnsmasq` re-enables it, and the uninstall then only *restarts* it — leaving it `enabled`/`active` on `0.0.0.0:53` exactly as before the fix. Cycle 1: `no` → disabled → `:53 FREE` → next install PASSES. Cycle 2: `yes` → left running → **cycle-3 byo preflight FAILED, exit 1**. The improved refusal does fire and names the command, so the customer is not stranded — but they must run by hand a cleanup Felhom could have run itself, on a box Felhom does own | **READY (M) — NEW 2026-08-12, RANK 2** | R-300 | Distinguish "the package pre-dates Felhom" from "a previous **Felhom** install left it" — e.g. keep the ownership record outside `state.json` (which the uninstall deletes), or record Felhom-installed-ness where the agent installs it. **Do not fix by purging on hosts we may not own** | CC |
 | **R-306** | **`--preflight-only` says "no state written" and writes state — with an answer that can be wrong.** `_state_put` short-circuits on `DRY_RUN` only (`felhom-host-install.sh:418`), so a preflight-only run creates `/var/lib/felhom-install/state.json`. Observed live: after a run whose banner read `PRE-FLIGHT PASS (mode=byo) — no state written, no install step executed`, the file existed containing `{"completed": [], "dnsmasq_preexisting": "yes"}`. Both the banner and the flag's own comment at line 226 assert the opposite. **The harm is not the file, it is the value**: the runbook recommends preflight-only first, then the same command without the flag, so on a box carrying a Felhom leftover the wrong ownership answer is baked in before the real install begins | **READY (S) — NEW 2026-08-12, RANK 3** | R-300, R-305 | Either make `_state_put` a no-op under `PREFLIGHT_ONLY` (and record ownership at install instead), or correct both claims. A comment asserting an invariant needs a test pinning it | CC |
 | **R-307** | **`demo-felhom` carries a LIVE abandon countdown that this drill did not start — and the end state says there should be none.** `settings.json`: `abandon_started_at 2026-08-10T08:06:31Z`, `abandon_at 2026-08-24T08:06:31Z`, `abandon_repo_path /home/felhom-repo.orphaned-20260810`. The drill's fence forbade starting, shortening or triggering a countdown, and none was; but its required end state was *"no abandon countdown anywhere"*, and one exists. **The two ways to satisfy that are not equivalent and the choice is the operator's:** cancel it (the orphaned repository is kept indefinitely — storage cost, no data risk) or let it run (on **2026-08-24** that repository is deleted, **irreversibly**). **Doing nothing selects deletion by default.** Deliberately not resolved by CC. **RULED 2026-08-12 (operator): KEEP the set-aside history — cancel the countdown.** Done the same evening through the product's own operator path (`--abandon-stop`, which refuses rather than silently no-opping), with the container stopped first so the running controller could not overwrite `settings.json` from memory. **Proved, not trusted to the exit code:** `abandon_started_at` and `abandon_at` are GONE (`AbandonStatus` returns `Active=false` when `AbandonAt` is empty — `offbox_abandon.go:111-113`, so no countdown renders); `abandon_repo_path` deliberately REMAINS as the pointer to the preserved store; the store still holds **36 snapshot objects** and its full `config/data/index/keys/locks/snapshots` structure; both repositories still present on the endpoint; **nothing deleted anywhere**. See R-313 — what was preserved cannot currently be opened by anyone | **CLOSED — countdown cancelled 2026-08-12 on the operator's ruling** | R-301, R-302, R-303, R-313 | — | CC |
-| **R-308** | **The stored controller password no longer opens `demo-felhom`.** `PASSWORD` in `~/.config/credentials` on DooPlex returns `Hibás jelszó` against `https://127.0.0.1:443/login` with `Host: felhom.demo-felhom.eu`. **NOT the known quoting trap** — the surrounding quotes were stripped before use and the stored value is unquoted (16 bytes for 15 chars), so this is a genuinely stale credential, not a mis-read. Cost during the drill: the dashboard was unreachable headlessly, so the offsite push had to be made with `restic` directly instead of the page's own button, which left the app-backup→unit→offsite chain unexercised. `R_DEMO-FELHOM` **was** correct and has since been rotated by this session's final ceremony (updated in place; prior file backed up alongside) | **READY (S) — NEW 2026-08-12, RANK 2** | — | Recover or reset the demo-felhom dashboard password and re-store it; note which boxes share one and which do not | operator |
+| **R-308** | **~~The stored controller password no longer opens `demo-felhom`~~ — WITHDRAWN 2026-08-12, this was MY BUG, not a defect.** The original diagnosis (`Hibás jelszó`) came from stripping only DOUBLE quotes off the `~/.config/credentials` value; the values are wrapped in **SINGLE** quotes, so a literal `'` was sent as part of the password. Correctly unquoted the value is 13 characters and logs in first try — **HTTP 302 with a `felhom_session` cookie**, measured. **This is the third time this project has produced a wrong 'the credential is stale' verdict from that exact trap**, and the standing memory about it is what caught it here — but only after the same mistake had also made the R-311 live test read as a FAILURE (HTTP 400) for twenty minutes. **The lesson is not 'strip quotes' but 'never let a shell decide what a secret is'**: parse the credentials file in Python, strip a matched pair of EITHER quote, and verify the length before use | **WITHDRAWN — not a defect (my error)** | — | Consider a tiny `read_credential.py` helper so no session hand-rolls the parse again | CC |
 | **R-309** | **The day-0 runbook says pushing the installer publishes it. It has not since R-110.** `documentation/runbooks/day0-install.md` §C.1: *"it is always the current `main` script. There is no release tag, no staging copy and no version selector; pushing `scripts/felhom-host-install.sh` publishes it."* R-110 moved `/scripts/` to the **installer tag's** tree — `manifests/webpage.yaml` carries **two** `--ref=installer-v…` pins (lines 327, 372) and `hostinstall_gates.py` gate 6 enforces that they name a tag. Confirmed live 2026-08-12: `https://felhom.eu/scripts/felhom-host-install.sh` served `SCRIPT_VERSION="1.25.0"` while `main` had `1.27.0`, three and a half hours after the push. **The runbook's version is the dangerous one to believe in both directions** — it would have an operator think a pushed fix is live when it is not, and think a pushed mistake is live when it is not | **READY (S) — NEW 2026-08-12, RANK 3** | R-110 | Correct §C.1 to name the tag-and-manifest publication step (as performed today) | CC |
 | **R-310** | **Two small edges on the installer, neither costing more than a moment.** (1) The R-297 operator-named refusal states the vouched version twice in consecutive sentences (*"…but the vouched golden is 0.213.0. The vouched golden is 0.213.0."*). (2) `--uninstall` reads its typed vmid confirmation from `/dev/tty` and `--force` deliberately does **not** bypass it, so teardown cannot be scripted without a pty — correct for an irreversible destroy, but undocumented; it surfaces as `line 891: /dev/tty: No such device or address` and an rc=1 that looks like a failure rather than a refusal to proceed unattended | **READY (S) — NEW 2026-08-12, RANK 4** | R-297 | Drop the duplicated sentence; add one runbook line naming the pty requirement | CC |
 | **R-311** | **A correct recovery code for a retained package stopped being reported as wrong.** The drill of 2026-08-12 proved the retained key WORKS (it opened a set-aside store and restored planted files byte-identical) while the product answered *"the recovery code did not open the sealed bundle"*. **What was already right:** the customer-facing screen did not bluntly accuse — R-222/R-226 hedged, naming both causes and the kept package. **What was wrong was the hedge's reason:** *„innen nem tudjuk megkülönböztetni őket"* was true only because NOTHING EVER LOOKED. **Shipped, three repos.** Hub v0.103.0: `GET /hosts/<id>/escrow/retained` — the first production caller `ListSupersededEscrow` has ever had — self-scoped, same recovery-mode gate, same audit event before the bytes leave, rows with a NULL `identity_blob` WITHHELD and returned as `unopenable_count` (they can never open anything, and serving them would let the screen promise recovery on exactly the boxes the original defect hurt), capped at 16. Agent v0.129.0: `FetchRetained` consulted ONLY after the current package refuses, `ErrCodeOpensRetained` + the supersession date, local API answers **422**, bounded at 6 attempts (~1 s of scrypt each), and fail-safe in every direction — a nil fetcher, a 404 from an older hub, a transport failure or a malformed package all leave the original refusal standing. Controller v0.214.0: class `RecoveryCodeOpensRetained` gated on `FeatureRetainedRecoveryClass` (MinAgent 0.129.0, SEPARATE from the R-224 gate because a box can sit between 0.126.0 and 0.129.0), and a message that says the code is correct, names the date, says the earlier package is kept, says the CURRENT backups are unaffected, and **deliberately promises no restore** — it routes to support, which can do it by hand. **The lie's true home was the agent:** removing the CONTROLLER case yields the neutral message (R-224's safe default catches it); removing the AGENT's retained lookup brings back the fail-closed wrong-code error in those exact words — both demonstrated, mutations asserted applied | **CLOSED — shipped hub v0.103.0 + agent v0.129.0 + controller v0.214.0** | R-199, R-222, R-224, R-226, R-304 | — | CC |