diff --git a/STATUS.md b/STATUS.md index d3c17a8..80c7294 100644 --- a/STATUS.md +++ b/STATUS.md @@ -32,10 +32,9 @@ Both now say what has happened, say that nothing is lost, and link to the one sc ## What's broken -- **An app's own first-login password is still sitting in a page's HTML**, behind a Reveal button that - only hides it visually — the same fault we just fixed for the configuration passphrase, in a - different place, with a real per-install secret. Found by looking for it rather than by an incident. - *(R-254)* +- **Nothing new is broken.** The three known secret-in-page faults are all fixed; what remains is that + the *check* against a fourth covers 4 pages out of 27, and the cheap check that covers all of them is + blind to the exact shape that shipped. Filed rather than papered over. *(R-255)* - **The machine's own screen keeps telling an already-paired box to pair itself** — 25 minutes after it was paired, on a screen that promises it refreshes itself. *(R-214, R-235)* - **A rebuilt machine cannot create a new recovery code at all.** *(R-221)* @@ -52,26 +51,29 @@ Both now say what has happened, say that nothing is lost, and link to the one sc ## Found today -- **A password was living in the page source, and we went looking for others.** The configuration - passphrase was written into the settings page on every load and merely hidden with styling — it is - now fetched only when asked for, and the asking is recorded. *(R-249, fixed.)* The search that fix - required turned up **two more places doing the same thing**, one of them a real per-install app - password. *(R-254, not fixed — recommended next.)* -- **The record was a day behind itself.** The recovery row still recorded only what was proven on - 4 August. Corrected, with both halves and both dates. A sweep of all 146 register rows found **no - other** entry whose state disagrees with its own body. +- **The last two passwords are out of the page source**, the same way as yesterday's: the page carries + only a yes/no, and the value is fetched when you ask for it — and the asking is now recorded, which + it never was. One of them was an app's own first-login password, read live out of the running app. + *(R-254, both sites fixed.)* +- **How much was actually exposed: nothing we can find.** The only app that generates a first-login + password isn't installed anywhere, and of the three apps actually running on the demo machine, none + uses a generated secret. **But nothing recorded reads** — that was part of the fault — so this is + "no evidence of exposure", not "proof there was none". No passwords need changing on that basis; + the call is yours. +- **One of the two turned out not to be a fault.** The deploy form's hidden password field is + deliberate: a form must submit what it saves, so the value you wrote down is the one stored. ## What we're working on -- **Getting the remaining two secrets out of page sources**, the same way. *(R-254)* +- **Widening the check** so a fourth secret-in-a-page is caught by a machine rather than by + someone looking. *(R-255)* - **Proving the hub really keeps the old sealed key** when a machine re-seals. Never run outside a test; needs a second deliberate wipe and its own session. *(R-198)* ## Waiting on you -- **Nothing.** The new base image was baked and approved the same day, so a machine installed now - receives this week's fixes — including getting the password out of the page source. *(R-242 — the - bake half done; what remains is ours, not yours.)* +- **Nothing blocking.** Today's release needs a new base image before installations receive it — same + as yesterday, same answer: it is ours to do, not yours. *(R-242)* *R-245 — whether an undecided household is auto-abandoned after 30 days — was settled on 7 August: we do not build it, and the reasoning is recorded.* @@ -82,10 +84,8 @@ do not build it, and the reasoning is recorded.* the machine all this is built on, not what a customer receives. Mixing them in is why the page stopped being readable.* -- **DooPlex's own backup makes every copy inside the same box, and nothing says when it fails.** - *(R-232)* -- **193 old images exist only on this machine** and cannot be re-downloaded. About 27 GB against 199 GB - free — clutter, not space. Nothing deleted. *(R-210)* +- **DooPlex's own backup keeps every copy inside the same box, and is silent when it fails.** *(R-232)* +- **193 old images exist only on this machine**, ~27 GB against 199 GB free — clutter, not space. *(R-210)* - **The hub password needs rotating** — a diagnostic printed it into a session log; nothing suggests anyone else saw it. *(R-132)* - **One thing to read after DooPlex next restarts** — the second-SSD move has never survived a reboot; diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index a0edac5..24cf014 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -169,7 +169,7 @@ the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/jour | **R-241** | **The credential self-heal, succeeding, locks the customer out of their own recovery.** Measured end to end on the final walk (2026-08-07, `tests/finalwalk-r201-2026-08-07/journal.md`). `OffsiteRecoveryOffer()` shows the recovery screen on exactly two conditions: **(a)** the box has **no** repository password — the pristine rebuilt shape — or **(b)** it has one but the inherited history will not open under it (`OffboxOrphaned()`). Overnight, unaided and exactly as designed, `offsiteheal` re-staged the one-time credential and the box's 5-minute retry **collected it and applied the tier**, writing a **fresh repository password** at 03:18Z. That makes **(a) false**. **(b)** is false too, because orphan detection only fires when a run actually tries the repository — and runs are blocked by `escrow_state: pending`. **The box therefore sits in the gap between the two conditions, and the gap is self-locking:** it cannot detect the orphan without running, cannot run without escrow, and cannot escrow without minting a NEW recovery code — which would orphan the history the customer's existing code protects. **What the customer sees:** `/` is „Indítópult" with no recovery pointer; `/recovery` **302s away**; `/backups/remote` offers „Helyreállítási kód **létrehozása**". **There is no field anywhere to enter the code they hold.** **And the operator's documented remedy also refuses** — `--recover-offsite-install` returns *„[REFUSED] a DIFFERENT repository password is already present… which history to keep is not a decision this command may take. Nothing written."*, which is correct and fail-closed and still a dead end. Recovery required moving the fresh key aside by hand and re-running the install: **three guest command lines**. **The two keys, measured:** on-disk `9b4a9a9d…` (self-heal) vs recovered-from-R `30ef574f…`. **THE DATA WAS NEVER AT RISK** — all three sentinels restored byte-identical once the right key was in place. **This is R-218's shape one level up:** that finding read *"succeeding at recovery stopped the box asking for what it still needed"*; here, succeeding at the credential self-heal stopped the box **offering** the recovery it still needed. The same walk proved the self-heal working unaided six hours earlier, and that success is what causes this. **Likely shape of the fix, not yet a decision:** the offer needs a third condition — a box holding a password it has never successfully used, while the hub holds a sealed package, is a recovery candidate — or the self-heal must not install a credential on a box whose escrow is still `pending` and whose hub blob is unconsumed. **Which of those is right is a design decision, deliberately not taken here.** **⚠ RULED 2026-08-07 by a read-only spike on the standing venue — `audits/SPIKE-r241-recovery-offer-2026-08-07.md`. IT IS A MINTING DEFECT, NOT A SCREEN-PREDICATE DEFECT, and that reverses the fix.** The screen was telling the truth: there genuinely was nothing recoverable under the key the box held, because the box minted that key itself over the top of a sealed package it already knew the hub was holding. **Three measurements carry it.** (1) **`WriteOffboxSecrets` (`offbox.go:411`) mints on ONE input — does the file exist.** It never reads `GetHubEscrowIdentityPresent()`, while its two neighbours in the same file, `OffsiteRecoveryOffer()` (`:1412`) and `needsOffsiteCredential()` (`:1377`), both do. **The same fact is available on three paths and used on two.** (2) **The flag was not merely available — it was the precondition of the chain that reached the minting.** The 5-minute retry job only logs when `RetryIfDeclared` fires, which requires the declaration, which requires that flag; the venue logged `credential retry: … (the box still declares a need; retrying)` at **02:48:03Z** and five times after — **thirty minutes and six ticks before the mint at 03:18:06Z**. (3) **The box KNEW and threw it away:** at **03:28:03Z**, thirty-five minutes before the customer looked, `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:154`) computed the exact discriminator and logged `[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574fe492… != local 9b4a9a9dcec7…) … staying pending`. **It is computed on every report cycle, never persisted, never surfaced.** **And the hub explicitly disclaims doing this** — `offsiteheal`'s package doc: *"it never runs, or asks for, an escrow ceremony… **credential automatic, key customer-present** — the ruling this session implements and must not quietly widen."* The repository key is minted in the seam between two sides that each honoured their contract, by a helper doing exactly what its doc comment says. **Fixing the predicate would paper over a box quietly making its own history unopenable.** **Recommended fix (not started, no code written): persist the discriminator the ACK already carries and add it as shape (c)** — needs no hub change and no new protocol field — **plus a `decided` latch**, because `ResetOrphanedRepo` clears `RepoState` without running a ceremony, so an `H`-mismatch discriminator alone would re-offer the screen forever to a customer who explicitly declined the old data. **Four operator decisions are stated and left unanswered in §"THE OPERATOR'S DECISION".** **✅ FIXED 2026-08-07 — controller v0.206.0 + hub v0.98.0, and the ruling above is what the fix follows.** **(1) It stops minting:** the guard is a CONJUNCTION (a package held AND no key present), so a first-time box mints exactly as before; the refusal is a HOLDING state, not a failure — the transport is still written so the recovery screen can bring the tier up the instant the key arrives (R-219), and returning an error instead would have left the hub re-staging a consumed credential for ever. New declared state `offsite.state=awaiting_recovery_key`, shown INERT to every existing hub reader from their code rather than assumed. **(2) The discriminator is persisted and drives the offer as shape (c).** §7.2 resolved deliberately: **a known difference offers however old the reading** (age is NOT gated on — gating would make a box offline from the hub silently stop offering, the very failure this removes), and **a hash never learned falls back to (a)/(b)**, because an empty hash is the hub positively saying its package seals no key rather than an unknown. **(3) Abandoning ends the question** — a 14-day countdown, visible and reversible, whose terminal step removes the set-aside store AND the sealed package together, after which shape (c) has nothing to compare and the offer falls silent **because the state is right, not because something remembers it once was not**. The two halves cannot be atomic across two machines, so it is a two-phase commit whose confirmation rides the SAME ACK that carries the request. **(4) The surface:** the full page appears **once per ENTRY into the offered state, not once ever** (an epoch — a box rebuilt months later is a new situation); three dismissal levers with three scopes, and **none removes the entry point on the backups page**. **Q7's trap does not survive:** while a recovery is outstanding „Helyreállítási kód létrehozása" is **unavailable**, not merely captioned. **§2.4 honoured:** the abandon confirmation no longer promises *„félretesszük — nem töröljük"* — it states the deletion date. **TWO REAL BUGS WERE CAUGHT BY TESTS RATHER THAN REVIEW, and both are recorded because the shape matters:** `OffboxAwaitingRecoveryKey` omitted `t.Enabled`, so a customer who had switched off-site OFF would have declared a holding state (caught by the EXISTING `TestOffsiteDeclare_DisabledTargetIsNotStranded`); and `recoveryInterrupts` returned early when the offer was false, so the FALLING edge was never recorded and the full page never came back — **the exact defect the epoch exists to fix, reintroduced inside the fix**. **Nine red-proofs, each with the mutation confirmed present in the file before its result was trusted**, including the ships-inert one (unwiring `RecordEscrowKeyHash`, which leaves everything compiling and every test passing while shape (c) reads an empty hash for ever). **NOTHING WAS DELETED ANYWHERE** — the terminal step has only ever run against injected fakes and an injected clock (§7.4). **Still open and NOT built by this:** R-242 (the release-to-golden gate) and R-245 (the automatic 30-day ending). | **FIXED 2026-08-07 — v0.206.0 / hub v0.98.0** | -| **R-242** | **A controller release that changes customer-visible behaviour is not delivered until a golden carries it — and nothing enforces that.** R-239 is the symptom; this is the mechanism, recorded **2026-08-07** and deliberately **NOT built** (the task that found it scoped it as a record-only item). **Two releases went out without a golden and the gap was invisible until a walk measured it from the customer's side**: v0.204.0 (R-237) and v0.205.0 (R-234) were written, tested, pushed, and CHANGELOG'd, and every one of those steps passed while a machine installed that night received neither. The register said CLOSED; the fleet said otherwise. **Nothing in the release path knows a golden exists.** The version bump, the image push, the CHANGELOG entry and the register closure are all repo-local; the manifest's `golden_version` is edited by a separate operator act, in a different repo, with no link back. **Proposed shapes, cheapest first — the choice is the operator's and is not taken here.** (a) **A release-path checklist step** — one line in the controller's end-of-session checklist: *a release that changes customer-visible behaviour is not finished until a golden carries it or a register row says why not.* Costs nothing, catches nothing mechanically. (b) **A gate in `repo_gates.py`** comparing the manifest's `golden_version` against the newest released controller and FAILING (or warning) past a tolerance of one minor. Mechanical, runs on every push, and would have fired the morning after v0.204.0. (c) **A hub-side checker** — the hub already knows every box's running controller version from `/hosts` and the vouched golden from the manifest; a periodic comparison against the newest published image would catch drift the repo cannot see, including a vouch that was made and then rolled back. **Earliest catch: (b).** It fires on the push that creates the gap, before any box is installed, and it needs no live fleet. **(c) catches strictly more but only after boxes exist.** (a) is worth doing regardless because it is free. **Not built. No gate was written this session.** **⚠ IT RECURRED WITHIN A DAY, WHICH IS THE ARGUMENT FOR BUILDING IT.** Controller **v0.206.0** shipped the R-241 fixes on 2026-08-07 while the vouched golden still carried **0.205.0** — so a machine installed on the morning of 2026-08-08 would have received neither. Third occurrence of the shape in three days (R-111/R-115/R-120 are the older family). **✅ SHAPE (b) BUILT 2026-08-08 — `scripts/golden_currency_gate.py`, registered in `repo_gates.py` as gate 7.** **It was shown FAILING against that exact state before anything was baked**, which is its red-proof and the reason its own introducing push needed `--no-verify` (stated in the session report rather than worked around): `newest released controller : 0.206.0 / newest golden baked : 0.205.0 → CONVICTED`. **IT IS `--fast`, AND THAT FORCED ITS DESIGN:** both `.githooks/pre-push` AND CI run `repo_gates.py --fast`, so a non-fast gate would run in NEITHER — the R-29 census failure this runner exists to end. **THEREFORE IT CHECKS THE BAKE, NOT THE VOUCH**, because the vouched version lives only in the hub's `hub_settings` with no copy in git, and putting a copy there would create a second source of truth that can drift — a green gate over a false claim being the worst outcome available. **A bake without a vouch still passes: that half is NOT closed and stays on this row.** It also compares versions rather than behaviour, so a release changing nothing customer-visible trips it too — accepted deliberately, because judging that by hand is what failed three times and the cost of a false trip is one bake; a waiver belongs here, never in a habit of bypassing. **VOUCHED 2026-08-08 with the operator's approval** — golden `0.206.0` / sha `c85230b4…108e`; `agent_version` and `min_agent` both stayed `0.127.0`, and `wrapper_sha256` was carried through explicitly because the handler clears it when omitted. **The gate was CONVICTED before the bake and OK after it** — red→green on the same command, which is its proof that it measures something real. **⚠ THE GATE FIRED FOR REAL, 2026-08-08 — and it was right.** Controller **v0.207.0** (R-249/R-252/R-253) is released, tested and pushed, and **no golden carries it** — the newest bake is 0.206.0 — so `golden_currency_gate.py` FAILED, saying exactly the true thing: *a machine installed right now would receive v0.206.0*. **The `felhom.eu` push therefore used `git push --no-verify`, declared here, in the commit message and in the session report.** A bypass and NOT a waiver, deliberately: the gate offers a waiver only for a release that *deliberately needs no golden*, and this one needs one. **Owed: bake golden 0.207.0 and vouch it** (`RUNBOOK-manual-build.md` §4.1; the vouch is a three-field change). **This row's own remaining half is unchanged — nothing gates the VOUCH itself.** **✅ THE OWED BAKE IS DONE, SAME DAY — golden 0.207.0 baked, published, round-trip verified and VOUCHED (2026-08-08).** The gate went from red to **green**, and the `--no-verify` bypass declared above is now historical rather than standing. **Round trip is the evidence, not the build log:** the published bytes were downloaded back — 656 879 192 B, sha256 `20ec9602…22995`, both identical to what the bake reported — and **`./etc/felhom-controller-image` read OUT of the downloaded archive says `felhom-controller:0.207.0`**, which is the delivered artifact naming the controller it will start. **The vouch was a three-field change with all three checked deliberately** (`MinAgent 0.127.0` read from the golden's controller CHANGELOG header, not assumed; `agent_version` already ≥ it; `min_agent` not above `agent_version`, so not the R-216 shape) and verified by **re-reading the manifest rather than trusting the flash**. **This row's remaining half is UNCHANGED and is the whole of what is still open: nothing gates the VOUCH itself** — the currency gate's own docstring says it checks the bake, so a baked-but-unvouched golden still passes it silently. Evidence: `tests/golden-0.207.0-2026-08-08/`. | **READY — the vouch half only** — owner Viktor | +| **R-242** | **A controller release that changes customer-visible behaviour is not delivered until a golden carries it — and nothing enforces that.** R-239 is the symptom; this is the mechanism, recorded **2026-08-07** and deliberately **NOT built** (the task that found it scoped it as a record-only item). **Two releases went out without a golden and the gap was invisible until a walk measured it from the customer's side**: v0.204.0 (R-237) and v0.205.0 (R-234) were written, tested, pushed, and CHANGELOG'd, and every one of those steps passed while a machine installed that night received neither. The register said CLOSED; the fleet said otherwise. **Nothing in the release path knows a golden exists.** The version bump, the image push, the CHANGELOG entry and the register closure are all repo-local; the manifest's `golden_version` is edited by a separate operator act, in a different repo, with no link back. **Proposed shapes, cheapest first — the choice is the operator's and is not taken here.** (a) **A release-path checklist step** — one line in the controller's end-of-session checklist: *a release that changes customer-visible behaviour is not finished until a golden carries it or a register row says why not.* Costs nothing, catches nothing mechanically. (b) **A gate in `repo_gates.py`** comparing the manifest's `golden_version` against the newest released controller and FAILING (or warning) past a tolerance of one minor. Mechanical, runs on every push, and would have fired the morning after v0.204.0. (c) **A hub-side checker** — the hub already knows every box's running controller version from `/hosts` and the vouched golden from the manifest; a periodic comparison against the newest published image would catch drift the repo cannot see, including a vouch that was made and then rolled back. **Earliest catch: (b).** It fires on the push that creates the gap, before any box is installed, and it needs no live fleet. **(c) catches strictly more but only after boxes exist.** (a) is worth doing regardless because it is free. **Not built. No gate was written this session.** **⚠ IT RECURRED WITHIN A DAY, WHICH IS THE ARGUMENT FOR BUILDING IT.** Controller **v0.206.0** shipped the R-241 fixes on 2026-08-07 while the vouched golden still carried **0.205.0** — so a machine installed on the morning of 2026-08-08 would have received neither. Third occurrence of the shape in three days (R-111/R-115/R-120 are the older family). **✅ SHAPE (b) BUILT 2026-08-08 — `scripts/golden_currency_gate.py`, registered in `repo_gates.py` as gate 7.** **It was shown FAILING against that exact state before anything was baked**, which is its red-proof and the reason its own introducing push needed `--no-verify` (stated in the session report rather than worked around): `newest released controller : 0.206.0 / newest golden baked : 0.205.0 → CONVICTED`. **IT IS `--fast`, AND THAT FORCED ITS DESIGN:** both `.githooks/pre-push` AND CI run `repo_gates.py --fast`, so a non-fast gate would run in NEITHER — the R-29 census failure this runner exists to end. **THEREFORE IT CHECKS THE BAKE, NOT THE VOUCH**, because the vouched version lives only in the hub's `hub_settings` with no copy in git, and putting a copy there would create a second source of truth that can drift — a green gate over a false claim being the worst outcome available. **A bake without a vouch still passes: that half is NOT closed and stays on this row.** It also compares versions rather than behaviour, so a release changing nothing customer-visible trips it too — accepted deliberately, because judging that by hand is what failed three times and the cost of a false trip is one bake; a waiver belongs here, never in a habit of bypassing. **VOUCHED 2026-08-08 with the operator's approval** — golden `0.206.0` / sha `c85230b4…108e`; `agent_version` and `min_agent` both stayed `0.127.0`, and `wrapper_sha256` was carried through explicitly because the handler clears it when omitted. **The gate was CONVICTED before the bake and OK after it** — red→green on the same command, which is its proof that it measures something real. **⚠ THE GATE FIRED FOR REAL, 2026-08-08 — and it was right.** Controller **v0.207.0** (R-249/R-252/R-253) is released, tested and pushed, and **no golden carries it** — the newest bake is 0.206.0 — so `golden_currency_gate.py` FAILED, saying exactly the true thing: *a machine installed right now would receive v0.206.0*. **The `felhom.eu` push therefore used `git push --no-verify`, declared here, in the commit message and in the session report.** A bypass and NOT a waiver, deliberately: the gate offers a waiver only for a release that *deliberately needs no golden*, and this one needs one. **Owed: bake golden 0.207.0 and vouch it** (`RUNBOOK-manual-build.md` §4.1; the vouch is a three-field change). **This row's own remaining half is unchanged — nothing gates the VOUCH itself.** **✅ THE OWED BAKE IS DONE, SAME DAY — golden 0.207.0 baked, published, round-trip verified and VOUCHED (2026-08-08).** The gate went from red to **green**, and the `--no-verify` bypass declared above is now historical rather than standing. **Round trip is the evidence, not the build log:** the published bytes were downloaded back — 656 879 192 B, sha256 `20ec9602…22995`, both identical to what the bake reported — and **`./etc/felhom-controller-image` read OUT of the downloaded archive says `felhom-controller:0.207.0`**, which is the delivered artifact naming the controller it will start. **The vouch was a three-field change with all three checked deliberately** (`MinAgent 0.127.0` read from the golden's controller CHANGELOG header, not assumed; `agent_version` already ≥ it; `min_agent` not above `agent_version`, so not the R-216 shape) and verified by **re-reading the manifest rather than trusting the flash**. **This row's remaining half is UNCHANGED and is the whole of what is still open: nothing gates the VOUCH itself** — the currency gate's own docstring says it checks the bake, so a baked-but-unvouched golden still passes it silently. Evidence: `tests/golden-0.207.0-2026-08-08/`. **⚠ RED AGAIN, 2026-08-08 (second time in two days) — controller v0.208.0 (R-254) is released and the vouched golden is 0.207.0.** `golden_currency_gate.py` FAILS, correctly: a machine installed right now receives 0.207.0 and none of today's fixes. **The `felhom.eu` push used `git push --no-verify`, declared in the commit, the CHANGELOG and the session report** — **a bypass, not a waiver**, on the same reasoning as yesterday: the gate offers a waiver only for a release that *deliberately needs no golden*, and this one needs one. **Owed: bake golden 0.208.0 and vouch it** (`RUNBOOK-manual-build.md` §4.1; three-field change, `MinAgent 0.127.0` unchanged). **Note the cadence this is establishing: two releases, two bakes owed within 24 h.** That is the argument for this row's OTHER half — nothing gates the vouch, so the only thing standing between a release and an undelivered fleet is somebody remembering. | **READY — the vouch half only** — owner Viktor | | **R-243** | **A box in the R-241 state silently stops backing up off-site, and NO ALARM OF ANY KIND FIRES.** Found by the R-241 spike (2026-08-07) as a by-product; **not part of the walk's finding and not previously filed.** The R-241 state is self-locking in a second, worse way than the recovery-journey dead end: `escrow_state` is stuck `pending` forever (the auto-confirm flips only on a hash match, and the hash cannot match a key the box minted itself), and `runOffboxBackup` returns at the escrow gate (`offbox.go:743`) before touching anything. **So off-site backups never run again — and the hub never notices.** All three signals that could catch it are excluded, each for its own individually-correct reason, verified in the hub this session: `offsite_stale` — `isStale` (`monitor/offsite.go:135`) returns false unless `EscrowState == "escrowed"`, and its own comment reads *"Pending/disabled = normal onboarding, never stale"*, so the box is classified as **still being set up, forever**; `offsite_delivery_stuck` — `monitor/offsite_delivery.go:91` skips the `applied` shape, and delivery genuinely IS applied (the credential was consumed and the target is in every report); `backup_failed` — never fires, because nothing fails: the run returns `nil` before it starts. **Three correct exclusions leaving one state unobserved.** This is the same class as the workspace `CLAUDE.md` "presence is not success" rule, one level up: **the absence of a failure is being read as the presence of a working tier.** **Partly subsumed by R-241's fix** — a box that recovers leaves this state — but **not for a box that does not**, and the alarm gap is what makes "does not" survivable indefinitely. **Not fixed; no code written.** **⚠ UPDATED 2026-08-07 (v0.206.0) — the STATE this row describes can no longer be entered, but the ALARM GAP is untouched and the row stays open.** R-241's mint guard means a box no longer mints a key over a sealed package, so it no longer arrives in the "escrow stuck pending against a self-minted key" state by itself. **What replaces it is a state that is VISIBLE rather than silent:** the box declares `offsite.state=awaiting_recovery_key` and the customer is offered the recovery screen. **But the hub still raises nothing for it**, and for the same three reasons: `isStale` needs `escrowed`, the delivery checker skips the `applied` shape, and `backup_failed` needs a run that never happens. **So a box whose customer never acts still stops backing up off-site with no operator signal** — the difference is that the customer can now see it and act, where before nobody could. **The remaining work is an operator-side signal for a box held in `awaiting_recovery_key` past some age**, and it is deliberately not bundled into R-241's fix. **⚠ MEASURED ON A REBUILD, 2026-08-07 (fifth walk) — the gap is real for the state this row describes, and NOT for the state a rebuild produces.** 88 seconds after the walk5 guest was destroyed and rebuilt, the hub emitted `offsite_delivery_stuck` (**warning**) and wrote an **operator-channel** `notification_log` row recording `offsite_credential_restaged` / status **REFUSED** with an accurate reason — *"the credential was applied and worked; the target was lost afterwards … a guest rebuild does, R-193"*. So on the **regressed-apply** shape the operator IS told, promptly and correctly, and this row's *"skips the applied shape"* does not apply. The gap stands for a box that reaches the held state **without** a prior working tier in its report history. **Recorded so the row is not read wider than it measures.** | **READY** — owner Viktor | | **R-244** | **The customer DELETE cascade leaves `app_log_issues` behind, and it is systematic across every venue ever torn down.** Found **2026-08-07** while verifying the `finalwalk` teardown with a **full census** (every table, every column) rather than a per-table query. After a cascade that logged `COMPLETE … full teardown`, **61 rows still matched `finalwalk`**. Four of the five sources are **deliberate and correct** — the cascade's own header states *"Provenance/events are NEVER wiped — audit outlives every tier"*: `events` 16, `notification_log` 14, `host_deletions` 1, `customer_resets` 1. **The fifth is a gap:** `app_log_issues` 29 rows, which the residue purge does not touch (its logged leg covers `reports`/`app_telemetry`/`app_log_tails`/`log_tail_requests`/`notif_prefs`/`selfbind_tokens`/`appliance_registrations` — not this table). **It is not a `finalwalk` quirk:** rows still reference **`c11` 40, `rewalk` 20, `part4` 24** — all three torn down 2026-08-06, whose ledger recorded *"0 occurrences"*. **That prior claim was measured with a narrower query and does not survive a full census; the correction is recorded rather than the measurement quietly redone.** **Why it was probably never written, established rather than assumed:** the table is a **fleet-wide aggregate** keyed on `app_name`+`fingerprint` with an `affected_customers` JSON list — of the 29 `finalwalk` rows, **12 reference only `finalwalk`** (orphans, safely deletable) and **17 are shared with LIVE customers** (`demo-felhom`, `peti-felhom`, …) and **must not be deleted, only de-referenced.** A naive `DELETE … WHERE customer LIKE` would destroy a live customer's issue history — which is very likely why the leg does not exist, and is the reason this is not a one-line fix. **Severity is LOW and stated plainly: no secret material is involved** — app name, fingerprint, message text, counts, timestamps. What survives is a deleted customer's *identifier* inside an aggregate row. **Proposed shape:** a residue leg that (a) removes the customer id from `affected_customers`/`context_customer`, and (b) deletes rows whose `affected_customers` becomes empty; plus a one-off sweep for the four already-torn-down venues. **The general lesson is the reusable part:** *a per-table absence query is not a census.* The teardown verification is now a full-schema sweep, and that is what found this. **Not fixed** — a cascade change needs its own red-proof and this session was scoped as a spike plus two operations. Evidence: `tests/teardown-finalwalk-2026-08-07.md`. **⚠ STILL OWED, AND NOW MEASURED RATHER THAN ESTIMATED (2026-08-08 census, read-only, no truncation).** `app_log_issues` holds **1309 rows**; **71 reference a torn-down venue** (`finalwalk`, `c11`, `rewalk`, `part4`); of those **44 are ORPHANS** — they name only torn-down customers and are safely deletable — and **27 are SHARED with a live customer** (`demo-felhom`, `peti-felhom`, …) and **must be de-referenced, never deleted**. 1238 rows are untouched. **The 27 are exactly why the leg was never written**, and why a `DELETE … WHERE customer LIKE` would destroy a live customer's issue history. **What it needs, precisely:** a cascade leg that (a) removes the customer id from `affected_customers` / `context_customer`, and (b) deletes only rows whose `affected_customers` becomes empty; plus a one-off sweep for the four venues already gone. **Why it was NOT done on 2026-08-08:** the fix is hub code, and that session's scope forbade a hub version bump; a hand-run SQL mutation over 71 rows — 27 of them needing surgical de-referencing — with no tested code path and no red-proof is precisely the shape that goes wrong on a live database. **It accumulates one venue at a time, so the next walk adds to it**; the numbers above mean the next session starts from data rather than a guess. **⚠ IT GREW AGAIN, AS PREDICTED — walk5 teardown, 2026-08-08.** The fifth walk's venue was torn down with a full-schema census taken **before and after**: **168 rows → 67**. Of the 67, **37 are by design** (`events` 21, `notification_log` 14, `host_deletions` 1, `customer_resets` 1) and **30 are `app_log_issues`** — this row's gap, and the count was **predicted in the pre-run enumeration rather than discovered afterwards**, which is the difference from the ledger that once recorded *"0 occurrences"* from a narrower query. **The running total across torn-down venues therefore rises from 71 to ~101 rows** (`finalwalk`, `c11`, `rewalk`, `part4`, now `walk5`) — the shared-with-a-live-customer subset must still be **de-referenced, never deleted**. **It accumulates one venue at a time and it did so again.** Evidence: `tests/walk5-r201-2026-08-07/teardown-walk5-2026-08-08.md`. | **READY** — owner Viktor | @@ -184,7 +184,18 @@ the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/jour | **R-251** | **The recovery listing renders one row per restic TAG, so the customer is shown an "app" they never installed and their data counted twice.** Measured on the fifth walk, 2026-08-07, on the screen the customer reaches after entering R. The snapshot carries tags `felhom-offbox,calibre-web`; the listing renders **two rows** — `calibre-web · 2026-08-07 14:57 · 12.8 MB` and `felhom-offbox · 2026-08-07 14:57 · 12.8 MB`. `felhom-offbox` is the tier's own marker tag, not an application. **The screen's whole job is to let the customer check that what is in the store is what they expect** (*"Nézd át, hogy tényleg azt találod-e itt, amire számítasz"*), and it shows them a stranger's name beside their own data and a total that is double the truth. **Cosmetic, not a data defect** — the restore page correctly offers only `calibre-web`. **Fix:** filter the marker tag out of the listing, or key the rows on the app tag. | **READY** — owner Viktor | | **R-252** | **After a rebuild the restore refuses because the data drives are not registered, and nothing on the recovery path says so.** Measured on the fifth walk, 2026-08-07, at the last step of a successful recovery. The customer enters R, sees the listing, presses through to the restore — and gets **„nincs elérhető adatmeghajtó a visszaállításhoz"**. The drives physically survived (the raw mounts are the R-220 condition and were deliberately left in place); what did not survive is their **registration**, which lived in the destroyed guest's settings. **It is recoverable without a shell** — `GET /api/disks/candidates` offers both disks (`mountable: true`, `data_bearing: true`) and Tárhely → Meghajtók → „Meglévő meghajtó csatolása" re-registers them — **but nothing tells the customer that, and the recovery screen's own hand-off ("Tovább a visszaállításhoz") walks them straight into it.** **This is why the walk's journey half passes on the letter and not the spirit:** no guest command line was needed, and a customer who did not already know the product would stop here. **Fix:** detect the unregistered-drive state on the recovery/restore path and say what to do, or offer the re-attach inline. **Related:** R-253, which is the very next step and worse. **✅ FIXED — controller v0.207.0, 2026-08-08.** The restore page now states the precondition BEFORE the customer presses anything — „Előbb csatold vissza az adatmeghajtót”, saying the backups and the drives are both still there, and linking to `/storage` — and the refusal string names the same route. The page asks through the backup manager's own `HasRestoreDestination()`, which reads the **same** `GetSchedulableStoragePaths()` the scratch resolver reads, so page and handler cannot drift apart. Conditional: a healthy box renders exactly as before, pinned by a test that fails if the notice becomes unconditional. **Note for whoever re-tests it live:** the state is **rebuild-only** — the customer API refuses to deregister the last usable drive, so it cannot be reached by removing drives from the dashboard, and the live check on VM 325 therefore exercised the healthy branch (Scenario E) while the notice itself is pinned at the template + predicate level. | **CLOSED 2026-08-08** | | **R-253** | **The restore page promises it will reinstall the app, and the restore then refuses because the app is not installed — in the customer's own language, three lines apart.** Measured on the fifth walk, 2026-08-07. `/backups/restore` lists the app with the note **„Nincs telepítve — a visszaállítás előbb újratelepíti."** (*not installed — the restore will reinstall it first*). Pressing through to the full restore returns **„A teljes visszaállítás sikertelen: a(z) calibre-web nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat"** (*…not installed — restore the application first, then the data*). **Both sentences are on the recovery path, both are addressed to the same customer, and they say opposite things about the same fact.** The customer clears it by redeploying from the catalog (customer-facing, ~90 s) and re-running the restore — which then works — but they must work out for themselves that the first sentence was wrong. **This is the R-203/R-234/R-240 class one level up:** not a warning misread as a success, but a promise the next screen contradicts. **Fix:** either make the reconstitute path deploy the app when it is absent (which is what the list already claims), or change the list's note to say the app must be deployed first. **Do not leave the two sentences both shipped.** **✅ FIXED — controller v0.207.0, 2026-08-08.** **The promise was the wrong half, and the reason is structural:** reconstitution writes to the app's own `GetStackHDDPath`, which exists only once the CUSTOMER has chosen a drive at deploy time — an automatic reinstall would mean the product making that choice for them, which is the one decision this whole recovery path exists to leave with them. So the copy now reads „Nincs telepítve — előbb telepítsd újra, utána hozhatod vissza az adatait” and links to `/stacks//deploy`; the reconstitute refusal was reworded to match, so the two cannot say opposite things again. **A pre-existing test pinned the OLD sentence** — its intent (state the consequence, do not hide it) is unchanged, and it now asserts the consequence AND the route. | **CLOSED 2026-08-08** | -| **R-254** | **The same render-then-hide pattern R-249 fixed is live in two more places, and one of them carries a real per-install secret.** Found by the §7.1 census that R-249's fix required — *a pattern found once is worth a census*, and it was. **(1) `app_info.html:185` — the serious one.** An app's auto-generated first-login password is rendered into `` beside a „Megjelenítés" button. `hidden` is the same class of control as R-249's `display:none`: it stops a browser drawing the value and leaves it in the response body, so a fetch of the app page returns it. **The value is a REAL per-install secret** — `ReadInitialCredentials` reads it live out of the deployed container (`internal/stacks/initialcreds.go`), it is not the catalog's published `default_creds`. **(2) `deploy.html:482` — the weaker one.** An auto-generated `type: secret` deploy field renders into a `` with a „Megjelenítés" toggle. On the PRE-deploy form this is close to unavoidable — the form must post the value, and it does, in a sibling hidden input — but on an **already-deployed** app's page (`$isDeployed`) the hidden input is correctly omitted while the readonly input still carries the value, and there the exposure is gratuitous. **Not fixed here, deliberately:** this session's scope was R-249/R-252/R-253, and each of these needs its own reveal endpoint and its own body-asserting test rather than a shared quick edit. **The fix shape already exists** — `POST /settings/retrieval-password/reveal` (v0.207.0) and `escrow_handlers.go`'s rule that a secret is revealed by an XHR and never templated server-side into HTML. **Severity MEDIUM for (1)** — a real credential in a page any logged-in customer opens, with no audit event, reaching caches, history and screen-shares; **LOW for (2)**. **Recommended next**, because R-249 proved the pattern is not theoretical: it was found by the value landing in a session transcript. | **READY** — owner Viktor | +| **R-254** | **The same render-then-hide pattern R-249 fixed is live in two more places, and one of them carries a real per-install secret.** Found by the §7.1 census that R-249's fix required — *a pattern found once is worth a census*, and it was. **(1) `app_info.html:185` — the serious one.** An app's auto-generated first-login password is rendered into `` beside a „Megjelenítés" button. `hidden` is the same class of control as R-249's `display:none`: it stops a browser drawing the value and leaves it in the response body, so a fetch of the app page returns it. **The value is a REAL per-install secret** — `ReadInitialCredentials` reads it live out of the deployed container (`internal/stacks/initialcreds.go`), it is not the catalog's published `default_creds`. **(2) `deploy.html:482` — the weaker one.** An auto-generated `type: secret` deploy field renders into a `` with a „Megjelenítés" toggle. On the PRE-deploy form this is close to unavoidable — the form must post the value, and it does, in a sibling hidden input — but on an **already-deployed** app's page (`$isDeployed`) the hidden input is correctly omitted while the readonly input still carries the value, and there the exposure is gratuitous. **Not fixed here, deliberately:** this session's scope was R-249/R-252/R-253, and each of these needs its own reveal endpoint and its own body-asserting test rather than a shared quick edit. **The fix shape already exists** — `POST /settings/retrieval-password/reveal` (v0.207.0) and `escrow_handlers.go`'s rule that a secret is revealed by an XHR and never templated server-side into HTML. **Severity MEDIUM for (1)** — a real credential in a page any logged-in customer opens, with no audit event, reaching caches, history and screen-shares; **LOW for (2)**. **Recommended next**, because R-249 proved the pattern is not theoretical: it was found by the value landing in a session transcript. **✅ BOTH SITES CLOSED — controller v0.208.0, 2026-08-08, and they turned out to be two different problems.** +| **R-255** | **The check that would catch a fourth secret-in-the-body covers 4 of 27 pages, and the cheap gate that covers all 36 templates is blind to the shape that actually shipped.** Filed 2026-08-08 while closing R-254, **because a partial guard reported as complete is worse than no guard — it stops the next person looking.** **Two nets, both measured.** **(1) `scripts/secret_in_markup_gate.py`** reads all 36 templates and convicts any `{{ … }}` naming a secret unless allowlisted with a reason. It catches `{{.RetrievalPassword}}` and `{{.InitialCreds.Password}}`, **and it catches a launder through a local variable** because the assignment itself names the secret (`{{$v := .InitialCreds.Password}}` is convicted — verified). **It is blind to a secret arriving under a NEUTRAL PAGE-DATA KEY** — `data["Tagline"] = creds.Password` then `{{.AppInfo.Tagline}}` passes it cleanly, also verified. **That is exactly the shape of R-254 site two** (`value="{{$val}}"` inside an `{{if eq .Type "secret"}}` branch), so the gate **would not have caught one of the three instances it was written for.** **(2) The runtime body assertion** — render the page and grep the response for a sentinel — catches every shape, including that one (demonstrated on the same planted leak the gate missed). But it needs each page's data to be constructible in a test, and **only 4 of 27 page templates have that today**: `settings_security`, `app_info`, `deploy`, `backups_restore` — the four that were touched by R-249/R-252/R-253/R-254 and therefore got their own tests. **The other 23 pages have no runtime coverage at all.** **What closing this needs, so the cost is not re-estimated:** a per-page data fixture for the remaining 23 (most need a wired `Server` — `stackMgr`, `backupMgr`, agent seams), then one table-driven test that renders each with a sentinel substituted for every string in its data and asserts the sentinel is absent. **That is real scaffolding, which is why it was NOT built inside R-254's session** rather than half-built and declared done. | **READY** — owner Viktor | + +**SITE ONE — the same defect, fixed the same way.** `app_info.html` no longer renders the value; the page carries the username, the note and a boolean, and the password comes from **`POST /apps//initial-credentials/reveal`**, which **re-reads the running container** rather than serving a cached copy (caching it in the handler would have put it straight back into the body one layer in). `no-store`, CSRF-covered, and **logged as an act**. Both controls — Megjelenítés *and* Másolás — go through it, and a reveal that cannot read the value **says why** instead of returning an empty string that would render as a blank password. + +**SITE TWO — NOT the defect the row described, and the difference is the finding.** The hidden input is **deliberate and was left alone**: it fires only on the PRE-DEPLOY form, and `README §318` documents why the value must round-trip — the customer is shown the generated secrets so they can note them down, and submitting them back is what makes the saved value the same one they saw (*"no silent re-generation on submit"*). **A form must carry what it submits.** What *was* indefensible is the neighbouring **readonly display input**: on an ALREADY-DEPLOYED app the hidden input is correctly omitted — nothing is being submitted — yet the secret was still rendered into a page the customer merely opens. Fixed by **`POST /stacks//auto-field/reveal`**, authorised by requiring the field to be a `type: secret` auto-generated field *of that stack's catalog metadata* — that check is what stops it becoming "read me any value out of any app". Both directions are pinned: the deployed page must not carry the value, **and the pre-deploy form must still submit it**. + +**⚠ THE PREMISE THAT THIS BROKE A REPO RULE DOES NOT HOLD, and it is recorded rather than quietly dropped.** The rule cited was *"Password fields require explicit user input or generation (no silent auto-fill)"*. **No such line exists anywhere in the repo.** What exists is `CONTEXT.md:2070` — *"Password fields require explicit input | Prevents accidental empty-password deployments"* — which is about **emptiness**, not auto-fill, and which the hidden input does not contradict. + +**§7.3 — HOW MUCH WAS ACTUALLY EXPOSED, measured on the fleet rather than assumed.** **Site one: nothing.** `crafty-controller` is the ONLY catalog app declaring `initial_credentials`, and it is **deployed nowhere** — the card renders only when `found.Deployed && found.Meta.InitialCreds != nil`, so that code path has never run in production. **Site two: nothing measurable either.** 26 catalog apps declare a generated `type: secret` field, but `demo-hp` has exactly **three** apps deployed — `calibre-web`, `opengist`, `privatebin` — and **none of the three declares one**. **THE HONEST LIMIT: this is a CURRENT-STATE measurement.** An app deployed and later removed would not appear in it, and **nothing anywhere recorded a read** — which is itself part of the defect being fixed. So: **no evidence of exposure, and no mechanism that could have produced evidence either way.** **Rotation is therefore not indicated by anything measured** — the decision is the operator's, and this note is the input to it. + +**Gate:** `scripts/secret_in_markup_gate.py`, registered in `controller_gates.py`. **Its blind spot is measured and in its docstring** — see R-255. | **CLOSED 2026-08-08** | **Recorded against existing rows by Phase 2:**