R-234 root-caused and CLOSED; R-218's state field corrected

R-234 was filed as "toggling an app on leaves it without a bundle, so the first run
skips it". Measured on demo-hp: that state does not survive a run — the off-site
run's own pre-dump phase calls captureAllRecoveryUnits for every DEPLOYED stack,
through admitApp, before the push, and a unit moved aside was RECREATED.

The actual cause was the single-flight: the manual run was dropped because an
earlier one was still going, runOffboxBackup returned nil, the handler had already
answered "A tavoli mentes elindult", and the card then showed the PREVIOUS run's
green verdict. Fixed in controller v0.205.0 and proven live on demo-hp: a second
request while one is in flight now says "Mar fut egy tavoli mentes — ez a keres nem
inditott ujat. A most lathato eredmeny meg a korabbi futase", as a flash_error.

Independently, and a real gap on its own: a run that skipped an app the customer
selected is now `incomplete`, not `ok`. Selected+deployed with no unit counts;
selected-but-undeployed is named with what to do but does NOT count, because a box
left amber by an app somebody removed is a status nobody reads.

R-218's state field read REOPENED while the same row's body already recorded the
fix shipped in v0.203.0 and proven live. Corrected to CLOSED, keeping the
over-claim history — it is why the row is worded as it is.

Capability map: the off-site capture row's `incomplete` sentence widened to cover a
whole-app skip, and it still does not claim a newly-selected app is protected by
the next run — for a deployed app it is, for an undeployed one the card says so.

Still open, deliberately: R-213, R-202, R-214, R-235.

STATUS.md

@@ -24,6 +24,15 @@ code was wrong. *(CAMPAIGN 11)*
 
 ## What's broken
 
+- **A backup could report „✓ Rendben" while quietly leaving out an app the customer had just
+  chosen — FIXED 6 August.** Two things were wrong and only one had been guessed at. The machine's
+  own rule said *"a warning beside a success is read as a success"*, and it applied it to an app
+  missing a *folder* but not to an app left out *entirely*; that now counts too. And the real cause of
+  the case we saw: pressing „Távoli mentés most" while a backup was already running answered
+  „elindult" and then showed the **previous** run's green tick — so the customer read it as covering
+  their new app. It did not, and the restore refused minutes later. The button now says plainly that
+  it did not start anything.
+
 - **A machine installed today still gets the older in-house service, so it cannot open a recovery
   package until you approve the newer one.** It is no longer *lied to* — it says plainly that the
   machine cannot do this yet — but **approving the new service is one click from you**, and until then

documentation/backlog/OPEN-ITEMS.md

@@ -111,7 +111,7 @@ and `journal-phase24.md` (Phases 2/4). Campaign document:
 | ID | What | State |
 |---|---|---|
 | **R-216** | **A correct recovery code was reported to the customer as wrong.** The unseal needs agent v0.125.0; on an older agent the route 404s and the unlock was attempted anyway, producing *„…nem fogadtuk el. Ellenőrizd, hogy mind a tíz szót pontosan…"* in 0.134 s. **The default state** — the Day-0 manifest vouches 0.120.0, and a reinstall actively DOWNGRADES a hand-fixed box back to it. The hub's own guard could not catch it: `ResolveManagedFloor` compared against the GOLDEN's MinAgent while serving a FLOOR that pointed elsewhere (the 8th entry in `CLAUDE.md`'s comment-vs-code table, and the first where the false invariant was a guard) | **SHIPPED** (controller v0.201.0 + hub v0.97.0/0.97.1) — **but see R-223**: the feature does not work on a NEW box until the manifest vouches agent 0.125.0. Until then such a box is correctly HELD, not lied to |
-| **R-218** | **Succeeding at recovery stopped the box asking for what it still needed.** `needsOffsiteCredential` short-circuited on a repository password existing — and installing one is the recovery screen's whole job. 32 s after the hub re-staged the credential, the customer's success switched the mechanism off; the hub held an unconsumed credential the box had no reason to collect, and nothing ever asked again | **REOPENED 2026-08-06 — the fix covers the DECLARATION half only, and this row over-claimed it.** Measured on the R-201 re-walk: the box declared, and **`offsiteheal` re-staged the secret at 11:44:57** saying *"the box re-consumes on its next cycle"* — **the next cycle came and went** (`host-report` 11:55:46, `Received report` 11:55:54, a full cycle **with a positive control that it ran**) **and the credential was still not consumed.** 23 minutes after the re-stage the box's last off-site-apply attempt was still the pre-re-stage one. A census of the customer-reachable actions on `/backups/remote` (`config`, `reset`, `run`, `toggle`) found **none that fetches a staged credential**, and the only lever is `systemctl restart felhom-controller-bootstrap.service` **inside the guest** — which worked in **18 s** (Campaign 11 measured 17), confirming nothing was wrong with the credential, the target or the key: **the only thing missing is anything at all to trigger a retry.** **This is the FIRST of the two dead ends that keep the recovery journey failing** | **CLOSED 2026-08-06 — shipped in controller v0.203.0 and PROVEN LIVE on a genuinely rebuilt box.** `Bridge.RetryIfDeclared` re-runs the SAME reconcile on a 5-minute tick for exactly as long as the box's own published declaration (`OffboxReportStatus().State`) says it needs a credential — the very statement the hub acts on, so the two can never disagree. Measured on the Part 4 venue (`documentation/tests/part4-rewalk-2026-08-06/journal.md`): after the rebuild the retry ticked and logged its reason honestly, the hub's `offsiteheal` re-staged at **13:24:57Z**, and the box **collected it on a tick and configured the tier** — `[offsite-apply] offsite configured for u629488-sub6@…:/home/felhom-repo` — **with no guest command line and no operator action**. On a healthy box the job registers, ticks, completes in 0 s and says nothing, which is asserted. |
+| **R-218** | **Succeeding at recovery stopped the box asking for what it still needed.** `needsOffsiteCredential` short-circuited on a repository password existing — and installing one is the recovery screen's whole job. 32 s after the hub re-staged the credential, the customer's success switched the mechanism off; the hub held an unconsumed credential the box had no reason to collect, and nothing ever asked again | **CLOSED 2026-08-06 — controller v0.203.0, proven live.** *(State corrected 2026-08-06: this field read REOPENED while the body below already recorded the fix shipped and proven. The history of the over-claim is kept deliberately — it is why the row is worded as it is.)* **The over-claim, as it stood: the fix covered the DECLARATION half only.** Measured on the R-201 re-walk: the box declared, and **`offsiteheal` re-staged the secret at 11:44:57** saying *"the box re-consumes on its next cycle"* — **the next cycle came and went** (`host-report` 11:55:46, `Received report` 11:55:54, a full cycle **with a positive control that it ran**) **and the credential was still not consumed.** 23 minutes after the re-stage the box's last off-site-apply attempt was still the pre-re-stage one. A census of the customer-reachable actions on `/backups/remote` (`config`, `reset`, `run`, `toggle`) found **none that fetches a staged credential**, and the only lever is `systemctl restart felhom-controller-bootstrap.service` **inside the guest** — which worked in **18 s** (Campaign 11 measured 17), confirming nothing was wrong with the credential, the target or the key: **the only thing missing is anything at all to trigger a retry.** **This is the FIRST of the two dead ends that keep the recovery journey failing** | **CLOSED 2026-08-06 — shipped in controller v0.203.0 and PROVEN LIVE on a genuinely rebuilt box.** `Bridge.RetryIfDeclared` re-runs the SAME reconcile on a 5-minute tick for exactly as long as the box's own published declaration (`OffboxReportStatus().State`) says it needs a credential — the very statement the hub acts on, so the two can never disagree. Measured on the Part 4 venue (`documentation/tests/part4-rewalk-2026-08-06/journal.md`): after the rebuild the retry ticked and logged its reason honestly, the hub's `offsiteheal` re-staged at **13:24:57Z**, and the box **collected it on a tick and configured the tier** — `[offsite-apply] offsite configured for u629488-sub6@…:/home/felhom-repo` — **with no guest command line and no operator action**. On a healthy box the job registers, ticks, completes in 0 s and says nothing, which is asserted. |
 | **R-219** | **The listing the screen promises could never render on the shape it exists for.** Listing needs a target; a target cannot exist without a repository password; shape (a) is defined by having none. And placing the key flipped the offer false, so the unlock response was the only chance and was guaranteed not to contain it | **SHIPPED** (controller v0.201.0) — the unlock now places the key, brings the tier up, then lists |
 | **R-217** | **An unreadable store reported as "opened, with unattributable content".** The failure path passed `backup.OffsiteInventory{}`, whose `Empty=false` the template read as `InvUntagged`. The field built to prevent exactly this names the hazard in its own doc comment | **SHIPPED** (controller v0.201.0) — opened / empty / unreadable are three distinguishable states |
 | **R-222** | **Reaching for a RETAINED earlier package read as a wrong code.** The engine is right (it fails closed against the current package); the message was not. Proven live: the correct code for the orphaned history got *„check your ten words"* | **SHIPPED** (controller v0.201.0 + hub v0.97.0) — the ACK carries `superseded_present`/`superseded_at` and the screen names the situation. **It states what the hub knows and promises nothing** — the read path is still unbuilt (R-199's inventory) |
@@ -157,7 +157,7 @@ the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/jour
 | **R-231** | **`/opt/backup/scripts/` on DooPlex is unversioned host state** — found 2026-08-06 while adding the auto-memory store to the backup set. No repository tracks the scripts that protect the recovery chain, so the edit made that day (`CLAUDE_MEMORY_DIR` in `backup-config.sh`, multi-path restic call in `backup-data.sh`) exists only on the box. This is the same class the part-2 session was closing, found inside the fix for it; the change is transcribed in `felhom.eu/workspace/README.md` so it is at least *recorded*. **Two related facts, both understating current safety:** the backup destination (`/mnt/5_hdd/backup`) is on the **same physical disk** as the workspace it protects, and the DooPlex backup set has **no off-site leg** (`sync-hetzner-backups.sh` is jarrs.eu and pulls *from* Hetzner *to* DooPlex). Bringing a root-owned production backup script under version control, and deciding what installs it, is its own scoped change. | **READY** — owner Viktor |
 | **R-233** | **The golden bake's acceptance checks were a list of strings the script does not print** — found 2026-08-06 while baking golden 0.203.0 by following `runbooks/RUNBOOK-manual-build.md` §4.1 verbatim. Two of the three named pass markers **cannot ever match**: `overlay2 OK` is not in `build-golden.sh` at all (the line it means is `  docker OK (overlay2; data-root /var/lib/docker)`), and `including mount point … mp1` refers to a volume that stopped existing in `build-golden.sh` **v3.0.0**, when R-165 collapsed the two data volumes into one. The 404 pre-gate's URL was also wrong — the published filename is `golden.tar.zst`, not `felhom-golden-<VER>.tar.zst`, so the pre-gate would 404 for the wrong reason and pass **even when the version already existed**. This is the *"an instrument that can silently drop results is not a measurement"* class landing on the bake's own acceptance check: a grep for an impossible string reads `0` forever, and `0` is indistinguishable from failure. **The bake was never actually unguarded** — the script's own `[ "$drv" = "overlay2" ] || { echo FATAL; exit 1; }` is fail-closed and the run exited 0 with no `FATAL`. The **document** was the broken part, which is why nothing had ever gone wrong and nobody had noticed. **FIXED in the same session:** all markers re-captured from the real log rather than paraphrased, the corrected pre-gate URL, the token handling moved off the command line into an in-VM runner script (the old `--setenv=GITEA_TOKEN=$GT` form put the value where `systemctl show` prints it), a required **positive control** on the token-leak grep, and the vouch step rewritten as the three-field change it actually is. **The general lesson:** a runbook's pass markers must be **copied from a captured log, never written from memory** — §4.0 of that same file already learned this for the qemu launch line and says so; §4.1 had not. | **CLOSED 2026-08-06** — fixed in `RUNBOOK-manual-build.md` |
 
-| **R-234** | **An off-site run reports success while silently omitting an app the customer just switched on.** Found 2026-08-06 on the Part 4 venue (`part4`, VM 323), and found ONLY because the pre-destruction verification restore was run instead of trusting the green tick. Sequence, measured: a run with no app selected produced **1 snapshot**; `POST /backup/offbox/toggle` enabled `calibre-web` (HTTP 302, and the „Nincs távoli mentésre jelölt alkalmazás" warning disappeared, so the selection HAD landed); the next run finished in 30 s and reported **„✓ Rendben · 12.0 MB · 1 pillanatkép"** — still one snapshot. The app restore then refused: **„offbox: nincs pillanatkép a(z) calibre-web alkalmazáshoz"**. A THIRD run took the snapshot count to 2 and the same restore then succeeded. So a run that the customer sees as a green success did not carry the app they had just enabled, and **nothing in the card distinguishes that from a run that did**. The customer's belief ("my app is off-site") and the truth diverge silently, and they would discover it only at restore — the worst possible moment. The snapshot count is on the same card, which is what makes the omission detectable in hindsight and invisible in the moment. **Not yet root-caused**: the likely shape is that the run captured the app-selection set before the toggle committed, but that is a hypothesis, not a measurement. **This is the exact class the project already has a rule for** — "presence is not success": the run's timestamp and tick record that a run HAPPENED, not that it carried what the customer asked for. | **READY** — owner Viktor |
+| **R-234** | **An off-site run reports success while silently omitting an app the customer just switched on.** Found 2026-08-06 on the Part 4 venue (`part4`, VM 323), and found ONLY because the pre-destruction verification restore was run instead of trusting the green tick. Sequence, measured: a run with no app selected produced **1 snapshot**; `POST /backup/offbox/toggle` enabled `calibre-web` (HTTP 302, and the „Nincs távoli mentésre jelölt alkalmazás" warning disappeared, so the selection HAD landed); the next run finished in 30 s and reported **„✓ Rendben · 12.0 MB · 1 pillanatkép"** — still one snapshot. The app restore then refused: **„offbox: nincs pillanatkép a(z) calibre-web alkalmazáshoz"**. A THIRD run took the snapshot count to 2 and the same restore then succeeded. So a run that the customer sees as a green success did not carry the app they had just enabled, and **nothing in the card distinguishes that from a run that did**. The customer's belief ("my app is off-site") and the truth diverge silently, and they would discover it only at restore — the worst possible moment. The snapshot count is on the same card, which is what makes the omission detectable in hindsight and invisible in the moment. ****ROOT-CAUSED 2026-08-06, and it was NOT the hypothesis in this row.** Two things were measured on demo-hp rather than assumed. (1) **The filed mechanism cannot happen for a deployed app:** the off-site run's own pre-dump phase calls `captureAllRecoveryUnits` for every DEPLOYED stack, through `admitApp`, before the push — a recovery unit moved aside was RECREATED by the run, which then reported `ok`. So "the toggle leaves the app without a bundle and the first run skips it" is not a state that survives a run. (2) **The actual cause was the single-flight:** the manual run was dropped because an earlier run was still going, `runOffboxBackup` returned **nil**, the handler had already answered „A távoli mentés elindult”, and the card then showed the PREVIOUS run's „✓ Rendben” — read as covering the app just selected. **Fixed in controller v0.205.0** on both fronts: the single-flight decision is taken SYNCHRONOUSLY in the handler and a dropped request says so („Már fut egy távoli mentés — ez a kérés nem indított újat. A most látható eredmény még a korábbi futásé…”, proven live on demo-hp), and — independently, because it is a real gap in its own right — **a run that skipped an app the customer selected is now `incomplete`**, not `ok`. Which skips count: selected+deployed with no unit YES; selected but NOT deployed no (named, with what to do — a box left amber by an app somebody removed is a status nobody reads); disconnected/decommissioned drive no (own signal); nothing selected no. The nightly path still returns nil on purpose. §7.3 measured: a recovery unit is a few KB of compose+manifest, only ENUMERATES dumps, does not stop the app, and already runs inside the off-site run — so there was no wait to remove and nothing was built. **This is the exact class the project already has a rule for** — "presence is not success": the run's timestamp and tick record that a run HAPPENED, not that it carried what the customer asked for. | **CLOSED 2026-08-06** — controller v0.205.0 |
 | **R-235** | **The appliance console keeps telling an already-paired box to go and pair itself.** Measured 2026-08-06 on VM 323: **25 minutes after** the operator bind, with the guest provisioned, the controller reporting 0.203.0 and the agent ONLINE, the physical console still displayed „Felhom — a doboz készen áll, és a **párosításra vár**" together with the now-spent pairing code `US3-6GP` — and, in the same panel, the promise **„Ez a képernyő magától frissül — nincs teendő a doboznál"**. It does not refresh. A customer looking at their screen is told the setup has not happened, and is told the screen would have updated if it had. Cosmetic in mechanism, not in effect: it invites the customer to re-pair a working box, or to call for help about a box that is already fine. Same family as R-234 — a surface asserting a state that stopped being true. | **READY** — owner Viktor |
 
 | **R-236** | ~~**After a guest rebuild the hub never re-stages the off-site credential.**~~ **WITHDRAWN 2026-08-06 — THIS WAS WRONG, and the mechanism works.** Measured 2026-08-06 on the Part 4 venue. The unlock screen promises „A gép még várja a házon kívüli tárhely kapcsolódási adatait — amint megvannak, a mentéseid listája megjelenik… **Nincs teendőd**". What actually happens: the rebuilt box declares `needs_credential`, R-218's retry job ticks every 5 min exactly as designed and logs the honest reason — `[offsite-apply] credential retry: consume one-time password: no unconsumed offsite password (already consumed or none provisioned) (the box still declares a need; retrying)` — **forever**, because the one-time password was consumed by the guest that no longer exists and nothing mints a new one. The fix is a single operator action that already exists (`/configs/<id>/offsite-reissue`), and pressing it resolved the stall within one tick. **R-218's consume half is not at fault — it is the half that works**; the gap is upstream, in who re-stages after a rebuild. **What the hub log actually shows** (`kubectl logs`, 2026-08-06): `offsite-delivery` refused to self-heal once a minute, correctly, naming `internal/offsiteheal` as the owner of the remediation and warning that a second mechanism minting there would double-issue; then at **15:24:57 local (13:24:57Z)** — `offsiteheal: re-staged the stored one-time offsite secret for customer part4 (declared needs_credential across 2 reports) — no provider credential was minted`. **The reconciler fired on schedule and succeeded.** The operator Re-issue at **13:26:39Z** came **102 seconds LATER**, was redundant, and **minted an unnecessary provider credential** (`re-issued shared credentials for part4 (subaccount 284735)`) — exactly the double-issue the guard exists to prevent. **The error was mine: I did not wait for the documented two-report debounce** (~16 minutes after a rebuild, since reports are ~15 min apart and the reconciler ticks every 5), and attributed the recovery to my own button press. „Nincs teendőd" is TRUE, and the wait is far inside the „egy napon belül" the card promises. **Operational lesson, not a product defect:** after a rebuild, wait for the debounce — the hub says once a minute, in its own log, that it owns this remediation. | **CLOSED 2026-08-06 — not a defect** |