R-203 shipped: the app and its backup agree, and 'ok' means it — R-201 unblocked

STATUS.md

@@ -46,16 +46,15 @@ Proven end to end on real hardware.
   recovered key back*, reopens the old backup store with it, or **restores a single file**. Today
   proves the key survives and returns; it does not prove the backups do. *(R-199 closed; R-200 half;
   R-201 open)*
-- **A backup reported success while leaving out a folder the customer was told is protected.** Found
+- *(fixed 4 Aug)* ~~A backup reported success while leaving out a folder the customer was told is
-  this evening on the HP machine, while setting up the wipe-and-restore proof. We deployed an app whose
+  protected.~~ **Both halves fixed the same day.** The app was writing to one folder and the backup was
-  book folder is declared *mandatory* — the strongest protection class — put a marked file in it, and
+  looking in another — one directory apart, on machines whose apps live on the system disk. They now
-  ran an off-site backup. **The backup said OK. Three snapshots. The folder was not in any of them.**
+  resolve to the same place, from a single piece of code instead of the three near-copies that had
-  The machine knew: one warning line inside the container says the folder was skipped. Nothing else
+  quietly drifted. **And a backup that cannot capture a folder marked essential no longer reports
-  does — not the card, not the counters, not the hub, not you. This is the same shape as everything
+  success**: it reports *Hiányos* (incomplete), names the app and the folders, and tells you — while
-  else we have been fixing this month: *a path the customer thinks is protected is not in the copy.*
+  still recording what it genuinely did capture, because half a backup is not no backup. Proved on the
-  It is the reason the proof stopped before the wipe — wiping would have destroyed the marked file and
+  HP machine by listing the backup's own contents and finding the marked file there by name and size —
-  proven nothing. **The two apps that were already backing up off-site are unaffected** — they declare
+  not by trusting a green tick. *(R-203)*
-  no such folders. *(R-203)*
 - **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card
   says the old backups may later be restorable with the matching recovery code. From today that is true
   for machines that re-seal from now on and **false for anything already orphaned** — and the machine
@@ -104,9 +103,10 @@ Proven end to end on real hardware.
 
 ## What we're working on
 
-- **Now:** fixing the folder-left-out-of-the-backup problem above. The wipe-and-restore proof is
+- **Now:** the wipe-and-restore proof, which is **unblocked and staged**. The marked file now lands in
-  **staged and waiting on it** — the machine, the code, the working off-site store, the app and the
+  the off-site backup, so there is finally something to recover. Everything else is already in place on
-  marked file are all in place; only the missing folder blocks it. Nothing was wiped. Both honesty fixes shipped today: a changed backup key now
+  the HP machine — the recovery code you saved, a working off-site store, the app and the file. Nothing
+  has been wiped; that step waits for your go-ahead at a marked stop point. Both honesty fixes shipped today: a changed backup key now
   raises an alarm on the day, and the email that stated the opposite of what it measured now describes
   what it actually saw.
 - **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to
@@ -158,6 +158,10 @@ Proven end to end on real hardware.
 
 ## Changed since last update
 
+- **2026-08-04 (latest)** — **Fixed the folder-left-out-of-the-backup problem, both halves.** The app
+  and its backup now look in the same directory, and a backup that misses a folder marked essential
+  reports *incomplete* instead of success. Proved by listing the backup's own contents and finding the
+  marked file. The wipe-and-restore proof is unblocked. *(R-203, R-201)*
 - **2026-08-04 (late)** — Set up the wipe-and-restore proof on the HP machine and **stopped before the
   wipe**: a folder marked as protected was missing from the off-site backup while the backup reported
   success. Three things were proved on the way, all firsts: a rebuilt machine's off-site backup now

documentation/audits/DRILL-r201-offsite-recovery-2026-08-04.md

@@ -270,6 +270,36 @@ confirm the key exists in the credentials file; no unseal was performed on demo-
 
 ---
 
+## 7b. UPDATE, same day — the blocker is FIXED and the drill is ready to resume
+
+Controller **v0.197.0** shipped both halves of R-203:
+
+- **the paths agree.** On demo-hp the live bind moved from `/mnt/sys_drive/userdata/media/books` to
+  `/mnt/sys_drive/felhom-data/userdata/media/books` — the directory the capture set looks in. The
+  capture log went from `0 mandatory path(s)` to **`1 mandatory path(s)`**.
+- **`ok` means it.** A run that cannot capture a MANDATORY directory now reports **`incomplete`**,
+  names the app and the folders, and raises the operator digest — instead of `ok` with a warning
+  beside it.
+
+**And the sentinel is in the snapshot, listed by name:**
+
+```
+$ restic ls -l latest --tag calibre-web
+-rw-r--r--  1000  1000  181  2026-08-04 12:53:06  /mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt
+```
+
+sha256 **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`** — byte-identical to §1,
+verified after the fix's migration moved the file to the corrected directory.
+
+**One live check could NOT be reproduced, and is recorded rather than claimed.** Hiding the books
+directory to watch the `incomplete` verdict fire on hardware did not work: the running container's
+bind mount **recreated** the directory, so `os.Stat` succeeded and there was no gap. That is itself
+worth knowing — a bind-mounted directory cannot easily be "missing" while its app runs, so the
+mandatory-gap condition arises in practice when the path resolves somewhere the app never binds (the
+R-203 case), not when a live app's own directory vanishes. The verdict is proven by the run-level
+test, which drives the real `RunOffboxBackup` and asserts both `incomplete` and the operator signal.
+The fixture was restored and the sentinel re-verified at the same hash.
+
 ## 8. To resume the drill
 
 1. **Fix or scope R-203.** Until the mandatory userdata path lands in the snapshot, no sentinel can

documentation/backlog/ROADMAP.md

@@ -65,6 +65,7 @@
 | R-200 | **The password-injection seam had a handler and no form** | M | **plumbing SHIPPED (controller v0.195.0); the form is NOT built** | `--recover-offsite-check` compares by hash and refuses to install. The customer-facing shape (yell → recovery-code form → preview → proceed) is priced against a chain that now exists rather than one that was assumed |
 | R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN |
 | R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 |
+| R-203 | **The app and its backup looked in different directories, and a run that skipped a mandatory folder still said `ok`** | M | **SHIPPED + PROVEN-LIVE (controller v0.197.0, 2026-08-04)** | Adds the capability-map row *"off-site app-data capture covers MANDATORY paths on both drive layouts"* as PROVEN-LIVE, and corrects that row's predecessor, which was optimistic. Unblocks R-201 |
 | R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
 | R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
 | R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |