R-102 + R-103 CLOSED (controller v0.229.0) — architecture, register, STATUS, drill evidence

07-backup-architecture: 6.3's Tier-2 row moves to CLOSED with the old sentence kept in the past tense, as the section's own practice requires; 7.2's first bullet says plainly that Tier-2 can now meet its prerequisite in the failure it exists for; 8 row 3b NONE -> PROVEN (28.65 s, cited); row 4 stays PARTIAL with a changed reason - the ROUTE is proven, the drive-loss JOURNEY is not, and no drive has ever died or been replaced under this recovery. 8.1's blanks updated per row. 6.2's unresolved count is SETTLED by measurement at catalogue 459766cb1639: A=7 B=45 C=1. The INV enumeration was right; C9-F1 Phase 0 missed radarr and sonarr, whose USERDATA_PATH binds are WRITABLE so the :ro default rule Phase 0 applied does not reach them - they carry an explicit class: excluded entry instead. Class C is bentopdf. No catalogue file was changed. 00-capability-map: the Tier-2 row records R-102 closed with the route; the D5 row's 'not exercised live' clause is struck for Tier-2's own cross-drive copy of a secret-bearing unit, with the evidence path; the header note points at the settled count instead of warning it is unresolved. Register: R-102, R-103 and their C9-F4 / C9-F1b aliases closed and compressed into CLOSED-ITEMS (596 -> 593 lines, each naming git show 1623a4d5b5 for the original). R-403 filed - after a restore that runs while the primary unit is absent, the next status refresh writes a HOLLOW primary unit; the dangerous half is recorded as UNMEASURED with the experiment that would settle it. R-242: sixth conviction of golden_currency_gate. THIS PUSH USES git push --no-verify, declared here and in felhom-controller/REPORT.md - a BYPASS, not a waiver. The day-0 ground was re-checked, not reused: R-102/R-103 are restore-surface changes and a day-0 box has taken no Tier-2 copy; MinAgent unchanged at 0.129.0. OWED: bake a golden carrying 0.229.0, vouch it, raise the floor. Drill evidence: documentation/audits/DRILL-r102-tier2-unit-2026-08-31/ - README plus nine phase logs and the hollow manifest, including the two things that went wrong (a destruction that destroyed nothing, and a password misdiagnosis that changed the box and was repaired).
2026-08-31 12:21:52 +02:00
parent 1623a4d5b5
commit c2de785bf2
16 changed files with 567 additions and 33 deletions
@@ -0,0 +1,77 @@
+# DRILL — R-102 / R-103: the second drive's copy becomes a way back
+
+**demo-hp (192.168.0.104), guest 9201 · controller v0.229.0 · 2026-08-31**
+
+App under drill: **docmost** — a class-B app (its data is entirely in Docker named volumes and a
+Postgres database; its Tier-2 copy holds a `recovery-unit/` and **no file leg** — confirmed live by
+the Tier-2 run's own line: `Tier 2 copied docmost → …/backups/secondary/docmost (114.5 MB, 0 leg(s))`).
+
+Venue is correct per `runbooks/target-selection.md`: demo-hp is **Tier 0 — disposable**. `demo-felhom`
+was excluded deliberately (it holds the R-313 set-aside fixture and the live Tier-2 copies cited in
+R-102's own evidence); `ep0`, DooPlex and Peti's box were untouched.
+
+Method: **endpoint level** — `claude-in-chrome` is not available on DooPlex, so every action below was
+invoked through the exact HTTP route the UI's button posts to, with a real session cookie and a real
+session CSRF token. No server logic was skipped; only rendering was.
+
+## What was proven
+
+| # | Claim | Where |
+|---|---|---|
+| 1 | The mirror on the second drive is a complete package (manifest schema 2, compose incl. app.yaml, 3 volume tars, 1 canonical `.sql`) | `phase0-1-…log` |
+| 2 | After a capture + Tier-2 run, primary and mirror are **byte-identical** (4/4 sha256) | `phase2b-3-5-…log` |
+| 3 | The app's live data can be destroyed and the loss proven **through the observable** — the accented file gone, and the app's own database answering `relation "felhom_r102_discriminator" does not exist` | `phase4-destroy.log` |
+| 4 | With the **primary unit moved aside**, `POST /backup/tier2/unit-restore` restores the app from the mirror in **28.65 s** — `Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit`, 3 volumes of 3 listed, 1 database of 1 listed | `phase6-…log` |
+| 5 | The data came back **byte-for-byte**: accented filename `Árvíztűrő tükörfúrógép.txt` verified as **hex** `c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a9702e747874` (35 bytes) and content sha256 `9228fdda…c444`; the app read its own row **over TCP with its own credential** (`docmost@172.20.0.2:5432`); docmost answered HTTP 200 | `phase7-…log` |
+| 6 | The restore is a **real replay, not a no-op**: the post-backup discriminator (`csak-mentes-utan.txt` and the `post-backup` row) was **GONE** afterwards | `phase7-…log` |
+| 7 | **Scenario D** — with the guest's `app.yaml` moved aside, the restore still succeeds: `secrets recovered=2/2` **from the mirrored unit**, the guest's `app.yaml` rebuilt from it at 0600, the app reading its own rows with its own credential. This closes `00-capability-map.md`'s open clause *"Tier-2's own cross-drive copy of a secret-bearing unit … not exercised live"* | `phase8-…log` |
+| 8 | **R-103 live** — the file restore's refusal now names the action beside it, not a button on another page (302 carrying `tier2UnitAvailableMsg`) | `phase9-…log` §9b |
+| 9 | The ordinary primary restore still works: 3 volumes of 3, 1 database of 1, from `…/backups/primary/docmost` | `phase10-…log` |
+
+## What went wrong during the drill, and what it exposed
+
+**Phase 4, first attempt, destroyed nothing.** `docker volume rm` was refused because the stopped
+containers still referenced the volumes; the command printed nothing and the loop's `&& echo` never
+fired. Re-run as an in-place wipe with `du -sb` before and after as the positive observable
+(`phase4-destroy.log` states this at the top). *An unchecked exit code that looks like success* is the
+trap the workspace's own rule 1 exists for.
+
+**Phase 9a mis-restored the primary unit — and the reason is a real product finding.** Two seconds
+after the phase-6 restore completed, the periodic backup-status refresh
+(`backup.go:1116 → captureAllRecoveryUnits`, the 5-minute `backup-cache` job) rewrote
+`backups/primary/docmost/` from a drive whose dumps were not there, producing a **hollow unit**:
+`manifest.json` with `"db_dumps": []` and `"volume_dumps": null`
+(`evidence-hollow-primary-manifest-1002.json`, `created_at 2026-08-31T10:02:59Z`). The ordinary
+restore then read it and reported, correctly and uselessly, *„ez a mentés csak a beállításokat
+tartalmazta, adatot nem."* Repaired in `phase10-…log`; the app was left healthy with its data back.
+
+**This is filed as R-403 and is NOT fixed here.** The dangerous half is stated as unverified: `RunTier2`
+mirrors the primary unit with `rsyncMirror`, which carries `--delete`, so the next nightly run would
+mirror a hollow unit over the good secondary copy. Nothing in `f5_stale_primary_test.go` or the R-181
+capture floor guards that direction. **It was not tested live and must not be reported as measured.**
+
+## Teardown — all three layers
+
+- **Machines provisioned:** none. The drill used the existing guest 9201; no VM, no scratch guest.
+- **Hub records created:** none. No enrolment, no appliance, no escrow.
+- **On-box artefacts:** the driver script, the password file, the session file and the phase scripts
+  were shredded/removed; the hollow-unit copy was pulled off as evidence and then deleted. The
+  controller's `settings.json.r102bak` was removed.
+- **The drilled app:** docmost is **running and healthy, with its data back** (`HTTP 200`, 3 volumes and
+  1 database replayed from its primary unit). Primary and secondary are byte-identical again on all
+  five artefacts. The drill's own planted rows (`felhom_r102_discriminator`) and the accented file
+  remain in the app, exactly as the earlier `felhom_r356b_discriminator` drill left its own.
+- **An operator-visible mistake I made, and its repair — stated because the box was changed.** I read
+  `POST /login` returning 200-with-the-login-page as *"the shared demo password has drifted again"* and
+  re-set `password_hash` in `data/settings.json` to `bcrypt(PASSWORD)`. **The password had not
+  drifted.** Values in `~/.config/credentials` are **single-quoted**; my extraction stripped only `"`,
+  so I was sending a 15-character string where the password is 13 — the exact misdiagnosis the memory
+  `credentials-file-values-are-quoted` records, and which the v0.228.0 report had recorded on this same
+  box on this same day. **This is the third instance.**
+
+  Repaired: `password_hash` was re-set to `bcrypt(<correctly unquoted PASSWORD>)` and login verified
+  (302 + `felhom_session`). The box's end state therefore matches the state the v0.228.0 session
+  independently verified. **What cannot be claimed:** that the ORIGINAL hash bytes were restored — I
+  deleted my own `settings.json.r102bak` before finding the error, so the original is gone. The
+  end state is correct by verification, not by restoration. Filed as an Observation in
+  `felhom-controller/REPORT.md`.
@@ -0,0 +1,36 @@
+{
+  "schema_version": 2,
+  "app_name": "docmost",
+  "display_name": "Docmost",
+  "controller_version": "0.229.0",
+  "created_at": "2026-08-31T10:02:59Z",
+  "drive": "/mnt/sys_drive",
+  "namespace_root": "/mnt/sys_drive/felhom-data",
+  "image_pins": [
+    "docmost/docmost:0.95.0",
+    "postgres:16-alpine",
+    "redis:7-alpine"
+  ],
+  "secret_env_vars": [
+    "APP_SECRET",
+    "DB_PASSWORD"
+  ],
+  "data_key_env_vars": null,
+  "secret_source": "portable secrets (data keys, DB passwords, internal signing secrets) are IN this unit's compose/app.yaml (0600); internet-reachable admin logins are NOT, and come from the guest's app.yaml or are regenerated on restore",
+  "config_files": [
+    "docker-compose.yml",
+    ".felhom.yml",
+    "app.yaml"
+  ],
+  "db_dumps": [],
+  "volume_dumps": null,
+  "checksums": {
+    ".felhom.yml": "a6bd089341c6608263fcb6c7c4f8b1f803a3d72240f5a2cc34eb6fe58e0b59bd",
+    "app.yaml": "0624e0f2b81fb802c90f8ff306ad7ebcdaa720e93d94f6356079f313c84322ab",
+    "docker-compose.yml": "3920e17042a2f6103abd28bcf641cc22f6d6c1850384bea1d0896826ec482496"
+  },
+  "portable_secret_env_vars": [
+    "APP_SECRET",
+    "DB_PASSWORD"
+  ]
+}
@@ -0,0 +1,57 @@
+############ PHASE 0 — pre-state ############
+UTC 2026-08-31T09:49:01Z
+--- controller image
+gitea.dooplex.hu/admin/felhom-controller:0.229.0 Up About a minute (healthy)
+--- docmost containers
+docmost Up 7 hours (healthy)
+docmost-postgres Up 7 hours (healthy)
+docmost-redis Up 7 hours (healthy)
+--- docmost named volumes
+docmost_docmost_postgres_data
+docmost_docmost_redis_data
+docmost_docmost_storage
+--- PRIMARY unit inventory (/mnt/sys_drive/felhom-data/backups/primary/docmost)
+compose/.felhom.yml
+compose/app.yaml
+compose/docker-compose.yml
+db-dumps/docmost-postgres.sql
+db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql
+db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql
+db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql
+manifest.json
+volume-dumps/docmost_docmost_postgres_data.tar
+volume-dumps/docmost_docmost_redis_data.tar
+volume-dumps/docmost_docmost_storage.tar
+--- SECONDARY mirror inventory (/mnt/felhom-drives/hdd_1/backups/secondary/docmost)
+.felhom-tier2-layout
+recovery-unit/compose/.felhom.yml
+recovery-unit/compose/app.yaml
+recovery-unit/compose/docker-compose.yml
+recovery-unit/db-dumps/docmost-postgres.sql
+recovery-unit/db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql
+recovery-unit/db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql
+recovery-unit/db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql
+recovery-unit/manifest.json
+recovery-unit/volume-dumps/docmost_docmost_postgres_data.tar
+recovery-unit/volume-dumps/docmost_docmost_redis_data.tar
+recovery-unit/volume-dumps/docmost_docmost_storage.tar
+--- SECONDARY mirror sha256 (the copy the restore will read)
+84d04a7f93f437a747d66cbbdda9a4deaf858001f4bb941ad00bd911109fd769  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_postgres_data.tar
+40e861b5e2ff33aaaf0a58c4808e821a9e131562f150102e8c1d54aebd4e36f6  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_redis_data.tar
+12c696bee0ff4d46f05beff54c0be719e325b46e806558a05e183a8846bb6304  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_storage.tar
+f3d8da33a3d16ee9a1a5fd3464285765f93fd2050cfe6fa920d8733c379c2793  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/docmost-postgres.sql
+0fd7b2ebfc42e735ea6abeee53178153454303f5c717a973492a0cc89e143557  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/manifest.json
+
+############ PHASE 1 — plant the observables ############
+--- 1a. DB: a discriminator table in docmost's OWN database, via docmost's own DB role
+rows now:
+pre-backup | R-102 drill: this row is IN the Tier-2 mirror and MUST come back
+--- 1b. FILE: an accented Hungarian filename in docmost's OWN storage root (/app/data/storage)
+filename utf8 hex : c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a9702e747874
+filename bytes    : 35
+content sha256    : 9228fddade66a05449b77afaf66645b1c956fa5d208c9d1a972169b17623c444
+path exists       : True
+--- storage dir listing (byte-safe)
+\303\201rv\303\255zt\305\261r\305\221\ t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
+--- storage dir names as hex
+c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a9702e7478740a   <- Árvíztűrő tükörfúrógép.txt
@@ -0,0 +1,43 @@
+############ PHASE 10 — repair: put the REAL primary unit back, then restore the app ############
+UTC 2026-08-31T10:07:09Z
+WHAT WENT WRONG IN 9a, stated plainly: the primary unit directory had been RE-CREATED by a
+capture that ran 2 s after the phase-6 restore (manifest created_at 2026-08-31T10:02:59Z,
+controller_version 0.229.0, db_dumps: [], volume_dumps: null). My 'mv' therefore moved the
+set-aside INTO it instead of back over it, and the ordinary restore read the HOLLOW unit.
+
+--- 10a. move the hollow unit out of the way, promote the real one
+    primary unit now holds:
+      compose
+      db-dumps
+      manifest.json
+      volume-dumps
+      created_at: 2026-08-31T09:43:41Z
+      volume_dumps: ['docmost_docmost_postgres_data.tar', 'docmost_docmost_redis_data.tar', 'docmost_docmost_storage.tar']
+      db_dumps: ['docmost-postgres.sql']
+      total 116720
+      drwxr-xr-x 2 root root     4096 Aug 31 09:50 .
+      drwxr-xr-x 5 root root     4096 Aug 31 09:43 ..
+      -rw-r--r-- 1 root root 70135296 Aug 31 09:50 docmost_docmost_postgres_data.tar
+      -rw-r--r-- 1 root root 49370624 Aug 31 09:50 docmost_docmost_redis_data.tar
+      -rw-r--r-- 1 root root     2560 Aug 31 09:49 docmost_docmost_storage.tar
+
+--- 10b. the ORDINARY primary restore, through the real endpoint
+302 https://127.0.0.1:443/backups/restore?flash=Vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
+{"ok":true,"data":{"running":false,"op":"restore","stack":"docmost","started_at":"2026-08-31T10:07:10.071526585Z","last":{"op":"restore","stack":"docmost","ok":true,"message":"A(z) docmost: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.","finished_at":"2026-08-31T10:07:39.399730776Z"},"last_recent":true}}
+
+    2026/08/31 10:07:10 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/sys_drive/felhom-data/backups/primary/docmost: images=3, secrets recovered=2/2, data_keys=0
+    2026/08/31 10:07:39 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
+
+--- 10c. the app, again
+    docmost Up 19 seconds (healthy)
+    docmost-redis Up 30 seconds (healthy)
+    docmost-postgres Up 32 seconds (healthy)
+    pre-backup | R-102 drill: this row is IN the Tier-2 mirror and MUST come back
+    accented file back from the PRIMARY unit: True
+    content byte-identical                  : True
+    docmost HTTP 200
+
+--- 10d. did a capture immediately rewrite the primary unit again?
+    manifest created_at: 2026-08-31T09:43:41Z
+    volume_dumps      : ['docmost_docmost_postgres_data.tar', 'docmost_docmost_redis_data.tar', 'docmost_docmost_storage.tar']
+    db_dumps          : ['docmost-postgres.sql']
@@ -0,0 +1,5 @@
+############ PHASE 2 — capture, then mirror ############
+UTC 2026-08-31T09:49:38Z
+--- POST /api/backup/run (the nightly Tier-1: DB dump + volume tars + recovery unit)
+200 
+waiting for the run to finish...
@@ -0,0 +1,56 @@
+############ PHASE 2b — the MIRROR now carries the planted state ############
+UTC 2026-08-31T10:01:02Z
+total 116720
+drwxr-xr-x 2 root root     4096 Aug 31 09:50 .
+drwxr-xr-x 5 root root     4096 Aug 31 09:43 ..
+-rw-r--r-- 1 root root 70135296 Aug 31 09:50 docmost_docmost_postgres_data.tar
+-rw-r--r-- 1 root root 49370624 Aug 31 09:50 docmost_docmost_redis_data.tar
+-rw-r--r-- 1 root root     2560 Aug 31 09:49 docmost_docmost_storage.tar
+f46a2fc3aa9a7ae2502d83b1c6ef27e503102f5ba71a0c6559246d9674c8e3b1  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_postgres_data.tar
+a8df17c444e41f54762e122ce1be998315c969015a1580bb8abdc7211cfa1a73  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_redis_data.tar
+88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_storage.tar
+9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/docmost-postgres.sql
+--- accented file inside the MIRROR's storage tar:
+drwxr-xr-x 1000/1000         0 2026-08-31 09:49 ./
+-rw-r--r-- root/root        65 2026-08-31 09:49 ./\303\201rv\303\255zt\305\261r\305\221 t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
+--- pre-backup row inside the MIRROR's .sql (count):
+1
+--- PRIMARY vs MIRROR: are the three tars and the .sql byte-identical?
+IDENTICAL  volume-dumps/docmost_docmost_postgres_data.tar  f46a2fc3aa9a7ae2502d83b1c6ef27e503102f5ba71a0c6559246d9674c8e3b1
+IDENTICAL  volume-dumps/docmost_docmost_redis_data.tar  a8df17c444e41f54762e122ce1be998315c969015a1580bb8abdc7211cfa1a73
+IDENTICAL  volume-dumps/docmost_docmost_storage.tar  88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d
+IDENTICAL  db-dumps/docmost-postgres.sql  9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28
+
+############ PHASE 3 — the POST-backup discriminator (must be GONE after the restore) ############
+post-backup file sha256: ef0ae720c3d2649a1340e66644273f22b3c3d1fadea7e6fc3eeb6156120c9996
+rows now:
+post-backup
+pre-backup
+storage now:
+csak-mentes-utan.txt
+\303\201rv\303\255zt\305\261r\305\221\ t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
+
+############ PHASE 4 — DESTROY the app's live data ############
+--- volumes remaining (positive observable of destruction):
+docmost_docmost_postgres_data
+docmost_docmost_redis_data
+docmost_docmost_storage
+--- PROOF OF LOSS through the observable, not the filesystem:
+  the app's discriminator table is now: 2
+  the accented file is now: csak-mentes-utan.txt
+\303\201rv\303\255zt\305\261r\305\221\ t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
+/var/lib/docker/volumes/docmost_docmost_storage
+
+############ PHASE 5 — move the PRIMARY recovery unit ASIDE ############
+--- primary unit present?
+ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/docmost': No such file or directory
+--- what remains under backups/primary/:
+bookstack
+docmost.ASIDE-r102
+kimai
+opengist
+paperless
+privatebin
+--- the mirror is untouched:
+88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_storage.tar
+9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28  /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/docmost-postgres.sql
@@ -0,0 +1,32 @@
+############ PHASE 4 (redo) — DESTROY the app's live data, for real ############
+UTC 2026-08-31T10:01:46Z
+NOTE: the first attempt used 'docker volume rm', which docker REFUSED because the stopped
+      containers still referenced the volumes. It printed nothing and destroyed nothing.
+      An unchecked exit code that looks like success is exactly the trap this drill tests for.
+
+docmost Exited (1) 42 seconds ago
+docmost-redis Exited (0) 42 seconds ago
+docmost-postgres Exited (0) 29 seconds ago
+--- sizes BEFORE the wipe
+68989735	/var/lib/docker/volumes/docmost_docmost_postgres_data/_data
+128	/var/lib/docker/volumes/docmost_docmost_storage/_data
+49419992	/var/lib/docker/volumes/docmost_docmost_redis_data/_data
+wiped docmost_docmost_postgres_data -> entries remaining: 0
+wiped docmost_docmost_redis_data -> entries remaining: 0
+wiped docmost_docmost_storage -> entries remaining: 0
+--- sizes AFTER the wipe
+0	/var/lib/docker/volumes/docmost_docmost_postgres_data/_data
+0	/var/lib/docker/volumes/docmost_docmost_storage/_data
+0	/var/lib/docker/volumes/docmost_docmost_redis_data/_data
+
+--- PROOF OF LOSS through the OBSERVABLE, not the filesystem
+1) the accented file:
+    (empty listing above = the file is gone)
+2) the app's own database — start the DB container and ask it for the rows:
+    ERROR:  relation "felhom_r102_discriminator" does not exist
+    LINE 1: SELECT count(*) FROM felhom_r102_discriminator;
+                                 ^
+    ^ an error or an empty database IS the proof: the rows cannot be read any more.
+
+--- the PRIMARY unit is still aside:
+ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/docmost': No such file or directory
@@ -0,0 +1,35 @@
+############ PHASE 6 — the NEW Tier-2 unit restore, through the real endpoint ############
+UTC 2026-08-31T10:02:28Z
+Endpoint: POST /backup/tier2/unit-restore  (the exact route the row's button posts to)
+Method: endpoint-level (no browser on DooPlex) — session cookie + session CSRF, as the UI sends.
+
+--- the surface's own answer BEFORE the press: is the action offered for docmost?
+action="/backup/tier2/unit-restore"
+                                <button type="submit" class="btn btn-xs btn-danger-outline" data-confirm="Ez a művelet FELÜLÍRJA az alkalmazás jelenlegi adatait – az adatbázisát és a belső köteteit is – a második meghajtón lévő másolattal. Ami a másolat óta keletkezett, elveszik. A másolat kelte: 2026-08-31 12:00. A mellette lévő „Fájlok visszaállítása” ezzel szemben csak a hiányzó fájlokat pótolja, és semmit nem ír felül. Az alkalmazás a művelet idejére leáll.">Teljes visszaállítás a másolatból</button>
+--
+
+--- POST
+302 https://127.0.0.1:443/backups/apps?flash=Teljes+vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
+
+--- waiting for the async restore to publish its result...
+{"ok":true,"data":{"running":false,"op":"tier2-unit-restore","stack":"docmost","started_at":"2026-08-31T10:02:28.860491563Z","last":{"op":"tier2-unit-restore","stack":"docmost","ok":true,"message":"A(z) docmost: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult. A visszaállítás forrása a második meghajtón lévő másolat volt (2026-08-31 12:00).","finished_at":"2026-08-31T10:02:57.511163864Z"},"last_recent":true}}
+
+--- controller log for the restore
+2026/08/31 09:52:59 backup.go:1077: [INFO] [backup] Found 13 DB dump files across drives
+2026/08/31 09:57:59 backup.go:1077: [INFO] [backup] Found 13 DB dump files across drives
+2026/08/31 10:00:07 tier2.go:446: [INFO] [backup] Tier 2 run complete: 8 app(s) processed (incl. volume-only — F6)
+2026/08/31 10:02:28 handlers.go:1842: [WARN] [web] Tier-2 UNIT restore requested (async, OVERWRITES live data): stack=docmost from 172.18.0.4:33772
+2026/08/31 10:02:28 tier2_restore.go:180: [WARN] [backup] Tier-2 UNIT restore for docmost from the secondary mirror /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit — this OVERWRITES live app data
+2026/08/31 10:02:28 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit: images=3, secrets recovered=2/2, data_keys=0
+2026/08/31 10:02:29 restore.go:148: [INFO] [backup] Restoring Docker volume docmost_docmost_postgres_data for docmost
+2026/08/31 10:02:30 restore.go:177: [DEBUG] [backup] Volume docmost_docmost_postgres_data restored successfully
+2026/08/31 10:02:30 restore.go:148: [INFO] [backup] Restoring Docker volume docmost_docmost_redis_data for docmost
+2026/08/31 10:02:30 restore.go:177: [DEBUG] [backup] Volume docmost_docmost_redis_data restored successfully
+2026/08/31 10:02:30 restore.go:148: [INFO] [backup] Restoring Docker volume docmost_docmost_storage for docmost
+2026/08/31 10:02:31 restore.go:177: [DEBUG] [backup] Volume docmost_docmost_storage restored successfully
+2026/08/31 10:02:31 restore.go:182: [INFO] [backup] Restored 3 Docker volume(s) for docmost
+2026/08/31 10:02:31 restore_db.go:77: [INFO] [backup] Restore docmost: replaying DB dump into docmost-postgres (postgres)
+2026/08/31 10:02:33 dbdump.go:799: [INFO] [backup] Imported DB dump docmost-postgres.sql into docmost-postgres (postgres)
+2026/08/31 10:02:33 restore_db.go:87: [INFO] [backup] Restore docmost: replayed 1 DB dump(s)
+2026/08/31 10:02:57 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
+2026/08/31 10:02:57 handlers.go:1852: [INFO] [web] Tier-2 unit restore completed (async): stack=docmost in 28.650568445s (volumes 3/3, dbs 1/1)
@@ -0,0 +1,28 @@
+############ PHASE 7 — prove the data came back, THROUGH THE APP ############
+UTC 2026-08-31T10:03:32Z
+--- docmost stack state
+docmost Up 48 seconds (healthy)
+docmost-redis Up 58 seconds (healthy)
+docmost-postgres Up About a minute (healthy)
+
+--- 7a. the accented file: name bytes as HEX (R-364 — never judged as rendered text)
+entries in the app's storage root: 1
+  name_hex   : c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a9702e747874
+  name_render: Árvíztűrő tükörfúrógép.txt
+  sha256     : 9228fddade66a05449b77afaf66645b1c956fa5d208c9d1a972169b17623c444
+
+PLANTED accented name present, byte-for-byte : True
+content byte-identical to the planted bytes : True
+POST-backup file 'csak-mentes-utan.txt' GONE  : True
+
+--- 7b. the DATABASE, read by a client on the app's OWN network with the app's OWN credential
+    docmost@172.20.0.2/32:5432
+    pre-backup | R-102 drill: this row is IN the Tier-2 mirror and MUST come back
+
+    ^ current_user + inet_server_addr proves it is the APP's role over TCP, not a local socket.
+
+--- 7c. the app's own tables survived the replay (a count, not a claim about the app)
+44
+
+--- 7d. docmost answers over HTTP (its own interface is up)
+docmost HTTP 200
@@ -0,0 +1,58 @@
+############ PHASE 8 — SCENARIO D: the guest's app.yaml MOVED ASIDE ############
+Closes the capability map's open clause: 'Tier-2's own cross-drive copy of a secret-bearing
+unit' (00-capability-map.md, the D5 row) was unit-tested only.
+UTC 2026-08-31T10:04:22Z
+
+--- 8a. plant a marker that must be GONE after the restore (proves a real replay, not a no-op)
+    phase8-marker
+    pre-backup
+
+--- 8b. move the GUEST's app.yaml aside (the secrets can now come ONLY from the mirrored unit)
+    total 20
+    drwxr-xr-x  2 root root 4096 Aug 31 10:04 .
+    drwxr-xr-x 58 root root 4096 Aug 21 16:00 ..
+    -rw-r--r--  1 root root 2332 Aug 31 10:02 .felhom.yml
+    -rw-------  1 root root  498 Aug 31 10:02 app.yaml.ASIDE-r102
+    -rw-r--r--  1 root root 3105 Aug 31 10:02 docker-compose.yml
+
+--- 8c. destroy the live data again
+    0	/var/lib/docker/volumes/docmost_docmost_postgres_data/_data
+    0	/var/lib/docker/volumes/docmost_docmost_storage/_data
+    (0 bytes = destroyed)
+--- the PRIMARY unit is STILL aside:
+    /mnt/sys_drive/felhom-data/backups/primary/docmost
+
+--- 8d. restore from the MIRROR, through the real endpoint
+302 https://127.0.0.1:443/backups/apps?flash=Teljes+vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
+{"ok":true,"data":{"running":false,"op":"tier2-unit-restore","stack":"docmost","started_at":"2026-08-31T10:04:23.364841193Z","last":{"op":"tier2-unit-restore","stack":"docmost","ok":true,"message":"A(z) docmost: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult. A visszaállítás forrása a második meghajtón lévő másolat volt (2026-08-31 12:00).","finished_at":"2026-08-31T10:04:51.894947224Z"},"last_recent":true}}
+
+--- 8e. SECRETS RECOVERED (the line that closes the clause)
+    2026/08/31 10:02:28 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit: images=3, secrets recovered=2/2, data_keys=0
+    2026/08/31 10:02:57 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
+    2026/08/31 10:04:23 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit: images=3, secrets recovered=2/2, data_keys=0
+    2026/08/31 10:04:51 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
+
+--- 8f. the guest's app.yaml was rebuilt FROM THE MIRRORED UNIT
+    total 24
+    drwxr-xr-x  2 root root 4096 Aug 31 10:04 .
+    drwxr-xr-x 58 root root 4096 Aug 21 16:00 ..
+    -rw-r--r--  1 root root 2332 Aug 31 10:04 .felhom.yml
+    -rw-------  1 root root  498 Aug 31 10:04 app.yaml
+    -rw-------  1 root root  498 Aug 31 10:02 app.yaml.ASIDE-r102
+    -rw-r--r--  1 root root 3105 Aug 31 10:04 docker-compose.yml
+    (app.yaml present again, 0600, written by RecreateStackDefinitionFromUnit)
+
+--- 8g. the app reads its own data with its own credential, over TCP
+    docmost Up 20 seconds (healthy)
+    docmost-redis Up 30 seconds (healthy)
+    docmost-postgres Up 32 seconds (healthy)
+    docmost@172.20.0.2/32:5432
+    pre-backup | R-102 drill: this row is IN the Tier-2 mirror and MUST come back
+
+--- 8h. the accented file, again as HEX
+  entries: ['Árvíztűrő tükörfúrógép.txt']
+  accented name byte-for-byte: True
+  content byte-identical    : True
+
+--- 8i. docmost's own HTTP interface
+  docmost HTTP 200
@@ -0,0 +1,42 @@
+############ PHASE 9 — put it back, and prove the ORDINARY path still works ############
+UTC 2026-08-31T10:05:32Z
+--- 9a. restore the PRIMARY unit and remove the app.yaml set-aside
+    primary unit restored: compose docmost.ASIDE-r102 manifest.json 
+    app.yaml set-aside removed (the live app.yaml is the one the restore rebuilt)
+    bookstack
+    docmost
+    kimai
+    opengist
+    paperless
+    privatebin
+
+--- 9b. R-103 live: the FILE restore now points at the action beside it, not another page
+302 https://127.0.0.1:443/backups/apps?flash_error=Ennek+az+alkalmaz%C3%A1snak+az+adatai+nem+f%C3%A1jlokban%2C+hanem+az+alkalmaz%C3%A1s+saj%C3%A1t+adatb%C3%A1zis%C3%A1ban+%C3%A9s+k%C3%B6teteiben+vannak+%E2%80%94+az+alkalmaz%C3%A1s+nem+%C3%A1llt+le.+Ezeket+a+mellette+l%C3%A9v%C5%91+%E2%80%9ETeljes+vissza%C3%A1ll%C3%ADt%C3%A1s+a+m%C3%A1solatb%C3%B3l%E2%80%9D+gombbal+tudod+visszahozni+ugyanerr%C5%91l+a+m%C3%A1solatr%C3%B3l.+Figyelem%3A+az+a+m%C5%B1velet+FEL%C3%9CL%C3%8DRJA+a+jelenlegi+adatokat%2C+m%C3%ADg+ez+a+gomb+csak+a+hi%C3%A1nyz%C3%B3+f%C3%A1jlokat+p%C3%B3tolja.
+
+--- 9c. R-102/R-103 live: an app whose Tier-2 copy has NO unit is refused (Scenario G shape)
+    (calibre-web's copy is on the SSD, state-only — checking what the surface offers per app)
+8
+    ^ apps offering the destructive action
+
+--- 9d. destroy again, then the ORDINARY PRIMARY restore (POST /backup/restore)
+    destroyed: 0	/var/lib/docker/volumes/docmost_docmost_postgres_data/_data
+302 https://127.0.0.1:443/backups/restore?flash=Vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
+{"ok":true,"data":{"running":false,"op":"restore","stack":"docmost","started_at":"2026-08-31T10:05:33.081614501Z","last":{"op":"restore","stack":"docmost","ok":true,"message":"A(z) docmost: a beállítások visszaálltak — az alkalmazás újraindult. FIGYELEM: ez a mentés csak a beállításokat tartalmazta, adatot nem. Az alkalmazás adatai NEM álltak vissza ebből a mentésből.","finished_at":"2026-08-31T10:05:57.630738384Z"},"last_recent":true}}
+
+--- 9e. which unit did the ORDINARY restore read?
+    2026/08/31 10:02:28 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit: images=3, secrets recovered=2/2, data_keys=0
+    2026/08/31 10:02:57 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
+    2026/08/31 10:04:23 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit: images=3, secrets recovered=2/2, data_keys=0
+    2026/08/31 10:04:51 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
+    2026/08/31 10:05:33 restore_unit.go:313: [INFO] [backup] Restoring docmost from recovery unit /mnt/sys_drive/felhom-data/backups/primary/docmost: images=3, secrets recovered=2/2, data_keys=0
+    2026/08/31 10:05:57 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: docmost — 0 volume(s) of 0 listed, 0 database(s) of 0 listed
+
+--- 9f. final state
+    docmost Up 14 seconds (healthy)
+    docmost-postgres Up 24 seconds (healthy)
+    docmost-redis Up 24 seconds (healthy)
+    ERROR:  relation "felhom_r102_discriminator" does not exist
+    LINE 1: select id||chr(32)||chr(124)||chr(32)||note from felhom_r102...
+                                                             ^
+    accented file back from the PRIMARY unit: False
+    docmost HTTP 200