R-403 drill evidence + the credential reader that ends a three-time mistake
gates / gates (push) Successful in 16s
gates / gates (push) Successful in 16s
The drill: the loss reproduced on the shipped v0.229.0 before anything was built. 120 082 104 B -> 7 036 B in one Tier-2 run, recorded as a success. Phases 1a (before), 1b (the hollow primary, produced through the R-102 restore path exactly as the 2026-08-31 observation was), 1c (the loss), 1d (repair). scripts/read_credential.py is Part 4's rider, and it exists because a note did not work three times: 2026-07-20 a Failed login was diagnosed as a stale password and written into memory; 2026-08-31 the same misreading recurred and was caught; 2026-08-31, hours later, it recurred AGAIN and rewrote a live box's password hash. Between them the project already had a memory file stating the rule, a worked recipe in it, and a session report describing the mistake. The rule now lives in the code path: one matching quote pair is unwrapped, the result is REFUSED if it still carries a quote, and --expect-length gives the caller a second opinion. The value goes file->file at 0600 and stdout gets only its length. test_read_credential.py asserts each refusal by its reason, with a positive control before believing the not-in-stdout result. Red-proof E1: remove the final quote assertion -> three cases fail by name.
This commit is contained in:
+51
@@ -0,0 +1,51 @@
|
||||
######## R-403 PHASE 1a — the BEFORE state, on the SHIPPED 0.229.0 build ########
|
||||
UTC 2026-08-31T11:29:29Z
|
||||
controller under test: gitea.dooplex.hu/admin/felhom-controller:0.229.0
|
||||
|
||||
=== SECONDARY copy — the thing that must survive ===
|
||||
--- full listing
|
||||
.felhom-tier2-layout
|
||||
recovery-unit/compose/.felhom.yml
|
||||
recovery-unit/compose/app.yaml
|
||||
recovery-unit/compose/docker-compose.yml
|
||||
recovery-unit/db-dumps/docmost-postgres.sql
|
||||
recovery-unit/db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql
|
||||
recovery-unit/db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql
|
||||
recovery-unit/db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql
|
||||
recovery-unit/manifest.json
|
||||
recovery-unit/volume-dumps/docmost_docmost_postgres_data.tar
|
||||
recovery-unit/volume-dumps/docmost_docmost_redis_data.tar
|
||||
recovery-unit/volume-dumps/docmost_docmost_storage.tar
|
||||
--- db-dumps/ and volume-dumps/ with sha256 (THE material at risk)
|
||||
9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/docmost-postgres.sql
|
||||
73917ba6bc3072dfc7b5be9c6df4f8361da7e987230f5d56f7b62f397fe15ef1 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql
|
||||
4c134c2ced74df26f49ef1910694cbbd25f2598549bb4b5bb145aac054935949 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql
|
||||
13e5a864701966d9e4053b5bb7dd800cca3d77ebb07f4fd2f32c86389422af25 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql
|
||||
f46a2fc3aa9a7ae2502d83b1c6ef27e503102f5ba71a0c6559246d9674c8e3b1 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_postgres_data.tar
|
||||
a8df17c444e41f54762e122ce1be998315c969015a1580bb8abdc7211cfa1a73 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_redis_data.tar
|
||||
88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_storage.tar
|
||||
--- counts
|
||||
db-dumps files: 4
|
||||
volume-dumps files: 3
|
||||
secondary unit size: 120082104 bytes
|
||||
|
||||
=== PRIMARY unit — currently COMPLETE ===
|
||||
created_at : 2026-08-31T09:43:41Z
|
||||
db_dumps : ['docmost-postgres.sql']
|
||||
volume_dumps: ['docmost_docmost_postgres_data.tar', 'docmost_docmost_redis_data.tar', 'docmost_docmost_storage.tar']
|
||||
/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/:
|
||||
total 568
|
||||
drwxr-xr-x 2 root root 4096 Aug 31 09:49 .
|
||||
drwxr-xr-x 5 root root 4096 Aug 31 09:43 ..
|
||||
-rw-r--r-- 1 root root 142277 Aug 31 09:49 docmost-postgres.sql
|
||||
-rw-r--r-- 1 root root 141363 Aug 22 16:23 pre-restore-20260822T162347Z-docmost-postgres.sql
|
||||
-rw-r--r-- 1 root root 141363 Aug 22 16:27 pre-restore-20260822T162708Z-docmost-postgres.sql
|
||||
-rw-r--r-- 1 root root 141363 Aug 22 21:54 pre-restore-20260822T215432Z-docmost-postgres.sql
|
||||
|
||||
/mnt/sys_drive/felhom-data/backups/primary/docmost/volume-dumps/:
|
||||
total 116720
|
||||
drwxr-xr-x 2 root root 4096 Aug 31 09:50 .
|
||||
drwxr-xr-x 5 root root 4096 Aug 31 09:43 ..
|
||||
-rw-r--r-- 1 root root 70135296 Aug 31 09:50 docmost_docmost_postgres_data.tar
|
||||
-rw-r--r-- 1 root root 49370624 Aug 31 09:50 docmost_docmost_redis_data.tar
|
||||
-rw-r--r-- 1 root root 2560 Aug 31 09:49 docmost_docmost_storage.tar
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
######## R-403 PHASE 1b — produce the hollow primary THE WAY THE DRILL DID ########
|
||||
UTC 2026-08-31T11:30:08Z
|
||||
--- 1b.0 SAFETY NET: copy the complete primary unit OUTSIDE every backup tree first
|
||||
(yesterday's mv landed inside a re-created directory; this path cannot be re-created over)
|
||||
stashed at /mnt/sys_drive/felhom-data/r403-safekeeping/docmost-unit:
|
||||
compose
|
||||
db-dumps
|
||||
manifest.json
|
||||
volume-dumps
|
||||
volume tars stashed: 3, db dumps stashed: 4
|
||||
|
||||
--- 1b.1 remove the primary unit (the R-102 scenario: the primary drive's package is gone)
|
||||
ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/docmost': No such file or directory
|
||||
|
||||
--- 1b.2 restore through the R-102 route, from the secondary mirror
|
||||
302 https://127.0.0.1:443/backups/apps?flash=Teljes+vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
|
||||
{"ok":true,"data":{"running":false,"op":"tier2-unit-restore","stack":"docmost","started_at":"2026-08-31T11:30:09.102644814Z","last":{"op":"tier2-unit-restore","stack":"docmost","ok":true,"message":"A(z) docmost: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult. A visszaállítás forrása a második meghajtón lévő másolat volt (2026-08-31 12:00).","finished_at":"2026-08-31T11:30:39.108378098Z"},"last_recent":true}}
|
||||
|
||||
--- 1b.3 wait for the 5-minute backup-cache job to write the primary unit
|
||||
primary manifest appeared after ~140s
|
||||
created_at : 2026-08-31T11:32:47Z
|
||||
db_dumps : []
|
||||
volume_dumps: None
|
||||
HOLLOW : True
|
||||
primary tree now:
|
||||
compose/.felhom.yml
|
||||
compose/app.yaml
|
||||
compose/docker-compose.yml
|
||||
manifest.json
|
||||
|
||||
--- 1b.4 the SECONDARY is still complete at this point (nothing has run against it yet)
|
||||
db-dumps: 4 volume-dumps: 3 size: 120082104
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
######## R-403 PHASE 1c — THE TIER-2 RUN. Does the empty package delete the good one? ########
|
||||
UTC 2026-08-31T11:33:14Z
|
||||
=== immediately BEFORE the run ===
|
||||
db-dumps : 4 files
|
||||
volume-dumps: 3 files
|
||||
unit size : 120082104 bytes
|
||||
9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/docmost-postgres.sql
|
||||
73917ba6bc3072dfc7b5be9c6df4f8361da7e987230f5d56f7b62f397fe15ef1 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql
|
||||
4c134c2ced74df26f49ef1910694cbbd25f2598549bb4b5bb145aac054935949 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql
|
||||
13e5a864701966d9e4053b5bb7dd800cca3d77ebb07f4fd2f32c86389422af25 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql
|
||||
f46a2fc3aa9a7ae2502d83b1c6ef27e503102f5ba71a0c6559246d9674c8e3b1 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_postgres_data.tar
|
||||
a8df17c444e41f54762e122ce1be998315c969015a1580bb8abdc7211cfa1a73 /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_redis_data.tar
|
||||
88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit/volume-dumps/docmost_docmost_storage.tar
|
||||
|
||||
=== POST /api/backup/tier2 (the real nightly path) ===
|
||||
200
|
||||
2026/08/31 11:33:14 tier2.go:402: [INFO] [backup] Tier 2 copied docmost → /mnt/felhom-drives/hdd_1/backups/secondary/docmost (14.9 KB, 0 leg(s), 0s)
|
||||
2026/08/31 11:33:15 tier2.go:446: [INFO] [backup] Tier 2 run complete: 8 app(s) processed (incl. volume-only — F6)
|
||||
|
||||
=== immediately AFTER the run ===
|
||||
db-dumps : 0 files
|
||||
volume-dumps: 0 files
|
||||
unit size : 7036 bytes
|
||||
full listing:
|
||||
.felhom-tier2-layout
|
||||
recovery-unit/compose/.felhom.yml
|
||||
recovery-unit/compose/app.yaml
|
||||
recovery-unit/compose/docker-compose.yml
|
||||
recovery-unit/manifest.json
|
||||
|
||||
=== VERDICT ===
|
||||
LOSS CONFIRMED — the empty package on the primary drive DELETED the complete copy
|
||||
on the second drive. 4 database dumps and 3 volume tars are gone.
|
||||
@@ -0,0 +1,26 @@
|
||||
######## R-403 PHASE 1d — repair the box before building the fix ########
|
||||
The first repair attempt used `rsync -a --delete` INSIDE the guest and silently did nothing:
|
||||
rsync in the guest: NOT-INSTALLED
|
||||
rsync lives in the CONTROLLER CONTAINER, not in guest 9201 — which is why Tier-2 (which shells out
|
||||
from inside the container) works while a guest-side script does not. The script ran with
|
||||
`set -uo pipefail` and no `-e`, so a missing binary continued as if it had succeeded. Recorded rather
|
||||
than quietly re-run: an unchecked exit code that looks like success is the same trap Phase 4 of the
|
||||
R-102 drill hit yesterday, in a different disguise.
|
||||
|
||||
Repaired with `cp -a` from the safety net:
|
||||
primary created_at: 2026-08-31T09:43:41Z
|
||||
db_dumps : ['docmost-postgres.sql']
|
||||
volume_dumps: ['docmost_docmost_postgres_data.tar','docmost_docmost_redis_data.tar','docmost_docmost_storage.tar']
|
||||
|
||||
Secondary rebuilt by a real Tier-2 run through POST /api/backup/tier2:
|
||||
db-dumps: 4 volume-dumps: 3 size: 120082104 bytes (identical to the phase-1a BEFORE state)
|
||||
f46a2fc3aa9a7ae2502d83b1c6ef27e503102f5ba71a0c6559246d9674c8e3b1 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
a8df17c444e41f54762e122ce1be998315c969015a1580bb8abdc7211cfa1a73 volume-dumps/docmost_docmost_redis_data.tar
|
||||
88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d volume-dumps/docmost_docmost_storage.tar
|
||||
9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28 db-dumps/docmost-postgres.sql
|
||||
docmost / docmost-redis / docmost-postgres: all healthy
|
||||
|
||||
The safety net at /mnt/sys_drive/felhom-data/r403-safekeeping/docmost-unit was intact throughout and
|
||||
is what made the repair possible. It was deliberately placed OUTSIDE every backup tree, because
|
||||
yesterday's set-aside was placed inside backups/primary/ and was swallowed by a directory the product
|
||||
re-created underneath it.
|
||||
Reference in New Issue
Block a user