DRILL 2026-08-21: the off-site restore never replays named volumes (R-354..R-365)
gates / gates (push) Successful in 16s
gates / gates (push) Successful in 16s
Diagnostic only — no code changed, no version bumped, nothing deployed.
The verdict is a mixture. The unit and the off-site snapshot HOLD the data, proven
by identity in both storage classes including two Hungarian accented filenames. The
loss is in the last leg: ReconstituteFromOffsite skips every isUnit placement and the
volume tars live inside the unit, so the off-site full restore has no named-volume
leg at all — while the local restore-from-unit does, and returned the same tar
byte-identical minutes later.
Twelve rows opened, ceiling R-353 -> R-365. Three HIGH:
R-354 off-site restore never replays volume dumps
R-355 paperless-ngx's Postgres is dumped under a non-existent stack, so its unit
has no DB dump, no safety dump is taken, and the customer is told it has none
R-356 the off-site restore refuses for all 40 no-drive apps saying the running app
"is not installed", with a remedy those apps make impossible
R-353's instruction (2) is satisfied and annotated: the 40-class DOES reach the
off-site tier. Its instruction (1) stands and is now larger. R-329 confirmed still
live and now the only bad-severity emit fleet-wide.
Evidence: documentation/audits/DRILL-backup-truth-2026-08-21/evidence/
This commit is contained in:
+42
@@ -0,0 +1,42 @@
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:33 offsite-restore/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:33 offsite-restore/opengist/
|
||||
drwxr-xr-x root/root 0 2026-08-03 08:11 offsite-restore/opengist/mnt/
|
||||
drwxr-xr-x root/root 0 2026-08-04 14:51 offsite-restore/opengist/mnt/sys_drive/
|
||||
drwxr-xr-x root/root 0 2026-08-03 08:29 offsite-restore/opengist/mnt/sys_drive/felhom-data/
|
||||
drwxr-xr-x root/root 0 2026-08-04 23:13 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/
|
||||
drwxr-xr-x root/root 0 2026-08-06 22:02 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/
|
||||
drwxr-xr-x root/root 0 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/
|
||||
drwxr-xr-x root/root 0 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/
|
||||
-rw-r--r-- root/root 1750 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/.felhom.yml
|
||||
-rw-r--r-- root/root 1260 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/docker-compose.yml
|
||||
-rw------- root/root 287 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/app.yaml
|
||||
drwxr-xr-x root/root 0 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/volume-dumps/
|
||||
-rw-r--r-- root/root 182272 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/volume-dumps/opengist_opengist_data.tar
|
||||
-rw-r--r-- root/root 1121 2026-08-09 10:30 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/manifest.json
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:30 offsite-restore/calibre-web/
|
||||
drwxr-xr-x root/root 0 2026-08-03 08:11 offsite-restore/calibre-web/mnt/
|
||||
drwxr-xr-x root/root 0 2026-08-04 14:51 offsite-restore/calibre-web/mnt/sys_drive/
|
||||
drwxr-xr-x root/root 0 2026-08-03 08:29 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/
|
||||
drwxrwsr-x root/1000 0 2026-08-04 18:45 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/
|
||||
drwxr-sr-x root/1000 0 2026-08-04 18:50 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/
|
||||
drwxrwsr-x 1000/1000 0 2026-08-09 10:15 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/
|
||||
-rw-r--r-- 1000/1000 181 2026-08-04 14:53 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt
|
||||
drwxrwxr-x 1000/1000 0 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/
|
||||
-rw-rw-r-- 1000/1000 21 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/plain.txt
|
||||
-rw-rw-r-- 1000/1000 3145728 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/binary-3mb.bin
|
||||
drwxrwxr-x 1000/1000 0 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/nested/
|
||||
-rw-rw-r-- 1000/1000 25 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/nested/\305\221szibarack.md
|
||||
-rw-rw-r-- 1000/1000 59 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/\303\241rv\303\255zt\305\261r\305\221-t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
|
||||
-rw-r--r-- 1000/1000 32768 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/metadata.db-shm
|
||||
-rw-r--r-- 1000/1000 0 2026-08-09 04:15 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/metadata.db-wal
|
||||
-rw-r--r-- 1000/1000 413696 2026-08-04 14:52 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/metadata.db
|
||||
drwxr-xr-x root/root 0 2026-08-04 23:13 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/
|
||||
drwxr-xr-x root/root 0 2026-08-06 22:02 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/
|
||||
drwxr-xr-x root/root 0 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/
|
||||
drwxr-xr-x root/root 0 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/
|
||||
-rw-r--r-- root/root 3035 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/.felhom.yml
|
||||
-rw-r--r-- root/root 2122 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/docker-compose.yml
|
||||
-rw------- root/root 317 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/app.yaml
|
||||
drwxr-xr-x root/root 0 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/volume-dumps/
|
||||
-rw-r--r-- root/root 368640 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/volume-dumps/calibre-web_calibre_web_config.tar
|
||||
-rw-r--r-- root/root 1157 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/manifest.json
|
||||
+1
@@ -0,0 +1 @@
|
||||
c9498bfba3dab7b8c59196be8a6c792f0044356d969b2d81ce4b00b61950aa96 documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase0-preexisting-scratch/offsite-restore-scratch.tar
|
||||
BIN
Binary file not shown.
+22
@@ -0,0 +1,22 @@
|
||||
DRILL 2026-08-21 — FULL/EMPTY contrast on demo-hp. All hashes sha256, byte-for-byte.
|
||||
Comparator positive control: one byte flipped at offset 500000 of binary-1mb.bin
|
||||
(af -> 00) => sha256sum -c FAILED rc=1; original PASSED rc=0. Mutant discarded.
|
||||
|
||||
Planted fixture (5 files, incl. two UTF-8 Hungarian accented names):
|
||||
SENTINEL.txt 0c23c8531214fe20cbc1ed177da22f51d65570c5aaedb39e380aa8d4d3e62991
|
||||
binary-1mb.bin 725763bbe679b22d5c231a14083ff13155f475c028c22ade4623955b50a2a84a
|
||||
nested/őszibarack.md a39ad6f623da67ac72f2e62a24245eef46c722c279ae89cd6b0c7c45afa6a4c4
|
||||
plain.txt 07e91a985809fc96752f97cddcf6523b211bc607c81a5808befabe35573926f1
|
||||
árvíztűrő-tükörfúrógép.txt 0d6a22ec56acf61b8a80e73eb12cb223d607acaa3684acfcfa7c3e34ecdfabea
|
||||
Name bytes (UTF-8 NFC):
|
||||
árvíztűrő-tükörfúrógép.txt = c3a1 72 76 c3ad 7a 74 c5b1 72 c591 2d 74 c3bc 6b c3b6 72 66 c3ba 72 c3b3 67 c3a9 70 2e747874
|
||||
őszibarack.md = c591 73 7a 69 62 61 72 61 63 6b 2e 6d 64
|
||||
|
||||
| class | data leg | in unit | in offsite snap | checking folder | OFF-SITE restore | LOCAL restore
|
||||
calibre-web FULL | drive | userdata files | n/a | YES 5/5 ident. | YES 5/5 ident. | YES 5/5 ident. | -
|
||||
calibre-web FULL | drive | named volume | YES 1.42MB | YES | YES | NO (silent) | -
|
||||
privatebin FULL | no-drive | named volume | YES 1.06MB | YES 5/5 ident.| YES 5/5 ident. | REFUSED (false) | YES 5/5 ident.
|
||||
opengist EMPTY | no-drive | named volume | YES 181KB skeleton | YES | YES | REFUSED (false) | -
|
||||
|
||||
CONCLUSION: the unit and the off-site snapshot HOLD the data, verified by identity.
|
||||
The off-site full restore has no named-volume leg at all; the local restore has one.
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
drwxr-xr-x root/root 0 2026-08-21 22:19 offsite-restore/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:33 offsite-restore/opengist/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:00 offsite-restore/opengist/mnt/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:01 offsite-restore/opengist/mnt/sys_drive/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:31 offsite-restore/opengist/mnt/sys_drive/felhom-data/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:31 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:31 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/
|
||||
drwxr-xr-x root/root 0 2026-08-21 22:17 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/
|
||||
drwxr-xr-x root/root 0 2026-08-21 22:17 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/
|
||||
-rw-r--r-- root/root 1750 2026-08-21 22:17 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/.felhom.yml
|
||||
-rw-r--r-- root/root 1260 2026-08-21 22:17 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/docker-compose.yml
|
||||
-rw------- root/root 287 2026-08-21 22:17 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/compose/app.yaml
|
||||
drwxr-xr-x root/root 0 2026-08-21 22:16 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/volume-dumps/
|
||||
-rw-r--r-- root/root 181248 2026-08-21 22:16 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/volume-dumps/opengist_opengist_data.tar
|
||||
-rw-r--r-- root/root 1121 2026-08-21 22:17 offsite-restore/opengist/mnt/sys_drive/felhom-data/backups/primary/opengist/manifest.json
|
||||
drwxr-xr-x root/root 0 2026-08-21 22:19 offsite-restore/privatebin/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:00 offsite-restore/privatebin/mnt/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:01 offsite-restore/privatebin/mnt/sys_drive/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:31 offsite-restore/privatebin/mnt/sys_drive/felhom-data/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:31 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:31 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/
|
||||
drwxr-xr-x root/root 0 2026-08-21 22:17 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/
|
||||
drwxr-xr-x root/root 0 2026-08-21 22:17 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/compose/
|
||||
-rw-r--r-- root/root 1723 2026-08-21 22:17 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/compose/.felhom.yml
|
||||
-rw-r--r-- root/root 1223 2026-08-21 22:17 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/compose/docker-compose.yml
|
||||
-rw------- root/root 288 2026-08-21 22:17 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/compose/app.yaml
|
||||
drwxr-xr-x root/root 0 2026-08-21 22:16 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/volume-dumps/
|
||||
-rw-r--r-- root/root 1055744 2026-08-21 22:16 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/volume-dumps/privatebin_privatebin_data.tar
|
||||
-rw-r--r-- root/root 1118 2026-08-21 22:17 offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/manifest.json
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:30 offsite-restore/calibre-web/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:00 offsite-restore/calibre-web/mnt/
|
||||
drwxr-xr-x nobody/nogroup 0 2026-08-21 18:26 offsite-restore/calibre-web/mnt/felhom-drives/
|
||||
drwxr-xr-x root/root 0 2026-07-22 03:30 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/
|
||||
drwxrwsr-x root/1000 0 2026-07-21 19:08 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/
|
||||
drwxrwsr-x root/1000 0 2026-07-26 08:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/
|
||||
drwxrwsr-x 1000/1000 0 2026-08-21 22:10 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/
|
||||
-rw-r--r-- 1000/1000 181 2026-08-04 14:53 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-SENTINEL.txt
|
||||
drwxrwxr-x 1000/1000 0 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/
|
||||
-rw-rw-r-- 1000/1000 35 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/plain.txt
|
||||
-rw-rw-r-- 1000/1000 1048576 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/binary-1mb.bin
|
||||
drwxrwxr-x 1000/1000 0 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/nested/
|
||||
-rw-rw-r-- 1000/1000 21 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/nested/\305\221szibarack.md
|
||||
-rw-rw-r-- 1000/1000 46 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/SENTINEL.txt
|
||||
-rw-rw-r-- 1000/1000 52 2026-08-21 22:09 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/\303\241rv\303\255zt\305\261r\305\221-t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
|
||||
drwxrwxr-x 1000/1000 0 2026-08-09 09:59 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/
|
||||
-rw-rw-r-- 1000/1000 21 2026-08-09 09:59 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/plain.txt
|
||||
-rw-rw-r-- 1000/1000 3145728 2026-08-09 09:59 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/binary-3mb.bin
|
||||
drwxrwxr-x 1000/1000 0 2026-08-09 09:59 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/nested/
|
||||
-rw-rw-r-- 1000/1000 25 2026-08-09 09:59 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/nested/\305\221szibarack.md
|
||||
-rw-rw-r-- 1000/1000 59 2026-08-09 09:59 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/\303\241rv\303\255zt\305\261r\305\221-t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
|
||||
-rw-r--r-- 1000/1000 32768 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/metadata.db-shm
|
||||
-rw-r--r-- 1000/1000 0 2026-08-09 04:15 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/metadata.db-wal
|
||||
-rw-r--r-- 1000/1000 413696 2026-08-04 14:52 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/metadata.db
|
||||
drwxr-xr-x root/root 0 2026-07-23 12:10 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/
|
||||
drwxr-xr-x root/root 0 2026-08-21 18:31 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/
|
||||
drwxr-xr-x root/root 0 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/
|
||||
drwxr-xr-x root/root 0 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/compose/
|
||||
-rw-r--r-- root/root 3035 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/compose/.felhom.yml
|
||||
-rw-r--r-- root/root 2122 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/compose/docker-compose.yml
|
||||
-rw------- root/root 327 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/compose/app.yaml
|
||||
drwxr-xr-x root/root 0 2026-08-21 22:16 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/volume-dumps/
|
||||
-rw-r--r-- root/root 1422848 2026-08-21 22:16 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/volume-dumps/calibre-web_calibre_web_config.tar
|
||||
-rw-r--r-- root/root 1165 2026-08-21 22:17 offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/manifest.json
|
||||
drwxr-xr-x root/root 0 2026-08-04 14:51 offsite-restore/calibre-web/mnt/sys_drive/
|
||||
drwxr-xr-x root/root 0 2026-08-03 08:29 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/
|
||||
drwxrwsr-x root/1000 0 2026-08-04 18:45 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/
|
||||
drwxr-sr-x root/1000 0 2026-08-04 18:50 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/
|
||||
drwxrwsr-x 1000/1000 0 2026-08-09 10:15 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/
|
||||
-rw-r--r-- 1000/1000 181 2026-08-04 14:53 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt
|
||||
drwxrwxr-x 1000/1000 0 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/
|
||||
-rw-rw-r-- 1000/1000 21 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/plain.txt
|
||||
-rw-rw-r-- 1000/1000 3145728 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/binary-3mb.bin
|
||||
drwxrwxr-x 1000/1000 0 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/nested/
|
||||
-rw-rw-r-- 1000/1000 25 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/nested/\305\221szibarack.md
|
||||
-rw-rw-r-- 1000/1000 59 2026-08-09 09:59 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/\303\241rv\303\255zt\305\261r\305\221-t\303\274k\303\266rf\303\272r\303\263g\303\251p.txt
|
||||
-rw-r--r-- 1000/1000 32768 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/metadata.db-shm
|
||||
-rw-r--r-- 1000/1000 0 2026-08-09 04:15 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/metadata.db-wal
|
||||
-rw-r--r-- 1000/1000 413696 2026-08-04 14:52 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/userdata/media/books/metadata.db
|
||||
drwxr-xr-x root/root 0 2026-08-04 23:13 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/
|
||||
drwxr-xr-x root/root 0 2026-08-06 22:02 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/
|
||||
drwxr-xr-x root/root 0 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/
|
||||
drwxr-xr-x root/root 0 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/
|
||||
-rw-r--r-- root/root 3035 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/.felhom.yml
|
||||
-rw-r--r-- root/root 2122 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/docker-compose.yml
|
||||
-rw------- root/root 317 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/compose/app.yaml
|
||||
drwxr-xr-x root/root 0 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/volume-dumps/
|
||||
-rw-r--r-- root/root 368640 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/volume-dumps/calibre-web_calibre_web_config.tar
|
||||
-rw-r--r-- root/root 1157 2026-08-09 10:30 offsite-restore/calibre-web/mnt/sys_drive/felhom-data/backups/primary/calibre-web/manifest.json
|
||||
+1
@@ -0,0 +1 @@
|
||||
7e59e57d6458d28f530dbaddbee0f2314ea1ef885052701531f57bad2529849d documentation/audits/DRILL-backup-truth-2026-08-21/evidence/phase3-experiment/checking-folders-after-full-restore.tar
|
||||
BIN
Binary file not shown.
+32
@@ -0,0 +1,32 @@
|
||||
VERBATIM customer-facing outcome strings, read from /api/backup/restore-status
|
||||
(the same value the wizard banner renders). Times CEST.
|
||||
|
||||
1) 22:21:51 off-site FULL RESTORE (reconstitute), app = privatebin [40-class, no HDD_PATH]
|
||||
ok = FALSE
|
||||
"A teljes visszaállítás sikertelen: a(z) privatebin nincs telepítve, ezért nincs hová
|
||||
visszaállítani az adatait. A mentése szerint az adatai itt voltak: /mnt/sys_drive.
|
||||
Telepítsd újra az alkalmazást (Alkalmazások) ugyanerre a helyre, utána ez a
|
||||
visszaállítás működni fog"
|
||||
FACT AT THAT MOMENT: privatebin deployed=true, state=running, container healthy.
|
||||
|
||||
2) 22:23:36 off-site FULL RESTORE (reconstitute), app = calibre-web [drive class]
|
||||
ok = TRUE
|
||||
"A(z) calibre-web: 5 fájl visszaállítva (mentés: 2026-08-21 22:16) — az alkalmazás
|
||||
újraindult. Ennek az alkalmazásnak nincs adatbázisa."
|
||||
FACT: the 5 declared-userdata files came back byte-identical.
|
||||
The named volume calibre-web_calibre_web_config was NOT restored, though its
|
||||
1,422,848-byte tar was in the snapshot, in the checking folder, and named in
|
||||
manifest.json volume_dumps. The message does not mention it.
|
||||
|
||||
3) 22:25:31 LOCAL restore from recovery unit, app = privatebin
|
||||
ok = TRUE
|
||||
"privatebin visszaállítva (helyi)."
|
||||
FACT: the named volume WAS restored, all 5 planted files byte-identical.
|
||||
The message carries no counts at all - it reads the same whatever happened.
|
||||
|
||||
4) 22:13:15 off-site backup run with ZERO apps selected
|
||||
log: "[offbox] backup run started (0 app(s) toggled)"
|
||||
"[offbox] backup OK: 0 app(s) backed up, 18 snapshot(s), 14s"
|
||||
card: badge "Aktív — nincs kijelölt alkalmazás" / "✓ Rendben" /
|
||||
"Sikeres — nincs mentésre jelölt alkalmazás" /
|
||||
"Nincs távoli mentésre jelölt alkalmazás — jelölj ki legalább egyet."
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
0c23c8531214fe20cbc1ed177da22f51d65570c5aaedb39e380aa8d4d3e62991 DRILL-2026-08-21/SENTINEL.txt
|
||||
725763bbe679b22d5c231a14083ff13155f475c028c22ade4623955b50a2a84a DRILL-2026-08-21/binary-1mb.bin
|
||||
a39ad6f623da67ac72f2e62a24245eef46c722c279ae89cd6b0c7c45afa6a4c4 DRILL-2026-08-21/nested/őszibarack.md
|
||||
07e91a985809fc96752f97cddcf6523b211bc607c81a5808befabe35573926f1 DRILL-2026-08-21/plain.txt
|
||||
0d6a22ec56acf61b8a80e73eb12cb223d607acaa3684acfcfa7c3e34ecdfabea DRILL-2026-08-21/árvíztűrő-tükörfúrógép.txt
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
ID Time Host Tags Paths
|
||||
------------------------------------------------------------------------------------------------------------------------------
|
||||
e6132ae5 2026-08-04 19:36:26 demo-hp felhom-offbox,calibre-web /mnt/sys_drive/felhom-data/backups/primary/calibre-web
|
||||
/mnt/sys_drive/felhom-data/userdata/media/books
|
||||
|
||||
9ac78c98 2026-08-04 19:36:31 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
92212ff8 2026-08-04 19:36:36 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
9d6d233f 2026-08-05 09:12:42 demo-hp felhom-offbox,calibre-web /mnt/sys_drive/felhom-data/backups/primary/calibre-web
|
||||
/mnt/sys_drive/felhom-data/userdata/media/books
|
||||
|
||||
29e7b245 2026-08-05 09:12:48 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
cd1db049 2026-08-05 09:12:53 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
0ef7a006 2026-08-06 20:00:44 demo-hp felhom-offbox,calibre-web /mnt/sys_drive/felhom-data/backups/primary/calibre-web
|
||||
/mnt/sys_drive/felhom-data/userdata/media/books
|
||||
|
||||
8662a8c1 2026-08-06 20:00:50 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
6dfa6602 2026-08-06 20:00:54 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
3635f945 2026-08-07 02:15:13 demo-hp felhom-offbox,calibre-web /mnt/sys_drive/felhom-data/backups/primary/calibre-web
|
||||
/mnt/sys_drive/felhom-data/userdata/media/books
|
||||
|
||||
7b1fa8b0 2026-08-07 02:15:17 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
38edf5b8 2026-08-07 02:15:21 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
a4d03ee3 2026-08-08 02:15:12 demo-hp felhom-offbox,calibre-web /mnt/sys_drive/felhom-data/backups/primary/calibre-web
|
||||
/mnt/sys_drive/felhom-data/userdata/media/books
|
||||
|
||||
f534bffe 2026-08-08 02:15:17 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
a685d30e 2026-08-08 02:15:22 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
41c830db 2026-08-09 08:30:38 demo-hp felhom-offbox,calibre-web /mnt/sys_drive/felhom-data/backups/primary/calibre-web
|
||||
/mnt/sys_drive/felhom-data/userdata/media/books
|
||||
|
||||
9e38b84c 2026-08-09 08:30:49 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
|
||||
78b93f04 2026-08-09 08:30:53 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
8c44bd4c 2026-08-21 21:00:52 demo-hp felhom-offbox,calibre-web /mnt/felhom-drives/hdd_1/backups/primary/calibre-web
|
||||
/mnt/felhom-drives/hdd_1/userdata/media/books
|
||||
|
||||
07bac5ad 2026-08-21 21:00:55 demo-hp felhom-offbox,privatebin /mnt/sys_drive/felhom-data/backups/primary/privatebin
|
||||
|
||||
7e4b703b 2026-08-21 21:00:58 demo-hp felhom-offbox,kimai /mnt/sys_drive/felhom-data/backups/primary/kimai
|
||||
|
||||
16cb8ce7 2026-08-21 21:01:08 demo-hp felhom-offbox,paperless-ngx /mnt/felhom-drives/hdd_1/appdata/paperless/media
|
||||
/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx
|
||||
|
||||
2a891149 2026-08-21 21:01:13 demo-hp felhom-offbox,romm /mnt/felhom-drives/hdd_1/backups/primary/romm
|
||||
|
||||
49b9f317 2026-08-21 21:01:22 demo-hp felhom-offbox,opengist /mnt/sys_drive/felhom-data/backups/primary/opengist
|
||||
------------------------------------------------------------------------------------------------------------------------------
|
||||
24 snapshots
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
paperless-ngx: the database is dumped into a directory for a stack that does not exist,
|
||||
so the recovery unit never contains it, and the restore then tells the customer the app
|
||||
has no database. Proven live 2026-08-21 22:39-22:45 CEST on demo-hp.
|
||||
|
||||
MECHANISM (source):
|
||||
internal/appbackup/dbdump.go:770 deriveStackName("paperless-postgres", known)
|
||||
1. candidate = suffixStripStackName("paperless-postgres") = "paperless" (line 801)
|
||||
2. known is non-empty, known["paperless"] is FALSE (the stack is "paperless-ngx")
|
||||
3. known["paperless-postgres"] is FALSE
|
||||
4. no known stack name is a prefix of "paperless-postgres" ("paperless-ngx" is not)
|
||||
5. FALLS THROUGH to `return candidate` (line 797) -> "paperless"
|
||||
An unresolved mapping is returned as if resolved. There is no warning and no refusal.
|
||||
|
||||
OBSERVED CONSEQUENCES (all live):
|
||||
a) the dump is written to
|
||||
/mnt/sys_drive/felhom-data/backups/primary/paperless/db-dumps/paperless-postgres.sql
|
||||
284,617 bytes, 72 tables, valid=true -- an orphan directory for a non-existent stack,
|
||||
on the SYSTEM drive, while the app's real unit is on /mnt/felhom-drives/hdd_1.
|
||||
b) the real unit /mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx/manifest.json
|
||||
records "db_dumps": null
|
||||
c) the off-site snapshot therefore carries no .sql at all
|
||||
(checking folder: `find ... -name "*.sql" | wc -l` = 0)
|
||||
d) writeSafetyDump (offbox_reconstitute.go:115) filters discovered DBs on
|
||||
db.StackName == stackName, so `mine` is empty -> returns ("", nil) -> hasDB = false.
|
||||
NO pre-restore safety dump is taken. Verified: `find /mnt -name "pre-restore-*"`
|
||||
returned nothing before AND after the destructive restore.
|
||||
e) the destructive restore ran to completion and reported SUCCESS:
|
||||
"A(z) paperless-ngx: 0 fájl visszaállítva (mentés: 2026-08-21 22:41)
|
||||
— az alkalmazás újraindult. Ennek az alkalmazásnak nincs adatbázisa."
|
||||
The controller had dumped that same database 5 minutes earlier.
|
||||
|
||||
The orphan directory is also invisible to the app's off-site push, because the push
|
||||
resolves paths from the app's own unit path -- so the only copy of that database dump
|
||||
is on the system drive of the machine it protects.
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
{"timestamp":"2026-08-21T20:38:46Z","level":"DEBUG","message":"DiscoverDatabases: paperless-postgres → stack=paperless, dbUser=paperless, dbName=paperless","source":"dbdump.go:150"}
|
||||
{"timestamp":"2026-08-21T20:39:37Z","level":"DEBUG","message":"DiscoverDatabases: paperless-postgres → stack=paperless, dbUser=paperless, dbName=paperless","source":"dbdump.go:150"}
|
||||
{"timestamp":"2026-08-21T20:39:37Z","level":"DEBUG","message":"DumpOne: starting dump for container=paperless-postgres, stack=paperless, dbType=postgres, dumpDir=/mnt/sys_drive/felhom-data/backups/primary/paperless/db-dumps","source":"dbdump.go:189"}
|
||||
{"timestamp":"2026-08-21T20:39:37Z","level":"DEBUG","message":"DumpOne: completed paperless-postgres → paperless-postgres.sql (size=277.9 KB, valid=true, tables=72, duration=313ms)","source":"dbdump.go:330"}
|
||||
{"timestamp":"2026-08-21T20:41:23Z","level":"DEBUG","message":"DiscoverDatabases: paperless-postgres → stack=paperless, dbUser=paperless, dbName=paperless","source":"dbdump.go:150"}
|
||||
{"timestamp":"2026-08-21T20:41:23Z","level":"DEBUG","message":"DumpOne: starting dump for container=paperless-postgres, stack=paperless, dbType=postgres, dumpDir=/mnt/sys_drive/felhom-data/backups/primary/paperless/db-dumps","source":"dbdump.go:189"}
|
||||
{"timestamp":"2026-08-21T20:41:23Z","level":"DEBUG","message":"DumpOne: completed paperless-postgres → paperless-postgres.sql (size=278.2 KB, valid=true, tables=72, duration=314ms)","source":"dbdump.go:330"}
|
||||
{"timestamp":"2026-08-21T20:43:46Z","level":"DEBUG","message":"DiscoverDatabases: paperless-postgres → stack=paperless, dbUser=paperless, dbName=paperless","source":"dbdump.go:150"}
|
||||
{"timestamp":"2026-08-21T20:44:33Z","level":"DEBUG","message":"DiscoverDatabases: paperless-postgres → stack=paperless, dbUser=paperless, dbName=paperless","source":"dbdump.go:150"}
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
total 288
|
||||
drwxr-xr-x 2 root root 4096 Aug 21 20:41 .
|
||||
drwxr-xr-x 3 root root 4096 Aug 21 20:39 ..
|
||||
-rw-r--r-- 1 root root 284902 Aug 21 20:41 paperless-postgres.sql
|
||||
--- real unit:
|
||||
{
|
||||
"schema_version": 2,
|
||||
"app_name": "paperless-ngx",
|
||||
"display_name": "Paperless-ngx",
|
||||
"controller_version": "0.217.0",
|
||||
"created_at": "2026-08-21T20:42:01Z",
|
||||
"drive": "/mnt/felhom-drives/hdd_1",
|
||||
"namespace_root": "/mnt/felhom-drives/hdd_1",
|
||||
"image_pins": [
|
||||
"ghcr.io/paperless-ngx/paperless-ngx:2.20.15",
|
||||
"postgres:16-alpine",
|
||||
"redis:7-alpine"
|
||||
],
|
||||
"secret_env_vars": [
|
||||
"DB_PASSWORD",
|
||||
"PAPERLESS_SECRET_KEY",
|
||||
"PAPERLESS_ADMIN_PASSWORD"
|
||||
],
|
||||
"data_key_env_vars": null,
|
||||
"secret_source": "portable secrets (data keys, DB passwords, internal signing secrets) are IN this unit's compose/app.yaml (0600); internet-reachable admin logins are NOT, and come from the guest's app.yaml or are regenerated on restore",
|
||||
"config_files": [
|
||||
"docker-compose.yml",
|
||||
".felhom.yml",
|
||||
"app.yaml"
|
||||
],
|
||||
"db_dumps": null,
|
||||
"volume_dumps": [
|
||||
"paperless-ngx_paperless_data.tar",
|
||||
"paperless-ngx_paperless_postgres_data.tar",
|
||||
"paperless-ngx_paperless_redis_data.tar"
|
||||
],
|
||||
"checksums": {
|
||||
".felhom.yml": "a7cce0a557fd151e6385a137f4721366dd2cd0aa3876783f0f9f0acc9a78dd23",
|
||||
"app.yaml": "8dfec452dfc00c7e3dce26864bf97165acac44f32470d2425c96e2bd86649cf0",
|
||||
"docker-compose.yml": "b112952565ef3928f192358ea58fdf4a5a26d788593bf2614e36050c77539b3d"
|
||||
},
|
||||
"portable_secret_env_vars": [
|
||||
"DB_PASSWORD",
|
||||
"PAPERLESS_SECRET_KEY"
|
||||
],
|
||||
"offsite_run_id": "20260821T204123Z",
|
||||
"dumps_at": "2026-08-21T20:41:23Z"
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
PART 4 — things we claim and have never watched. demo-hp, 2026-08-21, times CEST.
|
||||
|
||||
4.1 THE DESTRUCTIVE RESTORE
|
||||
(a) "nothing is ever deleted" -- PASS, both directions, calibre-web 22:26:59.
|
||||
POST-SNAPSHOT.txt, created after the snapshot, SURVIVED the restore.
|
||||
plain.txt, mutated after the snapshot, was OVERWRITTEN back to the snapshot's
|
||||
content (sha 07e91a98…). Copier is rsync -a, no --delete, no --ignore-existing
|
||||
(offbox_reconstitute.go:94).
|
||||
NOTE ON THE COUNT: the message says "2 fájl visszaállítva" because rsync counts
|
||||
TRANSFERS, not files restored. An identical restore reports "0 fájl visszaállítva",
|
||||
which is indistinguishable from a restore that did nothing.
|
||||
|
||||
(b) "a safety dump is taken and VERIFIED before anything is stopped, and the whole
|
||||
operation refuses if it cannot be"
|
||||
HAPPY PATH -- PASS, romm 23:02:44.
|
||||
21:02:47Z "[offbox] romm: pre-restore safety dump written →
|
||||
pre-restore-20260821T210246Z-romm-mariadb.sql (60.8 KB)"
|
||||
21:02:47Z "[stacks] StopStack romm: current state=running"
|
||||
The dump precedes the stop. Message correctly said
|
||||
"0 fájl és az adatbázis visszaállítva".
|
||||
THE REFUSAL -- PASS, romm 23:04:15.
|
||||
Safety dump made impossible by putting a regular FILE at the db-dumps path.
|
||||
Result: ok=FALSE,
|
||||
"A teljes visszaállítás sikertelen: a biztonsági mentés könyvtára nem hozható
|
||||
létre: mkdir /mnt/felhom-drives/hdd_1/backups/primary/romm/db-dumps:
|
||||
not a directory"
|
||||
Nothing changed: plain.txt kept my post-snapshot mutation (sha 3754dfc6…), and
|
||||
romm's container StartedAt was unchanged (21:03:08) -- the app was never stopped.
|
||||
JUDGEMENT: honest and it names the path, but it leaks a raw Go mkdir error into
|
||||
a customer surface.
|
||||
THE HOLE -- the guard only protects apps whose database the discovery resolves to
|
||||
the right stack. For paperless-ngx it concludes "no database", so hasDB is false
|
||||
and the refusal CANNOT fire: the undo is absent rather than refused. See
|
||||
../phase4-paperless/FINDING.txt.
|
||||
SIDE EFFECT, NOT PREVIOUSLY FILED -- the safety dump DESTROYS the unit's own DB dump.
|
||||
DumpOne writes the canonical `<stack>-<dbtype>.sql` (appbackup/dbdump.go:200-202),
|
||||
i.e. the app's real dump, and only THEN is it renamed to pre-restore-*.
|
||||
The comment at offbox_reconstitute.go:147-148 states it "can never overwrite the
|
||||
app's real dump". It does.
|
||||
PROVEN: romm's db-dumps held romm-mariadb.sql (62,270 B) at 22:59; after one
|
||||
reconstitute it held ONLY pre-restore-20260821T210246Z-romm-mariadb.sql.
|
||||
Consequence: until the next backup run the LOCAL restore-from-unit finds no .sql
|
||||
and tells the customer the app has no database.
|
||||
|
||||
4.3 A DAMAGED STORE -- MIXED
|
||||
Method: one byte flipped inside pack 967853d2… (offset 5,000,000) via the repo's own
|
||||
SFTP transport; the pack's name is its content hash, so this is genuine corruption.
|
||||
* `restic check` DOES detect it ("ciphertext verification failed",
|
||||
"Fatal: repository contains errors"). BUT the controller NEVER RUNS `restic check`:
|
||||
the only restic verbs in the whole controller are restore, snapshots, backup, unlock,
|
||||
stats, init, forget, prune, cat. The agent's restore-test is PBS-tier only.
|
||||
So the off-site store is never verified by any layer, at any time.
|
||||
* A restore that TOUCHES the damage fails honestly:
|
||||
ok=FALSE, "A visszaállítás sikertelen: offbox restore paperless-ngx: exit status 1:
|
||||
… ignoring error for …/documents/originals/0000011.pdf: ciphertext verification failed"
|
||||
* BUT the failure is not remembered. It left a PARTIAL scratch (78 MB, 54 files,
|
||||
15 of 16 originals). OffboxFullScratchReady (offbox_restore.go:305) only asks
|
||||
"does the directory exist and is it non-empty", so the wizard then offered all three
|
||||
actions including "Teljes visszaállítás indítása".
|
||||
* Pressing it ran the DESTRUCTIVE restore from that known-incomplete copy and reported
|
||||
SUCCESS: ok=TRUE, "A(z) paperless-ngx: 0 fájl visszaállítva … Ennek az alkalmazásnak
|
||||
nincs adatbázisa."
|
||||
REPO REPAIRED afterwards from the byte-identical originals; `restic check` now says
|
||||
"no errors were found".
|
||||
@@ -0,0 +1,190 @@
|
||||
# REPORT — RUNBOOK ep0: read the PBS changelog, then decide whether to upgrade (2026-08-18, midday)
|
||||
|
||||
**Outcome:** changelog read → **no connection-handling fix in the range**; operator ruled to upgrade
|
||||
anyway **for rehearsal value**; upgraded **4.2.2-1 → 4.2.5-1** cleanly; **the fd slope did not change,
|
||||
which is the predicted result.** Two dated checks filed as **R-341**.
|
||||
|
||||
**Both STOPs cleared by the operator.** No code changed in any repo; `documentation/` only.
|
||||
Evidence: `documentation/audits/evidence-ep0-pbs-upgrade-2026-08-18/`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Baselines re-confirmed on the machine
|
||||
|
||||
Not from the audit file — from `dpkg -l` and `apt-cache policy`, per the runbook.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Installed | `proxmox-backup-server` / `-client` **4.2.2-1** |
|
||||
| Candidate | **4.2.5-1** (4.2.3-1, 4.2.4-1 also available) |
|
||||
| Proxy PID / started | 542065, **03:54:53Z**, unrestarted since the incident |
|
||||
| Effective `open files` | **65536 / 65536** — the morning's drop-in in force |
|
||||
| `Recv-Q` / loopback | **0** / **`200` in 11 ms** |
|
||||
| Datastore | 3.7 G of 98 G, 4% |
|
||||
| felhom.eu `main` @ start | `435e044` — matches the runbook's stated baseline |
|
||||
|
||||
## 2. The before-slope — and a correction I owe the morning's report
|
||||
|
||||
Two independent windows on the same proxy generation:
|
||||
|
||||
| window | from → to | delta | rate |
|
||||
|---|---|---|---|
|
||||
| 31 min | 09:18:21Z fd=62 → 09:49:46Z fd=66 | +4 | **183/day** |
|
||||
| 5.64 h | 04:11:36Z fd=19 → 09:49:46Z fd=66 | +47 | **200/day** |
|
||||
|
||||
**This morning's incident note said ~85/day and "≈2 years of runway". Both were wrong.** They were
|
||||
extrapolated from a single 17-minute window whose delta was **one descriptor** — a sample of one
|
||||
cannot carry a daily rate, and the agreement with the historical ~73/day that made it feel solid was
|
||||
coincidence. **The real rate is ~185–200/day, ~2.6× what I published, and the runway is ~357 days,
|
||||
not two years.** Corrected in the incident document and in R-336 rather than left standing.
|
||||
|
||||
**The mechanism I named was also the minority one.** `CLOSE-WAIT` held flat at **1** across the
|
||||
window while `ESTAB` grew **45 → 49** — *all* the growth was established connections. At the wedge
|
||||
the split was **1011 ESTAB / 543 CLOSE-WAIT**, so ESTAB dominated there too. **R-336's fix must target
|
||||
connections the proxy never reaps, not just `CLOSE-WAIT` sockets.**
|
||||
|
||||
## 3. The changelog, verbatim — the run's primary deliverable
|
||||
|
||||
All three entries between 4.2.2-1 and 4.2.5-1 read in full (128 lines), then swept for
|
||||
`connection|file descriptor|fd|accept(|close_wait|keep-alive|socket|EMFILE|nofile|leak|proxy|listen|
|
||||
backlog|hyper|tokio`.
|
||||
|
||||
**Exactly one hit, and it is a false positive:**
|
||||
|
||||
> `* S3: config: allow editing the use-node-config flag that controls whether requests S3 endpoints`
|
||||
> ` honor the node's proxy settings or not`
|
||||
|
||||
HTTP-proxy configuration *for S3 requests* — not the `proxmox-backup-proxy` daemon.
|
||||
|
||||
What the range does contain. **4.2.5-1** — a security release hardening client-supplied manifests:
|
||||
|
||||
> `* backup: harden the handling of client supplied backup manifests:`
|
||||
> ` - only accept archive names that are plain file names carrying a server side type extension. A`
|
||||
> ` crafted name in a manifest could previously make a sync job read or write outside of the`
|
||||
> ` snapshot directory, running as the unprivileged 'backup' user.`
|
||||
> ` - keep an uploaded manifest in memory and only persist it on backup finish, checking that every`
|
||||
> ` archive it lists was really uploaded during that session and that the checksums match`
|
||||
|
||||
plus a sync/push chunk-reuse fix and a subscription-key architecture check. **4.2.4-1** — S3 rate
|
||||
limits, a file-locking user-lookup cache, the new `proxmox-enterprise-support-keyring` dependency,
|
||||
docs. **4.2.3-1** — UI/journal work, an LDAP search-filter escape, tape and timezone fixes.
|
||||
|
||||
**Nothing addresses descriptor lifetime or connection reaping. My recommendation was: do not upgrade
|
||||
for this reason.**
|
||||
|
||||
## 4. STOP 1 — the ruling
|
||||
|
||||
**Operator (Viktor) ruled: upgrade anyway, for rehearsal value** — *"see how that works for us, we
|
||||
need practice with that too"*. Legitimate and recorded as such: this was **a practice run of the
|
||||
upgrade procedure on a Tier-2 protected machine, not a fix for the leak**.
|
||||
|
||||
**The interpretation was fixed in writing before any numbers existed** (`stop1-ruling.txt`):
|
||||
unchanged slope = **expected**, not a failed upgrade; changed slope = a **surprise** needing
|
||||
explanation, not a confirmation. That file was written at the ruling, not afterwards, so neither
|
||||
outcome could be rationalised into a success.
|
||||
|
||||
## 5. STOP 2 — the snapshot, and what it does not cover
|
||||
|
||||
Snapshot **421440873** `felhom-hetzner-20260818`, 15.06 GB, status **Available** (complete, not
|
||||
merely started), server #147604682, project 15217960.
|
||||
|
||||
**It covers `/dev/sda` only.** `/mnt/pbs-datastore` is `/dev/sdb`, a separate 100 GB **Volume**, and
|
||||
Hetzner server snapshots exclude attached volumes — so this is a rollback for the *software* state
|
||||
(packages, unit files, the `LimitNOFILE` drop-ins, nftables, wg) and **not a backup of the backup
|
||||
data**. Fine for a package install that writes no datastore content; **it must not be remembered as
|
||||
datastore protection.** Taken on a running server, deliberately: powering off ep0 to guard a userspace
|
||||
package install would take the only off-premises copy offline.
|
||||
|
||||
## 6. The upgrade and its verification
|
||||
|
||||
Simulated first (`-s`): **0 to remove**, so the abort condition never triggered. Then
|
||||
`apt-get install --only-upgrade -y proxmox-backup-server`, **09:51:00→09:51:06Z, exit 0**. Upgraded
|
||||
server/client/docs to 4.2.5-1 plus one new dependency, `proxmox-enterprise-support-keyring 1.1` —
|
||||
**which the 4.2.4-1 changelog had declared**, a small real consistency check between what I read and
|
||||
what apt did.
|
||||
|
||||
| check | result |
|
||||
|---|---|
|
||||
| installed | server / client / docs **4.2.5-1** |
|
||||
| daemons | `proxmox-backup-proxy` **active running**, `proxmox-backup` **active running** |
|
||||
| proxy restarted | 542065 → **551655** @ 09:51:04 |
|
||||
| **effective `open files`** | **65536 / 65536** — survived the new package |
|
||||
| drop-ins on disk | both present, unmodified |
|
||||
| `Recv-Q` | **0** |
|
||||
| loopback | **`200` in 12 ms** |
|
||||
| `felhom-pve` over tunnel | **`200` in 0.103 s**, `felhom-pbs active` |
|
||||
| `demo-hp` over tunnel | **`200` in 0.096 s**, `felhom-pbs active` |
|
||||
| hub gauge, post-upgrade | `11:59:31 [INFO] PBS-DR box refreshed: 3.7% full (3.7 GB of 97.9 GB)` |
|
||||
|
||||
`proxmox-backup-manager version` now reads `4.2.5-1 running version: 4.2.5`. **This independently
|
||||
settles the morning's confusion**: that string was never reporting a stale daemon, and now that
|
||||
installed and running genuinely match, both halves agree.
|
||||
|
||||
**One false alarm, mine.** I queried `systemctl is-active proxmox-backup-api` and got `inactive`.
|
||||
**That unit does not exist** — `systemctl cat` returns *"No files found for
|
||||
proxmox-backup-api.service"*. The real pair is `proxmox-backup-proxy.service` ("API Proxy Server")
|
||||
and `proxmox-backup.service` ("API Server"), both active. A bad query, not a fault — recorded because
|
||||
for as long as it took to check, it looked exactly like one.
|
||||
|
||||
**No backup, restore or verify was triggered** to "prove" the endpoint, per the runbook: the reads
|
||||
above answer it without mutating a protected datastore.
|
||||
|
||||
## 7. The after-slope — unchanged, as predicted
|
||||
|
||||
| | window | delta | rate |
|
||||
|---|---|---|---|
|
||||
| before (PID 542065) | 09:18:21Z fd=62 → 09:49:46Z fd=66 | +4 / 1885 s | **183/day** |
|
||||
| after (PID 551655) | 09:51:22Z fd=17 → 10:23:21Z fd=22 | +5 / 1919 s | **225/day** |
|
||||
|
||||
**These are not distinguishable.** The windows differ by **one descriptor**; Poisson uncertainty on
|
||||
n=4 is ±2 and on n=5 is ±2.2, so both are consistent with a single unchanged rate. **The after-figure
|
||||
being numerically higher is noise, not a regression — and certainly not an improvement.** Composition
|
||||
repeats the pattern: **ESTAB 0 → 5, CLOSE-WAIT 0 → 1.**
|
||||
|
||||
**Thirty minutes cannot settle this in either direction, and nothing here claims it does.**
|
||||
|
||||
## 8. Register
|
||||
|
||||
- **R-341 filed, WATCHING** — the two dated checks: **+24 h (2026-08-19 ~10:00Z)** and
|
||||
**+7 d (2026-08-25 ~10:00Z)**, against new `t0` **fd=17 @ 09:51:22Z, PID 551655**, with the exact
|
||||
command and the instruction to **record the ESTAB/CLOSE-WAIT split, not just the total** — the
|
||||
split is what identifies which leak it is. If the PID has changed, the window is void.
|
||||
- **R-336 updated and STILL OPEN.** Its wrong 85/day baseline is corrected to 183–200/day, the
|
||||
mechanism is re-pointed at ESTAB, and the upgrade's null result is recorded. **It stays open on its
|
||||
own merits:** an upgrade that *had* fixed the leak still would not make ~85,000 requests/day to a
|
||||
weekly-write DR endpoint correct.
|
||||
- `documentation/runbooks/offsite-endpoint.md` — **no change made, correctly**: it states no PBS
|
||||
version literal, only package names, so there was nothing to update (and per `docs.md`, version
|
||||
literals do not belong in a current-state doc anyway).
|
||||
|
||||
## 9. Teardown
|
||||
|
||||
**This run provisioned nothing.** No `.deb` was downloaded — `apt-get changelog` served the text
|
||||
directly, so the runbook's `/tmp` cleanup was never needed. The Hetzner snapshot **is retained** as
|
||||
the rollback; deleting it is an operator decision, and it costs €0.018161/GB/month.
|
||||
|
||||
## 10. Observations, not acted on
|
||||
|
||||
- **`pvesm status` reports `felhom-pbs` with Total/Used/Available all `0`** on both boxes while
|
||||
status reads `active`. Consistent before and after the upgrade, and the hub's own gauge reads the
|
||||
real 3.7 GB / 97.9 GB, so nothing is broken — but the PVE-side numbers are not usable as a capacity
|
||||
signal. Likely the namespace-scoped token lacking datastore-level audit. Not filed; noted here.
|
||||
- **8 other packages are held back** (`8 not upgraded`), untouched deliberately — `full-upgrade` on
|
||||
this machine is forbidden by the runbook and would change the kernel and WireGuard alongside the
|
||||
thing under test.
|
||||
- **The morning's `--no-verify` situation is unchanged**: `golden-currency` still convicts on the
|
||||
inherited R-334 (controller 0.216.0 vs golden 0.214.0), untouched by this run.
|
||||
|
||||
## 11. CI, checked by run ID
|
||||
|
||||
**Run `351`, `head_sha 3e50902a9`, conclusion `failure`, elapsed 15 s** (10:27:00→10:27:15Z).
|
||||
Inherited, not caused: CI's only step is `python3 scripts/repo_gates.py --fast`, which convicts
|
||||
`golden-currency` on **R-334** (controller 0.216.0 vs newest golden 0.214.0) — files this run did not
|
||||
touch. 15 s is the workflow's own honest-failure band, not the R-265 reap band. The previous run
|
||||
`350` on `435e044` failed identically, before this run began.
|
||||
|
||||
**Expect one `[felhom CI] gates FAILED` mail for run 351.** Same cause as run 348 this morning; not a
|
||||
new fault, and not related to the upgrade.
|
||||
|
||||
`python3 scripts/unproven.py --summary` — **unchanged: 23 walked, 32 not walked of 55.** No number
|
||||
moved, correctly: this run proved an operational fact, not a product claim.
|
||||
Reference in New Issue
Block a user