R-403 CLOSED (controller v0.230.0), R-404 filed as a decision for Viktor
gates / gates (push) Failing after 17s

07-backup-architecture gains section 8.2, placed beside row 5 on purpose: the derived-copy rebuild
rule is UNCHANGED and section 8.2 names the single exception, so a future reader who finds RunTier2
skipping a leg does not fix it back. It carries the measurement (120 082 104 B -> 7 036 B on the
shipped v0.229.0), the four-case table, why hollow is a manifest question and not a size question,
why the data legs are deliberately not guarded, and why the capture job is not guarded either.

00-capability-map: the Tier-2 row's status does NOT move, stated explicitly rather than left
ambiguous. R-403 removes a way the route could be DESTROYED between uses; it does not change what
the route can be relied on for.

Register: R-403 CLOSED and compressed into CLOSED-ITEMS (594 -> 593 open lines). R-404 FILED as a
DECISION and deliberately not acted on - should a documents-only push be subject to the
golden-currency gate, now that it has been correctly bypassed six times? Both sides stated, plus
what happens if Viktor does nothing. The gate was NOT changed.

R-242: seventh conviction, and the FIRST where the day-0 ground does not apply - R-403 is a defect
in the nightly Tier-2 copy, which a day-0 box starts running on its first night. This push uses
git push --no-verify, declared here and in felhom-controller/REPORT.md. A golden carrying 0.230.0 is
owed and is more urgent than the previous six.

STATUS: the R-403 item moves out of 'Broken' into what works, in plain words; the delivery item now
says a golden is owed and that the fleet carries the defect; R-404 goes into the decide section with
its do-nothing outcome.

Drill evidence: nine phase logs, including the two things that went wrong (a repair whose rsync was
not installed in the guest and silently did nothing, and a session that expired mid-run so a POST
did nothing).
This commit is contained in:
2026-08-31 14:39:29 +02:00
parent 66156c619f
commit dddcc808be
10 changed files with 195 additions and 22 deletions
@@ -0,0 +1,18 @@
=== THE WARN, verbatim ===
2026/08/31 12:14:24 tier2.go:383: [WARN] [backup] Tier 2 docmost: unit leg SKIPPED — the recovery unit on the source drive lists no database dumps and no volume tars, while the existing copy at /mnt/felhom-drives/hdd_1/backups/secondary/docmost/recovery-unit does. The copy was PRESERVED rather than replaced with an empty one (R-403). The other legs continue.
2026/08/31 12:14:24 tier2.go:424: [INFO] [backup] Tier 2 copied docmost → /mnt/felhom-drives/hdd_1/backups/secondary/docmost (14.9 KB, 0 leg(s), 0s) [unit leg SKIPPED — existing package preserved, R-403]
=== the run summary line ===
2026/08/31 12:14:24 tier2.go:424: [INFO] [backup] Tier 2 copied docmost → /mnt/felhom-drives/hdd_1/backups/secondary/docmost (14.9 KB, 0 leg(s), 0s) [unit leg SKIPPED — existing package preserved, R-403]
=== AFTER the run ===
db-dumps: 4 volume-dumps: 3 size: 120082104
9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28 db-dumps/docmost-postgres.sql
73917ba6bc3072dfc7b5be9c6df4f8361da7e987230f5d56f7b62f397fe15ef1 db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql
4c134c2ced74df26f49ef1910694cbbd25f2598549bb4b5bb145aac054935949 db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql
13e5a864701966d9e4053b5bb7dd800cca3d77ebb07f4fd2f32c86389422af25 db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql
f46a2fc3aa9a7ae2502d83b1c6ef27e503102f5ba71a0c6559246d9674c8e3b1 volume-dumps/docmost_docmost_postgres_data.tar
a8df17c444e41f54762e122ce1be998315c969015a1580bb8abdc7211cfa1a73 volume-dumps/docmost_docmost_redis_data.tar
88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d volume-dumps/docmost_docmost_storage.tar
VERDICT: PRESERVED — all 7 files still there (v0.229.0 left 0)
@@ -0,0 +1,23 @@
######## SCENARIO B LIVE — the same state, on the FIXED build ########
UTC 2026-08-31T12:04:11Z
controller under test: gitea.dooplex.hu/admin/felhom-controller:0.230.0
--- recreate the hollow primary (hand-made this time; the R-102 path was proven in phase 1b)
hollow primary manifest written: db_dumps=[] volume_dumps=null
primary tree:
compose/.felhom.yml
compose/app.yaml
compose/docker-compose.yml
manifest.json
=== BEFORE the run ===
secondary db-dumps: 4 volume-dumps: 3 size: 120082104
9f676376f759733f5b62e590e4a2b31dddd66ff49990df3394332b790a092a28 db-dumps/docmost-postgres.sql
73917ba6bc3072dfc7b5be9c6df4f8361da7e987230f5d56f7b62f397fe15ef1 db-dumps/pre-restore-20260822T162347Z-docmost-postgres.sql
4c134c2ced74df26f49ef1910694cbbd25f2598549bb4b5bb145aac054935949 db-dumps/pre-restore-20260822T162708Z-docmost-postgres.sql
13e5a864701966d9e4053b5bb7dd800cca3d77ebb07f4fd2f32c86389422af25 db-dumps/pre-restore-20260822T215432Z-docmost-postgres.sql
f46a2fc3aa9a7ae2502d83b1c6ef27e503102f5ba71a0c6559246d9674c8e3b1 volume-dumps/docmost_docmost_postgres_data.tar
a8df17c444e41f54762e122ce1be998315c969015a1580bb8abdc7211cfa1a73 volume-dumps/docmost_docmost_redis_data.tar
88f21f491d0766aa7a1fc9eba5866e5fffd7a72fa640c55f7bccf575f2ba751d volume-dumps/docmost_docmost_storage.tar
=== POST /api/backup/tier2 ===
@@ -0,0 +1,10 @@
rows found: 8
app notice FIGYELEM skipped? package date in the confirm
bookstack False False False 2026-08-31 14:03
calibre-web False False False 2026-08-31 14:03
docmost True True True 2026-08-31 11:43
kimai False False False 2026-08-31 14:03
opengist False False False 2026-08-31 14:03
paperless-ngx False False False 2026-08-31 14:03
privatebin False False False 2026-08-31 14:03
romm False False False 2026-08-31 14:03
@@ -0,0 +1,19 @@
######## SCENARIO E LIVE — the primary is filled back in ########
UTC 2026-08-31T12:24:38Z
--- BEFORE: the primary is hollow (this is the state a Tier-2 unit restore leaves on 0.229.0)
created_at: 2026-08-31T12:08:49Z db_dumps: [] volume_dumps: None
compose/.felhom.yml
compose/app.yaml
compose/docker-compose.yml
manifest.json
--- the R-102 restore, through the real endpoint
302 https://127.0.0.1:443/backups/apps?flash=Teljes+vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
{"ok":true,"data":{"running":false,"op":"tier2-unit-restore","stack":"docmost","started_at":"2026-08-31T12:24:38.178701187Z","last":{"op":"tier2-unit-restore","stack":"docmost","ok":true,"message":"A(z) docmost: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult. A visszaállítás forrása a második meghajtón lévő másolat volt (2026-08-31 14:23).","finished_at":"2026-08-31T12:25:07.772790659Z"},"last_recent":true}}
--- IMMEDIATELY after the call returned (no waiting): is the primary a real package?
created_at: 2026-08-31T09:43:41Z db_dumps: ['docmost-postgres.sql'] volume_dumps: ['docmost_docmost_postgres_data.tar', 'docmost_docmost_redis_data.tar', 'docmost_docmost_storage.tar']
volume tars on the app drive: 3 db dumps: 4
--- the rehydrate log line, verbatim
2026/08/31 12:25:07 tier2_restore.go:253: [INFO] [backup] docmost: primary unit refilled from the secondary mirror (R-403) — 3 volume tar(s), 4 database dump(s) now on the app's own drive
@@ -0,0 +1,45 @@
######## TEARDOWN / final state ########
UTC 2026-08-31T12:33:57Z
--- a normal Tier-2 run now that BOTH sides are complete (the unit leg must NOT be skipped)
200
2026/08/31 12:33:57 tier2.go:424: [INFO] [backup] Tier 2 copied docmost → /mnt/felhom-drives/hdd_1/backups/secondary/docmost (114.5 MB, 0 leg(s), 0s)
unit-leg skips in this run: 0
--- both copies
PRIMARY : 3 tars, 4 dumps, 120082104 bytes
SECONDARY : 3 tars, 4 dumps, 120082104 bytes
IDENTICAL volume-dumps/docmost_docmost_postgres_data.tar
IDENTICAL db-dumps/docmost-postgres.sql
--- the surface is back to normal (no stale notice anywhere)
adatcsomagja occurrences: 0
--- remove the drill safety net and the driver
safekeeping removed
.bash_history
.bashrc
.docker
.profile
.ssh
settings.json.drill-backup
--- every app healthy?
docmost Up 9 minutes (healthy)
docmost-redis Up 9 minutes (healthy)
docmost-postgres Up 9 minutes (healthy)
romm Up 3 hours (healthy)
romm-db Up 3 hours (healthy)
romm-redis Up 3 hours (healthy)
privatebin Up 3 hours (healthy)
paperless-webserver Up 3 hours (healthy)
paperless-postgres Up 3 hours (healthy)
paperless-redis Up 3 hours (healthy)
opengist Up 3 hours (healthy)
kimai Up 3 hours (healthy)
kimai-db Up 3 hours (healthy)
calibre-web Up 3 hours (healthy)
bookstack Up 3 hours (healthy)
bookstack-db Up 3 hours (healthy)
filebrowser Up 9 days (healthy)
cloudflared Up 9 days
traefik Up 9 days