Restore and backup holes — fixed on main, proven on test boxes (2026-10-24 night, unattended)
Brief: „fix the restore and backup holes the review found, keep BYO hosts untouched, and pre-answer the review's
notes". Operator ruling recorded: D43 = A (by starting the run) — CONTEXT.md, 03, 11 §1.
Baselines (read at start, 16:47 UTC): felhom.eu 80f1ab31, felhom-controller 2449223 (v0.342.0 released), felhom-agent
f5c3d77 (v0.164.0), app-catalog 227a544. Register 207 rows. Architecture read first: 07 §6.3 / §8 / §8.2a, 03,
05, 11 §1.
What was fixed (commit only — nothing released, deployed, vouched or published)
| Row | Fix | Commit | CI |
|---|---|---|---|
| R-1251 (P1) | a pending put-back blocks every start: a copy on an unbound drive WAITS (restore hold putback_pending); the put-back runs first when the drive binds; a failed put-back holds (H2); Stop during a restore still gets the put-back (H7) |
controller bb2809d |
2301 / #482 success |
| R-1251 (found by the live proof) | the copy's folders are synced before the marker names the copy; the household is not told to run the restore again while the put-back waits | controller 062f9b4 |
2302 / #483 success |
| R-1252 (P2) | the second-drive reconcile runs only under a valid classification; a rejected block is said on the status and once a day to the operator; kept copies stay kept (H2); Remove drops kept records it deleted (H9) | controller 518a135 |
2303 / #484 success |
| R-1253 (P2) | a stop-first export refuses while another operation holds the app-stop marker | controller c9b4b82 |
2304 / #485 success |
| R-1254 (P2, hub half) | key, hash and identity blob in ONE transaction; a bad identity refused before any write | felhom.eu dcf4f84c |
2305 / #1256 success |
| R-1257 (P2, D43 = A) | the sshd drop-in only on an appliance; BYO: nothing written, our copy removed; X3 LAN addresses; X4 only the SSH step undone | felhom-agent c8d3c17 |
(final report) |
Every rule was red-proved: the rule taken out, the test seen failing, the rule put back (the commands are in the session
log; each test file names its red-proof). Images: controller dev images 0.343.0-dev.r1251a/b, r1252a, r1253a
were built locally and loaded straight into the test guests (never pushed); hub 0.170.0-rc.r1254 (67db584a45d7)
built locally from a clean clone at dcf4f84c, NOT pushed, NOT deployed.
The proofs — before (today's code) and after
| Proof | Before | After | Evidence |
|---|---|---|---|
| R-1251, Tester 1 (VM 341) | a cut restore + a late drive: the app ran on the cut data, a later start poured the old copy over new writes (review proof H1, lean clone) | run 2: held while the drive was away (Start/Update 409 with the sentence); the disk back → the put-back BEFORE the start; „gen2" (the newest) and the household's download there; a later restart poured nothing | P1-tester1/20…30 |
| R-1251 run 1 (a finding) | — | the cut left the journal as undo.json.tmp: the H2 hold kept the app stopped for support (never started on the cut data) → the folder-sync fix → run 2 |
P1-tester1/05…13 |
| R-1252, 9202 | one bogus path → „removed stale dest dir userdata/media" (review, 0.342.0, same box, same day) | two control runs kept the copy; the bogus path → „REJECTED … every existing copy … KEPT", the copy there, the status sentence set, the digest sent | P2-9202/ |
| R-1253, 9202 | a stop-first export during a restore replaced the restore's marker (static, review HUNT-1 H3) | the export refused (409) during a MeTube restore, uptime-kuma never stopped, the restore recorded its copy and finished; the same export after it ran (control) | P3-9202/ |
| R-1254 | — (static) | store and endpoint tests only (the hub is not deployed) | the test files |
| R-1257 | — (static) | the bundle harness (7 tests) | the test file |
How the late drive was made on Tester 1: the restore was cut by qm stop 341 2.1 s after the controller logged „the
restore copy is recorded" (tools/cut_on_line.sh); the data disk (nvme-scratch:341/vm-341-disk-2.raw, not the thin
pool) was detached with qm set 341 --delete scsi1, the VM started, and the disk re-attached with qm set 341 --scsi1 …;
the agent re-mounted it by itself („watchdog self-heal") and the boot recreate ran the put-back.
Teardown (three layers)
- Tester 1: MeTube removed through the product (Stop, then Remove with data and backups); my household file and the
media/metubefolder I made removed; my run-1 copy folder removed; the controller back on 0.342.0 and the two dev images removed; the data disk back onscsi1; no hold, no marker. One difference: privatebin's recreates left 3 empty anonymous volumes I cannot tell apart safely (196 → 199 after removing the 3 dangling ones my backup run made) — R-1267. (P1-tester1/00-as-found.txt,41-at-end.txt,42-anon-volumes.txt.) - 9202: the throwaway apps (audiobookshelf; MeTube and uptime-kuma) removed through the product, my marker and my
.fabremoved, the controller back on 0.342.0, the dev images removed; the dangling anonymous volumes my runs made removed by name. One difference: Immich's running container uses a new empty anonymous volume (R-1267). - Host (demo-hp): nothing provisioned; VM 341 config as found (
scsi1back). The catalog sweep's VM 9360 not touched. - Hub: nothing changed. The boxes pushed their normal events (Tester 1: the drive away and back, the interrupted restore).
Observations
FILED: R-1268— Remove with „remove backups" leaves a restore's undo copy (backups/restore-undo/<app>/) on the drive.FILED: R-1267(extended) — privatebin's image volumes leak like Immich's.NOT-A-FINDING:a support hold (restore_mixed) stays after the app is removed — by design (R-379: only a person clears it,-clear-restore-hold).NOT-A-FINDING:the boot sweep logged MeTube as held „by an app-data operation" while the marker waited — true (the kept marker), and the hold refused it as well.NOT-A-FINDING:a restore or the night dump replacing an EXPORT's marker — the replaced record only owed the exported app a restart, which the boot sweep gives every app the household wants running.