Files
felhom.eu/documentation/audits/restore-holes-2026-10-24
..

Restore and backup holes — fixed on main, proven on test boxes (2026-10-24 night, unattended)

Brief: „fix the restore and backup holes the review found, keep BYO hosts untouched, and pre-answer the review's notes". Operator ruling recorded: D43 = A (by starting the run) — CONTEXT.md, 03, 11 §1.

Baselines (read at start, 16:47 UTC): felhom.eu 80f1ab31, felhom-controller 2449223 (v0.342.0 released), felhom-agent f5c3d77 (v0.164.0), app-catalog 227a544. Register 207 rows. Architecture read first: 07 §6.3 / §8 / §8.2a, 03, 05, 11 §1.

What was fixed (commit only — nothing released, deployed, vouched or published)

Row Fix Commit CI
R-1251 (P1) a pending put-back blocks every start: a copy on an unbound drive WAITS (restore hold putback_pending); the put-back runs first when the drive binds; a failed put-back holds (H2); Stop during a restore still gets the put-back (H7) controller bb2809d 2301 / #482 success
R-1251 (found by the live proof) the copy's folders are synced before the marker names the copy; the household is not told to run the restore again while the put-back waits controller 062f9b4 2302 / #483 success
R-1252 (P2) the second-drive reconcile runs only under a valid classification; a rejected block is said on the status and once a day to the operator; kept copies stay kept (H2); Remove drops kept records it deleted (H9) controller 518a135 2303 / #484 success
R-1253 (P2) a stop-first export refuses while another operation holds the app-stop marker controller c9b4b82 2304 / #485 success
R-1254 (P2, hub half) key, hash and identity blob in ONE transaction; a bad identity refused before any write felhom.eu dcf4f84c 2305 / #1256 success
R-1257 (P2, D43 = A) the sshd drop-in only on an appliance; BYO: nothing written, our copy removed; X3 LAN addresses; X4 only the SSH step undone felhom-agent c8d3c17 (final report)

Every rule was red-proved: the rule taken out, the test seen failing, the rule put back (the commands are in the session log; each test file names its red-proof). Images: controller dev images 0.343.0-dev.r1251a/b, r1252a, r1253a were built locally and loaded straight into the test guests (never pushed); hub 0.170.0-rc.r1254 (67db584a45d7) built locally from a clean clone at dcf4f84c, NOT pushed, NOT deployed.

The proofs — before (today's code) and after

Proof Before After Evidence
R-1251, Tester 1 (VM 341) a cut restore + a late drive: the app ran on the cut data, a later start poured the old copy over new writes (review proof H1, lean clone) run 2: held while the drive was away (Start/Update 409 with the sentence); the disk back → the put-back BEFORE the start; „gen2" (the newest) and the household's download there; a later restart poured nothing P1-tester1/20…30
R-1251 run 1 (a finding) — the cut left the journal as undo.json.tmp: the H2 hold kept the app stopped for support (never started on the cut data) → the folder-sync fix → run 2 P1-tester1/05…13
R-1252, 9202 one bogus path → „removed stale dest dir userdata/media" (review, 0.342.0, same box, same day) two control runs kept the copy; the bogus path → „REJECTED … every existing copy … KEPT", the copy there, the status sentence set, the digest sent P2-9202/
R-1253, 9202 a stop-first export during a restore replaced the restore's marker (static, review HUNT-1 H3) the export refused (409) during a MeTube restore, uptime-kuma never stopped, the restore recorded its copy and finished; the same export after it ran (control) P3-9202/
R-1254 — (static) store and endpoint tests only (the hub is not deployed) the test files
R-1257 — (static) the bundle harness (7 tests) the test file

How the late drive was made on Tester 1: the restore was cut by qm stop 341 2.1 s after the controller logged „the restore copy is recorded" (tools/cut_on_line.sh); the data disk (nvme-scratch:341/vm-341-disk-2.raw, not the thin pool) was detached with qm set 341 --delete scsi1, the VM started, and the disk re-attached with qm set 341 --scsi1 …; the agent re-mounted it by itself („watchdog self-heal") and the boot recreate ran the put-back.

Teardown (three layers)

  • Tester 1: MeTube removed through the product (Stop, then Remove with data and backups); my household file and the media/metube folder I made removed; my run-1 copy folder removed; the controller back on 0.342.0 and the two dev images removed; the data disk back on scsi1; no hold, no marker. One difference: privatebin's recreates left 3 empty anonymous volumes I cannot tell apart safely (196 → 199 after removing the 3 dangling ones my backup run made) — R-1267. (P1-tester1/00-as-found.txt, 41-at-end.txt, 42-anon-volumes.txt.)
  • 9202: the throwaway apps (audiobookshelf; MeTube and uptime-kuma) removed through the product, my marker and my .fab removed, the controller back on 0.342.0, the dev images removed; the dangling anonymous volumes my runs made removed by name. One difference: Immich's running container uses a new empty anonymous volume (R-1267).
  • Host (demo-hp): nothing provisioned; VM 341 config as found (scsi1 back). The catalog sweep's VM 9360 not touched.
  • Hub: nothing changed. The boxes pushed their normal events (Tester 1: the drive away and back, the interrupted restore).

Observations

  • FILED: R-1268 — Remove with „remove backups" leaves a restore's undo copy (backups/restore-undo/<app>/) on the drive.
  • FILED: R-1267 (extended) — privatebin's image volumes leak like Immich's.
  • NOT-A-FINDING: a support hold (restore_mixed) stays after the app is removed — by design (R-379: only a person clears it, -clear-restore-hold).
  • NOT-A-FINDING: the boot sweep logged MeTube as held „by an app-data operation" while the marker waited — true (the kept marker), and the hold refused it as well.
  • NOT-A-FINDING: a restore or the night dump replacing an EXPORT's marker — the replaced record only owed the exported app a restart, which the boot sweep gives every app the household wants running.