66 lines
6.2 KiB
Markdown
66 lines
6.2 KiB
Markdown
# Restore and backup holes — fixed on main, proven on test boxes (2026-10-24 night, unattended)
|
|
|
|
Brief: „fix the restore and backup holes the review found, keep BYO hosts untouched, and pre-answer the review's
|
|
notes". **Operator ruling recorded:** D43 = A (by starting the run) — `CONTEXT.md`, `03`, `11` §1.
|
|
|
|
**Baselines (read at start, 16:47 UTC):** felhom.eu `80f1ab31`, felhom-controller `2449223` (v0.342.0 released), felhom-agent
|
|
`f5c3d77` (v0.164.0), app-catalog `227a544`. Register 207 rows. Architecture read first: `07` §6.3 / §8 / §8.2a, `03`,
|
|
`05`, `11` §1.
|
|
|
|
## What was fixed (commit only — nothing released, deployed, vouched or published)
|
|
|
|
| Row | Fix | Commit | CI |
|
|
|---|---|---|---|
|
|
| R-1251 (P1) | a pending put-back blocks every start: a copy on an unbound drive WAITS (restore hold `putback_pending`); the put-back runs first when the drive binds; a failed put-back holds (H2); Stop during a restore still gets the put-back (H7) | controller `bb2809d` | 2301 / #482 success |
|
|
| R-1251 (found by the live proof) | the copy's folders are synced before the marker names the copy; the household is not told to run the restore again while the put-back waits | controller `062f9b4` | 2302 / #483 success |
|
|
| R-1252 (P2) | the second-drive reconcile runs only under a valid classification; a rejected block is said on the status and once a day to the operator; kept copies stay kept (H2); Remove drops kept records it deleted (H9) | controller `518a135` | 2303 / #484 success |
|
|
| R-1253 (P2) | a stop-first export refuses while another operation holds the app-stop marker | controller `c9b4b82` | 2304 / #485 success |
|
|
| R-1254 (P2, hub half) | key, hash and identity blob in ONE transaction; a bad identity refused before any write | felhom.eu `dcf4f84c` | 2305 / #1256 success |
|
|
| R-1257 (P2, D43 = A) | the sshd drop-in only on an appliance; BYO: nothing written, our copy removed; X3 LAN addresses; X4 only the SSH step undone | felhom-agent `c8d3c17` | (final report) |
|
|
|
|
Every rule was red-proved: the rule taken out, the test seen failing, the rule put back (the commands are in the session
|
|
log; each test file names its red-proof). Images: controller dev images `0.343.0-dev.r1251a/b`, `r1252a`, `r1253a`
|
|
were built locally and loaded straight into the test guests (never pushed); hub `0.170.0-rc.r1254` (`67db584a45d7`)
|
|
built locally from a clean clone at `dcf4f84c`, NOT pushed, NOT deployed.
|
|
|
|
## The proofs — before (today's code) and after
|
|
|
|
| Proof | Before | After | Evidence |
|
|
|---|---|---|---|
|
|
| R-1251, Tester 1 (VM 341) | a cut restore + a late drive: the app ran on the cut data, a later start poured the old copy over new writes (review proof H1, lean clone) | run 2: held while the drive was away (Start/Update 409 with the sentence); the disk back → the put-back BEFORE the start; „gen2" (the newest) and the household's download there; a later restart poured nothing | `P1-tester1/20…30` |
|
|
| R-1251 run 1 (a finding) | — | the cut left the journal as `undo.json.tmp`: the H2 hold kept the app stopped for support (never started on the cut data) → the folder-sync fix → run 2 | `P1-tester1/05…13` |
|
|
| R-1252, 9202 | one bogus path → „removed stale dest dir userdata/media" (review, 0.342.0, same box, same day) | two control runs kept the copy; the bogus path → „REJECTED … every existing copy … KEPT", the copy there, the status sentence set, the digest sent | `P2-9202/` |
|
|
| R-1253, 9202 | a stop-first export during a restore replaced the restore's marker (static, review HUNT-1 H3) | the export refused (409) during a MeTube restore, uptime-kuma never stopped, the restore recorded its copy and finished; the same export after it ran (control) | `P3-9202/` |
|
|
| R-1254 | — (static) | store and endpoint tests only (the hub is not deployed) | the test files |
|
|
| R-1257 | — (static) | the bundle harness (7 tests) | the test file |
|
|
|
|
How the late drive was made on Tester 1: the restore was cut by `qm stop 341` 2.1 s after the controller logged „the
|
|
restore copy is recorded" (`tools/cut_on_line.sh`); the data disk (`nvme-scratch:341/vm-341-disk-2.raw`, not the thin
|
|
pool) was detached with `qm set 341 --delete scsi1`, the VM started, and the disk re-attached with `qm set 341 --scsi1 …`;
|
|
the agent re-mounted it by itself („watchdog self-heal") and the boot recreate ran the put-back.
|
|
|
|
## Teardown (three layers)
|
|
|
|
- **Tester 1:** MeTube removed through the product (Stop, then Remove with data and backups); my household file and the
|
|
`media/metube` folder I made removed; my run-1 copy folder removed; the controller back on 0.342.0 and the two dev
|
|
images removed; the data disk back on `scsi1`; no hold, no marker. One difference: privatebin's recreates left 3 empty
|
|
anonymous volumes I cannot tell apart safely (196 → 199 after removing the 3 dangling ones my backup run made) —
|
|
R-1267. (`P1-tester1/00-as-found.txt`, `41-at-end.txt`, `42-anon-volumes.txt`.)
|
|
- **9202:** the throwaway apps (audiobookshelf; MeTube and uptime-kuma) removed through the product, my marker and my
|
|
`.fab` removed, the controller back on 0.342.0, the dev images removed; the dangling anonymous volumes my runs made
|
|
removed by name. One difference: Immich's running container uses a new empty anonymous volume (R-1267).
|
|
- **Host (demo-hp):** nothing provisioned; VM 341 config as found (`scsi1` back). The catalog sweep's VM 9360 not touched.
|
|
- **Hub:** nothing changed. The boxes pushed their normal events (Tester 1: the drive away and back, the interrupted
|
|
restore).
|
|
|
|
## Observations
|
|
|
|
- `FILED: R-1268` — Remove with „remove backups" leaves a restore's undo copy (`backups/restore-undo/<app>/`) on the drive.
|
|
- `FILED: R-1267` (extended) — privatebin's image volumes leak like Immich's.
|
|
- `NOT-A-FINDING:` a support hold (`restore_mixed`) stays after the app is removed — by design (R-379: only a person
|
|
clears it, `-clear-restore-hold`).
|
|
- `NOT-A-FINDING:` the boot sweep logged MeTube as held „by an app-data operation" while the marker waited — true (the
|
|
kept marker), and the hold refused it as well.
|
|
- `NOT-A-FINDING:` a restore or the night dump replacing an EXPORT's marker — the replaced record only owed the exported
|
|
app a restart, which the boot sweep gives every app the household wants running.
|