Data-loss hunt 2026-10-20 C4: register 153 -> 169 (opened R-1175..R-1190: two P1 — shares second copy pruned by another drive's group, box rebuilt after the host delete cannot reach its off-site history; R-1177 proven LIVE on 9202 — a failed local restore destroys live data with no copy); STATUS: the P1 line and decisions D22, D23

2026-10-20 12:31:34 +02:00
1 parent 242584af4e
commit bffe2d9dea
@@ -15,6 +15,7 @@
 - The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15).
 ## What is broken
 - **P1 (the data-loss hunt, 2026-10-20 — still running):** shared folders on two drives lose their second copy every night, and a dead drive's copy is deleted the same night (R-1175); a box rebuilt after „delete the old host" cannot reach the household's off-site history (R-1176).
 - **P1:** about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
 - Two root-level weaknesses wait on your decisions D3 and D4 (R-985, R-982).
 - Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862).
@@ -22,7 +23,6 @@
 - The scratch box cannot use agent features (R-1083) or read the drill catalog (R-992).
 ## What we work on
 - The long rehearsal and data-loss hunt on a fresh box (another CC session, 2026-10-20 and 21). Its findings become rows.
 - Dated checks, run by CC. 2026-10-21: the hub's own off-site check (R-895), the weekly clean-up window (R-435), downloads out of the second-drive copy (R-1019). 2026-10-23: no app stop when the host is busy (R-1173).
 - A gate keeps this page short and true: it warns now and refuses from 2026-10-27 (R-1174). CC.
@@ -32,6 +32,8 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
 | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
 |---|---|---|---|---|---|---|
 | D23 | R-1181 | A missing app folder deletes its second-drive copy within one night. Keep the old copy until the household lets it go? | **Yes, keep it** (a deleted folder is the reason a backup exists; audiobooks, comics and ROMs have no other copy) | Keep as now: a deleted folder is gone from every local copy by the next morning | an accidental folder delete is unrecoverable after one night | — |
 | D22 | R-1177 | A LOCAL restore that fails destroys the live data and keeps no copy. Give it the off-site restore's „put back exactly" (your D20)? | **Yes** (proven live today: a failed restore left the app dead and the newest data gone) | No: a failed local restore can lose the household's newest data | the main restore button can lose data when a backup copy is damaged | — |
 | D21 | R-49 | Immich's off-site copy: leave out only the AI model files now (B), or also the raw database folder and Immich's own backups (A)? | **B now** (−62 %, no new risk); A's other parts after their proof | A: −92 %, but it needs a restore path we have never run | every Immich box sends about 12 times its real data off-site each night | — |
 | D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
 | D8 | R-206 | Install DooPlex's Docker settings from git (Docker restarts once), with a weekly build-cache prune? | **Yes, in a quiet window** (a rebuilt DooPlex then keeps the settings) | No: the live settings stay hand-made, and a rebuild loses them | git and the host can drift apart | — |
@@ -17,7 +17,7 @@ Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searche
 | D1b an AWAY drive is still chosen as the second-drive target | CONFIRMED-CODE (+ the same unit test: `hdd_3/iskola => hdd_1/...` after hdd_1 was marked disconnected) | defect | `tier2.go:151-180` loops `GetSchedulableStoragePaths()` (`settings.go:1593-1603` excludes only Decommissioned); no Disconnected check. On a real box the away drive's empty mount point is on the guest's root disk |
 | **K1** reinstall after „delete the old host" loses the route to the old escrow | **CONFIRMED** at the hub store (unit test against production code, `C3-K1-escrow-after-host-delete/`); the controller half (new key minted, history set aside on night one) CONFIRMED-CODE | defect (03 §8a / 07 §7.0 promise the route; D19+R-1154 removed it unrecorded) | hub `store.go:3164-3229` demotes into `host_escrow_superseded` under the OLD host_id and deletes the `hosts` row; `GetEscrowStatusForCustomer` `store.go:3499-3527` joins on `hosts` → nil for the new box; `customer_reset.go:48,155` and `store.go:4189` DO include `host_deletions`. Live proof needs Tester4 off-site ON (B7) |
 | K2 abandoning one set-aside copy purges every retained package | PLAUSIBLE | defect (breaks `store.go:4176-4179`'s own invariant) | `store.go:4185-4190` customer-wide delete read true |
-| **R1/D4** a failed LOCAL unit restore: no copy of the live state, app started anyway | CONFIRMED-CODE → live proof running (`C3-R1-local-restore-9202/`) | **design question** for the operator: decision 154 (R-638 A) chose „start anyway"; D20 (R-893 A) gave „put back exactly" to the OFF-SITE path only | `restore_unit.go:428-489` |
+| **R1/D4** a failed LOCAL unit restore: no copy of the live state, app started anyway | **CONFIRMED LIVE** on 9202 (`C3-R1-local-restore-9202/`): live row destroyed, DB crash-loops, no undo copy, raw English Docker text to the household | **design question** for the operator: decision 154 (R-638 A) chose „start anyway"; D20 (R-893 A) gave „put back exactly" to the OFF-SITE path only | `restore_unit.go:428-489` |
 | R2 a crash mid-restore starts the app on a mixed state; the undo copy is never read at start | PLAUSIBLE | defect | `appstop_marker.go:307`, `offbox_restore_undo.go:137` |
 | R3 a cut-off shares restore can be placed and reported restored | CONFIRMED-CODE | defect (R-358 class, fixed for apps, not shares) | `shares_restore.go:131-140` readiness = non-empty dir |
 | R4 `.fab` export drops a drive folder holding a symlink | PLAUSIBLE | defect | `appexport/fabplan.go:292`, `export.go:714` |
@@ -27,7 +27,7 @@ Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searche
 | R8 `.fab` import drops the family gate | PLAUSIBLE | defect | `cmd/controller/main.go:3517-3530`, `stacks/family_gate.go:173-190` |
 | R9–R11, R13, R14 | PLAUSIBLE | defect / design (R13) | low |
 | R12 interrupted whole restore leaves `*.felhom-restoring`, every retry fails | CONFIRMED-CODE | defect | `tier2_whole.go:86-87` O_EXCL; removed only on an in-process error |
-| **D2** a missing / emptied app folder deletes its second-drive copy within one night | CONFIRMED-CODE | defect-shaped corner of a choice: 07 §8.2 „data legs are NOT guarded" is about a leg LEAVING the class set; this is a declared leg that is MISSING, which the code itself logs as a gap (`tier2_capture.go:55-61`) and then reconciles away (`tier2.go:454`) | P1 for optional legs (no other tier holds them) |
+| **D2** a missing / emptied app folder deletes its second-drive copy within one night | **CONFIRMED** for a MISSING folder (unit test, production code + real copy seam: `C3-D2-missing-leg/` — mirror of `photos` deleted, run „ok", the warning says only „nem kerültek a másodlagos mentésbe"); the EMPTIED case CONFIRMED-CODE (`rsyncMirror` = `rsync -a --delete`, `tier2.go:744-751`; the test seam does not delete) | defect-shaped corner of a choice: 07 §8.2 „data legs are NOT guarded" is about a leg LEAVING the class set; this is a declared leg that is MISSING, which the code itself logs as a gap (`tier2_capture.go:55-61`) and then reconciles away (`tier2.go:454`) | P1 for optional legs (no other tier holds them) |
 | D3 / T11 a host clock ahead silences the whole-guest tier; the hub reads the same clock | PLAUSIBLE (B6 #8 on the rehearsal VM is its live test) | defect | agent `localapi/server.go:1178-1190` |
 | D5 a dump that fails its own check has already replaced the good one | CONFIRMED-CODE | defect for the ORDER | `appbackup/dbdump.go:388-422`: size check, rename (`:399`), then `ValidateDump` |
 | D6 kept-data page can list a running app's folder | PLAUSIBLE | defect | needs an unreadable app.yaml |
@@ -0,0 +1,8 @@
 # go test at controller 7ca59e2 (v0.336.0), scratch clone; the real copyTree mirror seam (rsync --delete stands in by copyTree for the emptied case — see note); 2026-10-20T10:19:17Z
 === RUN   TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror
    zz_rehearsal_d2_test.go:22: POSITIVE CONTROL after run 1: mirror photo=true doc=true
    zz_rehearsal_d2_test.go:33: run 2 err=<nil>; status="ok" warning="Figyelmeztetés: a(z) app alkalmazás egyes adatmappái nem kerültek a másodlagos mentésbe: appdata/app/photos."
    zz_rehearsal_d2_test.go:34: FAULT after run 2: mirror photo=false doc=true  (the second-drive copy is the household's only local copy of these files)
 --- PASS: TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror (0.02s)
 PASS
 ok  	gitea.dooplex.hu/admin/felhom-controller/internal/backup	0.030s
@@ -0,0 +1,35 @@
 package backup
 // THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D2). Never committed.
 import (
 	"os"
 	"path/filepath"
 	"testing"
 )
 func TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror(t *testing.T) {
 	m, src, target, prov := newTier2V2(t, "app")
 	prov.has["app"] = true
 	prov.binds["app"] = []ClassifiedBind{mHDD("appdata/app/photos"), mHDD("appdata/app/docs")}
 	mustWrite(t, filepath.Join(src, "appdata", "app", "photos", "family.jpg"), "PHOTO")
 	mustWrite(t, filepath.Join(src, "appdata", "app", "docs", "contract.pdf"), "PDF")
 	if err := m.RunTier2("app"); err != nil {
 		t.Fatal(err)
 	}
 	destBase := filepath.Join(target, "backups", "secondary", "app")
 	photo := filepath.Join(destBase, "hdd", "appdata", "app", "photos", "family.jpg")
 	doc := filepath.Join(destBase, "hdd", "appdata", "app", "docs", "contract.pdf")
 	t.Logf("POSITIVE CONTROL after run 1: mirror photo=%v doc=%v", exists(photo), exists(doc))
 	// The household (or an app bug, or fsck → lost+found) loses the live photos folder; the docs folder is emptied.
 	if err := os.RemoveAll(filepath.Join(src, "appdata", "app", "photos")); err != nil {
 		t.Fatal(err)
 	}
 	if err := os.Remove(filepath.Join(src, "appdata", "app", "docs", "contract.pdf")); err != nil {
 		t.Fatal(err)
 	}
 	err := m.RunTier2("app")
 	cd := m.settings.GetCrossDriveConfig("app")
 	t.Logf("run 2 err=%v; status=%q warning=%q", err, cd.LastStatus, cd.LastWarning)
 	t.Logf("FAULT after run 2: mirror photo=%v doc=%v  (the second-drive copy is the household's only local copy of these files)", exists(photo), exists(doc))
 }
@@ -0,0 +1,306 @@
 Tue Oct 20 10:09:14 UTC 2026
 filebrowser Up About an hour (healthy)
 felhom-controller Up About an hour (healthy)
 paperless-webserver Up 3 hours (healthy)
 paperless-postgres Up 3 hours (healthy)
 paperless-redis Up 3 hours (healthy)
 immich-server Up 3 hours (healthy)
 immich-postgres Up 3 hours (healthy)
 immich-machine-learning Up 3 hours (healthy)
 immich-redis Up 3 hours (healthy)
 traefik Up 4 days
 0a4fb1a71c72cf704d1afd44e1ab24a8a68e11b8d82ef1dcb71a75d61a9dc1a3
 0ce36c2e6db54e085f71a1436c4c89a9807a930c373198f451fa1665f90f0cab
 0e2ca5e8738b95134285a666b1a7622bfcabc4728fb9269cc972dfaccbd59d45
 0f4002ab75695fb312775a8203cc5213b256f1b01d255210cb5cdd0e50cd9070
 1a8fe48409be3defe01713e2f621091630b965e1805cbb5872d1738043efd35c
 1afe3420067df65e0d3e53baaec3debc39d2d42d333c7a3c266e4f6ff98e04f7
 1b41940af5c4fa82af59320d6c81ba622d97f2e43d8ec51c50dcf65c2fdcb1e8
 1f5120663962c7b03539bb993ab736be4800096e3d1f7147eb922d44b308ae6a
 01feef94eb7e38a3679921b66cfe206e08adfd9b360ed4a60414cde606de1643
 2a71e17dfa1e930ffe0c213cd1615c8d1adcbbf8d61615a38064d331c265c02e
 2aaab84bf843817d9e6f32b673277fd1e0c2c8c2d038b8126df44927b1b41006
 2ac49fc9cc2839f55c8abae76039950d15a5c301ea5b3918fa3e5d8715a15f42
 2bbdb463b29493d60d352bafefccbe57856cf80bd2cc193b61f9bcc1310fb45e
 2bc95b104191ffd31bdfdbe1e038d284d0624c04735094494c2c77d957e8e905
 2dcb4bd2c3c618d71f582d33d5b1243e6ab075d3b00abc238d4430668a2b6f55
 2e4c79db25c47436b44245376b1f6ff1991208ba081fddce1938c3897fc63c07
 2ef8994a563ae88024ba29db4e349d26e6e3889e0683d30d1fb258208c9a10b4
 2f08e882c5d23373e2df5486dce28d4728330f9f0131a2735ebc938bfb4d7b3a
 02f42b55ee403ceef63a472c3624da0b48fab9e6aa6dc0c7800ec68b2f2041ef
 3a3d7bc33a1ba21ae4a39c8cb345254cf66cd1dc60afae64e2299513b094b93b
 3ac9820f84d0a50f233c63a62ee1b63b92acac026d68c60a8227f370b0be00b6
 3d9a93769342e83a3c512d517b99cd64f8e794eb305c009e25c73a6f6627ad29
 3d9c1f23abbcef771a4c38a75d75865a12d76a4359fe83c1828652dba3f91710
 3f015f020f49d4967d2ac71b94060e6eaec86bc8852cf9a0b5f3dd8133dd528c
 3fa030697dbaffb6809991bb69a335dc4e9accfafd4ed5ee953ed676673912e6
 4a9ed2aed1fe75a0b02f688e29a20acc2bf7e081ba7518a7d75451be30fdc4e1
 4bbaae5ae8cc8cc3330a73bb872b16e1420805197fa9bad903f7db9a1f358041
 4c54a4cda3c5280d5b13facf3089f8a0710c4570876aaeb679025aabf92df99f
 4e052b4ef5602a68eddcea049bbc4e79b306012cf8b7f0c9cfc3e6c39af64fce
 4ea780981a54c3e87aa72eb80436e02f3c6b3aaeab96ecf869602f5f5b0b546e
 4f2ebb779645de5cf9d24e93c6fbd9d95d358b390dec94eed6f0223016785e42
 4f2266bb0b8a9b902eaa439ae9225963d6fe4359fe30569c2d8afa926b582585
 004ae946f5997471f4554ab7158a3b982b72391b59c59be40ea79b501eb1dab9
 5b84763ca63a28ae79c1393d700c534e7dd6e610f890f540981e49972e3854e9
 5c7dd57f445d87a39eea1e7b299e1a0acbbc946d6bfc92ba165c4af8933b322a
 5d47a8950cfd56492e0f8b4ed6bfbf470a2e22811206ac0da0b79e520890b41e
 5def44013bf255d4cd97c3829ac7ae895c607333a3ee586a4090bda128d379fd
 5ec44fe3223976727132521b3055187e88783716a1f5fd07d1625205acff9888
 6a84846fd8e476d206eafb8f5a529ecbdf6b1021daab890429077e32ef4c2b6b
 6aef4d0808bef53a15fc6d19614b641a5e36c941f2729572cfef8cfc8e36bbfd
 6b5a55f08cc63514e70a93a3eb5d294e9b109e990521bb0ddcea4818a8df1456
 6b8acdd640ae067c2ee5235ddea8ad4c0e1eec2aaef1277b426923f56303e07d
 6bcaa587af8bf282f83605713213c20275b1c8178c28de97b154af25bb6a674f
 6bdf1f89b902e5a7134a87e0269229531ec6c8391256ea1bf466d2d6b66dd8c9
 6bf7012e27730a047bff7d69ceadef9c06433783d0b9582faefa9c9d0f966c86
 6ce965b12082d21ed8ade00884de6477faea9f97cf7a85e4b5ab3302d3517fce
 6ddb2a450b19b940661c677b20936e92e8de54b8bc76f8dbacc56398d7d13a38
 6e21c6398b13ebac045a11b136fbb2d289e6abe99401eed9108b5464754a4c87
 6e0521e4314af444375b56e348a5e3d34aceb88d2d5e2f48b6986c27250e395e
 6f19aa563c4e3e2000ed997157bc7f5bc8a45ee1d77c2cdcd1fd239117a62705
 7b5530f7f440ff6477f2ee5885b37ff9cfbf933d7908225998875b233b4d2a2a
 7bfe063b0155297f5bca22c7ed02ae7ef9ce1b968670607415f7720e1e8649f0
 7ef24d6bf9fb190db2572505b489e45731a326c5c9c4ae6e36ffd7874560bbab
 7fc41ddc47424ea11d4940f7062cdae69b92af645e6759f8759f47b689a19cb7
 8b44a3824395c930d0db784e5650ba7b58663090c3bd72a44f9dc716008a9642
 8bdcbc26d645ae3bc60742debe32650510bcfb7c69ad4416688af5b1815e5645
 8c2a7cf3f809218d20576368d4d419a63a07a89c3b12cc389fb2f3424fb45b73
 8c4d45c90f336c0ffd5d619765d8d2a45e802d619548f187bb3f2672b3534969
 8cae3c834dd9284d1d0069de042c2d50e49a138a78d93a59eb752272e4a5e4f4
 8d2ab3e77a72ebd708630e22f2eefc0a66f73cd94ed2abbfc4f666c504a3f172
 008dfd2a2e98c6b6673ee471d72b9e150cfe02859eb3a3bbd31a3c582b5a3a46
 9a2e6958ba1ea4443ade7619d326c84c50dbc444810b47904ff625969725a091
 9a86729a66d00d8be95f93062381b9bd91b16f94b654636681553fec6c7c7985
 9abdd18e6a7b07800067237c93d48448d49d9ad2388a91d35653323c2bed4389
 9c59fdcefb7ed258e615f8cb688334a69aebdb8e38fd1075a60d6fd98167065a
 9c033896f0b00023816a49cea882a342c9b62ca4528640969a0d22c21f507505
 9db05859a329f9544f5a48cbd987f02fd00e48f160dcf6ee8331d8accfb423b4
 9e9fa98cb2344c5eda9bae7417557f2bc2503de1eed8219fdc2d2d378e1883fe
 9e06437718bd8f9a27240eff7b425406aae50f061ea47a6991b9830ee2af9f59
 9ee6f261789b97ea0887a6a905bf5bba61a7070cc91ff64cda0091923d9cdf25
 09ea67268ce8c0442357ba316f6431cd96143da6bb54008d3c342be8b6ddb169
 13b744ec13f00bd3e5e2ecaa37bf5385af38cf0ff94fd05e660cfed3339a0c05
 13d76d014354ffb4e5aabfe00eeed5206b98c502dcfd515545fdc7210316bf61
 18e51d43a77ac282e108fd8a8e55207282cdabd2219af1e47c2c214117179f51
 20ee459190ea3442ab7cf0c288182033fb84b75a15bdfc14540e80ab60dec542
 22ab14c474cdb6cf9f873dcb6c8ce85dfc588abb46e2bd0925cdeff384437c46
 24a30fa240cec28f9a8b6d02a47ad1fda39674f094ae095b0d6ec3cbf099ed44
 24da0da42dc2eb9b6d35dcb7cd513b1e5c7d2ee38b6d59d9a1d51f0d01f6f920
 27a56be2953f040882ecd07fd8b91937f8cb17c25d1ceed56656ed2075518f86
 31f3b426743b1cff7952948995915586480486ed7fd06eef3379cf54b8fb3d9a
 38ba44b0ff38f8df925512a477b8b5b4cba216a8ec0bd51b7463a86a88c42fea
 43d6928bc4bb7a1e097122424e35f22393a0ccf2e03863d7dc3cce44163c8ae8
 49d0cebfc3199cef395ae27a6ffa479340c1c2f69da9248a777d7bff9b6b9ca9
 49dbe15539d7170822958bb0699d140c4d493215236b32679340588710d5452b
 054cfa9b87316799ecd24ecfb480a38aa04b5f85a478a3db95c41706e5b63523
 57e9dc8d409d939d877a513023c95146400dd10078919ab9402d8144d5f30ca0
 62e86d5d6aded4a8ba536696c0df0f0af89f2cd7b4b893a7383e3abc00eeadc5
 63ddb5c60e6cf39dea412903cf1d6c0164a1146bcf4ab98ee94c9b1f9e45c6bb
 63e2655474549ef03aaf7d9d741502287d2f01170c5ca008d7b09b8e00b1be66
 67eff4c5c60d493dc2a9f7841f7f138b610dc389c32793a27a62ddc13d417a1c
 68d18a8d3519b6ea6928bbca40f9db7500b700f66e6eab236e7c32a214de2cc7
 68f83c9613471fe0e046a1f9252b71e86aa50d36315c675e92dc44a7cdd0267e
 71cb6156444948894aec343c9a2c0c7674aca84d292d090c7f786d13146709ed
 72ed18f5962e48dd5fb54e7d91d2fd0479017c27b2c60fc58a5d6b6e55242757
 74c6b39b7b926dd56db16f175026c2bf9e80a912916f273c122227672f181d03
 075b39bc7ae8c56a6d99923e4938ddbbb4174df6f0e653abf9c99b420b401a31
 78c01e39d1997eb54344f02f14dc02c7230cfb14ba58abc1a11feab2e9af85be
 78e7acb9228c1df4ca4c706d459b97c8e79e8399fe06476ce72bd919a3b2b77f
 83a8287d1e8700aa7e207b2c617a68557ce6f83f373f2c3c6d1443ae50032826
 83bc388f3ae110e72ba51a8f0b68f7bbdec3685907c1ff08095b63a1759502af
 85d6f7309436e59cce9e373b0aa1626fa5e12d8d81163773f5f2c45da9d56543
 88c6573974e81f1fff96999bf54b80c66bc6f1e9af6746b3cd6801616198e0fd
 92e2e73e3aef6e5330d2ce8aed3c58e7e3632a6f191f2b03c997ccabd245bb65
 93c2b9d0a1f54c2e6355d206d93bf2309f81076628aae9d868406c288b0f6bf4
 98a80f5f7903ef45f92f74a91f34dc01d527f8a2fe98b4fbe76711f5259416a5
 99f7aaac2646c66dd0eeeebc157b8563b56299c27ab4bf64e13920d089ca7156
 104e16e4796c52fe212c04b06417161e0bab20fc01632bfcf71421b73b979fbc
 122fe66bacc9f04b3f3e4f6d0a3b15abdef8321eed36637a1284e3bd145463b8
 158c590779b033e48b5efcf895fd0bad4c6f277303d72b963cd0802ecac1f440
 218a186ab13a38d514cf9f523e5798452ae4e5ec76ecf26fe1a46ae22b47a508
 283e9d53742a13c471e5f59f53a57c179d461f0017ad5f8db27881cdd994cd04
 305b75817cf5c06105b19dc8c56ff0d796dc6e267ae5bcfc6018aa3d12037744
 325ec2aa2a6c307bb82d4dcb600fa3ff127aa6d1f35dc683ab61124ef5f0f5b3
 338f7535ae9ef91754e76604238ca7652273ffce9e6f0a1300de0b1edb61da77
 343d41a3b38b5a7eac10750eb0213b81780edd3075e9a1fdf3a2c479f4d2474c
 346eaddf98e91d6d6e7b8c526074c27847e4a18be01b41027bec93abc2b7b2fb
 355ba69bab94a95e00e87c6c4808587881b9675699083e087917ec8492bebc65
 426ccc08685def1c9025c94d343de0e59713073984634ee226e69634d6a18fb8
 552a4b340eb6be03c188cb17628aee0f916b544142465f8e52e7e9d2325206ab
 556c805068afd3f6784bda2eb59642d7dc03756580e959e1c75262711a30d4d9
 0642eb9eee222e235de9310b9bd86b42493e5f4d71d430619b5832484dd09cd6
 645d70ab53013564fd243eae61db209e3793b826a9eeffd77395b18e27458789
 670e6447721f667df5222cb06a68c2682eb92803184a44faead8aea0dff44df6
 0717b5f7615c345fe47ab8212c06f506e08585551d93c6a8083b56a216adb71b
 734e6b63f955d0b508dea39a8637755e75fbb353d54430956a0010f607eb727d
 748a1560ef6216c8bf80545dbeadd29f99fd5c984b6f34bb678febf8b369adcc
 748cce6aa136bdc0b3111c73a7c9434508200f8c231933b925d42b37b9cca17c
 760cd2f9203dc3dc3f933e4a34bbaae857dee0afed66e727846a4c7f750ab5a8
 762e15e05b2f84f035a7011e10c64558dc6637c9fdde1846dc77f4056f3962d2
 787dc06078f3718da6faaba82269a7d21b213482c9e45025315bdb6cf59010b5
 866fc46ac37407bbf74761cecd21fa6322c4e7621f3dfbe963a85671d5753e5c
 921d92a1c692ea0fd42cbe500bbdb1f3defcd5fb9b51df216451aa260d95ad45
 932bae2149e79ef113eb4929614a895c642960160d5f5cb2f2e9afac7dfa6af2
 968a5d7a54fb1d8ef4163f287e128cbd541ab46a2cc54db721dd3eafe1397b65
 991a5d344112325fb2c40700cd0ee96a784ce2150cba0a7b0be3ed6eb2411878
 2469a8c88a452ed226dd6d1082f0cbfbe3107d76f919b930b6bc7ce3b839ed90
 2634a60c3c37d5eba4be4827c08c0e99a951b934604c29d116fcddc27808e480
 2788da1d8f917a9d5b8acab2c7fe7336b6a7810f1c7edab63ed327a6022ceddf
 3000c7863b2e80a72329ca1fa0dd03d2fd3994a2391e82450a8c92199cbb56b6
 3461d48cd9e88caf5363b0e4d96e799f6759ab8b8bc6ba4c148492a1637c01f0
 4681b8c67ad20fdd3ad04db895084d37a9296ac76b74ea7bc9465bb3ac93df8f
 4690cfe7c61625190732c028ad4b918930a65e295005171d48114aa8f27b567b
 5206b60d49110c01e05dc24034e12b767efa739e2ad21c252ab83db47e56d526
 5234fcace55ee7e44631499ed1f95fd0289918a84bd9f5cf6d9f761d43c34f17
 07402fbe22b572302f3317958f8cc4c6ca30317e0ee5bbbe9ff51c60b3da5a05
 7458b4c4b975debbd2bb705e9575d2d1ff5e9308ed9e70c302f5643136cd0cc0
 8097f97f4ae00b654b1c0b4408ad585f5bfce7999e482f244bd698268e1f6061
 8158fe1f757d26cadb484ed9690ad4d20d4cb382eeec7c9b0627b365afd6576f
 9917d608ac0ad05856d31873432d641828ef519825a3f451e6a1ab2519dbf0d1
 13175cc97740c834e2404c11ba9fa9418d0c390d27a87e6e869977d6a86c20c0
 36571aa758bc335acc463c3ea074422a5123c309a37e1a7fc818a2069941c493
 59438a5f6378d218a73d3106ea1984d1c160fe4fcbbb7ec1177067f17430dcb3
 60496cd3a1e1ad1dcaed30427b70a57b8b14a122bae7f2cd882f8154a4549978
 61544bee8e2c1cd65e999121bb63011a9c1b4139b20b04fa32e3a850ca8c82b7
 84046f48a922cd337284bc2448a5d2682ea6a2e3479e71531122aff9ee4cbc93
 86022b0cd1043facf7b1c933c3b6a6ff36277a76d50bdb2da6a5690d57381b9e
 98129d711a1e807ab7c3ae4c97187ab8505cebf1012218e8530d0829a2ad436f
 177451abc7ed0606468250a7793e018725e8c7f370ee8937d4941a11571dcb9d
 0217869eb1be5181b7053b993443dac73d9cd0804cd5bedb604edeb43f911702
 219118eb01fa7be03fcce2124b74708cacafe66aea617fd2bafa32f0b2d4fd80
 394985d0c3db0791b8c7569fccabd36959d00deb285075f9852568c4c81d32eb
 722380f945884d4dea921ffb958513e02a4802483ec0886712e5ed3fa28d751c
 774878c58b6a8cae6fbecdf656f2c1f09f37a89190068663e923763dbd6969ce
 925226a74657c7ed0fcdbcfa9b504a47f7e21d0982c44f7ed9576140fb417146
 963079a9efb903a575b363b27e15927312569528b9066511b4c282e01e1ec931
 3830913f0478ec59a7a31d538f858f8ef612e1b2ce3bb9058a2d18f7b43abe9c
 8579968bcd362a1aa9dcf14e22956b2fd15fd74a2fc2e7369ffb1457c2ce6693
 8678479fa55ea3871598eb6914eb8428ba599da4c619c62f231e6931f0cb1f54
 13694424a3895e8e73cc1e06af29aae0c3179c3cb5587b9daadeadcdb29b7cc7
 67711784cb23efbc0762e83f03365b837ec5216c38f3ba682a82eaff6a7863db
 92236992ea73ecd9befe3330e88e96a7aedacd9cbca0bb8c32c31d0e2c914e5b
 134995217d29dc47fb0ca1a274ceb5ea74225d36a98c4bbe6e07bf8c8d6898c7
 481706314b835ab042583c3107e168932844a162a2adf3b9ef2fa27433a7fd1d
 4098005900d38c71b152787ce4bdedfb8e2b2b8b061f87645b2b4a14e0004335
 4699188665aa6eb965b7bb9f4938187f377e7a4ad55fc97e44802ea76f3612e9
 04762469942ae24c353f929fced777f245bb9f2f888a9f2044ff27fc45933665
 a6c297b9282b39caca7e513724219af096a3cf6fd453c3736ee7c4c833ef417d
 a22cb584ba3444bc1dbd90b3723b1ad9424352251f2218a92b1884ef74e63b9f
 a68fad1bcb1f1030af3811590cc18d74eb1da3c67190af4459fa69118aa8f035
 a74b9b2333ee936c17edd0525b49fec1337768ccab68f5576a39486c875ff968
 a74b41238ed9ded9e866679ab00f85813d800f92e4bb2642d3f9de28a1655b81
 a84ff849188b2bab211fefe8bc61a70f58f48f6a2a6ca25c2645e0bb5c384cbf
 a2815ec8af59e0a137aa0cdf2baa48ca6d1b86e291c5728fab9449280d15fa75
 aa5cf2d08c68b8d3dfe3330861935fd97ff8fb9169645f6c632d970ad71b575c
 aadb8f0be78890f1e08630b7a762a82853dc1402326bd16cd2a2844fa63fef1d
 ac48d92438e8d19cf546764cc051172af82092223daad8ccfbfb4cb05100f5d6
 ac78e0a5ac10c7d0239e0e1ba3b604cd6fd418d87a6f32b66f3ce98c674164d8
 ad79ebffc7d8225cfbee6ad5fec4092ba7376f38dca912d6df8ea932f1eba20b
 ad7084f002b17a1a27f70c796e3d7424a064b2ef5b71eadb7b3b08dcc9f59042
 ae2a94eac2960e95f944c9a4a0649c94c405fb0e518adfba2e6fa0f80f2a5af2
 ae9d5c15cba1adf881044a4449eaf72f5f2ed753c301edc1e43bcfa4560e8d31
 ae5499adb73000b6f460e63d6e045fe492702bd73187f868d9988690cb72a453
 b0e341cb8fb7f2fb42f7a8512a9b922591cfa40bd3522f31a422a9fdb4586008
 b4bc9ff2d9b533e896dfc5082ba865a4ea7676d71259d73aa33d515a5a60dd91
 b8d2fbe7e6afc931581bf5f1edacaefb9071bcc9f709e294a3f5ead104f73890
 b8dd8697ad52f98dc44ed3c940752aa52e16d6ab7563d36b1c9e221db22d1e11
 b21ec6b73a9f1f51623355f2bf8229b5f6f9da1484e1ce105cf8e0363475a5b8
 b081c3329ed821e22b0ee9cecbc2f52a5e9be70be2b1cb83bb262c21c84640d1
 b274ece5f6e1d506ca0646077964797c3e7b4413c9a9f1e453d9257e6b4d09f9
 b676fcf4f5272015b942c8589135a18fa16204c5aec457f1d0898ce6c84638eb
 b1591e2a2604b6404d0d18e4b2b7599aafbcfb2bbacba06b5de5ff14abf14be3
 b7000ea518cd327f7dfb42858434fdb80c451aecf0a0325b6ccbff1f30545803
 beb17a9d330ce361923600aac8e13f8b10e62d0b4067380ad7a95c4d77f0110e
 bfb91f5666855fd57228801ed0a4294981bd2a6665a93d1093bf3c4a0e89cb2c
 c0fb7358aabef2c38e6921e2ed06b7fdddddecd29d530814545968323eba964e
 c1d4fc53203c4b65abb1ff76f9505a46175c57afcd9c95cf445c6a3c443839fb
 c3fbcb924a928bca859b69c75e4df275eb5e0abf03029785247313b0fa1cdd27
 c4bccf1c99f4990b0a23848b6fa58977dea0bdbf6ab69b7c0c2b90f7f0a2f5c3
 c5e875c34f4e4f1fd4edb1c6dbf4862a6543e26387ee480eeba0f5e3f44af7b6
 c8bb03639a4d7c41c233fa28d5a1c06dee874fe00614f764df9db3df14da86b5
 c16aa92f949bb89a034c6ef22ea8ece859883ab309086a204e43cedced8dd059
 c31f2070f92b36708a62c1d53a7aea495fd997440f2c01c851ad83c66357e379
 c35e0b961694a882dcc4c7bdb8e657d4b0685687cb5c4469559ce5078c95302a
 c56e78b468dfe3991fc787ab7fe30b4282a73d7fd2e79e4dfc9305bb064685ea
 c67a888239bc5e11ed44ca31307f3b46f7f76698d706ffc764024110d1a69ceb
 c424d546895a59e0f88e2296a665c03f4618ac304ef19742b6f1c543665b9bb3
 c716ebc8ba8fa1556e1dddfaae98c7019618d39ece0e78dbd1b9f58dd0760d94
 c797d99e68b0efce9b1337dc0e3e9cbf42806bef03021ade362cb4e4d1de4222
 c5708e0a13ababf73203b1e297b3b73229cd6370e3fec1653b7cf1dc143a0d0a
 c52417cfa3d923ce7168b7c2b082eb3ab31bc1750ff3f3edfc2c1cd59fb74f3d
 c53655b36c7abc28db3020511a0ebdffe8fa37d86412753fca86f4caa7ae7a6b
 c782734d8be184e0721d0a984b6ac3e8e0052206a9e46d22188a7fb2f923face
 cb71ab036375acb25c69afd915261acf65f919008fbd59417989f6ebef110c7a
 cb613ceec1bf86b718059ef3e864dce7ed60e5b6a7bd5ab9f7c633350df48f3a
 cb10732ef556d77ec0a0a5d7b29c0789b47a9757b67bdb3fa9d5be22809debd0
 cf799a4b43f87d63eba6042a73b9eba3f17b354be1fd2d94dfac2048e13f3fce
 d3c941cd28b5a692c2532f1381afdc57740d603fbba5eddbe3539f561f26a0ca
 d5bd7d95fbb1f56d7617d96a5a6deb65417b4448692cb45f4a0bedd824165fc1
 d6a74fc39944848c79012bb06d915c609b92c1099881b90b4cda59074bd1124f
 d58b42f643175b6272bb579d354a9510bc34fc88e4e70dda08346997ba3cec13
 d256b9490d623a136093e459b9f466dedadaa0d00235d6934044e6591ee7e2c5
 d352bb11678cf6cb47a16740ee9b79cfbf4c19d4da23f4fb95621bbdea6b91b2
 d053441be51f569ac393c26c220b6369ae3455ae76f3627eba223e8b4e40eacd
 daa717bf6363a3afe8dbcd3ab863a81ca737e0999242bd99c618daa0e4fa085b
 db0c37c3ce9df2f2090c622eb4194e6860e26846398d4b4a092b9c4c21441af4
 db2434b8bc6fb7858aa5eea51b23eb3fe17ed8e8ac025596ebb4f857d00f9ba9
 db6910df9d6429a8ecffced74f22c26f45a11d48efea51e48e8097bf43331540
 dbcfeebe1162c457fdf6f7cbfff66c971dfaca8a1a8ae2fb5472fb9007f83ddc
 dc5c95c9a8f2f01a8eab63c7d591a7d008dbaf5f4b6157380faeeaeab168cd59
 de81766fb8e806f93c97771ae1a0ff529831bc5e7f50e2187d41dd606b8e23f1
 e0e25d3ed1797b0092c897ad904a4ac7a2bd77e12b96e1790a939aa009851803
 e0ee75c206f1b9be41e608c6de05f8fad8532e3bda240cfad0640c673ae46395
 e1ae949c5339eca43334dd6720b49df19082036b8f53151b8dfac97f2d2c0b6d
 e2ca0579185aa6d7324f144357323b4d5521c1e94e8dc7929f0ffa9a73e1e4cd
 e5f9a9a8f7e4b6d097aef1b0493180a547dbf19006920433e7b80aa8aac0026b
 e6a40ba40f9ac5f43ba2276b3cb6110bb4c0e20003e866a55ca28c954ddd0e9a
 e9e1b0601616187e8934c7212e2fde5b590b50308aa9c8c5dd12e419c6bebc18
 e45c71e0680d2856e2945881fd01ad6320868e0a36427c6bca637c53dfecc625
 e62af03dd8d599dfbdbecb8c6516d95e582ee803f2ddd3798479fbc769229030
 e72bfb25201770edb1889f678f5a844a57c9f5308acd57a86c5f4936b800d441
 e311fa91eca24a97e0c558e91d06dca353aa9348a7ea4b111e694e181868fc27
 e339a495e0c048e30483e4fd3bef66186712975f3c33c90ef63a3c9686da3160
 e668b1c1f44dae28ca34f530e313c1e544751b63bf0c480300be8e242683fecb
 e4638cbb5b12548a76a2cb6e87c7f2ada75d61d3c7d1062535ef0bf2b3cde05d
 e7945baeafe0ac6d9a1e6761786c35fddad76c81d86e544d369188c186a0389b
 ea35c704202ee36415d4f7a0873bcb254d30268450d7c9ed2dc611c1e2f4d5d1
 eaeb108b23d455d01ec0c9e44d39ff395522942794d26f3c149e29615be9bf1f
 eb420ae07f9c459abc101c1264fa8cf779002ef20748ef97abe7c3764c24f032
 ec30ca07d0b0af16afe60f217dedf15573045ed9adc45e21c3592ee6716fbe39
 ed02b9c56e48d34910bc97ebff834f1cb082c23ebcae97b0f51f3a2633664466
 edcc77bb3d279e3dfe7d687fe30e22f95f5d9391b9908e61b2206a6cf309ea91
 eea963e8499939d36d09e42d8d5cd86798e81e264c4d4c012191e42926e508b8
 f1b01ee5efb2117033ddcad5f5eafe8e6f65ad6fc3e933902bfc494c13a7d3c0
 f01b151ee48f9a3a631457069ece3a2ac3e111df28110f3c06f221ee7c54eff8
 f2ec30f9f836ed26cb13fdb80064d1dcabf953d1952fea2d85bb6410f4cb2271
 f8a87cab6bc0890c7a2e36f73e533eac1a1cabb13b886b8ed71afa4c8b7d90be
 f9b0fe025aedb49e94d80119a8a8dc0f85800bc592e568dc95b0f1e5439c54fc
 f56a1ab986e83e118ff174e9805f6151e47ff693c17a2f475fcedd83cb98e051
 f82a956615af08264b64e051e09c1441152d92371d4d2db8d5c0ac4ab47a7182
 f287b03a768126d72af210dceab12e329573779e7aad56c123ac0dbb14fc4f4e
 f347e779fb8c09daba2a757dfc3ee7c415f8f5a2eaace19535d363cc3ab3171a
 f547c232f5e1785aa11200ac03dcfda6f056400bf1456a3388db6d73a8158ca0
 f6822a239614fc82462535db5b2806dd616696c9552e1d469922f4b7cd2bf56e
 fa453f6fb451a564d31e704bbca0bcbcc9ad1109a25a8e5ad5c704789a424c4b
 fb5ad60698efb6ee9f45fd440e56c060ba01ba03a854f556e42f8090b2057c7e
 fbc407de7b12a4975b60dac518ac287f076659213cce708871162342419ab383
 fbf633eb758b7b66ab761c34eeebc759791e2f73020e839b3cfc2d692c9ba21f
 fc1bddcbef8c02474db71bf11dace70e1e51d94f3c3fe6d9d45f4a669ac4a9fd
 fd453dd47dc82ff7625bd4093c62140fe14a348814f094b43b519f7f59293af7
 fdc05b3bfda3101bec7d66b5e0cfb589e06f1f4b4d6b2f9d84dac470b0abf53b
 fdebde6f8bcdd5281c50bcb4669c45dc54e33563bec872f8d23d76d2fe3ebfdf
 fe503d3e9a34b113d4c56b59c5986e13c44b6e401b6e6681eb62712d24368329
 felhom-controller-data
 filebrowser_filebrowser_data
 immich_immich_ml_cache
 immich_immich_postgres_data
 immich_immich_redis_data
 paperless-ngx_paperless_data
 paperless-ngx_paperless_postgres_data
 paperless-ngx_paperless_redis_data
 privatebin_privatebin_data
 ls: cannot access '/mnt/felhom-drives/scratch_hdd/felhom/backups/primary/': No such file or directory
 Filesystem      Size  Used Avail Use% Mounted on
 /dev/loop1       69G   11G   54G  17% /var/lib/felhom
 /dev/nvme0n1    938G  210G  681G  24% /mnt/felhom-drives/scratch_hdd
@@ -0,0 +1,104 @@
 10:10:26Z == 1 deploy
 10:10:27Z deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
 10:11:12Z state: running
 10:11:42Z == 2 marker A + backup
 10:11:46Z CREATE TABLE
 INSERT 0 1
 10:11:49Z [2] DB markers: 'A'
 10:11:49Z POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'}
 10:12:59Z backup finished: {"db_dump": {"count": 3, "duration": "1m9.055678483s", "last_run": "2026-10-20T10:12:58.690358712Z", "success": true}, "enabled": true, "running": false}
 10:13:04Z unit on disk:
 UNIT /mnt/sys_drive/felhom-data/backups/primary/rallly
 2026-10-20 10:12 487 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/app.yaml
 2026-10-20 10:12 4071 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/docker-compose.yml
 2026-10-20 10:12 11894 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/.felhom.yml
 2026-10-20 10:12 52228608 /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar
 2026-10-20 10:11 111371 /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql
 2026-10-20 10:12 3195 /mnt/sys_drive/felhom-data/backups/primary/rallly/manifest.json
 2026-10-20 10:12 1191 /mnt/sys_drive/felhom-data/backups/primary/rallly/data-stamps.json
 10:13:04Z == 3 POSITIVE CONTROL: marker B (live) then a normal unit restore
 10:13:09Z INSERT 0 1
 10:13:12Z [3-before] DB markers: 'A,B'
 10:13:12Z [3] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
 10:13:12Z [3] POST /backup/restore -> HTTP/2 302  ['location: /backups/restore?flash=flash.restore.started']
 10:13:12Z [3]   restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "offbox-reconstitute", "stack": "rallly", "ok": true, "message": "A(z) rallly: 0 fájl és 1 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 20. 09:30) — az alkalmazás újraindult.", "finished_at": "2026-10-20T07:36:57.852528534Z"}}
 10:13:40Z [3]   restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true}
 10:14:00Z [3] after: state=running hold=None
 10:14:18Z [3-after] DB markers: 'A'
 10:14:18Z == 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half
 10:14:22Z INSERT 0 1
 10:14:25Z [4-live] DB markers: 'A,C'
 10:14:29Z TRUNCATED /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar 52228608 -> 26114304
 10:14:29Z == 5 the unit restore over the damaged copy
 10:14:29Z [5] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
 10:14:29Z [5] POST /backup/restore -> HTTP/2 302  ['location: /backups/restore?flash=flash.restore.started']
 10:14:29Z [5]   restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true}
 10:14:44Z [5]   restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": false, "message": "Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1\nstderr: ...twork rallly_rallly-internal Creating \n Network rallly_rallly-internal Created \n Network rallly_rallly-internal Created \n Container rallly-postgres Creating \n Container rallly-postgres Created \n Container rallly Creating \n Container rallly Created \n Container rallly-postgres Starting \n Container rallly-p
 10:15:04Z [5] after: state=degraded hold=None
 10:15:38Z [5-after] DB markers: 'Error response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running'
 10:15:41Z containers:
 rallly Created
 rallly-postgres Restarting (1) 5 seconds ago
 10:15:44Z db container log tail:
 Data page checksums are enabled.
 initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty
 initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker".
 The files belonging to this database system will be owned by user "postgres".
 This user must also own the server process.
 The database cluster will be initialized with locale "en_US.utf8".
 The default database encoding has accordingly been set to "UTF8".
 The default text search configuration will be set to "english".
 Data page checksums are enabled.
 initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty
 initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker".
 10:15:47Z any undo / safety copy of the live state on disk?
 /mnt/sizing-bench/backups/secondary/paperless-ngx/recovery-unit/db-dumps/pre-restore-20260927T094411Z-paperless-ngx-postgres.sql
 /mnt/sys_drive/felhom-data/backups/primary/rallly
 /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar
 /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql
 10:15:51Z controller log (restore lines):
 Container rallly Created 
 Container rallly-postgres Starting 
 Container rallly-postgres Started 
 Container rallly-postgres Waiting 
 Container rallly-postgres Error dependency rallly-postgres failed to start
 dependency failed to start: container rallly-postgres is unhealthy
 2026/10/20 10:14:42 manager.go:1362: [ERROR] [stacks] Stack rallly start failed after 1.1s: exit code 1
 stderr: ...twork rallly_rallly-internal Creating 
 Network rallly_rallly-internal Created 
 Network rallly_rallly-internal Created 
 Container rallly-postgres Creating 
 Container rallly-postgres Created 
 Container rallly Creating 
 Container rallly Created 
 Container rallly-postgres Starting 
 Container rallly-postgres Started 
 Container rallly-postgres Waiting 
 Container rallly-postgres Error dependency rallly-postgres failed to start
 dependency failed to start: container rallly-postgres is unhealthy
 2026/10/20 10:14:42 handlers.go:1881: [ERROR] [web] Restore failed (async): stack=rallly: starting rallly after restore from unit: starting stack rallly: exit code 1
 stderr: ...twork rallly_rallly-internal Creating 
 Network rallly_rallly-internal Created 
 Network rallly_rallly-internal Created 
 Container rallly-postgres Creating 
 Container rallly-postgres Created 
 Container rallly Creating 
 Container rallly Created 
 Container rallly-postgres Starting 
 Container rallly-postgres Started 
 Container rallly-postgres Waiting 
 Container rallly-postgres Error dependency rallly-postgres failed to start
 dependency failed to start: container rallly-postgres is unhealthy
 2026/10/20 10:14:49 manager.go:920: [DEBUG] [stacks] restart-policy of down member "rallly" = "unless-stopped"
 2026/10/20 10:14:59 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
 2026/10/20 10:15:09 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
 2026/10/20 10:15:09 main.go:4054: [INFO] [deadapp] rallly crossed the crash_loop threshold (8 in 10m0s) while a backup/quiesce/update holds it — NOT stopped
 2026/10/20 10:15:19 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
 2026/10/20 10:15:29 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
 2026/10/20 10:15:39 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
 2026/10/20 10:15:49 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
 exit 0
@@ -0,0 +1,18 @@
 2026/10/20 10:14:29 handlers.go:1873: [WARN] [web] Restore requested (async): stack=rallly, snapshot=helyi from 172.18.0.3:36322
 2026/10/20 10:14:29 restore_dbcreds.go:114: [INFO] [backup] Restore rallly: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106)
 2026/10/20 10:14:29 restore_unit.go:393: [INFO] [backup] Restoring rallly from recovery unit /mnt/sys_drive/felhom-data/backups/primary/rallly: images=2, secrets recovered=2/2, data_keys=0
 2026/10/20 10:14:29 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
 2026/10/20 10:14:40 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.8s)
 2026/10/20 10:14:40 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
 2026/10/20 10:14:40 restore.go:246: [ERROR] [backup] Failed to populate volume rallly_rallly_postgres_data: tar: short read — exit status 1
 2026/10/20 10:14:40 restore_unit.go:448: [ERROR] [backup] volume restore for rallly: failed to restore 1 volume(s): [rallly_rallly_postgres_data]
 2026/10/20 10:14:40 [INFO] [stacks] SaveAppConfig: saved config for rallly
 2026/10/20 10:14:40 deploy.go:883: [INFO] [stacks] Redeploying rallly from recovery unit with 4 env vars
 2026/10/20 10:14:40 pin.go:93: [INFO] [stacks] pin rallly: rallly=lukevella/rallly:4.15.3, rallly-postgres=postgres:18-alpine
 2026/10/20 10:14:40 [INFO] [stacks] SaveAppConfig: saved config for rallly
 2026/10/20 10:14:40 restore_unit.go:467: [ERROR] [backup] rallly: NOT replaying the database copy — the volume restore failed, so the database is not known to be the copy's own
 2026/10/20 10:14:40 manager.go:1351: [INFO] [stacks] Starting stack: rallly
 2026/10/20 10:14:42 manager.go:1704: [ERROR] [stacks] Command failed: docker compose up -d (in /opt/docker/stacks/rallly) — exit code 1 (took 1.1s)
 2026/10/20 10:14:42 manager.go:1712: [ERROR] [stacks] stderr (last part): ...twork rallly_rallly-internal Creating 
 2026/10/20 10:14:42 manager.go:1362: [ERROR] [stacks] Stack rallly start failed after 1.1s: exit code 1
 2026/10/20 10:14:42 handlers.go:1881: [ERROR] [web] Restore failed (async): stack=rallly: starting rallly after restore from unit: starting stack rallly: exit code 1
@@ -0,0 +1,28 @@
 200
 {
 "ok": true,
 "message": "Stack rallly stop completed"
 }
 200
 {
 "ok": true,
 "data": {
  "removed": "rallly",
  "volumes_removed": [
   "rallly_rallly_postgres_data"
  ],
  "hdd_paths_removed": [],
  "hdd_paths_preserved": [],
  "hdd_note": "Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott nem volt mit törölni.",
  "userdata_kept": [],
  "backup_paths_removed": [
   "/mnt/sys_drive/felhom-data/backups/primary/rallly (26M)"
  ],
  "verified": true
 },
 "message": "Stack rallly removed"
 }
 0
 0
 ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/rallly': No such file or directory
 /opt/docker/stacks/rallly
@@ -0,0 +1,25 @@
 # C3 — R1/D4 proven live: a failed LOCAL restore leaves a broken app and no copy of what it replaced
 Scratch guest 9202 on demo-hp, controller **0.331.0** (the local-restore code is unchanged since v0.328.0 —
 `git log` on `restore_unit.go`/`restore.go`; 0.336.0 runs the same lines). Hub off on 9202. The dashboard password was
 reset with the controller's own escape hatch (`--print-reset-code` → `/claim`), held in `sec` only.
 Architecture read: `07` §6.3 (the ladder „replay → rollback → hold"), §8; `09` §3 decision 154 (R-638 A), D20 (R-893 A).
 | step (UTC) | act | observed |
 |---|---|---|
 | 10:10 | Rallly 4.15.3 + postgres:18 deployed through `POST /api/stacks/rallly/deploy` | running |
 | 10:11 | marker A (psql in the app's own DB container); „Mentés most" (`POST /api/backup/run`) | unit written 10:12 (`rallly_rallly_postgres_data.tar` 52 MB + `rallly-postgres.sql`) |
 | 10:13 | **positive control:** marker B, then the household's restore (`POST /backup/restore`, point „helyi") | „1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult"; markers **A** (B rewound, as designed) |
 | 10:14:22 | marker C — the household's newest row | live: **A,C** |
 | 10:14:29 | **fault:** the unit's DB volume tar truncated to half (a torn / damaged copy) | 52 228 608 → 26 114 304 B |
 | 10:14:29 | the same restore | stop 10.8 s → `volume rm` → `tar: short read` → `[ERROR] volume restore … failed` → definition rewritten → „NOT replaying the database copy" → start → **postgres crash-loops** (`initdb: directory … exists but is not empty`) |
 | 10:14:44 | what the household reads | „Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1 stderr: ...twork rallly_rallly-internal Creating …" — **raw English Docker output** |
 | 10:15:47 | any copy of the live state (marker C)? | **none** — no undo dump, no live-volume copy for Rallly (`find` of `*restore-undo*`/`pre-restore-*` finds only a September paperless file) |
 | 10:15 | the dead-app guard | `rallly crossed the crash_loop threshold … while a backup/quiesce/update holds it — NOT stopped` |
 **So:** the household's live data (marker C) was destroyed by the restore itself, the app does not start, and nothing
 on the box can put the pre-restore state back. The off-site restore got exactly this protection in v0.336.0 (D20 = A:
 „put back exactly as it was"); the LOCAL path — 07 §3's main customer restore — did not. Logs: `01-run.log`,
 `02-controller-log-fault-window.txt`. Teardown through the product (`03-teardown.txt`): Rallly removed with its data
 and backups; no container, no volume, no unit (the `/opt/docker/stacks/rallly` folder is the catalog template every
 app has).
@@ -0,0 +1,153 @@
 #!/usr/bin/env python3
 """r1_proof.py — C3 proof of candidate R1/D4 on scratch guest 9202 (data-loss hunt 2026-10-20).
 Claim under test: a LOCAL (Tier-1 unit) restore that fails after it has started replacing the app's volumes
 restarts the app on whatever is left, and keeps NO copy of the live state it replaced — so the household's newer
 data is gone and nothing can put it back (unlike the off-site path, which got „put back exactly" in v0.336.0).
 Method (mechanism proof — markers are written with psql inside the app's own database container, as the
 2026-10-19 R-893 measurement did; the household-facing acts are the dashboard's own endpoints):
  1 deploy Rallly (throwaway) through POST /api/stacks/rallly/deploy
  2 marker A; „Mentés most" = POST /api/backup/run (whole box, scratch); wait
  3 POSITIVE CONTROL: marker B (live), unit restore (POST /backup/restore) -> A present, B gone (expected rewind)
  4 marker C (the household's newest live row); THE FAULT: truncate the unit's DB volume tar to half
  5 unit restore again -> observe: the answer, the app's state, what the DB holds, any undo/safety copy on disk
  6 teardown: remove Rallly with its data and backups through the product; verify nothing left
 Env: CTL_BASE, CTL_DOMAIN, CTL_PW_ENV (see tools/ctl.py). Prints a timestamped log (no secrets).
 """
 import json, os, re, subprocess, sys, time
 sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/rehearsal-2026-10-20/tools")
 import ctl  # noqa: E402
 APP = "rallly"
 DOMAIN = os.environ["CTL_DOMAIN"]
 def say(*a):
    print(time.strftime("%H:%M:%S", time.gmtime()) + "Z " + " ".join(str(x) for x in a), flush=True)
 def guest(script, timeout=600):
    r = subprocess.run(["ssh", "demo-hp", "export LC_ALL=C; cat > /tmp/r1p.sh; pct push 9202 /tmp/r1p.sh /tmp/r1p.sh;"
                        " pct exec 9202 -- bash /tmp/r1p.sh; pct exec 9202 -- rm -f /tmp/r1p.sh; rm -f /tmp/r1p.sh"],
                       input=script, capture_output=True, text=True, timeout=timeout)
    return (r.stdout + r.stderr).strip()
 def psql(sql):
    return guest(f"""C=$(docker ps --format '{{{{.Names}}}}' | grep -E '^rallly.*(postgres|db)' | head -1)
 U=$(docker exec $C printenv POSTGRES_USER); D=$(docker exec $C printenv POSTGRES_DB)
 docker exec -i $C psql -v ON_ERROR_STOP=1 -U "$U" -d "$D" -tA <<'SQL'
 {sql}
 SQL""")
 def stack():
    code, d = ctl.call("GET", f"/api/stacks/{APP}")
    return (d.get("data") or {}) if isinstance(d, dict) else {}
 def wait_backup(t_cap=1200):
    t0 = time.time()
    time.sleep(5)
    while time.time() - t0 < t_cap:
        code, d = ctl.call("GET", "/api/backup/status")
        dd = (d.get("data") or {}) if isinstance(d, dict) else {}
        if not dd.get("running"):
            return dd
        time.sleep(5)
    return {"timeout": True}
 def restore(tag):
    code, d = ctl.call("GET", f"/api/backup/snapshots?stack={APP}")
    data = d.get("data") if isinstance(d, dict) else None
    snaps = data if isinstance(data, list) else next((data[k] for k in ("snapshots", "items", "restore_points")
                                                      if isinstance(data, dict) and isinstance(data.get(k), list)), [])
    say(f"[{tag}] restore points offered: {json.dumps(snaps, ensure_ascii=False)[:400]}")
    sid = snaps[0].get("id") or snaps[0].get("snapshot_id") or snaps[0].get("short_id")
    out = subprocess.run(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", ctl.HOST, "-H", "Cookie: " + ctl._SESS,
                          "-X", "POST", "--data-urlencode", "_csrf=" + ctl._CSRF, "--data-urlencode", "stack_name=" + APP,
                          "--data-urlencode", "snapshot_id=" + sid, ctl.BASE + "/backup/restore"],
                         capture_output=True, text=True).stdout
    say(f"[{tag}] POST /backup/restore -> {out.splitlines()[0] if out else '?'} "
        f"{[l for l in out.splitlines() if l.lower().startswith('location')][:1]}")
    t0, last = time.time(), None
    while time.time() - t0 < 900:
        code, d = ctl.call("GET", "/api/backup/restore-status")
        dd = (d.get("data") or {}) if isinstance(d, dict) else {}
        cur = json.dumps(dd, ensure_ascii=False)[:600]
        if cur != last:
            say(f"[{tag}]   restore-status: {cur}")
            last = cur
        if not dd.get("running") and time.time() - t0 > 8:
            break
        time.sleep(3)
    time.sleep(20)
    st = stack()
    say(f"[{tag}] after: state={st.get('state')} hold={st.get('hold_reason')!r}")
 def markers(tag):
    say(f"[{tag}] DB markers: {psql('SELECT string_agg(m, \',\' ORDER BY m) FROM r1_marker;')!r}")
 def main():
    say("== 1 deploy")
    if not stack().get("deployed"):
        code, d = ctl.call("GET", f"/api/stacks/{APP}/deploy-fields")
        fields = (((d.get("data") or {}).get("metadata") or {}).get("deploy_fields")) or []
        vals = {"DOMAIN": DOMAIN, "SUBDOMAIN": "r1poll"}
        for f in fields:
            if f.get("env_var") in vals:
                continue
            if f.get("type") == "password":
                import secrets
                vals[f["env_var"]] = "R1-" + secrets.token_hex(10)
            elif f.get("required") and f.get("default"):
                vals[f["env_var"]] = f["default"]
        code, d = ctl.call("POST", f"/api/stacks/{APP}/deploy", {"values": vals})
        say("deploy ->", code, str(d)[:160])
        for _ in range(90):
            time.sleep(5)
            st = stack()
            if st.get("deployed") and (st.get("app_config") or {}).get("pinned_images") and st.get("state") in ("running", "unhealthy", "degraded"):
                break
        say("state:", stack().get("state"))
        time.sleep(30)
    say("== 2 marker A + backup")
    say(psql("CREATE TABLE IF NOT EXISTS r1_marker(m text primary key); INSERT INTO r1_marker VALUES ('A') ON CONFLICT DO NOTHING;"))
    markers("2")
    code, d = ctl.call("POST", "/api/backup/run")
    say("POST /api/backup/run ->", code, str(d)[:200])
    say("backup finished:", json.dumps(wait_backup(), ensure_ascii=False)[:500])
    unit = guest(f"for d in /mnt/felhom-drives/*/backups/primary/{APP} /mnt/sys_drive/felhom-data/backups/primary/{APP}; do "
                 f"[ -d $d ] && echo UNIT $d && find $d -maxdepth 3 -type f -printf '%TY-%Tm-%Td %TH:%TM %s %p\\n'; done")
    say("unit on disk:\n" + unit)
    say("== 3 POSITIVE CONTROL: marker B (live) then a normal unit restore")
    say(psql("INSERT INTO r1_marker VALUES ('B');"))
    markers("3-before")
    restore("3")
    time.sleep(15)
    markers("3-after")
    say("== 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half")
    say(psql("INSERT INTO r1_marker VALUES ('C');"))
    markers("4-live")
    say(guest(f"for d in /mnt/felhom-drives/*/backups/primary/{APP} /mnt/sys_drive/felhom-data/backups/primary/{APP}; do "
              f"[ -d $d ] || continue; f=$(find $d -name '*.tar' -path '*volume*' | grep -iE 'postgres|db' | head -1); "
              f"[ -n \"$f\" ] || f=$(find $d -name '*.tar' | head -1); s=$(stat -c %s \"$f\"); "
              f"truncate -s $((s/2)) \"$f\"; echo \"TRUNCATED $f $s -> $(stat -c %s \"$f\")\"; done"))
    say("== 5 the unit restore over the damaged copy")
    restore("5")
    time.sleep(30)
    markers("5-after")
    say("containers:\n" + guest("docker ps -a --format '{{.Names}} {{.Status}}' | grep -i rallly"))
    say("db container log tail:\n" + guest("C=$(docker ps -a --format '{{.Names}}' | grep -E '^rallly.*(postgres|db)' | head -1); docker logs --tail 15 $C 2>&1"))
    say("any undo / safety copy of the live state on disk?\n" + guest(
        "find / -xdev \\( -path /proc -o -path /sys \\) -prune -o \\( -iname '*restore-undo*' -o -iname 'pre-restore-*' \\) -print 2>/dev/null | head; "
        "for d in /mnt/felhom-drives/*/backups /mnt/sys_drive/felhom-data/backups; do find $d -maxdepth 4 -iname '*rallly*' -newer /etc/hostname 2>/dev/null | head; done"))
    say("controller log (restore lines):\n" + guest("docker logs --since 30m felhom-controller 2>&1 | grep -iE 'restore|rallly' | grep -vi 'GET /api' | tail -40"))
 if __name__ == "__main__":
    main()
@@ -151,13 +151,21 @@ stopping line that lies.
 | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
 | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
-## Backup & restore — 26 rows (P2 3, P3 11, P4 12)
+## Backup & restore — 41 rows (P1 2, P2 9, P3 18, P4 12)
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
 | **R-1175** | Backup & restore | P1 | **Shared folders on two drives that use the same second drive: every night one drive's shares lose their second copy, and the night a drive dies its share copy is deleted.** The shares second-drive job runs one group per source drive; each group's clean-up walks EVERY drive key on the target and removes what is not its own (`internal/backup/tier2_shares.go:201-230`, `:266-267` → `tier2.go:299-329`). PROVEN against production code (fake copy seam): shares on hdd_2 and hdd_3 both target hdd_1 — run 1 logs `removed stale dest dir hdd_2/b_share`, so after EVERY run only the last group's shares have a copy while the status says ok; then hdd_3 goes away — run 2 logs `removed stale dest dir hdd_3/csaladi_fotok`: the dead drive's only second copy is deleted that night. Data-loss hunt 2026-10-20 (C1 reviewer 2, C2, C3: `audits/dataloss-hunt-2026-10-20/C3-D1-shares-tier2/`). | OPEN | — | Fix: reconcile only the group's OWN key dir (and keep a key whose drive is away/decommissioned until the household releases it); test with the C3 cases as red-proofs; release by the normal train | CC |
 | **R-1176** | Backup & restore | P1 | **A box rebuilt after „delete the old host first" cannot reach the household's off-site history: the recovery code is never asked for, a new key is minted, and the old history is set aside on night one.** D19 (`hub/internal/api/passphrase_scope.go:58`) makes the reinstall stop until the operator deletes the old host; the delete demotes the escrow under the OLD host id and deletes the host row (`hub/internal/store/store.go:3164-3229`); the new box's status query joins on live hosts only (`store.go:3499-3527`), so it reports no package and no retained package (both flags false) → the controller mints a fresh repo password (`offbox.go:542-555`), offers no recovery (`:1788`) and on its first night sets the old repository aside (`:905-906`); the retained-package API 404s for a deleted host (`handler.go:1691`). The reset and purge paths DO read `host_deletions` (`customer_reset.go:48,155`; `store.go:4189`) — the status path does not. PROVEN at the hub store against production code (`audits/dataloss-hunt-2026-10-20/C3-K1-escrow-after-host-delete/`: control = present; after delete + new host = nil). 03 §8a and 07 §7.0 promise this route; D19 + R-1154 removed it without a recorded choice. Data-loss hunt 2026-10-20 (C1 reviewer 3). | OPEN | the rehearsal's B7 is the live proof (needs Tester4 off-site ON) | Decide the shape (status query includes `host_deletions` for the same customer, and the retained GET answers for a deleted host of the caller's customer); red-proof with the C3 test; hub + controller release by the normal train | CC |
 | **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/` **LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest | — | — | operator |
 | **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):** `_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the first verdict lands at 07:10 hub time (05:10 UTC), after the night session that reached it ended; that brief did not include this read. Nothing is lost: the hub keeps its listing; read the 2026-10-20 07:10 lines on 10-21 (or later on 10-20). | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC |
 | **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). | operator: print the sheet | Operator: print + store the sheet; then close | operator |
 | **R-1177** | Backup & restore | P2 | **A local restore that fails destroys the household's live data and keeps no copy of it: the app is left broken.** PROVEN LIVE on scratch 9202 (`audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/`): Rallly + Postgres; control restore OK; then a live row C, the unit's DB volume tar damaged (half), the household's restore → the live volume removed, `tar: short read`, the DB replay correctly skipped, the app started → Postgres crash-loops (`initdb: directory … exists but is not empty`); row C gone; NO undo or safety copy anywhere; the household reads „Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1 stderr: …" (raw English Docker output). Mechanism `internal/backup/restore_unit.go:428-489`, `restore.go:215-246`. The OFF-SITE path got „put back exactly as it was" in v0.336.0 (D20 = A, R-893); the LOCAL path — 07 §3's main customer restore — did not. Two reviewers found it independently (R1, D4). | WAITING-ON-OPERATOR | decision: extend D20's put-back (safety dump + live-volume copy before the first change, rollback on failure) to the local and second-drive unit restores? | Operator: answer the D-question in STATUS; then CC builds it with the R-893 harness | operator |
 | **R-1178** | Backup & restore | P2 | **The second drive's „Teljes visszaállítás" can change files, then refuse the database half, leave the app STOPPED, and tell the household „the app and its data are untouched".** `tier2_whole.go:301-340` stops the app and merges the files first, then calls the unit restore, whose refusals (cut-off dump, mixed versions, allow-list, data key — `restore_unit.go:366-423`) all return before its own stop/start. PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-R5-whole-restore/`): 1 file restored + 1 replaced (the live one kept beside), calls `[stop]`, no start, message „…A visszaállítás biztonsági okból nem indult el — az alkalmazás és az adatai érintetlenek." Data-loss hunt 2026-10-20 (C1 reviewer 1). | OPEN | — | Fix: run the unit's pre-checks BEFORE the file half (or start the app and say what changed on a refusal); red-proof with the C3 test | CC |
 | **R-1179** | Backup & restore | P2 | **A drive marked „away" can still be chosen as the second-drive backup target.** `tier2.go:151-180` loops `GetSchedulableStoragePaths()`, which drops only decommissioned drives (`settings.go:1593-1603`); nothing checks `Disconnected`. Seen in the R-1175 unit test (a share copied to a drive marked disconnected). On a real box the away drive's mount point is an empty folder on the guest's root disk (or the host's root via the bind), so the night's copy fills the wrong disk and is not where the household thinks. CONFIRMED in code; not run live. Data-loss hunt 2026-10-20. | OPEN | — | Fix: skip a disconnected target like a decommissioned one; live check on a scratch box with a removed drive | CC |
 | **R-1180** | Backup & restore | P2 | **The nightly second-drive mirror (`rsync --delete`) can run DURING a restore from that same mirror and delete files not yet copied back.** `RunAllTier2` (`tier2.go:479-520`) takes no backup flag and does not ask about a running restore; the whole restore holds only the backup flag (`tier2_whole.go:301`), the night leg only `LegLock` (`cmd/controller/main.go:1348-1352`). A long restore of a photo library at 03:30, or 15 min after a missed night, overlaps. CONFIRMED in code; not run live. Data-loss hunt 2026-10-20 (C1 reviewer 4, T1). | OPEN | — | Fix: the mirror leg skips (and says so) while any restore runs; test with the existing tier-2 seams | CC |
 | **R-1181** | Backup & restore | P2 | **A missing or emptied app folder deletes its second-drive copy within one night; for OPTIONAL folders (audiobooks, comics, ROMs, Immich's external library) that copy was the only one.** A missing declared leg is skipped (`tier2_capture.go:55-61`, silent for optional legs) and then reconciled away (`tier2.go:454`); an emptied one is mirrored with `rsync -a --delete` (`:744-751`). PROVEN for a missing folder against production code (`audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/`): the mirror's `photos` deleted, run „ok", warning only „nem kerültek a másodlagos mentésbe". 07 §8.2 chose „data legs are not guarded" for a leg that LEAVES the class set; a declared leg that is MISSING is not that case. Optional legs are in no off-site tier (07 §6.2). | WAITING-ON-OPERATOR | decision: keep a missing leg's mirror (and say so) until the household releases it? And: does an accidental folder deletion need a second-drive history for optional legs? | Operator: answer in STATUS | operator |
 | **R-1182** | Backup & restore | P2 | **Abandoning ONE set-aside off-site copy deletes EVERY retained key package of the customer, including ones that open other set-aside copies still stored.** hub `api/handler.go:626-627` → `PurgeSupersededEscrowForCustomer` (`store.go:4185-4190`) deletes customer-wide; the comment right above (`:4176-4179`) states the invariant this breaks. PLAUSIBLE (code read by reviewer and main session; not run). Data-loss hunt 2026-10-20 (C1 reviewer 3, K2). | VERIFY | — | Prove with a hub store test (two superseded rows, one abandon); then purge only the package that covers the abandoned path | CC |
 | **R-49** | Backup & restore | P3 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker.** **READY 2026-10-20 — measured and planned, nothing changed; the decision is the operator's (STATUS D21).** Re-measured on 9202 (Immich v3.2.4): off-site ~1.33 GB = model cache 824 MB (62 %), raw database folder 283 MB (21 %), Immich's own backups 115 MB (9 %), logical dump 54 MB, originals 49 MB, thumbnails 2 MB — 102 MB must survive. A: leave out all three re-fetchable parts (−92 %; needs a never-run restore from the dump alone on the vectorchord image). B: only the model cache (−62 %; needs one restore with an empty cache). Pick: B now, A's other parts after their proof (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`). | — | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app | CC |
 | **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC |
 | **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC |
@@ -181,6 +189,13 @@ stopping line that lies.
 | **R-1171** | Backup & restore | P3 | **demo-hp's LOCAL whole-guest copy is never restore-tested: the box keeps one copy (keep-daily 1) and the restore test wants a copy 24 h old, so the only copy turns 24 h old exactly when the next night replaces it.** Read 2026-10-20 on demo-hp (agent journal): every 6-hourly evaluation since 2026-10-18 says `local: no settled archive yet — nothing to prove`; the local storage holds one 9201 archive; the last local proof was 2026-10-14 03:42 (`settle=24h`, `keep_last=1`, `prune=keep-daily=1`). The hub's `restore_test_stale` (168 h) will alarm from ~2026-10-21 03:42. A real household with retention 1 would be the same. | OPEN — filed 2026-10-20 | demo-hp `journalctl -u felhom-agent` („restore-test evaluated"), the hub's host report `restore_tests` | Agent: a tier that keeps ONE copy proves its newest archive once it is older than the settle time measured from the BACKUP's end, not 24 h — or a settle below the cadence (e.g. 6 h) for a 1-copy tier; a test; then read the next evaluation | CC |
 | **R-1172** | Backup & restore | P4 | **After option A (R-893): a failed off-site restore that was PUT BACK is logged at ERROR, and an empty `backups/restore-undo/` folder stays on disk.** Measured on 9202 2026-10-20 (rc.1): the web handler logs `[ERROR] [web] off-box reconstitute rallly (async): … Nem változott semmi…` — good news at ERROR; the put-back removes the per-app folder but not the empty `restore-undo/` parent. Small; 0.336.0 was already pushed (a version is pushed once). | OPEN — filed 2026-10-20 | `audits/night2-2026-10-20/P2/live/11-controller-log-window.txt`, `12-after-failed-restore.txt` | Next controller release: the put-back outcome logged at WARN (its own sentence is the household's), the empty parent removed; a test | CC |
 | **R-1173** | Backup & restore | P3 | **R-921's consequence (no app stop for an off-site tier the agent will refuse) has not been seen live.** Agent 0.161.0 serves `host_busy` (proven live on demo-hp 2026-10-20 during a press: busy `backup:local`), and controller ≥ 0.335.0 reads it — but the case only arises on the night schedule (a manual press backs up the local tier only), so the no-stop branch is proven by its red-proved test alone. | OPEN — filed 2026-10-20 | `audits/night2-2026-10-20/P1/1b-host-busy-poll.txt` | Read demo-hp's controller log after the night its off-site (pbs) tier is due (~2026-10-22): a `host is busy … no app is stopped (R-921)` line, or the off-site tier starting with nothing in flight | CC |
 | **R-1184** | Backup & restore | P3 | **A cut-off shared-folder restore can be „placed" and reported restored while files are missing; the download is then deleted.** Readiness is only „the scratch folder is not empty" (`internal/backup/shares_restore.go:131-140`) — the R-358 class fixed for apps, not for shares; 30-min download window (`web/offbox_handlers.go:716`); fixed 2 GiB free-space gate that fails open (`:107`). CONFIRMED in code. Data-loss hunt 2026-10-20 (C1 reviewer 1, R3). | OPEN | — | Fix: a completion marker after restic succeeds, as for apps | CC |
 | **R-1185** | Backup & restore | P3 | **A database dump that fails its own content check has already replaced last night's good dump, and the second drive copies it an hour later.** `appbackup/dbdump.go:388-422`: size check, rename over the good file (`:399`), then `ValidateDump` (`:419`). Volume tars are promoted with no content check either (`backup.go:902`). Tier 1 is one point per app by design (07 §6.1); the ORDER is the defect. CONFIRMED in code. Data-loss hunt 2026-10-20 (D5). | OPEN | — | Fix: validate the temp file before the rename; keep the last good dump on a failed check | CC |
 | **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC |
 | **R-1187** | Backup & restore | P3 | **`.fab` export/import candidates from the data-loss hunt, not yet proven:** (a) an export silently drops a drive folder that holds a symlink (`appexport/fabplan.go:292`, `export.go:714`); (b) the import never sets file owners and drops symlinks (`restore.go:1157-1176`); (c) import into a STOPPED app merges into the existing volumes, and the default export (stop ticked) carries no SQL dump (`restore.go:361-374`, `export.go:262,366`, `appbackup/dbdump.go:142`); (d) no concurrency guard, no undo, no record (`restore.go:181-186`, `web/handler_export.go:312-353`); (e) the import drops the family gate, the sign-up lock and the version pin (`cmd/controller/main.go:3517-3530`, `stacks/family_gate.go:173-190`); (f) it trusts the bundle's HDD_PATH (`restore.go:589-631`); (g) the DB import runs psql without ON_ERROR_STOP and misses postgis (`restore.go:1052-1053, 928-937`). Read by reviewer 1; main session spot-checked (a), (e). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Prove (e) and (c) first on 9202 with a throwaway app; split into rows only what reproduces | CC |
 | **R-1188** | Backup & restore | P3 | **Restore-path candidates from the data-loss hunt, not yet proven (one confirmed in code):** (a) a crash or power cut mid-restore restarts the app on a mixed state and never reads the undo copy (`appstop_marker.go:307`, `offbox_restore_undo.go:137`, `bootrecon/bootrecon.go:210-213`); (b) a failed `volume rm` is ignored before the untar (`restore.go:218`); (c) CONFIRMED in code: an interrupted second-drive whole restore leaves `<file>.felhom-restoring`, and every retry fails „file exists" (`tier2_whole.go:86-87`, O_EXCL); (d) the full off-site download has a 30-min cap per press and the household is not told to retry (`web/offbox_handlers.go:455`); (e) place and full restore re-read the LATEST snapshot instead of the downloaded one (`offbox_restore.go:675-679`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Prove (a) with the B6 power-cut-mid-restore scenario; (c) with a kill test on 9202 | CC |
 | **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt, not yet proven:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it, and the hub reads the same clock (agent `localapi/server.go:1178-1192`, `backup/runner.go:607-629`, hub `monitor/deadline.go:243`; R-973 cap counts future runs `backup_cap.go:61`); (b) the whole-guest start creeps ~5 min/night until a night is skipped silently (`server.go:1184`, `quiesce.go:150, 807-818`); (c) the whole-guest quiesce stops apps mid-restore or mid-update (`quiesce.go:213-275, 565`); (d) the off-site proof and integrity check never run on a box that is off at 05:30–06:00 (`cmd/controller/main.go:1575-1619`, comment says the opposite); (e) hub false „no DB dump" for a late or evening backup time (`monitor/deadline.go:416-419, 556-565`); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — design question; (g) spring forward 2027-03-28 fires dump and second copy together; (h) self-update busy check before the pull only (`updater.go:532-545, 764`); (i) a double-clicked restore shows failed while it runs (`web/handlers.go:1853-1874`). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`. | VERIFY | — | (a) is B6 #8's live test; (b) read the agent journal on demo-hp; the rest by unit tests | CC |
 | **R-1190** | Backup & restore | P3 | **Key and retention candidates from the data-loss hunt, not yet proven:** (a) a recovery ceremony while off-site is not configured seals a package with no repo password and supersedes the good one (`escrow_handlers.go:217`, agent `main.go:2975-2981`, hub `store.go:3392`); (b) the global „create your recovery code" bar links to the ceremony the backups page hides during a recovery offer (`escrow_banner.go:23-33` vs `backups_remote.html:144-160`); (c) after a reuse-path reinstall the new PBS key K is never escrowed and nothing compares fingerprints (`felhom-pbs-apply:99`, `escrow_confirm.go:182-205`); (d) whole-guest off-site copies run before any escrow exists (design question vs 07 §6.1's Tier-3 gate); (e) the escrow PUT is two statements; (f) the operator's off-site re-issue resets the provider password before saving it (hub `offsite.go:217-245`); (g) „Megőrzött adatok" can list a running app's live folder when its app.yaml is unreadable (`kept.go:220-235, 330-402`); (h) a unit restore can point an app back at an old drive and F5 then prunes the current drive's unit (`backup.go:1517-1574`). Evidence `audits/dataloss-hunt-2026-10-20/C1-keys-candidates.md`, `C1-delete-retention-candidates.md`. | VERIFY | — | Unit tests first for (a), (e), (g) | CC |
 ## Storage & devices — 9 rows (P3 3, P4 6)
@@ -252,7 +267,7 @@ stopping line that lies.
 | **R-1170** | Monitoring & notifications | P3 | **A box whose tunnel container disappears raises no alarm, even when its tunnel is configured.** The hub alarms on `not_running` only (`checkTunnel`). A guest with no cloudflared container read `unknown` since Docker 29 (lower-case „no such object", R-1070) and reads `absent` with the next agent release — neither alarms. On a box whose customer has a tunnel token, a missing container means the box is unreachable from outside. Found 2026-10-19 while fixing R-1070. | OPEN — filed 2026-10-19 | `hub/internal/api/handler.go` `checkTunnel` | Hub: treat `absent` as down when the customer's config carries a tunnel token (two reports in a row, as today); a test | CC |
-## Hub & operator — 11 rows (P3 4, P4 7)
+## Hub & operator — 12 rows (P3 5, P4 7)
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -260,6 +275,7 @@ stopping line that lies.
 | **R-31** | Hub & operator | P3 | **[P2-HIGH] Offsite provisioning is synchronous with no status affordance.** Save runs the Hetzner sync in-request, so the request can hit the nginx 504 **while succeeding server-side**: the operator cannot tell failed from slow, and a retry races the first attempt. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: operator-only; a known workaround (click once, wait, verify) exists.** **2026-10-06 night: the race half fixed on felhom.eu main (hub, unreleased):** a second Save while the first still provisions is refused with 409 („already running — wait about a minute, then reload; do not save again"); nothing saved, nothing created; per customer, in memory. `TestProvision_R31_*`, red-proof `audits/night-burndown-2026-10-06/hub/R-31-red.txt`. LEFT: the async save with a status card (the escrow-card idiom). | — | Direction: make it async + a status card, reusing the proven **awaiting-card/poll idiom** (v0.138.0 escrow card). **Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click.** | CC |
 | **R-882** | Hub & operator | P3 | **Longhorn on DooPlex could not grow a volume online: its `instance-manager` (116 days up) called a host process that no longer existed** — `nsenter: cannot open /host/proc/196610/ns/mnt` on every expansion retry, and an offline growth was blocked by the expansion's own attachment ticket (found 2026-10-05 growing `hub-data` to 2 Gi). A restart of the instance-manager (operator-approved) fixed it: 77/77 volumes back `attached/healthy` in 110 s. **Why the cached PID went stale was not established** (likely a containerd/k3s or iscsid restart after the instance-manager started), so it will recur after the next such restart and stay invisible until a volume needs to grow. `audits/hub-db-offsite-2026-10-05/partA/step1-*.txt` | **OPEN** | — | Find which host process the PID was and whether Longhorn 1.10.x re-resolves it; until then, before growing any volume, check the instance-manager's age against the last k3s/containerd restart | operator |
 | **R-883** | Hub & operator | P3 | **8 DooPlex workloads run an image by a moving tag (`:latest` or none), so any pod restart is a silent upgrade.** Measured 2026-10-05: the Longhorn restart restarted zipline on `ghcr.io/diced/zipline:latest` (pull Always), which pulled 4.8.0; 4.8.0 refused its database (`cannot safely migrate from prisma to drizzle: expected migration 20260508022000 … was not applied`) and crash-looped. Fixed for zipline by pinning `4.7.0` (homelab-manifests `90f60e4`, `4c8ec7a`; 4.7.0 applied the four missing migrations; a dump from before is kept out-of-band). The other 7 were counted, not named or changed. **Checked from source 2026-10-05 (burn-down round 2):** homelab-manifests@87dfc29 still has moving tags (grep image lines without a numeric tag): admin-system/toolbox.yaml:12 nicolaka/netshoot:latest (a bare Pod); calibre-system/cwa.yaml:826 calibre-web-automated:dev; outline-system/outline.yaml:270 minio/minio:latest; tandoor-system/recipe-importer.yaml:26 gitea.dooplex.hu/admin/recipe-importer:latest (pull Always :27); adventurelog-system/adventurelog.yaml:100 and :256 adventurelog-backend/frontend:latest (pull Always :101,:257); jarrs-system/jarr- | **OPEN** | — | List the 7 (`kubectl get deploy,sts -A` images without a fixed tag), pin each to the running version, and let Renovate move them | operator |
 | **R-1183** | Hub & operator | P3 | **On Sunday 2026-10-25 (the clocks go back) every hub daily job after 02:00 runs TWICE: the alarm check at 04:00 and 05:00, the hub database snapshot twice (keep 2 → the previous day's copy pruned a day early), the prune and the key audit twice.** `hub/cmd/hub/main.go:1057-1062` adds 24 h to an absolute time; the controller fixed this (`scheduler.go:511-516`), the hub did not. PROVEN by replaying the hub's exact arithmetic (`audits/dataloss-hunt-2026-10-20/C3-T9-hub-dst/replay.txt`). On 2027-03-28 each runs once, an hour late. No architecture document mentions DST. Data-loss hunt 2026-10-20 (T9). | OPEN | the frozen run (no hub deploy unattended, decision 162) | Expect duplicate operator mails on Sunday morning; fix with the controller's day+1 `time.Date` and a DST test; deploy with the operator | CC |
 | **R-264** | Hub & operator | P4 | **Twenty-one facts the boxes report that the hub can now decode nowhere, each allowlisted with a reason rather than silently skipped — and for these the reason is "no consumer today, and one is arguably owed".** Split out of R-260 on 2026-08-08 so that closing the CLASS (gated) and fixing its sharpest instance (`operator_key_configured`) could not be mistaken for having decided what the hub should do with the rest. **The list, grouped by what a consumer would be for.** **(a) Guest-network health — `guest_net` and its seven children** (`checked_at`, `has_route`, `dhclient_alive`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`). The R-54 watchdog reports per-guest network state and self-heal counts every cycle and the hub — the component that emails the operator — models none of it. There is a live incident in this project's own record where a killed `dhclient` took a tunnel down for 1 h 15 m (`audits/INCIDENT-guest-dhclient-killed-2026-07-20.md`); a recurring-heal signal is exactly what would have surfaced it. **This is the strongest candidate of the twenty-one.** **(b) `selfupdate_pending` + `selfupdate_pending_version`** — an agent that has flipped its binary and never committed reports pending on every heartbeat so that "the operator sees WHY the version isn't advancing", and no operator can see it. **(c) `mgmt_plane.healed_recently`** — bounded: the hub DOES alarm on the `privsep_healed_at` timestamp beside it, so the recurring-clobber signal is not lost, only this flag. **(d) `restore_tests.mount_parity` + `mount_inventory`** — R-262's subject; the verdict is not lost (a mismatch fails before `Pass` is set) but the hub cannot tell a full-fidelity pass from a boot-only one. **(e) `pbs_dr.applied_at`.** **(f) Controller-side: `config_hash`, `reporting_disabled`, `stacks`, `storage.migrated_to`, `backup.last_db_dump`, `backup.last_integrity_check`** — the last two are backup-integrity timestamps, which is the "presence is not success" neighbourhood. **For each the question is the same and is NOT answered here: is it wanted? If the hub should act on it, model it and name what consults it. If it should not, the honest end is that the emitter stops sending it** — a fact emitted forever and consumed nowhere is a future false green waiting for someone to write a check against it. **Deliberately not decided in the G-1 session**, whose scope was the gate plus the operator-access instance; unilaterally removing emitters would also break the byte-identical cross-repo host-report golden and is a coordinated two-repo change. **⚠ DISPOSITIONS RECORDED 2026-08-13 — and the first thing to say is that they were NOT in this register before today.** The rulings were made on 2026-08-12; this row still read **READY — owner Viktor** and the gate's twenty entries still all said *"arguably owed"*, so a session told to *"re-read the dispositions from the register"* would have found none. They are written down now, which is the point of writing them down. **THE COUNT WAS ALSO WRONG:** this row says *twenty-one*; the gate's allowlist held **twenty**, measured. Twenty is the number the dispositions below account for, exactly. **(1) BUILD A READER — four groups, fourteen facts.** (a) guest-network health, (b) the staged-update-pending pair, (d) the restore-test depth pair, (f-part) the two backup-integrity timestamps. **(2) NO READER WANTED — five facts, now recorded as `not consumed, DELIBERATELY` with the ruling and its date** in `scripts/wire_contract_gate.py`, each with its own reason rather than a bare refusal: `mgmt_plane.healed_recently` (the hub already alarms on the timestamp beside it), `pbs_dr.applied_at` (`pbs_dr.state` is the verdict; the timestamp alone is the attempt-read-as-result trap), `config_hash` (the hub authors the config and knows its own generation), `stacks` (the app view is built from the purpose-built `app_telemetry` wire), `storage.migrated_to` (box-local bookkeeping with no hub-side intent to reconcile against). **The emitters are deliberately left alone** — removing one is a coordinated two-repo change and breaks the host-report golden; the honest end here is a recorded decision, not a deletion. The gate grew a THIRD entry kind to carry them, because an undecided fact and a decided one must not read alike. **(3) `reporting_disabled`, decided on its own merits: RECLASSIFIED `redundant`** — `health.status = "disabled"` travels in the same minimal report, is decoded into `reports.health_status`, and IS rendered. **The decision surfaced a real defect the flag would not have fixed: the staleness checker is age-only, so a deliberately-silent box still alarms → R-321.** **PROGRESS, 2026-08-13: the first reader is BUILT — guest-network health, R-319.** Its eight allowlist entries are **removed** (an allowlisted tag is skipped, so leaving them would have meant the new reader's fields were never checked); the gate's checked-tLine truncated
 | **R-451** | Hub & operator | P4 | **[P3-LOW] UPDATE ARC SLICE 7 — a fleet sweep: the operator can SEE, and MOVE, how far behind every box is.** Slices 1 and 2 make one box's state visible on that box's own pages. The operator has no fleet view, and **it is not derivable from what is already reported: the hub's report payload carries container name, state, CPU and memory, and NO image field at all** (spike §5, which is why Peti's box could only be recorded UNKNOWN). So this is a hub-side change as well as a controller one. Rank LOW today because the fleet is two enrolled boxes; it rises with the fleet. `architecture/09-update-architecture.md` §6, §8.4 **BOTH SIDES VERIFIED 2026-09-21, and it is cheaper than this row implies.** The controller's payload carries no image (`internal/report/types.go` L98–103) and the hub's `Store.SaveReport` (`hub/internal/store/store.go:965`) denormalises only container **counts** — but **the hub stores the raw report JSON whole**, so a new controller field lands there the day it is sent. What is missing is the denormalisation and the page, not the transport. Shape in `09` §6.2–6.3; the payload question is `09` §3b **Q7**. **-- RULED 2026-09-23 (`09` §3 decision 18):** the report carries, per compose service (database included), the installed reference, the catalog reference and the badge state. **Built later, when the fleet grows** — Q7's recommendation, confirmed. | **DEFERRED** (2026-10-03 triage: the row's verdict was finished, but it names open work no other row carries — the fleet sweep (report field, hub denormalisation, fleet page) is ruled and not built — deferred until the fleet grows (`09` §6.3)) — **RULED — build deferred until the fleet grows; owner: CC** | — | — | CC |
 | **R-814** | Hub & operator | P4 | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | **VERIFY** (2026-10-03 triage: a July watch row with no id; given R-814. WAITING-ON-OPERATOR — no record found that the box was deleted.) — WAITING-ON-OPERATOR | operator console | Delete the box | operator |