From bffe2d9dea8e090cee464422a329705bfe786d2d Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 20 Oct 2026 12:31:34 +0200 Subject: [PATCH] =?UTF-8?q?Data-loss=20hunt=202026-10-20=20C4:=20register?= =?UTF-8?q?=20153=20->=20169=20(opened=20R-1175..R-1190:=20two=20P1=20?= =?UTF-8?q?=E2=80=94=20shares=20second=20copy=20pruned=20by=20another=20dr?= =?UTF-8?q?ive's=20group,=20box=20rebuilt=20after=20the=20host=20delete=20?= =?UTF-8?q?cannot=20reach=20its=20off-site=20history;=20R-1177=20proven=20?= =?UTF-8?q?LIVE=20on=209202=20=E2=80=94=20a=20failed=20local=20restore=20d?= =?UTF-8?q?estroys=20live=20data=20with=20no=20copy);=20STATUS:=20the=20P1?= =?UTF-8?q?=20line=20and=20decisions=20D22,=20D23?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- STATUS.md | 4 +- .../dataloss-hunt-2026-10-20/C2-verdicts.md | 4 +- .../C3-D2-missing-leg/go-test-output.txt | 8 + .../C3-D2-missing-leg/zz_rehearsal_d2_test.go | 35 ++ .../C3-R1-local-restore-9202/00-as-found.txt | 306 ++++++++++++++++++ .../C3-R1-local-restore-9202/01-run.log | 104 ++++++ .../02-controller-log-fault-window.txt | 18 ++ .../C3-R1-local-restore-9202/03-teardown.txt | 28 ++ .../C3-R1-local-restore-9202/README.md | 25 ++ .../C3-R1-local-restore-9202/r1_proof.py | 153 +++++++++ documentation/backlog/OPEN-ITEMS.md | 20 +- 11 files changed, 700 insertions(+), 5 deletions(-) create mode 100644 documentation/audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/go-test-output.txt create mode 100644 documentation/audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/zz_rehearsal_d2_test.go create mode 100644 documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/00-as-found.txt create mode 100644 documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/01-run.log create mode 100644 documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/02-controller-log-fault-window.txt create mode 100644 documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/03-teardown.txt create mode 100644 documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/README.md create mode 100644 documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/r1_proof.py diff --git a/STATUS.md b/STATUS.md index 21fddf2b..defc0a5e 100644 --- a/STATUS.md +++ b/STATUS.md @@ -15,6 +15,7 @@ - The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15). ## What is broken +- **P1 (the data-loss hunt, 2026-10-20 — still running):** shared folders on two drives lose their second copy every night, and a dead drive's copy is deleted the same night (R-1175); a box rebuilt after „delete the old host" cannot reach the household's off-site history (R-1176). - **P1:** about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours). - Two root-level weaknesses wait on your decisions D3 and D4 (R-985, R-982). - Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). @@ -22,7 +23,6 @@ - The scratch box cannot use agent features (R-1083) or read the drill catalog (R-992). ## What we work on -- The long rehearsal and data-loss hunt on a fresh box (another CC session, 2026-10-20 and 21). Its findings become rows. - Dated checks, run by CC. 2026-10-21: the hub's own off-site check (R-895), the weekly clean-up window (R-435), downloads out of the second-drive copy (R-1019). 2026-10-23: no app stop when the host is busy (R-1173). - A gate keeps this page short and true: it warns now and refuses from 2026-10-27 (R-1174). CC. @@ -32,6 +32,8 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| +| D23 | R-1181 | A missing app folder deletes its second-drive copy within one night. Keep the old copy until the household lets it go? | **Yes, keep it** (a deleted folder is the reason a backup exists; audiobooks, comics and ROMs have no other copy) | Keep as now: a deleted folder is gone from every local copy by the next morning | an accidental folder delete is unrecoverable after one night | — | +| D22 | R-1177 | A LOCAL restore that fails destroys the live data and keeps no copy. Give it the off-site restore's „put back exactly" (your D20)? | **Yes** (proven live today: a failed restore left the app dead and the newest data gone) | No: a failed local restore can lose the household's newest data | the main restore button can lose data when a backup copy is damaged | — | | D21 | R-49 | Immich's off-site copy: leave out only the AI model files now (B), or also the raw database folder and Immich's own backups (A)? | **B now** (−62 %, no new risk); A's other parts after their proof | A: −92 %, but it needs a restore path we have never run | every Immich box sends about 12 times its real data off-site each night | — | | D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — | | D8 | R-206 | Install DooPlex's Docker settings from git (Docker restarts once), with a weekly build-cache prune? | **Yes, in a quiet window** (a rebuilt DooPlex then keeps the settings) | No: the live settings stay hand-made, and a rebuild loses them | git and the host can drift apart | — | diff --git a/documentation/audits/dataloss-hunt-2026-10-20/C2-verdicts.md b/documentation/audits/dataloss-hunt-2026-10-20/C2-verdicts.md index b4107545..0b7e7428 100644 --- a/documentation/audits/dataloss-hunt-2026-10-20/C2-verdicts.md +++ b/documentation/audits/dataloss-hunt-2026-10-20/C2-verdicts.md @@ -17,7 +17,7 @@ Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searche | D1b an AWAY drive is still chosen as the second-drive target | CONFIRMED-CODE (+ the same unit test: `hdd_3/iskola => hdd_1/...` after hdd_1 was marked disconnected) | defect | `tier2.go:151-180` loops `GetSchedulableStoragePaths()` (`settings.go:1593-1603` excludes only Decommissioned); no Disconnected check. On a real box the away drive's empty mount point is on the guest's root disk | | **K1** reinstall after „delete the old host" loses the route to the old escrow | **CONFIRMED** at the hub store (unit test against production code, `C3-K1-escrow-after-host-delete/`); the controller half (new key minted, history set aside on night one) CONFIRMED-CODE | defect (03 §8a / 07 §7.0 promise the route; D19+R-1154 removed it unrecorded) | hub `store.go:3164-3229` demotes into `host_escrow_superseded` under the OLD host_id and deletes the `hosts` row; `GetEscrowStatusForCustomer` `store.go:3499-3527` joins on `hosts` → nil for the new box; `customer_reset.go:48,155` and `store.go:4189` DO include `host_deletions`. Live proof needs Tester4 off-site ON (B7) | | K2 abandoning one set-aside copy purges every retained package | PLAUSIBLE | defect (breaks `store.go:4176-4179`'s own invariant) | `store.go:4185-4190` customer-wide delete read true | -| **R1/D4** a failed LOCAL unit restore: no copy of the live state, app started anyway | CONFIRMED-CODE → live proof running (`C3-R1-local-restore-9202/`) | **design question** for the operator: decision 154 (R-638 A) chose „start anyway"; D20 (R-893 A) gave „put back exactly" to the OFF-SITE path only | `restore_unit.go:428-489` | +| **R1/D4** a failed LOCAL unit restore: no copy of the live state, app started anyway | **CONFIRMED LIVE** on 9202 (`C3-R1-local-restore-9202/`): live row destroyed, DB crash-loops, no undo copy, raw English Docker text to the household | **design question** for the operator: decision 154 (R-638 A) chose „start anyway"; D20 (R-893 A) gave „put back exactly" to the OFF-SITE path only | `restore_unit.go:428-489` | | R2 a crash mid-restore starts the app on a mixed state; the undo copy is never read at start | PLAUSIBLE | defect | `appstop_marker.go:307`, `offbox_restore_undo.go:137` | | R3 a cut-off shares restore can be placed and reported restored | CONFIRMED-CODE | defect (R-358 class, fixed for apps, not shares) | `shares_restore.go:131-140` readiness = non-empty dir | | R4 `.fab` export drops a drive folder holding a symlink | PLAUSIBLE | defect | `appexport/fabplan.go:292`, `export.go:714` | @@ -27,7 +27,7 @@ Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searche | R8 `.fab` import drops the family gate | PLAUSIBLE | defect | `cmd/controller/main.go:3517-3530`, `stacks/family_gate.go:173-190` | | R9–R11, R13, R14 | PLAUSIBLE | defect / design (R13) | low | | R12 interrupted whole restore leaves `*.felhom-restoring`, every retry fails | CONFIRMED-CODE | defect | `tier2_whole.go:86-87` O_EXCL; removed only on an in-process error | -| **D2** a missing / emptied app folder deletes its second-drive copy within one night | CONFIRMED-CODE | defect-shaped corner of a choice: 07 §8.2 „data legs are NOT guarded" is about a leg LEAVING the class set; this is a declared leg that is MISSING, which the code itself logs as a gap (`tier2_capture.go:55-61`) and then reconciles away (`tier2.go:454`) | P1 for optional legs (no other tier holds them) | +| **D2** a missing / emptied app folder deletes its second-drive copy within one night | **CONFIRMED** for a MISSING folder (unit test, production code + real copy seam: `C3-D2-missing-leg/` — mirror of `photos` deleted, run „ok", the warning says only „nem kerültek a másodlagos mentésbe"); the EMPTIED case CONFIRMED-CODE (`rsyncMirror` = `rsync -a --delete`, `tier2.go:744-751`; the test seam does not delete) | defect-shaped corner of a choice: 07 §8.2 „data legs are NOT guarded" is about a leg LEAVING the class set; this is a declared leg that is MISSING, which the code itself logs as a gap (`tier2_capture.go:55-61`) and then reconciles away (`tier2.go:454`) | P1 for optional legs (no other tier holds them) | | D3 / T11 a host clock ahead silences the whole-guest tier; the hub reads the same clock | PLAUSIBLE (B6 #8 on the rehearsal VM is its live test) | defect | agent `localapi/server.go:1178-1190` | | D5 a dump that fails its own check has already replaced the good one | CONFIRMED-CODE | defect for the ORDER | `appbackup/dbdump.go:388-422`: size check, rename (`:399`), then `ValidateDump` | | D6 kept-data page can list a running app's folder | PLAUSIBLE | defect | needs an unreadable app.yaml | diff --git a/documentation/audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/go-test-output.txt b/documentation/audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/go-test-output.txt new file mode 100644 index 00000000..06a20735 --- /dev/null +++ b/documentation/audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/go-test-output.txt @@ -0,0 +1,8 @@ +# go test at controller 7ca59e2 (v0.336.0), scratch clone; the real copyTree mirror seam (rsync --delete stands in by copyTree for the emptied case — see note); 2026-10-20T10:19:17Z +=== RUN TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror + zz_rehearsal_d2_test.go:22: POSITIVE CONTROL after run 1: mirror photo=true doc=true + zz_rehearsal_d2_test.go:33: run 2 err=; status="ok" warning="Figyelmeztetés: a(z) app alkalmazás egyes adatmappái nem kerültek a másodlagos mentésbe: appdata/app/photos." + zz_rehearsal_d2_test.go:34: FAULT after run 2: mirror photo=false doc=true (the second-drive copy is the household's only local copy of these files) +--- PASS: TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror (0.02s) +PASS +ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.030s diff --git a/documentation/audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/zz_rehearsal_d2_test.go b/documentation/audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/zz_rehearsal_d2_test.go new file mode 100644 index 00000000..8a3398e6 --- /dev/null +++ b/documentation/audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/zz_rehearsal_d2_test.go @@ -0,0 +1,35 @@ +package backup + +// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D2). Never committed. +import ( + "os" + "path/filepath" + "testing" +) + +func TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror(t *testing.T) { + m, src, target, prov := newTier2V2(t, "app") + prov.has["app"] = true + prov.binds["app"] = []ClassifiedBind{mHDD("appdata/app/photos"), mHDD("appdata/app/docs")} + mustWrite(t, filepath.Join(src, "appdata", "app", "photos", "family.jpg"), "PHOTO") + mustWrite(t, filepath.Join(src, "appdata", "app", "docs", "contract.pdf"), "PDF") + if err := m.RunTier2("app"); err != nil { + t.Fatal(err) + } + destBase := filepath.Join(target, "backups", "secondary", "app") + photo := filepath.Join(destBase, "hdd", "appdata", "app", "photos", "family.jpg") + doc := filepath.Join(destBase, "hdd", "appdata", "app", "docs", "contract.pdf") + t.Logf("POSITIVE CONTROL after run 1: mirror photo=%v doc=%v", exists(photo), exists(doc)) + + // The household (or an app bug, or fsck → lost+found) loses the live photos folder; the docs folder is emptied. + if err := os.RemoveAll(filepath.Join(src, "appdata", "app", "photos")); err != nil { + t.Fatal(err) + } + if err := os.Remove(filepath.Join(src, "appdata", "app", "docs", "contract.pdf")); err != nil { + t.Fatal(err) + } + err := m.RunTier2("app") + cd := m.settings.GetCrossDriveConfig("app") + t.Logf("run 2 err=%v; status=%q warning=%q", err, cd.LastStatus, cd.LastWarning) + t.Logf("FAULT after run 2: mirror photo=%v doc=%v (the second-drive copy is the household's only local copy of these files)", exists(photo), exists(doc)) +} diff --git a/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/00-as-found.txt b/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/00-as-found.txt new file mode 100644 index 00000000..62ac5910 --- /dev/null +++ b/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/00-as-found.txt @@ -0,0 +1,306 @@ +Tue Oct 20 10:09:14 UTC 2026 +filebrowser Up About an hour (healthy) +felhom-controller Up About an hour (healthy) +paperless-webserver Up 3 hours (healthy) +paperless-postgres Up 3 hours (healthy) +paperless-redis Up 3 hours (healthy) +immich-server Up 3 hours (healthy) +immich-postgres Up 3 hours (healthy) +immich-machine-learning Up 3 hours (healthy) +immich-redis Up 3 hours (healthy) +traefik Up 4 days +0a4fb1a71c72cf704d1afd44e1ab24a8a68e11b8d82ef1dcb71a75d61a9dc1a3 +0ce36c2e6db54e085f71a1436c4c89a9807a930c373198f451fa1665f90f0cab +0e2ca5e8738b95134285a666b1a7622bfcabc4728fb9269cc972dfaccbd59d45 +0f4002ab75695fb312775a8203cc5213b256f1b01d255210cb5cdd0e50cd9070 +1a8fe48409be3defe01713e2f621091630b965e1805cbb5872d1738043efd35c +1afe3420067df65e0d3e53baaec3debc39d2d42d333c7a3c266e4f6ff98e04f7 +1b41940af5c4fa82af59320d6c81ba622d97f2e43d8ec51c50dcf65c2fdcb1e8 +1f5120663962c7b03539bb993ab736be4800096e3d1f7147eb922d44b308ae6a +01feef94eb7e38a3679921b66cfe206e08adfd9b360ed4a60414cde606de1643 +2a71e17dfa1e930ffe0c213cd1615c8d1adcbbf8d61615a38064d331c265c02e +2aaab84bf843817d9e6f32b673277fd1e0c2c8c2d038b8126df44927b1b41006 +2ac49fc9cc2839f55c8abae76039950d15a5c301ea5b3918fa3e5d8715a15f42 +2bbdb463b29493d60d352bafefccbe57856cf80bd2cc193b61f9bcc1310fb45e +2bc95b104191ffd31bdfdbe1e038d284d0624c04735094494c2c77d957e8e905 +2dcb4bd2c3c618d71f582d33d5b1243e6ab075d3b00abc238d4430668a2b6f55 +2e4c79db25c47436b44245376b1f6ff1991208ba081fddce1938c3897fc63c07 +2ef8994a563ae88024ba29db4e349d26e6e3889e0683d30d1fb258208c9a10b4 +2f08e882c5d23373e2df5486dce28d4728330f9f0131a2735ebc938bfb4d7b3a +02f42b55ee403ceef63a472c3624da0b48fab9e6aa6dc0c7800ec68b2f2041ef +3a3d7bc33a1ba21ae4a39c8cb345254cf66cd1dc60afae64e2299513b094b93b +3ac9820f84d0a50f233c63a62ee1b63b92acac026d68c60a8227f370b0be00b6 +3d9a93769342e83a3c512d517b99cd64f8e794eb305c009e25c73a6f6627ad29 +3d9c1f23abbcef771a4c38a75d75865a12d76a4359fe83c1828652dba3f91710 +3f015f020f49d4967d2ac71b94060e6eaec86bc8852cf9a0b5f3dd8133dd528c +3fa030697dbaffb6809991bb69a335dc4e9accfafd4ed5ee953ed676673912e6 +4a9ed2aed1fe75a0b02f688e29a20acc2bf7e081ba7518a7d75451be30fdc4e1 +4bbaae5ae8cc8cc3330a73bb872b16e1420805197fa9bad903f7db9a1f358041 +4c54a4cda3c5280d5b13facf3089f8a0710c4570876aaeb679025aabf92df99f +4e052b4ef5602a68eddcea049bbc4e79b306012cf8b7f0c9cfc3e6c39af64fce +4ea780981a54c3e87aa72eb80436e02f3c6b3aaeab96ecf869602f5f5b0b546e +4f2ebb779645de5cf9d24e93c6fbd9d95d358b390dec94eed6f0223016785e42 +4f2266bb0b8a9b902eaa439ae9225963d6fe4359fe30569c2d8afa926b582585 +004ae946f5997471f4554ab7158a3b982b72391b59c59be40ea79b501eb1dab9 +5b84763ca63a28ae79c1393d700c534e7dd6e610f890f540981e49972e3854e9 +5c7dd57f445d87a39eea1e7b299e1a0acbbc946d6bfc92ba165c4af8933b322a +5d47a8950cfd56492e0f8b4ed6bfbf470a2e22811206ac0da0b79e520890b41e +5def44013bf255d4cd97c3829ac7ae895c607333a3ee586a4090bda128d379fd +5ec44fe3223976727132521b3055187e88783716a1f5fd07d1625205acff9888 +6a84846fd8e476d206eafb8f5a529ecbdf6b1021daab890429077e32ef4c2b6b +6aef4d0808bef53a15fc6d19614b641a5e36c941f2729572cfef8cfc8e36bbfd +6b5a55f08cc63514e70a93a3eb5d294e9b109e990521bb0ddcea4818a8df1456 +6b8acdd640ae067c2ee5235ddea8ad4c0e1eec2aaef1277b426923f56303e07d +6bcaa587af8bf282f83605713213c20275b1c8178c28de97b154af25bb6a674f +6bdf1f89b902e5a7134a87e0269229531ec6c8391256ea1bf466d2d6b66dd8c9 +6bf7012e27730a047bff7d69ceadef9c06433783d0b9582faefa9c9d0f966c86 +6ce965b12082d21ed8ade00884de6477faea9f97cf7a85e4b5ab3302d3517fce +6ddb2a450b19b940661c677b20936e92e8de54b8bc76f8dbacc56398d7d13a38 +6e21c6398b13ebac045a11b136fbb2d289e6abe99401eed9108b5464754a4c87 +6e0521e4314af444375b56e348a5e3d34aceb88d2d5e2f48b6986c27250e395e +6f19aa563c4e3e2000ed997157bc7f5bc8a45ee1d77c2cdcd1fd239117a62705 +7b5530f7f440ff6477f2ee5885b37ff9cfbf933d7908225998875b233b4d2a2a +7bfe063b0155297f5bca22c7ed02ae7ef9ce1b968670607415f7720e1e8649f0 +7ef24d6bf9fb190db2572505b489e45731a326c5c9c4ae6e36ffd7874560bbab +7fc41ddc47424ea11d4940f7062cdae69b92af645e6759f8759f47b689a19cb7 +8b44a3824395c930d0db784e5650ba7b58663090c3bd72a44f9dc716008a9642 +8bdcbc26d645ae3bc60742debe32650510bcfb7c69ad4416688af5b1815e5645 +8c2a7cf3f809218d20576368d4d419a63a07a89c3b12cc389fb2f3424fb45b73 +8c4d45c90f336c0ffd5d619765d8d2a45e802d619548f187bb3f2672b3534969 +8cae3c834dd9284d1d0069de042c2d50e49a138a78d93a59eb752272e4a5e4f4 +8d2ab3e77a72ebd708630e22f2eefc0a66f73cd94ed2abbfc4f666c504a3f172 +008dfd2a2e98c6b6673ee471d72b9e150cfe02859eb3a3bbd31a3c582b5a3a46 +9a2e6958ba1ea4443ade7619d326c84c50dbc444810b47904ff625969725a091 +9a86729a66d00d8be95f93062381b9bd91b16f94b654636681553fec6c7c7985 +9abdd18e6a7b07800067237c93d48448d49d9ad2388a91d35653323c2bed4389 +9c59fdcefb7ed258e615f8cb688334a69aebdb8e38fd1075a60d6fd98167065a +9c033896f0b00023816a49cea882a342c9b62ca4528640969a0d22c21f507505 +9db05859a329f9544f5a48cbd987f02fd00e48f160dcf6ee8331d8accfb423b4 +9e9fa98cb2344c5eda9bae7417557f2bc2503de1eed8219fdc2d2d378e1883fe +9e06437718bd8f9a27240eff7b425406aae50f061ea47a6991b9830ee2af9f59 +9ee6f261789b97ea0887a6a905bf5bba61a7070cc91ff64cda0091923d9cdf25 +09ea67268ce8c0442357ba316f6431cd96143da6bb54008d3c342be8b6ddb169 +13b744ec13f00bd3e5e2ecaa37bf5385af38cf0ff94fd05e660cfed3339a0c05 +13d76d014354ffb4e5aabfe00eeed5206b98c502dcfd515545fdc7210316bf61 +18e51d43a77ac282e108fd8a8e55207282cdabd2219af1e47c2c214117179f51 +20ee459190ea3442ab7cf0c288182033fb84b75a15bdfc14540e80ab60dec542 +22ab14c474cdb6cf9f873dcb6c8ce85dfc588abb46e2bd0925cdeff384437c46 +24a30fa240cec28f9a8b6d02a47ad1fda39674f094ae095b0d6ec3cbf099ed44 +24da0da42dc2eb9b6d35dcb7cd513b1e5c7d2ee38b6d59d9a1d51f0d01f6f920 +27a56be2953f040882ecd07fd8b91937f8cb17c25d1ceed56656ed2075518f86 +31f3b426743b1cff7952948995915586480486ed7fd06eef3379cf54b8fb3d9a +38ba44b0ff38f8df925512a477b8b5b4cba216a8ec0bd51b7463a86a88c42fea +43d6928bc4bb7a1e097122424e35f22393a0ccf2e03863d7dc3cce44163c8ae8 +49d0cebfc3199cef395ae27a6ffa479340c1c2f69da9248a777d7bff9b6b9ca9 +49dbe15539d7170822958bb0699d140c4d493215236b32679340588710d5452b +054cfa9b87316799ecd24ecfb480a38aa04b5f85a478a3db95c41706e5b63523 +57e9dc8d409d939d877a513023c95146400dd10078919ab9402d8144d5f30ca0 +62e86d5d6aded4a8ba536696c0df0f0af89f2cd7b4b893a7383e3abc00eeadc5 +63ddb5c60e6cf39dea412903cf1d6c0164a1146bcf4ab98ee94c9b1f9e45c6bb +63e2655474549ef03aaf7d9d741502287d2f01170c5ca008d7b09b8e00b1be66 +67eff4c5c60d493dc2a9f7841f7f138b610dc389c32793a27a62ddc13d417a1c +68d18a8d3519b6ea6928bbca40f9db7500b700f66e6eab236e7c32a214de2cc7 +68f83c9613471fe0e046a1f9252b71e86aa50d36315c675e92dc44a7cdd0267e +71cb6156444948894aec343c9a2c0c7674aca84d292d090c7f786d13146709ed +72ed18f5962e48dd5fb54e7d91d2fd0479017c27b2c60fc58a5d6b6e55242757 +74c6b39b7b926dd56db16f175026c2bf9e80a912916f273c122227672f181d03 +075b39bc7ae8c56a6d99923e4938ddbbb4174df6f0e653abf9c99b420b401a31 +78c01e39d1997eb54344f02f14dc02c7230cfb14ba58abc1a11feab2e9af85be +78e7acb9228c1df4ca4c706d459b97c8e79e8399fe06476ce72bd919a3b2b77f +83a8287d1e8700aa7e207b2c617a68557ce6f83f373f2c3c6d1443ae50032826 +83bc388f3ae110e72ba51a8f0b68f7bbdec3685907c1ff08095b63a1759502af +85d6f7309436e59cce9e373b0aa1626fa5e12d8d81163773f5f2c45da9d56543 +88c6573974e81f1fff96999bf54b80c66bc6f1e9af6746b3cd6801616198e0fd +92e2e73e3aef6e5330d2ce8aed3c58e7e3632a6f191f2b03c997ccabd245bb65 +93c2b9d0a1f54c2e6355d206d93bf2309f81076628aae9d868406c288b0f6bf4 +98a80f5f7903ef45f92f74a91f34dc01d527f8a2fe98b4fbe76711f5259416a5 +99f7aaac2646c66dd0eeeebc157b8563b56299c27ab4bf64e13920d089ca7156 +104e16e4796c52fe212c04b06417161e0bab20fc01632bfcf71421b73b979fbc +122fe66bacc9f04b3f3e4f6d0a3b15abdef8321eed36637a1284e3bd145463b8 +158c590779b033e48b5efcf895fd0bad4c6f277303d72b963cd0802ecac1f440 +218a186ab13a38d514cf9f523e5798452ae4e5ec76ecf26fe1a46ae22b47a508 +283e9d53742a13c471e5f59f53a57c179d461f0017ad5f8db27881cdd994cd04 +305b75817cf5c06105b19dc8c56ff0d796dc6e267ae5bcfc6018aa3d12037744 +325ec2aa2a6c307bb82d4dcb600fa3ff127aa6d1f35dc683ab61124ef5f0f5b3 +338f7535ae9ef91754e76604238ca7652273ffce9e6f0a1300de0b1edb61da77 +343d41a3b38b5a7eac10750eb0213b81780edd3075e9a1fdf3a2c479f4d2474c +346eaddf98e91d6d6e7b8c526074c27847e4a18be01b41027bec93abc2b7b2fb +355ba69bab94a95e00e87c6c4808587881b9675699083e087917ec8492bebc65 +426ccc08685def1c9025c94d343de0e59713073984634ee226e69634d6a18fb8 +552a4b340eb6be03c188cb17628aee0f916b544142465f8e52e7e9d2325206ab +556c805068afd3f6784bda2eb59642d7dc03756580e959e1c75262711a30d4d9 +0642eb9eee222e235de9310b9bd86b42493e5f4d71d430619b5832484dd09cd6 +645d70ab53013564fd243eae61db209e3793b826a9eeffd77395b18e27458789 +670e6447721f667df5222cb06a68c2682eb92803184a44faead8aea0dff44df6 +0717b5f7615c345fe47ab8212c06f506e08585551d93c6a8083b56a216adb71b +734e6b63f955d0b508dea39a8637755e75fbb353d54430956a0010f607eb727d +748a1560ef6216c8bf80545dbeadd29f99fd5c984b6f34bb678febf8b369adcc +748cce6aa136bdc0b3111c73a7c9434508200f8c231933b925d42b37b9cca17c +760cd2f9203dc3dc3f933e4a34bbaae857dee0afed66e727846a4c7f750ab5a8 +762e15e05b2f84f035a7011e10c64558dc6637c9fdde1846dc77f4056f3962d2 +787dc06078f3718da6faaba82269a7d21b213482c9e45025315bdb6cf59010b5 +866fc46ac37407bbf74761cecd21fa6322c4e7621f3dfbe963a85671d5753e5c +921d92a1c692ea0fd42cbe500bbdb1f3defcd5fb9b51df216451aa260d95ad45 +932bae2149e79ef113eb4929614a895c642960160d5f5cb2f2e9afac7dfa6af2 +968a5d7a54fb1d8ef4163f287e128cbd541ab46a2cc54db721dd3eafe1397b65 +991a5d344112325fb2c40700cd0ee96a784ce2150cba0a7b0be3ed6eb2411878 +2469a8c88a452ed226dd6d1082f0cbfbe3107d76f919b930b6bc7ce3b839ed90 +2634a60c3c37d5eba4be4827c08c0e99a951b934604c29d116fcddc27808e480 +2788da1d8f917a9d5b8acab2c7fe7336b6a7810f1c7edab63ed327a6022ceddf +3000c7863b2e80a72329ca1fa0dd03d2fd3994a2391e82450a8c92199cbb56b6 +3461d48cd9e88caf5363b0e4d96e799f6759ab8b8bc6ba4c148492a1637c01f0 +4681b8c67ad20fdd3ad04db895084d37a9296ac76b74ea7bc9465bb3ac93df8f +4690cfe7c61625190732c028ad4b918930a65e295005171d48114aa8f27b567b +5206b60d49110c01e05dc24034e12b767efa739e2ad21c252ab83db47e56d526 +5234fcace55ee7e44631499ed1f95fd0289918a84bd9f5cf6d9f761d43c34f17 +07402fbe22b572302f3317958f8cc4c6ca30317e0ee5bbbe9ff51c60b3da5a05 +7458b4c4b975debbd2bb705e9575d2d1ff5e9308ed9e70c302f5643136cd0cc0 +8097f97f4ae00b654b1c0b4408ad585f5bfce7999e482f244bd698268e1f6061 +8158fe1f757d26cadb484ed9690ad4d20d4cb382eeec7c9b0627b365afd6576f +9917d608ac0ad05856d31873432d641828ef519825a3f451e6a1ab2519dbf0d1 +13175cc97740c834e2404c11ba9fa9418d0c390d27a87e6e869977d6a86c20c0 +36571aa758bc335acc463c3ea074422a5123c309a37e1a7fc818a2069941c493 +59438a5f6378d218a73d3106ea1984d1c160fe4fcbbb7ec1177067f17430dcb3 +60496cd3a1e1ad1dcaed30427b70a57b8b14a122bae7f2cd882f8154a4549978 +61544bee8e2c1cd65e999121bb63011a9c1b4139b20b04fa32e3a850ca8c82b7 +84046f48a922cd337284bc2448a5d2682ea6a2e3479e71531122aff9ee4cbc93 +86022b0cd1043facf7b1c933c3b6a6ff36277a76d50bdb2da6a5690d57381b9e +98129d711a1e807ab7c3ae4c97187ab8505cebf1012218e8530d0829a2ad436f +177451abc7ed0606468250a7793e018725e8c7f370ee8937d4941a11571dcb9d +0217869eb1be5181b7053b993443dac73d9cd0804cd5bedb604edeb43f911702 +219118eb01fa7be03fcce2124b74708cacafe66aea617fd2bafa32f0b2d4fd80 +394985d0c3db0791b8c7569fccabd36959d00deb285075f9852568c4c81d32eb +722380f945884d4dea921ffb958513e02a4802483ec0886712e5ed3fa28d751c +774878c58b6a8cae6fbecdf656f2c1f09f37a89190068663e923763dbd6969ce +925226a74657c7ed0fcdbcfa9b504a47f7e21d0982c44f7ed9576140fb417146 +963079a9efb903a575b363b27e15927312569528b9066511b4c282e01e1ec931 +3830913f0478ec59a7a31d538f858f8ef612e1b2ce3bb9058a2d18f7b43abe9c +8579968bcd362a1aa9dcf14e22956b2fd15fd74a2fc2e7369ffb1457c2ce6693 +8678479fa55ea3871598eb6914eb8428ba599da4c619c62f231e6931f0cb1f54 +13694424a3895e8e73cc1e06af29aae0c3179c3cb5587b9daadeadcdb29b7cc7 +67711784cb23efbc0762e83f03365b837ec5216c38f3ba682a82eaff6a7863db +92236992ea73ecd9befe3330e88e96a7aedacd9cbca0bb8c32c31d0e2c914e5b +134995217d29dc47fb0ca1a274ceb5ea74225d36a98c4bbe6e07bf8c8d6898c7 +481706314b835ab042583c3107e168932844a162a2adf3b9ef2fa27433a7fd1d +4098005900d38c71b152787ce4bdedfb8e2b2b8b061f87645b2b4a14e0004335 +4699188665aa6eb965b7bb9f4938187f377e7a4ad55fc97e44802ea76f3612e9 +04762469942ae24c353f929fced777f245bb9f2f888a9f2044ff27fc45933665 +a6c297b9282b39caca7e513724219af096a3cf6fd453c3736ee7c4c833ef417d +a22cb584ba3444bc1dbd90b3723b1ad9424352251f2218a92b1884ef74e63b9f +a68fad1bcb1f1030af3811590cc18d74eb1da3c67190af4459fa69118aa8f035 +a74b9b2333ee936c17edd0525b49fec1337768ccab68f5576a39486c875ff968 +a74b41238ed9ded9e866679ab00f85813d800f92e4bb2642d3f9de28a1655b81 +a84ff849188b2bab211fefe8bc61a70f58f48f6a2a6ca25c2645e0bb5c384cbf +a2815ec8af59e0a137aa0cdf2baa48ca6d1b86e291c5728fab9449280d15fa75 +aa5cf2d08c68b8d3dfe3330861935fd97ff8fb9169645f6c632d970ad71b575c +aadb8f0be78890f1e08630b7a762a82853dc1402326bd16cd2a2844fa63fef1d +ac48d92438e8d19cf546764cc051172af82092223daad8ccfbfb4cb05100f5d6 +ac78e0a5ac10c7d0239e0e1ba3b604cd6fd418d87a6f32b66f3ce98c674164d8 +ad79ebffc7d8225cfbee6ad5fec4092ba7376f38dca912d6df8ea932f1eba20b +ad7084f002b17a1a27f70c796e3d7424a064b2ef5b71eadb7b3b08dcc9f59042 +ae2a94eac2960e95f944c9a4a0649c94c405fb0e518adfba2e6fa0f80f2a5af2 +ae9d5c15cba1adf881044a4449eaf72f5f2ed753c301edc1e43bcfa4560e8d31 +ae5499adb73000b6f460e63d6e045fe492702bd73187f868d9988690cb72a453 +b0e341cb8fb7f2fb42f7a8512a9b922591cfa40bd3522f31a422a9fdb4586008 +b4bc9ff2d9b533e896dfc5082ba865a4ea7676d71259d73aa33d515a5a60dd91 +b8d2fbe7e6afc931581bf5f1edacaefb9071bcc9f709e294a3f5ead104f73890 +b8dd8697ad52f98dc44ed3c940752aa52e16d6ab7563d36b1c9e221db22d1e11 +b21ec6b73a9f1f51623355f2bf8229b5f6f9da1484e1ce105cf8e0363475a5b8 +b081c3329ed821e22b0ee9cecbc2f52a5e9be70be2b1cb83bb262c21c84640d1 +b274ece5f6e1d506ca0646077964797c3e7b4413c9a9f1e453d9257e6b4d09f9 +b676fcf4f5272015b942c8589135a18fa16204c5aec457f1d0898ce6c84638eb +b1591e2a2604b6404d0d18e4b2b7599aafbcfb2bbacba06b5de5ff14abf14be3 +b7000ea518cd327f7dfb42858434fdb80c451aecf0a0325b6ccbff1f30545803 +beb17a9d330ce361923600aac8e13f8b10e62d0b4067380ad7a95c4d77f0110e +bfb91f5666855fd57228801ed0a4294981bd2a6665a93d1093bf3c4a0e89cb2c +c0fb7358aabef2c38e6921e2ed06b7fdddddecd29d530814545968323eba964e +c1d4fc53203c4b65abb1ff76f9505a46175c57afcd9c95cf445c6a3c443839fb +c3fbcb924a928bca859b69c75e4df275eb5e0abf03029785247313b0fa1cdd27 +c4bccf1c99f4990b0a23848b6fa58977dea0bdbf6ab69b7c0c2b90f7f0a2f5c3 +c5e875c34f4e4f1fd4edb1c6dbf4862a6543e26387ee480eeba0f5e3f44af7b6 +c8bb03639a4d7c41c233fa28d5a1c06dee874fe00614f764df9db3df14da86b5 +c16aa92f949bb89a034c6ef22ea8ece859883ab309086a204e43cedced8dd059 +c31f2070f92b36708a62c1d53a7aea495fd997440f2c01c851ad83c66357e379 +c35e0b961694a882dcc4c7bdb8e657d4b0685687cb5c4469559ce5078c95302a +c56e78b468dfe3991fc787ab7fe30b4282a73d7fd2e79e4dfc9305bb064685ea +c67a888239bc5e11ed44ca31307f3b46f7f76698d706ffc764024110d1a69ceb +c424d546895a59e0f88e2296a665c03f4618ac304ef19742b6f1c543665b9bb3 +c716ebc8ba8fa1556e1dddfaae98c7019618d39ece0e78dbd1b9f58dd0760d94 +c797d99e68b0efce9b1337dc0e3e9cbf42806bef03021ade362cb4e4d1de4222 +c5708e0a13ababf73203b1e297b3b73229cd6370e3fec1653b7cf1dc143a0d0a +c52417cfa3d923ce7168b7c2b082eb3ab31bc1750ff3f3edfc2c1cd59fb74f3d +c53655b36c7abc28db3020511a0ebdffe8fa37d86412753fca86f4caa7ae7a6b +c782734d8be184e0721d0a984b6ac3e8e0052206a9e46d22188a7fb2f923face +cb71ab036375acb25c69afd915261acf65f919008fbd59417989f6ebef110c7a +cb613ceec1bf86b718059ef3e864dce7ed60e5b6a7bd5ab9f7c633350df48f3a +cb10732ef556d77ec0a0a5d7b29c0789b47a9757b67bdb3fa9d5be22809debd0 +cf799a4b43f87d63eba6042a73b9eba3f17b354be1fd2d94dfac2048e13f3fce +d3c941cd28b5a692c2532f1381afdc57740d603fbba5eddbe3539f561f26a0ca +d5bd7d95fbb1f56d7617d96a5a6deb65417b4448692cb45f4a0bedd824165fc1 +d6a74fc39944848c79012bb06d915c609b92c1099881b90b4cda59074bd1124f +d58b42f643175b6272bb579d354a9510bc34fc88e4e70dda08346997ba3cec13 +d256b9490d623a136093e459b9f466dedadaa0d00235d6934044e6591ee7e2c5 +d352bb11678cf6cb47a16740ee9b79cfbf4c19d4da23f4fb95621bbdea6b91b2 +d053441be51f569ac393c26c220b6369ae3455ae76f3627eba223e8b4e40eacd +daa717bf6363a3afe8dbcd3ab863a81ca737e0999242bd99c618daa0e4fa085b +db0c37c3ce9df2f2090c622eb4194e6860e26846398d4b4a092b9c4c21441af4 +db2434b8bc6fb7858aa5eea51b23eb3fe17ed8e8ac025596ebb4f857d00f9ba9 +db6910df9d6429a8ecffced74f22c26f45a11d48efea51e48e8097bf43331540 +dbcfeebe1162c457fdf6f7cbfff66c971dfaca8a1a8ae2fb5472fb9007f83ddc +dc5c95c9a8f2f01a8eab63c7d591a7d008dbaf5f4b6157380faeeaeab168cd59 +de81766fb8e806f93c97771ae1a0ff529831bc5e7f50e2187d41dd606b8e23f1 +e0e25d3ed1797b0092c897ad904a4ac7a2bd77e12b96e1790a939aa009851803 +e0ee75c206f1b9be41e608c6de05f8fad8532e3bda240cfad0640c673ae46395 +e1ae949c5339eca43334dd6720b49df19082036b8f53151b8dfac97f2d2c0b6d +e2ca0579185aa6d7324f144357323b4d5521c1e94e8dc7929f0ffa9a73e1e4cd +e5f9a9a8f7e4b6d097aef1b0493180a547dbf19006920433e7b80aa8aac0026b +e6a40ba40f9ac5f43ba2276b3cb6110bb4c0e20003e866a55ca28c954ddd0e9a +e9e1b0601616187e8934c7212e2fde5b590b50308aa9c8c5dd12e419c6bebc18 +e45c71e0680d2856e2945881fd01ad6320868e0a36427c6bca637c53dfecc625 +e62af03dd8d599dfbdbecb8c6516d95e582ee803f2ddd3798479fbc769229030 +e72bfb25201770edb1889f678f5a844a57c9f5308acd57a86c5f4936b800d441 +e311fa91eca24a97e0c558e91d06dca353aa9348a7ea4b111e694e181868fc27 +e339a495e0c048e30483e4fd3bef66186712975f3c33c90ef63a3c9686da3160 +e668b1c1f44dae28ca34f530e313c1e544751b63bf0c480300be8e242683fecb +e4638cbb5b12548a76a2cb6e87c7f2ada75d61d3c7d1062535ef0bf2b3cde05d +e7945baeafe0ac6d9a1e6761786c35fddad76c81d86e544d369188c186a0389b +ea35c704202ee36415d4f7a0873bcb254d30268450d7c9ed2dc611c1e2f4d5d1 +eaeb108b23d455d01ec0c9e44d39ff395522942794d26f3c149e29615be9bf1f +eb420ae07f9c459abc101c1264fa8cf779002ef20748ef97abe7c3764c24f032 +ec30ca07d0b0af16afe60f217dedf15573045ed9adc45e21c3592ee6716fbe39 +ed02b9c56e48d34910bc97ebff834f1cb082c23ebcae97b0f51f3a2633664466 +edcc77bb3d279e3dfe7d687fe30e22f95f5d9391b9908e61b2206a6cf309ea91 +eea963e8499939d36d09e42d8d5cd86798e81e264c4d4c012191e42926e508b8 +f1b01ee5efb2117033ddcad5f5eafe8e6f65ad6fc3e933902bfc494c13a7d3c0 +f01b151ee48f9a3a631457069ece3a2ac3e111df28110f3c06f221ee7c54eff8 +f2ec30f9f836ed26cb13fdb80064d1dcabf953d1952fea2d85bb6410f4cb2271 +f8a87cab6bc0890c7a2e36f73e533eac1a1cabb13b886b8ed71afa4c8b7d90be +f9b0fe025aedb49e94d80119a8a8dc0f85800bc592e568dc95b0f1e5439c54fc +f56a1ab986e83e118ff174e9805f6151e47ff693c17a2f475fcedd83cb98e051 +f82a956615af08264b64e051e09c1441152d92371d4d2db8d5c0ac4ab47a7182 +f287b03a768126d72af210dceab12e329573779e7aad56c123ac0dbb14fc4f4e +f347e779fb8c09daba2a757dfc3ee7c415f8f5a2eaace19535d363cc3ab3171a +f547c232f5e1785aa11200ac03dcfda6f056400bf1456a3388db6d73a8158ca0 +f6822a239614fc82462535db5b2806dd616696c9552e1d469922f4b7cd2bf56e +fa453f6fb451a564d31e704bbca0bcbcc9ad1109a25a8e5ad5c704789a424c4b +fb5ad60698efb6ee9f45fd440e56c060ba01ba03a854f556e42f8090b2057c7e +fbc407de7b12a4975b60dac518ac287f076659213cce708871162342419ab383 +fbf633eb758b7b66ab761c34eeebc759791e2f73020e839b3cfc2d692c9ba21f +fc1bddcbef8c02474db71bf11dace70e1e51d94f3c3fe6d9d45f4a669ac4a9fd +fd453dd47dc82ff7625bd4093c62140fe14a348814f094b43b519f7f59293af7 +fdc05b3bfda3101bec7d66b5e0cfb589e06f1f4b4d6b2f9d84dac470b0abf53b +fdebde6f8bcdd5281c50bcb4669c45dc54e33563bec872f8d23d76d2fe3ebfdf +fe503d3e9a34b113d4c56b59c5986e13c44b6e401b6e6681eb62712d24368329 +felhom-controller-data +filebrowser_filebrowser_data +immich_immich_ml_cache +immich_immich_postgres_data +immich_immich_redis_data +paperless-ngx_paperless_data +paperless-ngx_paperless_postgres_data +paperless-ngx_paperless_redis_data +privatebin_privatebin_data +ls: cannot access '/mnt/felhom-drives/scratch_hdd/felhom/backups/primary/': No such file or directory +Filesystem Size Used Avail Use% Mounted on +/dev/loop1 69G 11G 54G 17% /var/lib/felhom +/dev/nvme0n1 938G 210G 681G 24% /mnt/felhom-drives/scratch_hdd diff --git a/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/01-run.log b/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/01-run.log new file mode 100644 index 00000000..558ff040 --- /dev/null +++ b/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/01-run.log @@ -0,0 +1,104 @@ +10:10:26Z == 1 deploy +10:10:27Z deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +10:11:12Z state: running +10:11:42Z == 2 marker A + backup +10:11:46Z CREATE TABLE +INSERT 0 1 +10:11:49Z [2] DB markers: 'A' +10:11:49Z POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'} +10:12:59Z backup finished: {"db_dump": {"count": 3, "duration": "1m9.055678483s", "last_run": "2026-10-20T10:12:58.690358712Z", "success": true}, "enabled": true, "running": false} +10:13:04Z unit on disk: +UNIT /mnt/sys_drive/felhom-data/backups/primary/rallly +2026-10-20 10:12 487 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/app.yaml +2026-10-20 10:12 4071 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/docker-compose.yml +2026-10-20 10:12 11894 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/.felhom.yml +2026-10-20 10:12 52228608 /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar +2026-10-20 10:11 111371 /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql +2026-10-20 10:12 3195 /mnt/sys_drive/felhom-data/backups/primary/rallly/manifest.json +2026-10-20 10:12 1191 /mnt/sys_drive/felhom-data/backups/primary/rallly/data-stamps.json +10:13:04Z == 3 POSITIVE CONTROL: marker B (live) then a normal unit restore +10:13:09Z INSERT 0 1 +10:13:12Z [3-before] DB markers: 'A,B' +10:13:12Z [3] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}] +10:13:12Z [3] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +10:13:12Z [3] restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "offbox-reconstitute", "stack": "rallly", "ok": true, "message": "A(z) rallly: 0 fájl és 1 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 20. 09:30) — az alkalmazás újraindult.", "finished_at": "2026-10-20T07:36:57.852528534Z"}} +10:13:40Z [3] restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true} +10:14:00Z [3] after: state=running hold=None +10:14:18Z [3-after] DB markers: 'A' +10:14:18Z == 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half +10:14:22Z INSERT 0 1 +10:14:25Z [4-live] DB markers: 'A,C' +10:14:29Z TRUNCATED /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar 52228608 -> 26114304 +10:14:29Z == 5 the unit restore over the damaged copy +10:14:29Z [5] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}] +10:14:29Z [5] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +10:14:29Z [5] restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true} +10:14:44Z [5] restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": false, "message": "Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1\nstderr: ...twork rallly_rallly-internal Creating \n Network rallly_rallly-internal Created \n Network rallly_rallly-internal Created \n Container rallly-postgres Creating \n Container rallly-postgres Created \n Container rallly Creating \n Container rallly Created \n Container rallly-postgres Starting \n Container rallly-p +10:15:04Z [5] after: state=degraded hold=None +10:15:38Z [5-after] DB markers: 'Error response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running' +10:15:41Z containers: +rallly Created +rallly-postgres Restarting (1) 5 seconds ago +10:15:44Z db container log tail: +Data page checksums are enabled. + +initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty +initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker". +The files belonging to this database system will be owned by user "postgres". +This user must also own the server process. + +The database cluster will be initialized with locale "en_US.utf8". +The default database encoding has accordingly been set to "UTF8". +The default text search configuration will be set to "english". + +Data page checksums are enabled. + +initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty +initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker". +10:15:47Z any undo / safety copy of the live state on disk? +/mnt/sizing-bench/backups/secondary/paperless-ngx/recovery-unit/db-dumps/pre-restore-20260927T094411Z-paperless-ngx-postgres.sql +/mnt/sys_drive/felhom-data/backups/primary/rallly +/mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar +/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql +10:15:51Z controller log (restore lines): +Container rallly Created + Container rallly-postgres Starting + Container rallly-postgres Started + Container rallly-postgres Waiting + Container rallly-postgres Error dependency rallly-postgres failed to start +dependency failed to start: container rallly-postgres is unhealthy +2026/10/20 10:14:42 manager.go:1362: [ERROR] [stacks] Stack rallly start failed after 1.1s: exit code 1 +stderr: ...twork rallly_rallly-internal Creating + Network rallly_rallly-internal Created + Network rallly_rallly-internal Created + Container rallly-postgres Creating + Container rallly-postgres Created + Container rallly Creating + Container rallly Created + Container rallly-postgres Starting + Container rallly-postgres Started + Container rallly-postgres Waiting + Container rallly-postgres Error dependency rallly-postgres failed to start +dependency failed to start: container rallly-postgres is unhealthy +2026/10/20 10:14:42 handlers.go:1881: [ERROR] [web] Restore failed (async): stack=rallly: starting rallly after restore from unit: starting stack rallly: exit code 1 +stderr: ...twork rallly_rallly-internal Creating + Network rallly_rallly-internal Created + Network rallly_rallly-internal Created + Container rallly-postgres Creating + Container rallly-postgres Created + Container rallly Creating + Container rallly Created + Container rallly-postgres Starting + Container rallly-postgres Started + Container rallly-postgres Waiting + Container rallly-postgres Error dependency rallly-postgres failed to start +dependency failed to start: container rallly-postgres is unhealthy +2026/10/20 10:14:49 manager.go:920: [DEBUG] [stacks] restart-policy of down member "rallly" = "unless-stopped" +2026/10/20 10:14:59 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)] +2026/10/20 10:15:09 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)] +2026/10/20 10:15:09 main.go:4054: [INFO] [deadapp] rallly crossed the crash_loop threshold (8 in 10m0s) while a backup/quiesce/update holds it — NOT stopped +2026/10/20 10:15:19 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)] +2026/10/20 10:15:29 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)] +2026/10/20 10:15:39 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)] +2026/10/20 10:15:49 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)] +exit 0 diff --git a/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/02-controller-log-fault-window.txt b/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/02-controller-log-fault-window.txt new file mode 100644 index 00000000..f48a522f --- /dev/null +++ b/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/02-controller-log-fault-window.txt @@ -0,0 +1,18 @@ +2026/10/20 10:14:29 handlers.go:1873: [WARN] [web] Restore requested (async): stack=rallly, snapshot=helyi from 172.18.0.3:36322 +2026/10/20 10:14:29 restore_dbcreds.go:114: [INFO] [backup] Restore rallly: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106) +2026/10/20 10:14:29 restore_unit.go:393: [INFO] [backup] Restoring rallly from recovery unit /mnt/sys_drive/felhom-data/backups/primary/rallly: images=2, secrets recovered=2/2, data_keys=0 +2026/10/20 10:14:29 manager.go:1440: [INFO] [stacks] Stopping stack: rallly +2026/10/20 10:14:40 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.8s) +2026/10/20 10:14:40 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly +2026/10/20 10:14:40 restore.go:246: [ERROR] [backup] Failed to populate volume rallly_rallly_postgres_data: tar: short read — exit status 1 +2026/10/20 10:14:40 restore_unit.go:448: [ERROR] [backup] volume restore for rallly: failed to restore 1 volume(s): [rallly_rallly_postgres_data] +2026/10/20 10:14:40 [INFO] [stacks] SaveAppConfig: saved config for rallly +2026/10/20 10:14:40 deploy.go:883: [INFO] [stacks] Redeploying rallly from recovery unit with 4 env vars +2026/10/20 10:14:40 pin.go:93: [INFO] [stacks] pin rallly: rallly=lukevella/rallly:4.15.3, rallly-postgres=postgres:18-alpine +2026/10/20 10:14:40 [INFO] [stacks] SaveAppConfig: saved config for rallly +2026/10/20 10:14:40 restore_unit.go:467: [ERROR] [backup] rallly: NOT replaying the database copy — the volume restore failed, so the database is not known to be the copy's own +2026/10/20 10:14:40 manager.go:1351: [INFO] [stacks] Starting stack: rallly +2026/10/20 10:14:42 manager.go:1704: [ERROR] [stacks] Command failed: docker compose up -d (in /opt/docker/stacks/rallly) — exit code 1 (took 1.1s) +2026/10/20 10:14:42 manager.go:1712: [ERROR] [stacks] stderr (last part): ...twork rallly_rallly-internal Creating +2026/10/20 10:14:42 manager.go:1362: [ERROR] [stacks] Stack rallly start failed after 1.1s: exit code 1 +2026/10/20 10:14:42 handlers.go:1881: [ERROR] [web] Restore failed (async): stack=rallly: starting rallly after restore from unit: starting stack rallly: exit code 1 diff --git a/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/03-teardown.txt b/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/03-teardown.txt new file mode 100644 index 00000000..129de6e4 --- /dev/null +++ b/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/03-teardown.txt @@ -0,0 +1,28 @@ +200 +{ + "ok": true, + "message": "Stack rallly stop completed" +} +200 +{ + "ok": true, + "data": { + "removed": "rallly", + "volumes_removed": [ + "rallly_rallly_postgres_data" + ], + "hdd_paths_removed": [], + "hdd_paths_preserved": [], + "hdd_note": "Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott nem volt mit törölni.", + "userdata_kept": [], + "backup_paths_removed": [ + "/mnt/sys_drive/felhom-data/backups/primary/rallly (26M)" + ], + "verified": true + }, + "message": "Stack rallly removed" +} +0 +0 +ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/rallly': No such file or directory +/opt/docker/stacks/rallly diff --git a/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/README.md b/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/README.md new file mode 100644 index 00000000..008ff797 --- /dev/null +++ b/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/README.md @@ -0,0 +1,25 @@ +# C3 — R1/D4 proven live: a failed LOCAL restore leaves a broken app and no copy of what it replaced + +Scratch guest 9202 on demo-hp, controller **0.331.0** (the local-restore code is unchanged since v0.328.0 — +`git log` on `restore_unit.go`/`restore.go`; 0.336.0 runs the same lines). Hub off on 9202. The dashboard password was +reset with the controller's own escape hatch (`--print-reset-code` → `/claim`), held in `sec` only. +Architecture read: `07` §6.3 (the ladder „replay → rollback → hold"), §8; `09` §3 decision 154 (R-638 A), D20 (R-893 A). + +| step (UTC) | act | observed | +|---|---|---| +| 10:10 | Rallly 4.15.3 + postgres:18 deployed through `POST /api/stacks/rallly/deploy` | running | +| 10:11 | marker A (psql in the app's own DB container); „Mentés most" (`POST /api/backup/run`) | unit written 10:12 (`rallly_rallly_postgres_data.tar` 52 MB + `rallly-postgres.sql`) | +| 10:13 | **positive control:** marker B, then the household's restore (`POST /backup/restore`, point „helyi") | „1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult"; markers **A** (B rewound, as designed) | +| 10:14:22 | marker C — the household's newest row | live: **A,C** | +| 10:14:29 | **fault:** the unit's DB volume tar truncated to half (a torn / damaged copy) | 52 228 608 → 26 114 304 B | +| 10:14:29 | the same restore | stop 10.8 s → `volume rm` → `tar: short read` → `[ERROR] volume restore … failed` → definition rewritten → „NOT replaying the database copy" → start → **postgres crash-loops** (`initdb: directory … exists but is not empty`) | +| 10:14:44 | what the household reads | „Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1 stderr: ...twork rallly_rallly-internal Creating …" — **raw English Docker output** | +| 10:15:47 | any copy of the live state (marker C)? | **none** — no undo dump, no live-volume copy for Rallly (`find` of `*restore-undo*`/`pre-restore-*` finds only a September paperless file) | +| 10:15 | the dead-app guard | `rallly crossed the crash_loop threshold … while a backup/quiesce/update holds it — NOT stopped` | + +**So:** the household's live data (marker C) was destroyed by the restore itself, the app does not start, and nothing +on the box can put the pre-restore state back. The off-site restore got exactly this protection in v0.336.0 (D20 = A: +„put back exactly as it was"); the LOCAL path — 07 §3's main customer restore — did not. Logs: `01-run.log`, +`02-controller-log-fault-window.txt`. Teardown through the product (`03-teardown.txt`): Rallly removed with its data +and backups; no container, no volume, no unit (the `/opt/docker/stacks/rallly` folder is the catalog template every +app has). diff --git a/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/r1_proof.py b/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/r1_proof.py new file mode 100644 index 00000000..58825331 --- /dev/null +++ b/documentation/audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/r1_proof.py @@ -0,0 +1,153 @@ +#!/usr/bin/env python3 +"""r1_proof.py — C3 proof of candidate R1/D4 on scratch guest 9202 (data-loss hunt 2026-10-20). + +Claim under test: a LOCAL (Tier-1 unit) restore that fails after it has started replacing the app's volumes +restarts the app on whatever is left, and keeps NO copy of the live state it replaced — so the household's newer +data is gone and nothing can put it back (unlike the off-site path, which got „put back exactly" in v0.336.0). + +Method (mechanism proof — markers are written with psql inside the app's own database container, as the +2026-10-19 R-893 measurement did; the household-facing acts are the dashboard's own endpoints): + 1 deploy Rallly (throwaway) through POST /api/stacks/rallly/deploy + 2 marker A; „Mentés most" = POST /api/backup/run (whole box, scratch); wait + 3 POSITIVE CONTROL: marker B (live), unit restore (POST /backup/restore) -> A present, B gone (expected rewind) + 4 marker C (the household's newest live row); THE FAULT: truncate the unit's DB volume tar to half + 5 unit restore again -> observe: the answer, the app's state, what the DB holds, any undo/safety copy on disk + 6 teardown: remove Rallly with its data and backups through the product; verify nothing left +Env: CTL_BASE, CTL_DOMAIN, CTL_PW_ENV (see tools/ctl.py). Prints a timestamped log (no secrets). +""" +import json, os, re, subprocess, sys, time +sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/rehearsal-2026-10-20/tools") +import ctl # noqa: E402 + +APP = "rallly" +DOMAIN = os.environ["CTL_DOMAIN"] + + +def say(*a): + print(time.strftime("%H:%M:%S", time.gmtime()) + "Z " + " ".join(str(x) for x in a), flush=True) + + +def guest(script, timeout=600): + r = subprocess.run(["ssh", "demo-hp", "export LC_ALL=C; cat > /tmp/r1p.sh; pct push 9202 /tmp/r1p.sh /tmp/r1p.sh;" + " pct exec 9202 -- bash /tmp/r1p.sh; pct exec 9202 -- rm -f /tmp/r1p.sh; rm -f /tmp/r1p.sh"], + input=script, capture_output=True, text=True, timeout=timeout) + return (r.stdout + r.stderr).strip() + + +def psql(sql): + return guest(f"""C=$(docker ps --format '{{{{.Names}}}}' | grep -E '^rallly.*(postgres|db)' | head -1) +U=$(docker exec $C printenv POSTGRES_USER); D=$(docker exec $C printenv POSTGRES_DB) +docker exec -i $C psql -v ON_ERROR_STOP=1 -U "$U" -d "$D" -tA <<'SQL' +{sql} +SQL""") + + +def stack(): + code, d = ctl.call("GET", f"/api/stacks/{APP}") + return (d.get("data") or {}) if isinstance(d, dict) else {} + + +def wait_backup(t_cap=1200): + t0 = time.time() + time.sleep(5) + while time.time() - t0 < t_cap: + code, d = ctl.call("GET", "/api/backup/status") + dd = (d.get("data") or {}) if isinstance(d, dict) else {} + if not dd.get("running"): + return dd + time.sleep(5) + return {"timeout": True} + + +def restore(tag): + code, d = ctl.call("GET", f"/api/backup/snapshots?stack={APP}") + data = d.get("data") if isinstance(d, dict) else None + snaps = data if isinstance(data, list) else next((data[k] for k in ("snapshots", "items", "restore_points") + if isinstance(data, dict) and isinstance(data.get(k), list)), []) + say(f"[{tag}] restore points offered: {json.dumps(snaps, ensure_ascii=False)[:400]}") + sid = snaps[0].get("id") or snaps[0].get("snapshot_id") or snaps[0].get("short_id") + out = subprocess.run(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", ctl.HOST, "-H", "Cookie: " + ctl._SESS, + "-X", "POST", "--data-urlencode", "_csrf=" + ctl._CSRF, "--data-urlencode", "stack_name=" + APP, + "--data-urlencode", "snapshot_id=" + sid, ctl.BASE + "/backup/restore"], + capture_output=True, text=True).stdout + say(f"[{tag}] POST /backup/restore -> {out.splitlines()[0] if out else '?'} " + f"{[l for l in out.splitlines() if l.lower().startswith('location')][:1]}") + t0, last = time.time(), None + while time.time() - t0 < 900: + code, d = ctl.call("GET", "/api/backup/restore-status") + dd = (d.get("data") or {}) if isinstance(d, dict) else {} + cur = json.dumps(dd, ensure_ascii=False)[:600] + if cur != last: + say(f"[{tag}] restore-status: {cur}") + last = cur + if not dd.get("running") and time.time() - t0 > 8: + break + time.sleep(3) + time.sleep(20) + st = stack() + say(f"[{tag}] after: state={st.get('state')} hold={st.get('hold_reason')!r}") + + +def markers(tag): + say(f"[{tag}] DB markers: {psql('SELECT string_agg(m, \',\' ORDER BY m) FROM r1_marker;')!r}") + + +def main(): + say("== 1 deploy") + if not stack().get("deployed"): + code, d = ctl.call("GET", f"/api/stacks/{APP}/deploy-fields") + fields = (((d.get("data") or {}).get("metadata") or {}).get("deploy_fields")) or [] + vals = {"DOMAIN": DOMAIN, "SUBDOMAIN": "r1poll"} + for f in fields: + if f.get("env_var") in vals: + continue + if f.get("type") == "password": + import secrets + vals[f["env_var"]] = "R1-" + secrets.token_hex(10) + elif f.get("required") and f.get("default"): + vals[f["env_var"]] = f["default"] + code, d = ctl.call("POST", f"/api/stacks/{APP}/deploy", {"values": vals}) + say("deploy ->", code, str(d)[:160]) + for _ in range(90): + time.sleep(5) + st = stack() + if st.get("deployed") and (st.get("app_config") or {}).get("pinned_images") and st.get("state") in ("running", "unhealthy", "degraded"): + break + say("state:", stack().get("state")) + time.sleep(30) + say("== 2 marker A + backup") + say(psql("CREATE TABLE IF NOT EXISTS r1_marker(m text primary key); INSERT INTO r1_marker VALUES ('A') ON CONFLICT DO NOTHING;")) + markers("2") + code, d = ctl.call("POST", "/api/backup/run") + say("POST /api/backup/run ->", code, str(d)[:200]) + say("backup finished:", json.dumps(wait_backup(), ensure_ascii=False)[:500]) + unit = guest(f"for d in /mnt/felhom-drives/*/backups/primary/{APP} /mnt/sys_drive/felhom-data/backups/primary/{APP}; do " + f"[ -d $d ] && echo UNIT $d && find $d -maxdepth 3 -type f -printf '%TY-%Tm-%Td %TH:%TM %s %p\\n'; done") + say("unit on disk:\n" + unit) + say("== 3 POSITIVE CONTROL: marker B (live) then a normal unit restore") + say(psql("INSERT INTO r1_marker VALUES ('B');")) + markers("3-before") + restore("3") + time.sleep(15) + markers("3-after") + say("== 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half") + say(psql("INSERT INTO r1_marker VALUES ('C');")) + markers("4-live") + say(guest(f"for d in /mnt/felhom-drives/*/backups/primary/{APP} /mnt/sys_drive/felhom-data/backups/primary/{APP}; do " + f"[ -d $d ] || continue; f=$(find $d -name '*.tar' -path '*volume*' | grep -iE 'postgres|db' | head -1); " + f"[ -n \"$f\" ] || f=$(find $d -name '*.tar' | head -1); s=$(stat -c %s \"$f\"); " + f"truncate -s $((s/2)) \"$f\"; echo \"TRUNCATED $f $s -> $(stat -c %s \"$f\")\"; done")) + say("== 5 the unit restore over the damaged copy") + restore("5") + time.sleep(30) + markers("5-after") + say("containers:\n" + guest("docker ps -a --format '{{.Names}} {{.Status}}' | grep -i rallly")) + say("db container log tail:\n" + guest("C=$(docker ps -a --format '{{.Names}}' | grep -E '^rallly.*(postgres|db)' | head -1); docker logs --tail 15 $C 2>&1")) + say("any undo / safety copy of the live state on disk?\n" + guest( + "find / -xdev \\( -path /proc -o -path /sys \\) -prune -o \\( -iname '*restore-undo*' -o -iname 'pre-restore-*' \\) -print 2>/dev/null | head; " + "for d in /mnt/felhom-drives/*/backups /mnt/sys_drive/felhom-data/backups; do find $d -maxdepth 4 -iname '*rallly*' -newer /etc/hostname 2>/dev/null | head; done")) + say("controller log (restore lines):\n" + guest("docker logs --since 30m felhom-controller 2>&1 | grep -iE 'restore|rallly' | grep -vi 'GET /api' | tail -40")) + + +if __name__ == "__main__": + main() diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index a87c49b9..4c6ae096 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -151,13 +151,21 @@ stopping line that lies. | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC | | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC | -## Backup & restore — 26 rows (P2 3, P3 11, P4 12) +## Backup & restore — 41 rows (P1 2, P2 9, P3 18, P4 12) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| +| **R-1175** | Backup & restore | P1 | **Shared folders on two drives that use the same second drive: every night one drive's shares lose their second copy, and the night a drive dies its share copy is deleted.** The shares second-drive job runs one group per source drive; each group's clean-up walks EVERY drive key on the target and removes what is not its own (`internal/backup/tier2_shares.go:201-230`, `:266-267` → `tier2.go:299-329`). PROVEN against production code (fake copy seam): shares on hdd_2 and hdd_3 both target hdd_1 — run 1 logs `removed stale dest dir hdd_2/b_share`, so after EVERY run only the last group's shares have a copy while the status says ok; then hdd_3 goes away — run 2 logs `removed stale dest dir hdd_3/csaladi_fotok`: the dead drive's only second copy is deleted that night. Data-loss hunt 2026-10-20 (C1 reviewer 2, C2, C3: `audits/dataloss-hunt-2026-10-20/C3-D1-shares-tier2/`). | OPEN | — | Fix: reconcile only the group's OWN key dir (and keep a key whose drive is away/decommissioned until the household releases it); test with the C3 cases as red-proofs; release by the normal train | CC | +| **R-1176** | Backup & restore | P1 | **A box rebuilt after „delete the old host first" cannot reach the household's off-site history: the recovery code is never asked for, a new key is minted, and the old history is set aside on night one.** D19 (`hub/internal/api/passphrase_scope.go:58`) makes the reinstall stop until the operator deletes the old host; the delete demotes the escrow under the OLD host id and deletes the host row (`hub/internal/store/store.go:3164-3229`); the new box's status query joins on live hosts only (`store.go:3499-3527`), so it reports no package and no retained package (both flags false) → the controller mints a fresh repo password (`offbox.go:542-555`), offers no recovery (`:1788`) and on its first night sets the old repository aside (`:905-906`); the retained-package API 404s for a deleted host (`handler.go:1691`). The reset and purge paths DO read `host_deletions` (`customer_reset.go:48,155`; `store.go:4189`) — the status path does not. PROVEN at the hub store against production code (`audits/dataloss-hunt-2026-10-20/C3-K1-escrow-after-host-delete/`: control = present; after delete + new host = nil). 03 §8a and 07 §7.0 promise this route; D19 + R-1154 removed it without a recorded choice. Data-loss hunt 2026-10-20 (C1 reviewer 3). | OPEN | the rehearsal's B7 is the live proof (needs Tester4 off-site ON) | Decide the shape (status query includes `host_deletions` for the same customer, and the retained GET answers for a deleted host of the caller's customer); red-proof with the C3 test; hub + controller release by the normal train | CC | | **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/` **LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest | — | — | operator | | **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):** `_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the first verdict lands at 07:10 hub time (05:10 UTC), after the night session that reached it ended; that brief did not include this read. Nothing is lost: the hub keeps its listing; read the 2026-10-20 07:10 lines on 10-21 (or later on 10-20). | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC | | **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). | operator: print the sheet | Operator: print + store the sheet; then close | operator | +| **R-1177** | Backup & restore | P2 | **A local restore that fails destroys the household's live data and keeps no copy of it: the app is left broken.** PROVEN LIVE on scratch 9202 (`audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/`): Rallly + Postgres; control restore OK; then a live row C, the unit's DB volume tar damaged (half), the household's restore → the live volume removed, `tar: short read`, the DB replay correctly skipped, the app started → Postgres crash-loops (`initdb: directory … exists but is not empty`); row C gone; NO undo or safety copy anywhere; the household reads „Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1 stderr: …" (raw English Docker output). Mechanism `internal/backup/restore_unit.go:428-489`, `restore.go:215-246`. The OFF-SITE path got „put back exactly as it was" in v0.336.0 (D20 = A, R-893); the LOCAL path — 07 §3's main customer restore — did not. Two reviewers found it independently (R1, D4). | WAITING-ON-OPERATOR | decision: extend D20's put-back (safety dump + live-volume copy before the first change, rollback on failure) to the local and second-drive unit restores? | Operator: answer the D-question in STATUS; then CC builds it with the R-893 harness | operator | +| **R-1178** | Backup & restore | P2 | **The second drive's „Teljes visszaállítás" can change files, then refuse the database half, leave the app STOPPED, and tell the household „the app and its data are untouched".** `tier2_whole.go:301-340` stops the app and merges the files first, then calls the unit restore, whose refusals (cut-off dump, mixed versions, allow-list, data key — `restore_unit.go:366-423`) all return before its own stop/start. PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-R5-whole-restore/`): 1 file restored + 1 replaced (the live one kept beside), calls `[stop]`, no start, message „…A visszaállítás biztonsági okból nem indult el — az alkalmazás és az adatai érintetlenek." Data-loss hunt 2026-10-20 (C1 reviewer 1). | OPEN | — | Fix: run the unit's pre-checks BEFORE the file half (or start the app and say what changed on a refusal); red-proof with the C3 test | CC | +| **R-1179** | Backup & restore | P2 | **A drive marked „away" can still be chosen as the second-drive backup target.** `tier2.go:151-180` loops `GetSchedulableStoragePaths()`, which drops only decommissioned drives (`settings.go:1593-1603`); nothing checks `Disconnected`. Seen in the R-1175 unit test (a share copied to a drive marked disconnected). On a real box the away drive's mount point is an empty folder on the guest's root disk (or the host's root via the bind), so the night's copy fills the wrong disk and is not where the household thinks. CONFIRMED in code; not run live. Data-loss hunt 2026-10-20. | OPEN | — | Fix: skip a disconnected target like a decommissioned one; live check on a scratch box with a removed drive | CC | +| **R-1180** | Backup & restore | P2 | **The nightly second-drive mirror (`rsync --delete`) can run DURING a restore from that same mirror and delete files not yet copied back.** `RunAllTier2` (`tier2.go:479-520`) takes no backup flag and does not ask about a running restore; the whole restore holds only the backup flag (`tier2_whole.go:301`), the night leg only `LegLock` (`cmd/controller/main.go:1348-1352`). A long restore of a photo library at 03:30, or 15 min after a missed night, overlaps. CONFIRMED in code; not run live. Data-loss hunt 2026-10-20 (C1 reviewer 4, T1). | OPEN | — | Fix: the mirror leg skips (and says so) while any restore runs; test with the existing tier-2 seams | CC | +| **R-1181** | Backup & restore | P2 | **A missing or emptied app folder deletes its second-drive copy within one night; for OPTIONAL folders (audiobooks, comics, ROMs, Immich's external library) that copy was the only one.** A missing declared leg is skipped (`tier2_capture.go:55-61`, silent for optional legs) and then reconciled away (`tier2.go:454`); an emptied one is mirrored with `rsync -a --delete` (`:744-751`). PROVEN for a missing folder against production code (`audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/`): the mirror's `photos` deleted, run „ok", warning only „nem kerültek a másodlagos mentésbe". 07 §8.2 chose „data legs are not guarded" for a leg that LEAVES the class set; a declared leg that is MISSING is not that case. Optional legs are in no off-site tier (07 §6.2). | WAITING-ON-OPERATOR | decision: keep a missing leg's mirror (and say so) until the household releases it? And: does an accidental folder deletion need a second-drive history for optional legs? | Operator: answer in STATUS | operator | +| **R-1182** | Backup & restore | P2 | **Abandoning ONE set-aside off-site copy deletes EVERY retained key package of the customer, including ones that open other set-aside copies still stored.** hub `api/handler.go:626-627` → `PurgeSupersededEscrowForCustomer` (`store.go:4185-4190`) deletes customer-wide; the comment right above (`:4176-4179`) states the invariant this breaks. PLAUSIBLE (code read by reviewer and main session; not run). Data-loss hunt 2026-10-20 (C1 reviewer 3, K2). | VERIFY | — | Prove with a hub store test (two superseded rows, one abandon); then purge only the package that covers the abandoned path | CC | | **R-49** | Backup & restore | P3 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker.** **READY 2026-10-20 — measured and planned, nothing changed; the decision is the operator's (STATUS D21).** Re-measured on 9202 (Immich v3.2.4): off-site ~1.33 GB = model cache 824 MB (62 %), raw database folder 283 MB (21 %), Immich's own backups 115 MB (9 %), logical dump 54 MB, originals 49 MB, thumbnails 2 MB — 102 MB must survive. A: leave out all three re-fetchable parts (−92 %; needs a never-run restore from the dump alone on the vectorchord image). B: only the model cache (−62 %; needs one restore with an empty cache). Pick: B now, A's other parts after their proof (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`). | — | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app | CC | | **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC | | **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC | @@ -181,6 +189,13 @@ stopping line that lies. | **R-1171** | Backup & restore | P3 | **demo-hp's LOCAL whole-guest copy is never restore-tested: the box keeps one copy (keep-daily 1) and the restore test wants a copy 24 h old, so the only copy turns 24 h old exactly when the next night replaces it.** Read 2026-10-20 on demo-hp (agent journal): every 6-hourly evaluation since 2026-10-18 says `local: no settled archive yet — nothing to prove`; the local storage holds one 9201 archive; the last local proof was 2026-10-14 03:42 (`settle=24h`, `keep_last=1`, `prune=keep-daily=1`). The hub's `restore_test_stale` (168 h) will alarm from ~2026-10-21 03:42. A real household with retention 1 would be the same. | OPEN — filed 2026-10-20 | demo-hp `journalctl -u felhom-agent` („restore-test evaluated"), the hub's host report `restore_tests` | Agent: a tier that keeps ONE copy proves its newest archive once it is older than the settle time measured from the BACKUP's end, not 24 h — or a settle below the cadence (e.g. 6 h) for a 1-copy tier; a test; then read the next evaluation | CC | | **R-1172** | Backup & restore | P4 | **After option A (R-893): a failed off-site restore that was PUT BACK is logged at ERROR, and an empty `backups/restore-undo/` folder stays on disk.** Measured on 9202 2026-10-20 (rc.1): the web handler logs `[ERROR] [web] off-box reconstitute rallly (async): … Nem változott semmi…` — good news at ERROR; the put-back removes the per-app folder but not the empty `restore-undo/` parent. Small; 0.336.0 was already pushed (a version is pushed once). | OPEN — filed 2026-10-20 | `audits/night2-2026-10-20/P2/live/11-controller-log-window.txt`, `12-after-failed-restore.txt` | Next controller release: the put-back outcome logged at WARN (its own sentence is the household's), the empty parent removed; a test | CC | | **R-1173** | Backup & restore | P3 | **R-921's consequence (no app stop for an off-site tier the agent will refuse) has not been seen live.** Agent 0.161.0 serves `host_busy` (proven live on demo-hp 2026-10-20 during a press: busy `backup:local`), and controller ≥ 0.335.0 reads it — but the case only arises on the night schedule (a manual press backs up the local tier only), so the no-stop branch is proven by its red-proved test alone. | OPEN — filed 2026-10-20 | `audits/night2-2026-10-20/P1/1b-host-busy-poll.txt` | Read demo-hp's controller log after the night its off-site (pbs) tier is due (~2026-10-22): a `host is busy … no app is stopped (R-921)` line, or the off-site tier starting with nothing in flight | CC | +| **R-1184** | Backup & restore | P3 | **A cut-off shared-folder restore can be „placed" and reported restored while files are missing; the download is then deleted.** Readiness is only „the scratch folder is not empty" (`internal/backup/shares_restore.go:131-140`) — the R-358 class fixed for apps, not for shares; 30-min download window (`web/offbox_handlers.go:716`); fixed 2 GiB free-space gate that fails open (`:107`). CONFIRMED in code. Data-loss hunt 2026-10-20 (C1 reviewer 1, R3). | OPEN | — | Fix: a completion marker after restic succeeds, as for apps | CC | +| **R-1185** | Backup & restore | P3 | **A database dump that fails its own content check has already replaced last night's good dump, and the second drive copies it an hour later.** `appbackup/dbdump.go:388-422`: size check, rename over the good file (`:399`), then `ValidateDump` (`:419`). Volume tars are promoted with no content check either (`backup.go:902`). Tier 1 is one point per app by design (07 §6.1); the ORDER is the defect. CONFIRMED in code. Data-loss hunt 2026-10-20 (D5). | OPEN | — | Fix: validate the temp file before the rename; keep the last good dump on a failed check | CC | +| **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC | +| **R-1187** | Backup & restore | P3 | **`.fab` export/import candidates from the data-loss hunt, not yet proven:** (a) an export silently drops a drive folder that holds a symlink (`appexport/fabplan.go:292`, `export.go:714`); (b) the import never sets file owners and drops symlinks (`restore.go:1157-1176`); (c) import into a STOPPED app merges into the existing volumes, and the default export (stop ticked) carries no SQL dump (`restore.go:361-374`, `export.go:262,366`, `appbackup/dbdump.go:142`); (d) no concurrency guard, no undo, no record (`restore.go:181-186`, `web/handler_export.go:312-353`); (e) the import drops the family gate, the sign-up lock and the version pin (`cmd/controller/main.go:3517-3530`, `stacks/family_gate.go:173-190`); (f) it trusts the bundle's HDD_PATH (`restore.go:589-631`); (g) the DB import runs psql without ON_ERROR_STOP and misses postgis (`restore.go:1052-1053, 928-937`). Read by reviewer 1; main session spot-checked (a), (e). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Prove (e) and (c) first on 9202 with a throwaway app; split into rows only what reproduces | CC | +| **R-1188** | Backup & restore | P3 | **Restore-path candidates from the data-loss hunt, not yet proven (one confirmed in code):** (a) a crash or power cut mid-restore restarts the app on a mixed state and never reads the undo copy (`appstop_marker.go:307`, `offbox_restore_undo.go:137`, `bootrecon/bootrecon.go:210-213`); (b) a failed `volume rm` is ignored before the untar (`restore.go:218`); (c) CONFIRMED in code: an interrupted second-drive whole restore leaves `.felhom-restoring`, and every retry fails „file exists" (`tier2_whole.go:86-87`, O_EXCL); (d) the full off-site download has a 30-min cap per press and the household is not told to retry (`web/offbox_handlers.go:455`); (e) place and full restore re-read the LATEST snapshot instead of the downloaded one (`offbox_restore.go:675-679`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Prove (a) with the B6 power-cut-mid-restore scenario; (c) with a kill test on 9202 | CC | +| **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt, not yet proven:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it, and the hub reads the same clock (agent `localapi/server.go:1178-1192`, `backup/runner.go:607-629`, hub `monitor/deadline.go:243`; R-973 cap counts future runs `backup_cap.go:61`); (b) the whole-guest start creeps ~5 min/night until a night is skipped silently (`server.go:1184`, `quiesce.go:150, 807-818`); (c) the whole-guest quiesce stops apps mid-restore or mid-update (`quiesce.go:213-275, 565`); (d) the off-site proof and integrity check never run on a box that is off at 05:30–06:00 (`cmd/controller/main.go:1575-1619`, comment says the opposite); (e) hub false „no DB dump" for a late or evening backup time (`monitor/deadline.go:416-419, 556-565`); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — design question; (g) spring forward 2027-03-28 fires dump and second copy together; (h) self-update busy check before the pull only (`updater.go:532-545, 764`); (i) a double-clicked restore shows failed while it runs (`web/handlers.go:1853-1874`). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`. | VERIFY | — | (a) is B6 #8's live test; (b) read the agent journal on demo-hp; the rest by unit tests | CC | +| **R-1190** | Backup & restore | P3 | **Key and retention candidates from the data-loss hunt, not yet proven:** (a) a recovery ceremony while off-site is not configured seals a package with no repo password and supersedes the good one (`escrow_handlers.go:217`, agent `main.go:2975-2981`, hub `store.go:3392`); (b) the global „create your recovery code" bar links to the ceremony the backups page hides during a recovery offer (`escrow_banner.go:23-33` vs `backups_remote.html:144-160`); (c) after a reuse-path reinstall the new PBS key K is never escrowed and nothing compares fingerprints (`felhom-pbs-apply:99`, `escrow_confirm.go:182-205`); (d) whole-guest off-site copies run before any escrow exists (design question vs 07 §6.1's Tier-3 gate); (e) the escrow PUT is two statements; (f) the operator's off-site re-issue resets the provider password before saving it (hub `offsite.go:217-245`); (g) „Megőrzött adatok" can list a running app's live folder when its app.yaml is unreadable (`kept.go:220-235, 330-402`); (h) a unit restore can point an app back at an old drive and F5 then prunes the current drive's unit (`backup.go:1517-1574`). Evidence `audits/dataloss-hunt-2026-10-20/C1-keys-candidates.md`, `C1-delete-retention-candidates.md`. | VERIFY | — | Unit tests first for (a), (e), (g) | CC | ## Storage & devices — 9 rows (P3 3, P4 6) @@ -252,7 +267,7 @@ stopping line that lies. | **R-1170** | Monitoring & notifications | P3 | **A box whose tunnel container disappears raises no alarm, even when its tunnel is configured.** The hub alarms on `not_running` only (`checkTunnel`). A guest with no cloudflared container read `unknown` since Docker 29 (lower-case „no such object", R-1070) and reads `absent` with the next agent release — neither alarms. On a box whose customer has a tunnel token, a missing container means the box is unreachable from outside. Found 2026-10-19 while fixing R-1070. | OPEN — filed 2026-10-19 | `hub/internal/api/handler.go` `checkTunnel` | Hub: treat `absent` as down when the customer's config carries a tunnel token (two reports in a row, as today); a test | CC | -## Hub & operator — 11 rows (P3 4, P4 7) +## Hub & operator — 12 rows (P3 5, P4 7) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -260,6 +275,7 @@ stopping line that lies. | **R-31** | Hub & operator | P3 | **[P2-HIGH] Offsite provisioning is synchronous with no status affordance.** Save runs the Hetzner sync in-request, so the request can hit the nginx 504 **while succeeding server-side**: the operator cannot tell failed from slow, and a retry races the first attempt. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: operator-only; a known workaround (click once, wait, verify) exists.** **2026-10-06 night: the race half fixed on felhom.eu main (hub, unreleased):** a second Save while the first still provisions is refused with 409 („already running — wait about a minute, then reload; do not save again"); nothing saved, nothing created; per customer, in memory. `TestProvision_R31_*`, red-proof `audits/night-burndown-2026-10-06/hub/R-31-red.txt`. LEFT: the async save with a status card (the escrow-card idiom). | — | Direction: make it async + a status card, reusing the proven **awaiting-card/poll idiom** (v0.138.0 escrow card). **Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click.** | CC | | **R-882** | Hub & operator | P3 | **Longhorn on DooPlex could not grow a volume online: its `instance-manager` (116 days up) called a host process that no longer existed** — `nsenter: cannot open /host/proc/196610/ns/mnt` on every expansion retry, and an offline growth was blocked by the expansion's own attachment ticket (found 2026-10-05 growing `hub-data` to 2 Gi). A restart of the instance-manager (operator-approved) fixed it: 77/77 volumes back `attached/healthy` in 110 s. **Why the cached PID went stale was not established** (likely a containerd/k3s or iscsid restart after the instance-manager started), so it will recur after the next such restart and stay invisible until a volume needs to grow. `audits/hub-db-offsite-2026-10-05/partA/step1-*.txt` | **OPEN** | — | Find which host process the PID was and whether Longhorn 1.10.x re-resolves it; until then, before growing any volume, check the instance-manager's age against the last k3s/containerd restart | operator | | **R-883** | Hub & operator | P3 | **8 DooPlex workloads run an image by a moving tag (`:latest` or none), so any pod restart is a silent upgrade.** Measured 2026-10-05: the Longhorn restart restarted zipline on `ghcr.io/diced/zipline:latest` (pull Always), which pulled 4.8.0; 4.8.0 refused its database (`cannot safely migrate from prisma to drizzle: expected migration 20260508022000 … was not applied`) and crash-looped. Fixed for zipline by pinning `4.7.0` (homelab-manifests `90f60e4`, `4c8ec7a`; 4.7.0 applied the four missing migrations; a dump from before is kept out-of-band). The other 7 were counted, not named or changed. **Checked from source 2026-10-05 (burn-down round 2):** homelab-manifests@87dfc29 still has moving tags (grep image lines without a numeric tag): admin-system/toolbox.yaml:12 nicolaka/netshoot:latest (a bare Pod); calibre-system/cwa.yaml:826 calibre-web-automated:dev; outline-system/outline.yaml:270 minio/minio:latest; tandoor-system/recipe-importer.yaml:26 gitea.dooplex.hu/admin/recipe-importer:latest (pull Always :27); adventurelog-system/adventurelog.yaml:100 and :256 adventurelog-backend/frontend:latest (pull Always :101,:257); jarrs-system/jarr- | **OPEN** | — | List the 7 (`kubectl get deploy,sts -A` images without a fixed tag), pin each to the running version, and let Renovate move them | operator | +| **R-1183** | Hub & operator | P3 | **On Sunday 2026-10-25 (the clocks go back) every hub daily job after 02:00 runs TWICE: the alarm check at 04:00 and 05:00, the hub database snapshot twice (keep 2 → the previous day's copy pruned a day early), the prune and the key audit twice.** `hub/cmd/hub/main.go:1057-1062` adds 24 h to an absolute time; the controller fixed this (`scheduler.go:511-516`), the hub did not. PROVEN by replaying the hub's exact arithmetic (`audits/dataloss-hunt-2026-10-20/C3-T9-hub-dst/replay.txt`). On 2027-03-28 each runs once, an hour late. No architecture document mentions DST. Data-loss hunt 2026-10-20 (T9). | OPEN | the frozen run (no hub deploy unattended, decision 162) | Expect duplicate operator mails on Sunday morning; fix with the controller's day+1 `time.Date` and a DST test; deploy with the operator | CC | | **R-264** | Hub & operator | P4 | **Twenty-one facts the boxes report that the hub can now decode nowhere, each allowlisted with a reason rather than silently skipped — and for these the reason is "no consumer today, and one is arguably owed".** Split out of R-260 on 2026-08-08 so that closing the CLASS (gated) and fixing its sharpest instance (`operator_key_configured`) could not be mistaken for having decided what the hub should do with the rest. **The list, grouped by what a consumer would be for.** **(a) Guest-network health — `guest_net` and its seven children** (`checked_at`, `has_route`, `dhclient_alive`, `heal_succeeded`, `heals_last_hour`, `last_heal_at`, `damped`). The R-54 watchdog reports per-guest network state and self-heal counts every cycle and the hub — the component that emails the operator — models none of it. There is a live incident in this project's own record where a killed `dhclient` took a tunnel down for 1 h 15 m (`audits/INCIDENT-guest-dhclient-killed-2026-07-20.md`); a recurring-heal signal is exactly what would have surfaced it. **This is the strongest candidate of the twenty-one.** **(b) `selfupdate_pending` + `selfupdate_pending_version`** — an agent that has flipped its binary and never committed reports pending on every heartbeat so that "the operator sees WHY the version isn't advancing", and no operator can see it. **(c) `mgmt_plane.healed_recently`** — bounded: the hub DOES alarm on the `privsep_healed_at` timestamp beside it, so the recurring-clobber signal is not lost, only this flag. **(d) `restore_tests.mount_parity` + `mount_inventory`** — R-262's subject; the verdict is not lost (a mismatch fails before `Pass` is set) but the hub cannot tell a full-fidelity pass from a boot-only one. **(e) `pbs_dr.applied_at`.** **(f) Controller-side: `config_hash`, `reporting_disabled`, `stacks`, `storage.migrated_to`, `backup.last_db_dump`, `backup.last_integrity_check`** — the last two are backup-integrity timestamps, which is the "presence is not success" neighbourhood. **For each the question is the same and is NOT answered here: is it wanted? If the hub should act on it, model it and name what consults it. If it should not, the honest end is that the emitter stops sending it** — a fact emitted forever and consumed nowhere is a future false green waiting for someone to write a check against it. **Deliberately not decided in the G-1 session**, whose scope was the gate plus the operator-access instance; unilaterally removing emitters would also break the byte-identical cross-repo host-report golden and is a coordinated two-repo change. **⚠ DISPOSITIONS RECORDED 2026-08-13 — and the first thing to say is that they were NOT in this register before today.** The rulings were made on 2026-08-12; this row still read **READY — owner Viktor** and the gate's twenty entries still all said *"arguably owed"*, so a session told to *"re-read the dispositions from the register"* would have found none. They are written down now, which is the point of writing them down. **THE COUNT WAS ALSO WRONG:** this row says *twenty-one*; the gate's allowlist held **twenty**, measured. Twenty is the number the dispositions below account for, exactly. **(1) BUILD A READER — four groups, fourteen facts.** (a) guest-network health, (b) the staged-update-pending pair, (d) the restore-test depth pair, (f-part) the two backup-integrity timestamps. **(2) NO READER WANTED — five facts, now recorded as `not consumed, DELIBERATELY` with the ruling and its date** in `scripts/wire_contract_gate.py`, each with its own reason rather than a bare refusal: `mgmt_plane.healed_recently` (the hub already alarms on the timestamp beside it), `pbs_dr.applied_at` (`pbs_dr.state` is the verdict; the timestamp alone is the attempt-read-as-result trap), `config_hash` (the hub authors the config and knows its own generation), `stacks` (the app view is built from the purpose-built `app_telemetry` wire), `storage.migrated_to` (box-local bookkeeping with no hub-side intent to reconcile against). **The emitters are deliberately left alone** — removing one is a coordinated two-repo change and breaks the host-report golden; the honest end here is a recorded decision, not a deletion. The gate grew a THIRD entry kind to carry them, because an undecided fact and a decided one must not read alike. **(3) `reporting_disabled`, decided on its own merits: RECLASSIFIED `redundant`** — `health.status = "disabled"` travels in the same minimal report, is decoded into `reports.health_status`, and IS rendered. **The decision surfaced a real defect the flag would not have fixed: the staleness checker is age-only, so a deliberately-silent box still alarms → R-321.** **PROGRESS, 2026-08-13: the first reader is BUILT — guest-network health, R-319.** Its eight allowlist entries are **removed** (an allowlisted tag is skipped, so leaving them would have meant the new reader's fields were never checked); the gate's checked-tag count rose **182 → 190** and skipped fell **88 → 80**, which is the positive control that the wiring is real. **WHERE THE TWENTY NOW STAND: 8 read · 5 deliberately unread · 1 redundant · 6 still owed a reader** (`selfupdate_pending`, `selfupdate_pending_version`, `restore_tests.mount_parity`, `restore_tests.mount_inventory`, `backup.last_db_dump`, `backup.last_integrity_check`) — counts measured from the allowlist, not estimated. **Only ONE reader was built on purpose:** four at once is a design session pretending to be an implementation, and this one now tells us what the other three cost | **READY** — operator ruling 2026-10-12 (S7, `09` §3 decision 202): CC owns the six owed readers and builds them when the queue is idle, one group per session, attended hub deploys. **OPEN — 6 of 20 still owed a reader; dispositions recorded 2026-08-13, first reader shipped (R-319)** — owner Viktor | — | Last in the 2026-10-12 arc | CC | | **R-451** | Hub & operator | P4 | **[P3-LOW] UPDATE ARC SLICE 7 — a fleet sweep: the operator can SEE, and MOVE, how far behind every box is.** Slices 1 and 2 make one box's state visible on that box's own pages. The operator has no fleet view, and **it is not derivable from what is already reported: the hub's report payload carries container name, state, CPU and memory, and NO image field at all** (spike §5, which is why Peti's box could only be recorded UNKNOWN). So this is a hub-side change as well as a controller one. Rank LOW today because the fleet is two enrolled boxes; it rises with the fleet. `architecture/09-update-architecture.md` §6, §8.4 **BOTH SIDES VERIFIED 2026-09-21, and it is cheaper than this row implies.** The controller's payload carries no image (`internal/report/types.go` L98–103) and the hub's `Store.SaveReport` (`hub/internal/store/store.go:965`) denormalises only container **counts** — but **the hub stores the raw report JSON whole**, so a new controller field lands there the day it is sent. What is missing is the denormalisation and the page, not the transport. Shape in `09` §6.2–6.3; the payload question is `09` §3b **Q7**. **-- RULED 2026-09-23 (`09` §3 decision 18):** the report carries, per compose service (database included), the installed reference, the catalog reference and the badge state. **Built later, when the fleet grows** — Q7's recommendation, confirmed. | **DEFERRED** (2026-10-03 triage: the row's verdict was finished, but it names open work no other row carries — the fleet sweep (report field, hub denormalisation, fleet page) is ruled and not built — deferred until the fleet grows (`09` §6.3)) — **RULED — build deferred until the fleet grows; owner: CC** | — | — | CC | | **R-814** | Hub & operator | P4 | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | **VERIFY** (2026-10-03 triage: a July watch row with no id; given R-814. WAITING-ON-OPERATOR — no record found that the box was deleted.) — WAITING-ON-OPERATOR | operator console | Delete the box | operator |