From 3add386aa9002d98f1e2d6f3e201729aa589e105 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 24 Oct 2026 21:08:38 +0200 Subject: [PATCH] Restore-holes run (2026-10-24 night): D43 = A recorded; 07/05/03/11 carry the R-1251, R-1252, R-1254 and D43 rules; register 207 -> 208 (R-1268 opened; R-1251/1252/1253/1257 VERIFY, R-1254 NARROWED); STATUS Monday list + D44/D45 picks; evidence (Tester 1, 9202) --- CONTEXT.md | 10 + REPORT-restore-holes-2026-10-24.md | 29 ++ STATUS.md | 15 +- documentation/architecture/03-host-agent.md | 7 + .../architecture/05-hub-architecture.md | 6 + .../architecture/07-backup-architecture.md | 37 ++ documentation/architecture/11-os-updates.md | 4 + .../P1-tester1/00-as-found.txt | 347 ++++++++++++++++++ .../P1-tester1/01-deploy.txt | 8 + .../P1-tester1/02-gen1.txt | 12 + .../P1-tester1/03-backup.txt | 25 ++ .../P1-tester1/04-gen2.txt | 1 + .../P1-tester1/05-restore-log-until-cut.txt | 51 +++ .../05-restore-log-until-cut.txt.cut | 1 + .../P1-tester1/06-disk-off-and-boot.txt | 9 + .../P1-tester1/07-boot-without-drive.txt | 16 + .../P1-tester1/08-start-refused.txt | 7 + .../09-controller-log-before-replug.txt | 148 ++++++++ .../P1-tester1/10-replug.txt | 5 + .../P1-tester1/11-drive-returns.txt | 30 ++ .../P1-tester1/12-after-replug-state.txt | 15 + .../P1-tester1/13-controller-log-run1-end.txt | 191 ++++++++++ .../P1-tester1/14-run1-teardown.txt | 6 + .../P1-tester1/15-run2-setup.txt | 8 + .../P1-tester1/20-run2-deploy.txt | 17 + .../P1-tester1/21-run2-log-until-cut.txt | 10 + .../P1-tester1/21-run2-log-until-cut.txt.cut | 1 + .../P1-tester1/22-run2-disk-off-and-boot.txt | 6 + .../P1-tester1/23-run2-boot-without-drive.txt | 19 + .../P1-tester1/24-run2-start-refused.txt | 3 + .../P1-tester1/25-run2-log-before-replug.txt | 146 ++++++++ .../P1-tester1/26-run2-replug.txt | 5 + .../P1-tester1/27-run2-drive-returns.txt | 49 +++ .../P1-tester1/28-run2-later-restart.txt | 4 + .../P1-tester1/29-run2-log-after-restart.txt | 122 ++++++ .../P1-tester1/30-run2-agent-journal.txt | 20 + .../P1-tester1/40-teardown.txt | 36 ++ .../P1-tester1/41-at-end.txt | 45 +++ .../P1-tester1/42-anon-volumes.txt | 14 + .../P2-9202/00-as-found.txt | 43 +++ .../P2-9202/01-switch.txt | 6 + .../P2-9202/02-control-runs.txt | 52 +++ .../P2-9202/03-break.txt | 3 + .../P2-9202/04-fault-run.txt | 29 ++ .../P2-9202/05-unbreak-run.txt | 27 ++ .../P2-9202/06-teardown.txt | 5 + .../P2-9202/07-at-end.txt | 43 +++ .../P2-9202/08-anon.txt | 8 + .../P3-9202/00-as-found.txt | 25 ++ .../P3-9202/01-setup.txt | 16 + .../P3-9202/02-race.txt | 26 ++ .../P3-9202/03-control-and-teardown.txt | 38 ++ .../P3-9202/04-at-end.txt | 25 ++ .../audits/restore-holes-2026-10-24/README.md | 65 ++++ .../tools/chain9202.sh | 9 + .../tools/cut_on_line.sh | 13 + .../restore-holes-2026-10-24/tools/t1.py | 99 +++++ .../D44-D45-picks.md | 59 +++ documentation/backlog/OPEN-ITEMS.md | 17 +- documentation/backlog/STATUS-HISTORY.md | 5 + 60 files changed, 2082 insertions(+), 16 deletions(-) create mode 100644 REPORT-restore-holes-2026-10-24.md create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/00-as-found.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/01-deploy.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/02-gen1.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/03-backup.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/04-gen2.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/05-restore-log-until-cut.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/05-restore-log-until-cut.txt.cut create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/06-disk-off-and-boot.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/07-boot-without-drive.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/08-start-refused.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/09-controller-log-before-replug.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/10-replug.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/11-drive-returns.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/12-after-replug-state.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/13-controller-log-run1-end.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/15-run2-setup.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/20-run2-deploy.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/21-run2-log-until-cut.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/21-run2-log-until-cut.txt.cut create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/22-run2-disk-off-and-boot.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/23-run2-boot-without-drive.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/24-run2-start-refused.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/25-run2-log-before-replug.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/26-run2-replug.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/27-run2-drive-returns.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/28-run2-later-restart.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/29-run2-log-after-restart.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/30-run2-agent-journal.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/40-teardown.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/41-at-end.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P2-9202/00-as-found.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P2-9202/01-switch.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P2-9202/02-control-runs.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P2-9202/03-break.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P2-9202/04-fault-run.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P2-9202/05-unbreak-run.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P2-9202/06-teardown.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P2-9202/07-at-end.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P2-9202/08-anon.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P3-9202/00-as-found.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P3-9202/01-setup.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P3-9202/02-race.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P3-9202/03-control-and-teardown.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/P3-9202/04-at-end.txt create mode 100644 documentation/audits/restore-holes-2026-10-24/README.md create mode 100755 documentation/audits/restore-holes-2026-10-24/tools/chain9202.sh create mode 100755 documentation/audits/restore-holes-2026-10-24/tools/cut_on_line.sh create mode 100644 documentation/audits/restore-holes-2026-10-24/tools/t1.py create mode 100644 documentation/audits/review-round-2-2026-10-24/D44-D45-picks.md diff --git a/CONTEXT.md b/CONTEXT.md index 309c02cd..cf769758 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -63,6 +63,16 @@ > STATUS:** the default limit's size (CC: 50 % of the measured upload) and who is told (CC: the family only about the > qBittorrent limit they can change; the unexplained alarms to the operator only — the R-388 rule). +> **2026-10-24 (night, unattended) — the restore and backup holes; operator ruling D43 = A** +> (`audits/restore-holes-2026-10-24/README.md`). **Ruling D43 = A (operator, 2026-10-24 evening, by starting the run):** +> the root-key-only sshd drop-in applies only in appliance mode; BYO hosts keep their SSH settings (`11` §1, `03`). +> Fixed on main, nothing released: controller `bb2809d`/`062f9b4` (R-1251: a pending put-back blocks every start — a +> restore hold `putback_pending`; the put-back runs first when the drive binds; a failed put-back holds; the copy synced +> before the marker names it), `518a135` (R-1252: reconcile only under a valid classification), `c9b4b82` (R-1253: a +> stop-first export refuses while another operation holds the marker); hub `dcf4f84c` (R-1254: escrow in one +> transaction; image built locally, not pushed); agent `c8d3c17` (R-1257: D43). Proven: R-1251 on Tester 1 (power cut, +> disk off, disk back), R-1252 and R-1253 on 9202. D44/D45 prepared: `audits/review-round-2-2026-10-24/D44-D45-picks.md`. + > **2026-10-24 (evening, unattended) — review round 2** (`audits/review-round-2-2026-10-24/README.md`). The 17 architecture > documents checked against `main`: facts corrected at `file:line` (EDITS.md), rulings and promises left for the operator > (R-1266, D45), undocumented decisions written as notes (R-1265, D44). New: **R-1251 (P1)** a cut restore of a drive-backed diff --git a/REPORT-restore-holes-2026-10-24.md b/REPORT-restore-holes-2026-10-24.md new file mode 100644 index 00000000..8345e273 --- /dev/null +++ b/REPORT-restore-holes-2026-10-24.md @@ -0,0 +1,29 @@ +# REPORT — the restore and backup holes, D43 = A, and the D44/D45 picks (2026-10-24 night, unattended) + +**Plain words first.** +- A power cut during a restore of an app on a drive no longer brings old data back later (R-1251, the P1). Proven on + Tester 1 with a real power-off and a drive that came back late. +- One wrong line in an app's backup settings no longer deletes its second-drive copy (R-1252). Proven on 9202. +- An export no longer overwrites a running restore's record (R-1253, proven on 9202); the hub saves a recovery package + in one step (R-1254, tested, not deployed). +- BYO hosts keep their own SSH settings (your D43 = A, R-1257). Tested in the bundle's harness. +- The review's 32 notes have one line each with a pick: answer „all picks" or name the ones you disagree with. +- Nothing is released. Monday: controller v0.343.0, agent v0.165.0 + your vouch, hub v0.170.0. + +**Evidence and tables:** `documentation/audits/restore-holes-2026-10-24/README.md`; the picks: +`documentation/audits/review-round-2-2026-10-24/D44-D45-picks.md`. + +**Commits:** controller `bb2809d`, `062f9b4`, `518a135`, `c9b4b82`, `e1613ae`; felhom.eu `dcf4f84c` (hub) and this +commit (documents, register, STATUS, CONTEXT, evidence); felhom-agent `c8d3c17`, `4a1a849`. CI by hash: in the final +report. + +**Register:** before 207, after 208; opened 1 (R-1268), closed 0. R-1251/R-1252/R-1253/R-1257 → VERIFY (fixed on main, +waits for the release); R-1254 → NARROWED (the hub half fixed); R-1265/R-1266 carry the picks; R-1267 extended. + +**Documents edited:** `07` (the R-1251 rule after D22's paragraph; the R-1252 rule in §8.2a; a pointer in §6.5), `05` +(the escrow transaction), `03` and `11` §1 (D43 = A). **System poster:** no fact on it changed (none of the four). +**Instruction files:** none edited. + +**Machines:** Tester 1 and 9202 put back as found except leftover empty anonymous volumes of the R-1267 class (named in +the audit). demo-hp: nothing provisioned; the catalog sweep's VM untouched. DooPlex, ep0, household boxes, the hub: not +changed. No release, deploy, vouch, publish or catalog push. diff --git a/STATUS.md b/STATUS.md index 155f2ed1..0f6a4722 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-10-24 (evening, unattended review).** The 17 design documents were checked against the code; about 130 old facts are corrected, no rule changed. Two new data-loss holes are proven; both need a fix in an attended run. Two small controller faults are fixed, not released. +**Updated 2026-10-24 (night, unattended fix run).** Four restore and backup holes are fixed on main (three proven on test boxes); BYO hosts keep their SSH settings (D43 = A). Nothing is released — see Monday's list. ## What works - **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241). @@ -15,10 +15,10 @@ - The three test boxes answer only over https; the hub's key opens nothing on a box; password guesses are counted (map E12, E13, E17). - A hacked app that uploads or computes more than it explains alarms only you; each box tests its upload weekly; qBittorrent gets half (map F13). - An urgent security fix reaches the boxes at night after one test on the test box; the box backs up first and undoes a failed update (map B23). -- The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15). +- The hub mails alarms and recoveries and keeps a weekly whole-box copy on ep0; CC uses its own limited token (map F1, F11, G15). ## What is broken -- **P1 (new):** a power cut while an app on a drive is restored can later pour old data over new data (R-1251); one wrong line in an app's backup settings deletes its second-drive copy (R-1252, P2). +- **P1 (fixed on main, NOT released):** a power cut while an app on a drive is restored can later pour old data over new data (R-1251); one wrong line in an app's backup settings deletes its second-drive copy (R-1252, P2). - **P1:** any hacked app on DooPlex can read the hub's nightly backups, and maybe the signing keys (R-1241, homelab session); about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours). - Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). The scratch box cannot use agent features (R-1083). - DooPlex: something sets the shared checkout to „bare" (R-935). @@ -32,9 +32,8 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| -| D45 | R-1266 | Six places where a design document says one thing and the code does another (two update rules, the golden check, „the hub never connects into a box", „the operator cannot open the off-site copy", the tunnel plan). Should the text follow the code? | **Yes — the text follows the code** (each is how the system works today; you read every changed sentence) | the code follows the text: six code changes, one of them to the box's door for you | the documents keep saying what is not true | — | -| D44 | R-1265 | Confirm about 25 behaviours the code has but no document named (now written as notes), and that MeTube's downloads are not backed up? | **Yes, all** (each was read in the code; none adds risk) | name the ones to change; each becomes a row | the notes stay marked „undocumented" | — | -| D43 | R-1257 | The SSH rule „root logs in by key only" also reaches a box the household owns (a BYO host), where we promised to change nothing on the host. Keep it there too? | **No — skip it on BYO hosts** (it is their machine; we said we would not touch it) | keep it: a BYO owner who uses a root password over SSH loses it at the next update, and the installer screen must say so | it reaches every BYO host at its next update | — | +| D45 | R-1266 | Seven places where a design document says one thing and the code does another. One line each, with a pick: `documentation/audits/review-round-2-2026-10-24/D44-D45-picks.md`. Answer „all picks" or name the ones you disagree with. | **Yes — the text follows the code** (each is how the system works today; you read every changed sentence) | the code follows the text: six code changes, one of them to the box's door for you | the documents keep saying what is not true | — | +| D44 | R-1265 | 25 behaviours the code has but no document named. One line each, with a pick (21 confirm, 4 small changes): the same file. Answer „all picks" or name the ones you disagree with. | **All picks** (each was read in the code) | name the ones to change; each becomes a row | the notes stay marked „undocumented" | — | | D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — | | D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — | | D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — | @@ -51,11 +50,11 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14. 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12. +**Monday 2026-10-26, release list (attended).** 1. Controller v0.343.0: the restore and backup fixes (R-1251–R-1253), R-1250. If not: the P1 stays open. 2. Agent v0.165.0 + your vouch: BYO keeps its SSH (R-1257). 3. Hub v0.170.0: the recovery package in one step (R-1254). **Day, 2026-10-23 — your do list.** 1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens. 2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef. If not: two dead entries alarm. -3. D37 and D38 in the table (two small yes/no). If not: Gitea may restart in release hours; no golden in the release copy. -4. Give the homelab session R-1241, R-1242, the 7-item list at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md`, the 5-item list at the end of `documentation/audits/dooplex-alarms-2026-10-24/README.md` (the database's two passwords), and an upload alarm on DooPlex's `eno1` (the line is ~100 Mbit/s; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups. +3. Give the homelab session R-1241, R-1242, the lists at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` and `documentation/audits/dooplex-alarms-2026-10-24/README.md`, and an upload alarm on DooPlex's `eno1` (~100 Mbit/s line; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups. **The rest** - System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy. - Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09. diff --git a/documentation/architecture/03-host-agent.md b/documentation/architecture/03-host-agent.md index b8e65467..0060587c 100644 --- a/documentation/architecture/03-host-agent.md +++ b/documentation/architecture/03-host-agent.md @@ -130,6 +130,13 @@ the one trust root; the agent's API token is untouched; a box gets the new rule 0.146.0 cannot update itself on that box). **[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — SHIPPED in agent v0.163.0 (`1109092`) and installer-v1.36.0 (`manifests/webpage.yaml:347`); corrected 2026-10-24] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag. +**[DESIGN, operator ruling D43 = A, 2026-10-24 — committed agent `c8d3c17`, unreleased; R-1257]** **Appliances only.** The +drop-in's bundle policy is `appliance`: written only when the root-owned install record (`/var/lib/felhom-install/state.json`) +says `appliance` (the installer records the mode before it installs the bundle); on a `byo` record nothing is written and +a copy WE wrote (this path, beginning with our header line) is removed, then `sshd -t` and a reload; a box with no +readable record is left as it is. The „in force" check asks `sshd -T` for each of the host's own global IPv4 addresses +(a `Match Address ` block no longer reads as in force); a failing `sshd -t` puts back ONLY the SSH file and keeps +the rest of the bundle (the report says `sshd_step_undone`). **[FACT, agent `a0c88e5`, released in v0.163.0 — corrected 2026-10-24] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b). diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index 06575018..e0b388d4 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -296,6 +296,12 @@ What remains: PBS repo and namespace in the **DR recipe** (`dr_recipe`, reported every cycle — R-106 fixed it to match the backup); the endpoint's PBS fingerprint in **tenantsync** (`hub/internal/tenantsync`); the wrapped key in **`host_escrow.blob`** (and the identity blob beside it). + **[DESIGN] 2026-10-24 (R-1254; committed felhom.eu `dcf4f84c`, hub next v0.170.0 — NOT deployed):** the escrow PUT + stores the blob, its sealed-password hash and the identity blob in ONE transaction (`Store.SaveHostEscrowWithIdentity`; + the superseded copy is made inside it). A failure leaves all of the old package or all of the new — never the new hash + beside the old identity package, which a box would read as „escrowed" while the written code opened nothing. A + malformed identity blob is refused before anything is stored. (The agent ceremony's „the previous code still works" + after a lost answer, R-1254's other half, stays open.) Both existing infra-backup tables retire — `infra_backup_versions` (the current/live one, all readers hit it) **and** `infra_backups` (the deprecated legacy mirror). (The planned slim DR record on `hosts` was never diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 830be65e..5cd1da35 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -778,6 +778,28 @@ than the unit), the app running, the record reads the D22 sentence. Known limit named volume the unit replaces (none in the catalog today — a drive-file app's unit restore is refused by R-538) is not covered by the volume copy. +**`[DESIGN]` 2026-10-24 — a pending put-back blocks EVERY start (R-1251; review round 2 HUNT-1 H1, H2, H7; committed in +controller `bb2809d` + `062f9b4`, unreleased — rides the next controller release).** For a drive-backed app the copy lives on +the app's drive, and at start the drive is often not bound yet. A copy that cannot be reached while its drive is not +bound is **waiting, never „gone"**: the crash recovery writes a restore hold (`putback_pending`, the copy's folder +in it) and starts nothing. The hold is the one store every start path already honours — the household's +Start/Restart/Update, the boot recreate, the drive return, the media restart, the boot sweep, the night chain, the +update leg — so no path has its own check. When the drive binds, the put-back runs FIRST (the boot recreate and the +drive-return restart; the start and every drive return sweep the waiting holds), the hold is lifted, then the app +starts. The household reads „…A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el…" on the +app and „Ne indítsd újra a visszaállítást…" in the restore record. A put-back that FAILS (at start or when the drive +returns), or a copy gone while its drive is bound, leaves the app **held** with the D20 hold („félúton megszakadt"), +the copy kept, and the marker lets go of the app — never stopped-and-unheld (H2). Stop pressed during the restore still +gets the put-back; only the start is skipped (H7). A removal clears a waiting put-back; a later restore that completes +drops it (the household chose that data). The copy's folders are synced to disk BEFORE the marker names the copy. +**`[FACT]` 2026-10-24 night, Tester 1 (VM 341; MeTube on the data drive, 1.5 GB in its volume; +`audits/restore-holes-2026-10-24/P1-tester1/`):** a local restore cut by powering the VM off 2.1 s after „the restore +copy is recorded", the data disk taken off, the box booted: held as waiting, Start and Update refused (409, the +sentence); the disk plugged back: the agent re-mounted it, the boot recreate put the copy back BEFORE the start, the +household's newest data („gen2") and its download on the drive were there, and a later controller restart poured +nothing. Run 1 of the same proof found the journal left as `undo.json.tmp` (the rename lived only in the page cache) — +the app was held for support, never started on the cut data; the folder sync above fixed it, run 2 passed. + **[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.** Recorded here rather than only in a closed register row, because a decision that survives only inside a closed work item is a decision nobody will find. @@ -888,6 +910,8 @@ run makes it longer. ### 6.5 Kept data — what a removed app leaves on the drive (controller v0.274.0, `09` §3 decision 36) +*Second-drive copies that are KEPT (D25) and the rule that a rejected backup block never deletes one: §8.2a (R-1252, 2026-10-24).* + **What it is.** "Remove the app, keep my data" leaves the app's drive folder (`/appdata/`, or the folder its definition binds through `${HDD_PATH}`) in place. A reinstall over it no longer runs silently into the old files (R-657): the install asks „A megőrzött adataimat használom" / "Use my kept data" (the database from the newest copy of @@ -1597,6 +1621,19 @@ or pattern. **Why:** the data-loss hunt of 2026-10-20 proved two nights that del (R-1175: one share group's reconcile removed the other groups' copies, and a dead drive's copy; R-1181: a missing or emptied folder took its copy within one night). Controller: `internal/backup/tier2_kept.go`. +**`[DESIGN]` 2026-10-24 — a rejected backup block never deletes a copy (R-1252; review round 2 HUNT-2 H1, H2, H9; +committed in controller `518a135`, unreleased).** The night's reconcile (deleting dest dirs the legs no longer name) +runs ONLY under a valid classification. An app whose block is rejected (one bad line rejects the whole block) or absent +falls back to the legacy legs, and nothing is reconciled that night — every existing copy stays. A REJECTED block puts +„Az alkalmazás mentési beállításai most nem olvashatók…" on the app's second-drive status and tells the operator once a +day (the operator-only digest `backup_run_failures`, leg `tier2-settings-unreadable`). A kept copy (D25) is always in +the keep set, also when its leg later leaves the declared set (H2); Remove with „remove backups" drops the kept-copy +records of what it deleted (H9). **`[FACT]` 2026-10-24 night, 9202 (`audits/restore-holes-2026-10-24/P2-9202/`):** +audiobookshelf with a household file in `media/audiobooks`; two control runs kept the copy; one bogus path in the +block → „the backup block is REJECTED … every existing copy on the second drive is KEPT", the copy there, the status +sentence set, the operator digest sent (9202 has no hub: logged as dropped); the review's run on 0.342.0 the same day +deleted that copy (`review-round-2-2026-10-24/proofs/H2-bad-backup-block/`). + ### 8.2b The second-drive copy runs in a memory-capped container that never outlives the copy [DESIGN + FACT, R-1214, 2026-10-22/23 nights; rides controller 0.340.0] The copy (`rsync -a --delete`) runs in a container of the controller's own image, capped at 512 MB, so its page cache diff --git a/documentation/architecture/11-os-updates.md b/documentation/architecture/11-os-updates.md index 54985599..7c0c347d 100644 --- a/documentation/architecture/11-os-updates.md +++ b/documentation/architecture/11-os-updates.md @@ -76,6 +76,10 @@ boxes installed. Nobody keeps a hand-written list. - **A BYO host** (the owner brought their own Proxmox). `[FACT]` The installer leaves its repositories alone: *"apt repo alignment skipped (byo — the owner manages repos)"* (`scripts/felhom-host-install.sh`, `align_apt_repos`). `[PROPOSAL]` On a BYO box we update the guest and Docker only, never the host. + **[DESIGN, operator ruling D43 = A, 2026-10-24 — committed agent `c8d3c17`, unreleased]** The host's root-key-only sshd + drop-in (D4, R-982) is an APPLIANCE-only file of the config bundle: written only when the root-owned install record + says `appliance`; on a `byo` record nothing is written and a copy the bundle wrote earlier (this path, our first line) + is removed; a box whose record proves nothing is left as it is (R-1257). - **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5). - **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/00-as-found.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/00-as-found.txt new file mode 100644 index 00000000..8cd2043e --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/00-as-found.txt @@ -0,0 +1,347 @@ +# Tester 1 as found — 2026-10-24T17:13:10Z +## host +felhom +felhom-agent 0.164.0 + 19:13:10 up 1 day, 1:40, 1 user, load average: 0.37, 0.39, 0.41 +├─/mnt/adatlemez /dev/sdb +│ └─/mnt/felhom-drives/adatlemez /dev/sdb[/felhom-data] +/dev/mapper/pve-root 59G 16G 41G 28% / +/dev/sdb 49G 31G 17G 66% /mnt/adatlemez +## guest +gitea.dooplex.hu/admin/felhom-controller:0.342.0 +bookstack lscr.io/linuxserver/bookstack:26.09.1 Up 14 hours (healthy) +bookstack-db mariadb:12.3 Up 14 hours (healthy) +cloudflared cloudflare/cloudflared:2026.9.3 Up 26 hours (healthy) +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 9 hours (healthy) +filebrowser gtstef/filebrowser:1.5.6-stable Up 26 hours (healthy) +paperless-postgres postgres:18-alpine Up 14 hours (healthy) +paperless-redis redis:7-alpine Up 14 hours (healthy) +paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 14 hours (healthy) +privatebin privatebin/pdo:2.0.6 Up 14 hours (healthy) +traefik traefik:v3.7.13 Up 26 hours +-- volumes +026676d188d863de11370e48d6decb4a3c9f18e291d47e6b6f93134c04fd8b8a +0307ed61b95007e0e6c16b610151f970d48ff44aaf27107e142642f1542b0aff +03fb88df97c2caadbad5b2939e66a2ac7405a6b70c3bc9015dcdee638f4f559c +0512f70d8466285288b2898f5e5a3d1894fa693da5bc437149ce8bb6e292dd3d +05be881c3d6df732ac8accd17b8b60ae528f3d4dc3e82dbcc581ec0fc11788fd +07e9e1a0abf55dcc8b862c1cfda992ae61c6ba6abe1cadbaadc6a88ba27389ba +087a9bace33db5ca4cb7ba5321c5179148b70d234bb89c79580cedc538269542 +0af1d7ba94faf62af4085edae6877f0eaf9e7d3327eea5c78ea7c1b374604e36 +0b2999097abdee1118b633751262f8cd305d964843d0ef7e5fcb13cd4943c28d +0c60ac019df853c9a423de4d8df848c9ba55997fe972df8daee0cadd9cd0a8e9 +0dbbd609c9ffebb17db835a8e78c5e5055f4d406fd391f0a98df8da2acdc1c80 +0e80fb67355307fe7a308c935143ebcffe92ef48c19c6ac8f683f1be2cdcaac4 +0ed0950de66f59e390312976d2dcbc482fc46ccc6116eaedffefc8385ee569cd +13cdc778636898ab0ae3f0608517b74b2d37f304ad27f38994c63bd2d1df88e1 +169495ece08dae7a0f7cd9db8644d848390d105ab7decac010f93f960b25770e +1736f9912df81d3baa0948c81e0c29d060257e5dc71bb48825e7ca0f2e332ec9 +17a70549ee795512a61f658effcf0d58782819edfc637cc7450a932809410800 +18d8583ed522380a43e6c3bd5ff8f71f3f182e1035cb58297d875efe0a2a9cdd +1a2ee8eb082b857bbe7faf0afb09302e71a7ddb759fd94d96bf33c6b1dfd595c +1b1cfa1031168bfe8f253b181cd596b2e26a0bc2d1f2bab62a06837dc7d4a6fd +1b290754d93a7b164751ab1a1adc8c33bfed608152945f0778a4309c988f156e +1bd7c60829f82c73bf745abcdbd3c638d6170759a787d356ed3d007f6f7ef44f +1d0dfc3bcdbf21fd53119721be0994da359573807740a24f681cdc26403f3849 +1dc145b6978bcf718f58b2870b56f3ccb4de31bdd46f492a1fe80b50b27c39c0 +1e97663bce6009c54dfc1f46397070fc89e5e93a5c9f2603cafcf69c0e6cd033 +1efb8232cca07e79814c3312fb01742a04b1b4e9f685f2a05850971ee266bcd7 +20c2493284ddbd104a18a8f46ed591ff9db77e5dac8c6c6dd5424606e0299fb3 +23c16e83962478de5f949ad9ca3c87e2bd4dc1411f93def1054bc99102211d0f +254bb12bbe1265abebbb46d929e9b9a20c76a2c4e51b248075ca8c4f553c8481 +29d4c05b0a58d8e3ddf299dfb77c312ef4604ce3580caf565f8b6effb03866ed +2ad544ce19a9fae74fd836bf867f20884a3c58cbcb27af67e8e2ff514c91b414 +2d6e67e3979b18ec184b09ec197bb175002f029c1ed59442c6f1817823236747 +2e00ffe61ba63c08c1937440678e26b3673807e3629523e0d6b873c6c337d989 +300b348c97bc82d526b924691f5c2ec2cde931a1b32e57f403b29a132ff60d69 +3083f29c152163f2aecc43c938f99c132cb2c94a3ea6a42e6c2735599eff6493 +31d03e714a361cbeb031105de688981c659c5195ae2eabacfc92f493d3c56ead +33ae418cf3f0923739bfafb632c10d89e55bdb1264f5b5d1a3060bcbdd8510a5 +341e4cdfefa6769d31b3e1bcf50f941a560fa91545fff410e1ba2e06d0362c91 +352d14b0639f21ebec9290dd0dd695f3b32007ff609f04088116306d9f382577 +3573efc924d1f6ff9a44e8e9bf455bbf1ca809b8c49cbc468c54d29ef711c054 +35eae9e23b7dbbe0068613f2a99308d3e8f5d81246b1956a14c84e932b7913de +3665996bb9b3d30e859cce10cd616d5030337d583b5ef98d80cebe0ea80d5ecc +36e89857418bdb0db5fee915bcc57a706bdae7e7ffad640d9d7fea0aa106755b +38c6960925ea3092357512de1d0783f5ecb458750e669f0fc5e2d686e4cd08d1 +3911a1c2d88aee190825bff3923dbdaf21e09aca1b2dd231ff886436c9f4e0f9 +3982c2639123f3bf7d27ee389618e9b8436c539e5db5e08cea76b2770aa50acb +3e03055905de6ed34652f654c9f06a00e4144fc4d1a9fd34537257667e46f62d +3fdce3fd3489379cafe338eff1f282b30d61e687ac883d11cbb4c16c9cfb6d68 +40adc1b526df2547bf7690b03f6b34655ede6496fc1fe3ba2b2a366e4c5d97a9 +40b9deceaca0f4d72070ed243fb8e0c230c473fe374a4e53c45b7ec74c96ab35 +43742e9f4b2112256104565572a219a767489b96a1a13d31994b97add17b18ed +4464ee2fd1ddf58e950a02d2ab0a8d08e01c616df16ebf34591c2cc927f09728 +448a8cf6b579abd1476ab3bb5c7e2d887daa2afeaed121417e49689d098d107b +48e77a53d9ce9159d511916d9811e8a1af878cbf946d55518819a69b614ec2d6 +49f595a534f50b0c4c6e9936cdee077aeca8a4f050c2924deef5e522be241e94 +4a9c6139cf88809d39a0988527a321864d52a5402e0bdd742fc454de1c394e05 +4b176b8b9860a157765395f3b2116bf6a8551a811d5e407b37ad607bb1c728e0 +4cf4d383b6f47c68734196ec255d53957bcb0c83ea7d1f1a18c666c9db9db293 +4eef3059d5c0d83ad0adf115d2c3c0b81329b1c21b6c4e13abf27c6bc3081919 +51dc5eaee47e0d902546941ae5462363a57bab30f604f950f4f2efdb6c06dc94 +523603f8305b77733af3080e7c3f346625afc21124bc59e9ccec78e5c679b36f +53d45c8f089f14a9fdc3153ec9d932198002b4be0be58db547f56a6cd8c8317a +5692d2745f6c5f3c01479ea934451fbc4ba8982f89e60d4fc6a44fc22c25b8e1 +58261c801d3c4f9b3c40481bb73513a12234f791c465469f23a8ae1af8b65069 +58f2f64dd8c4c12028c7360ef8663ef9f1ae426f30bce1d83a73a81054a3948b +597b93a8833088933ab7281c5df8b8a4e866620b1992e235d7b62bebe5209e6b +59e65f6aee1a077839c9ddc34902d3b18a6ec5491e6df8ab35bd23cc259c158e +59f0d572040c92a3d797d98fc49654a92b393318de5da9f9779229dfa400c065 +5ad6e65fcba390d193580bff949c448a2ca9434331fdf9a42157892ea4ad8a3b +5bd3cb958f8a42194a5980266c30b9eac522455d2d15832680282c85a77f24e6 +5c28d2f2ee2ecc47776f4c06004a937778ba4d76475bbeb6506f244c621b59e2 +5dd4cd85aef07c83a68ca496eccc97c6fa3460fb732889e96684537dd5e7350a +6131874f33d3c3887516e68de0368769a8cac5348ccacfceea5ac54273a36611 +61a81b7aa4ec3b4e7bd140a236cb0b2ce60e21b9f1330a8d609cb3ab062ea27e +6503b04e0922099bbab0c241c38701177cc1d75cfdc06e7e0b283145d6e66adb +6784c394d01cfcbe663502b836cae3a629f7f39f3eb649524fd62e06d8c86cf3 +68cfc2664c15490c3d89efcbd41093ba61eee8b218dc647c12d8ef62ec997fd4 +6aceb21ff40dd2bfe14b07836e44885c5bb65ae7ace30562e72074f2fcfa47cf +6e77a58f856657d4b3da205dd99884eb97555482fd917dd42747f6c321a93283 +6f06aeea4a112e6f6ee3ee401dd3acf2189f233ba8010d259c13706d643d521e +6f9ef861cc7e00994738e64d5ab640673a66f617d6933277c7110f1135beb5ae +7090e6a6fb4233d6c7e06dac2d9fa31591388a14e5ff9af22177692951131811 +7224dac5110be6bb73a640f813218c65497aca167f97423ec3ba7c5d573efa8e +745e2aa6027e5f1a9ff6f8cb69144f8aef2e77338a85068a04be893cbb87373f +77c9939676ec7e4c56d332d2f38b3454048758fbc9b70c144d255ebf78eff866 +7a29f347e972db28979f697dfd2107373cbe219633af4fb969f961cde572d492 +7b11f5e4e56fdaa4f41587ef61da287bd0f249d2d0974ed2f7521a04bd10a23b +80691462fec8717413857aa76148b63b07fadd71ff38f8dbfc90cf3317b84437 +8076fde747347156353daa1b46aa97fbc854d9e51c8754b3c833a6674fc8bcf2 +84d1bd03752af9ab2a9df5e3af9edc84cecb294253d4522b7241cd5e8a21b940 +84f50eff27db32f45be4549210f0935faa0bec618f34085d0af741a3f9b451cb +85212d642b9775b85822fa204aa87b9add826eba1c1216d46a103fb5c09795c2 +85a130c049861e25e1672c11abff79946997074dfdbb73f48006210d2f006772 +85a4484bb1ba82f6e38837a09e6923c20c36b5afa9dc130debd7bbed6efd3c36 +86e6ca57521ffe0bdb8354abcf8d769805333bd1abc55d6d5fa34a73f69540a8 +86f2ae52a9e75a8dc526a92bfca2aab397f462b17a933dca65692fe07246b41a +86fff796264e3da25bfa88156e47098cb2fd7626edfd53ed8cc46af8ad9c9fd4 +8768803887bb29d1f3e3f0031cf1f79a031fd4a118e58426a111c90e3b562b01 +8a7ea3a230023522d445d38125d701cf2d36245da4d7020b5d03e5f26583bfcc +8c3748404cd4c9f05f232ad3de41efd09c9787be38d3f40de2a97dfe3462319c +8c738b754e6adc44dcf89cbf54084d769c08e9bde2137b7da06de55e1565e945 +8d02518fe445ffe5073953f6c59a91a75447e0d824155356fdfb723cbfc2bcf9 +8db607d6cd685eb60ec45850abc21cb690e2c2e28138e34119293219eb6f84e0 +8e91c61a79f6898f3db44eff65b32b31cf1841ec2a7542941805c4b490e9f096 +8eb670e140bfa81eb00c2102ad23ea2033775e571f353db2ff1aad6f2535dda8 +91bf64330904fafce2bdb442cb48cff15f017f1b50e4170a0b493459889f15df +92f78e44d608c3a070de162939af9248acfa2a1a6111d1a63fa68f0bc2acc597 +930d24bdff15118065d54b712d82475d10c1a677886f47f7ac582b9a69a7ddaf +93e3db6eef0cf8c3ac5beb3c6f966a9beaf4f176197746073fd4b1371d0afc4c +95357e4c1efb7d954eaf3f8b7a308fcd65aae4eea22f1988c65bd84bb0898294 +9658d1100070dd5036b2bbd673e72b8a5e50ab8086f9e8f2234feceba73e6134 +96e018f984bc36e4249c0d65d2e8c33007d63ccbfec107a0b92445be17fe0ccf +99c2977efbdbfd8abdc20694df171b41e0a0774c9b2016ecca829643358a8808 +9a1404592c789382258ec7c1a2d6be74a28ebd4dc433c039b86c89497e561165 +9ac0907283e5874963a41116da3364cb0f313603dc6dae2788ba0131128e9fbc +9ac2217865c4c77eb51cf074da4885d278bede1158a9c7d8e31f074488a6bf3f +9d92ff119044918fb0b44e1dcab2c36139e5176bb918fedd601ff1bdf88c355e +9e2452e7d60e31de7b2c83ad26d0a702f7d32679ce5404752b49e60eb68318e9 +9ef356efb45c5eb06816bd9dcf31153d254ab95aed73ad73bd24700c529a72a1 +a0d3fed2d1504d92044b20e95223c3b365877f47ae95d49751c4f41e83cd9bd1 +a59452d57d553008c895847a4676f1e54df99412453f0440aa822c954b80310a +a5a5914b129ef814b6d933ac8c94527e387bcc972c463cf477346990e22ceaf0 +a714c2bbbad8419b5238fb0e6aede18b90e8c16e04fb739bc8ed3d5c1a2b2739 +a74bbcf91ea34ba1558e4f92c400d00d5d1b4654c032bb77240f226d99be3ac2 +a7e800d4dc55c7aa8de862f050cd64473bb5d7472780f011c3c08addca55c09f +a82b87e9c13274143e8622babf9ba27d908d4713b61445055e28dbee16203c45 +ac369736ccf2fbc61d5d5044fca171e84d5689c0105f889b16cbefc6e57b9894 +ad89ce83ef046ca4d1d9fee168dd9ea1cdf33afdec91bc658bb9cd6aabfafadf +adecf80e0947e89de653f5bdc49e3eca1001bbcf208fe8f6cca236436cfb6504 +ae1ea01423ad0fcf9d2c2e19e3b59179700625804db438644ff737eadc66203e +aed401ec2d1c9e4d70b7f0893dce625a8a956baa6be10b7b168e9cf8c048b2c4 +af2ef39dea8eac3871ee860874157389c1c509c3b70ff37c4b7d990f3cf0a9b7 +b1f696383cabdbc0709ca1f955e6dd4f6083bba1922c71295245429de1b18c34 +b22a97bce70026ce57c9b35a2397c00d883588c50b0f2011c36ae44643270590 +b4449dff437f068e99c52394132b65225e2da322ad09eb668476f06a4e42f440 +b4c8a01fdc0d6890c8112d9a270de3fed5915011431ad8aee41cd480772c0b12 +b521b88edc168b2f72bb4687ec53a92f5095276484ba215bc7b2ac0a69c03861 +b71d4c0f2639a08e4380dfb3816e08eb9fc41978ba0ca2be32624d1a82e2500c +b866b4bc219c3ebd57f965ec78fe878708cf5edf7d52d881971b7e68e86c6d01 +b92895aad000c10b55b645614ba9a3ca6390781d63a1bfa2ef11ea7e236f8784 +b93d6f8106d62cf711e2e2d4b1ee973a5a1ba1c4c08cff9c62a72fcfaf17aa3a +bb4cb8ad03b92e415724e7b9ea371fa9a0719c3b244cf38e3465dc025f23e712 +bookstack_bookstack_config +bookstack_bookstack_db_data +c0a1b3cb428ce0a10c88a92cb1fe7d33efbae86303e44ac0a61cc5d49b1c166e +c0f132dee2b24dbbafc6ff8ef94c40047973c328d50f19fbdc3c0d2a1009c1d0 +c0f33ced1f6bafa7ddafe86eb0b33082d005be39dc8000e4d9e0017071bccc2f +c116cbbc8a58794e88003a08d78cc86c938c344440ab3ee2874155710ed26644 +c4634a92ca630ad48193f2dbe645f3b0c5e435daf3b76ff1359770f706723593 +c8d34ded435496db30ee00525fe00ec46d73c63e24341bd8835c080995283325 +ca98a3f6471eb0efab615e78dcc496f17fed147cfc573af648c184ff7e0aafa5 +cd33ea68493c24cd7e78e7bd5d39a19da518585af06115a3a5fa31c5c91a8a8b +ce313d72e040dcc30a4df849212511a3688d0764ed2caa814afd55dd6f8cb525 +ce8c79b3a922c33549bc79ae6af5fe88e9a7200eee7915ae2b5811584261a320 +d1285cc85513b1a6de68e6190fa4788a2a5f1b44ccfa708cc717335df3b44fb9 +d1f18944c2ab0db265da296ef140ff2d0f45303fb5e65198107c0e3e26d6f79e +d51fe1ac2fc10b9abb6d6e333fda91bd794cf0b5a36fe2060e30b2758a5507d3 +d72d9e389c168c257944b3dde0e807ed795fe77bcd36d755598b0536965df44a +d8a67a033a39fe6f886903556f8f69c5d72cf13bf4860c2d50ecf808bcf84c81 +d8feb4aae5566bdb02e7c3f29ed5d994221ce2c0054df3fa38acc1fe84c271c6 +da85e9426537a6005df390aa45832fc2dfa9093fe1e245d4cf15de061eac2efe +dcc3f781ccbda058c8dd8104c9ec76629685530b3120bb80979108c602b2bdf7 +ddde08f176b72ff003e6451eb5af3384221de9287224c0421a80155e39a6ae91 +dee8a41031f9d7e4b6a20684d22317c1159dc9f66cacc4a9708208f43a8d3306 +e014c71a27c899b0c6456c53569a2dd6c9e77126a7361bef2c99761c1d231801 +e04e180ff6f066952eadb82c3965cd4cce04dabd7423f3ac60c5cfc774bb6e82 +e0a6cf7542476d85181416e3a5b54588e692d5e2a36cb7439a0d4fc05ea75cdf +e12bef6ed1435f6789b65273b37e4620ecf7a52fa4c056414cb637770e82213a +e19cbc63c47f6602cfe5c4844b311a7292db5c70e961e0fb2220b01a0fd8762e +e3e3b934d26f9d2737c64897d302e5f9f03594e312b85307c2f4e5d60ef1c58c +e4d4dd066ab630f703fe3753b5fb00f6c88fa47c344bf51214c598b88d8afca8 +e4d62b254a585664909f6814404cfb7f1a2803c6a043cf92925b79481640628f +e589c34e20e7d78b616fd1cfed16ec332dcd20d9a819f52bf808e6250e6371f5 +e65f710998dd172fc35db9136bd058572ca65a425b21eca5f9ed412d05e675f7 +e84951d8e332c6a76085c83a3be458bf078c9c3be8796d55ea70be37d4660f41 +e87ec0a62652b292400fe4e5f0b73c484cc2bc244ced64216589930bcfd0e63b +e9385bce883dd85ba21d07cb6bcf82d262655dd4889545cad7787cb2c34c4ed7 +eab31183ef7b453dc049d816fc372d893d8d69383e96f9628df02c7e34f42198 +eb2fd76f92ec9c0dae6407c79603e82fc785cba2721498bf38ca4ea67bb993fb +ed421f3cea361f2670c63349d25bdc481a8280b4bca294a2f6a1ff7cc7a09700 +ed89b25f9b78775c9744c10a0b05214ecc61ee40a268917e0ffc05755c969e6c +edaf98a9bd0a8272f16f5a123e8e89a847717097be8e0a2fe4a6f1965915d67d +eec0215eff753e61208f8ce89c328832e95b2045635216e771c59a4bbac64ae8 +f00f2d239580728a7e6e40378a0dd2078ea243c74e440739f3536db70f7799f2 +f0603085c01674274d6e4cc3871f9ad50768859aed42364e441f6262486d2086 +f2e2beccd3dd346bb19a82e7585e100c5ffcecf64e00402db1b938bde6b84ff9 +f353c9bff2ac802c271d08011700155ba8b13d5320f8d722b431aee3fb7d9326 +f4573daac82885e39585604297044e7677d4edff30eb4cdde2edbfb2b1b2d8ae +f514957d3ebce5d70a42be290cef385fc5326644e73581024552010f869e6f0f +f58282c0d4b95da155ffe4eefa3778fdb50617dd918a733c93c7eb8c50d39c56 +f8d6f1387935ae6316af476ffe7dba55288cfeb4fa95a30fed6b6fe4156647e2 +f92b58bff009e768c65bc67bd75d410fe063414261fc841dbda26c2bae82c95c +f93ea05339c0f6c48f5cd9bf18f14922555ce62b7fd06e7563946d4134f0c444 +fb1a557fd1e7a61643134e0cbc57d7c54306330a5ec4f673f7871488e8072556 +fb3c614084787aac24fc60190bbf6b79c5f1a9c3f93a6f53566691e130a573c1 +fd26547d4646042fead55ad21bafa7ac4ffa83b464adf7f0dba4385dabe7eb59 +fe1940040b28a09bcbe20ea3819d93b119247eb5543cba462370667cfae1c663 +felhom-controller-data +ff9fcd38ad6080875b5c121ea94957128f35c5e3300f0192521a40ccf055c0bd +filebrowser_filebrowser_data +paperless-ngx_paperless_data +paperless-ngx_paperless_postgres_data +paperless-ngx_paperless_redis_data +privatebin_privatebin_data +-- drive +/mnt/felhom-drives/adatlemez/backups/: +offsite-proof +primary +secondary + +/mnt/felhom-drives/adatlemez/backups/offsite-proof: + +/mnt/felhom-drives/adatlemez/backups/primary: +paperless-ngx + +/mnt/felhom-drives/adatlemez/backups/secondary: +bookstack +jellyfin +plex +privatebin +prowlarr +qbittorrent +radarr +sonarr +transmission +ls: cannot access '/mnt/felhom-drives/adatlemez/backups/restore-undo': No such file or directory +-- marker +ls: cannot access '/var/lib/felhom/data/appstop-state.json': No such file or directory +data mount: +-- holds +Traceback (most recent call last): + File "", line 1, in + import json,sys;d=json.load(open(sys.argv[1]));print(sorted((d.get("restore_holds") or {}).keys())) + ~~~~^^^^^^^^^^^^^ +FileNotFoundError: [Errno 2] No such file or directory: '/settings.json' +/dev/mapper/pve-vm--9201--disk--1 69G 4.8G 61G 8% /var/lib/docker +/dev/sdb 49G 31G 17G 66% /mnt/felhom-drives/adatlemez +appdata-run.json +catalog-cache +dashboard-sessions.json +debug-ring.log +disk-health-state.json +encryption.key +filebrowser-sso-header +image-retention-v2.done +metrics.db +metrics.db-shm +metrics.db-wal +night-ledger.json +offbox +settings.json +settings.json.bak +setup-gate.key +update-state.json +whole-guest-ledger.json +holds: [] +ls: cannot access '/var/lib/docker/volumes/felhom-controller-data/_data/data/appstop-state.json': No such file or directory +actualbudget +adventurelog +audiobookshelf +bentopdf +bookstack +calcom +calibre-web +claper +cloudflared +code-server +crafty-controller +dawarich +docmost +emby +filebrowser +ghost +gitea +glance +gokapi +grafana +gramps-web +grimmory +grocy +home-assistant +homebox +homepage +immich +jellyfin +karakeep +kimai +komga +lubelogger +mealie +metube +n8n +navidrome +nextcloud +nzbget +onlyoffice +opengist +outline +paperless-ngx +papra +plant-it +plex +privatebin +prowlarr +qbittorrent +radarr +radicale +rallly +recipe-importer +romm +sabnzbd +seerr +sonarr +sparkyfitness +tandoor +termix +traefik +transmission +uptime-kuma +vaultwarden +vikunja +wanderer +wger +wishlist +zipline diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/01-deploy.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/01-deploy.txt new file mode 100644 index 00000000..c25fde53 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/01-deploy.txt @@ -0,0 +1,8 @@ +{"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"} + +202 +metube Up 6 seconds (healthy) +local metube_metube_state + HDD_PATH: /mnt/felhom-drives/adatlemez + - HDD_PATH +desired_state: running diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/02-gen1.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/02-gen1.txt new file mode 100644 index 00000000..2a053489 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/02-gen1.txt @@ -0,0 +1,12 @@ +total 1572880 +drwxr-xr-x 2 1000 1000 4096 Oct 24 17:16 . +drwx-----x 3 root root 4096 Oct 24 17:15 .. +-rw-r--r-- 1 root root 1610612736 Oct 24 17:16 r1251-ballast.bin +-rw-r--r-- 1 root root 54 Oct 24 17:16 r1251-gen.txt +total 12 +drwxrwsr-x 2 1000 1000 4096 Oct 24 17:16 . +drwxrwsr-x 16 root 1000 4096 Oct 13 19:26 .. +-rw-r--r-- 1 root 1000 53 Oct 24 17:16 r1251-household-video.txt +{"ok":true,"message":"Mentés elindítva"} + +200 diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/03-backup.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/03-backup.txt new file mode 100644 index 00000000..14e98ab0 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/03-backup.txt @@ -0,0 +1,25 @@ +2026/10/24 17:16:08 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/24 17:16:08 [INFO] [scheduler] Job stack-scan completed (took 115ms) +2026/10/24 17:16:09 [INFO] [scheduler] Job agent-channel-health completed (took 554ms) +2026/10/24 17:16:57 [INFO] [backup] Stopping metube for safe volume dump +2026/10/24 17:16:57 [INFO] [stacks] Stopping stack: metube +2026/10/24 17:16:58 [INFO] [stacks] Stack metube stopped successfully (took 0.7s) +2026/10/24 17:17:11 [INFO] [scheduler] Job agent-channel-health completed (took 2.614s) +2026/10/24 17:17:21 [INFO] [backup] Volume dump: metube/metube_metube_state → 1.5 GB +2026/10/24 17:17:21 [INFO] [backup] Restarting metube after volume dump +2026/10/24 17:17:21 [INFO] [stacks] Starting stack: metube +2026/10/24 17:17:22 [INFO] [stacks] Stack metube started successfully (took 0.5s) +2026/10/24 17:17:25 [INFO] [stacks] Stack metube post-start status: +2026/10/24 17:17:25 [INFO] [stacks] metube ghcr.io/alexta69/metube:2026.09.29@sha256:8e2fe9beeefc55a02f6e1d04cad0fc3c22e8f067d309188f016551dc99ec27b3 running Up 3 seconds (health: starting) +2026/10/24 17:17:50 [INFO] [backup] App-data backup completed: 2 databases (528.6 KB total), 4 volume dump(s) (1m8.704s) +2026/10/24 17:17:50 [INFO] [backup] Recovery unit captured for metube → /mnt/felhom-drives/adatlemez/backups/primary/metube (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +/mnt/felhom-drives/adatlemez/backups/primary/metube +/mnt/felhom-drives/adatlemez/backups/primary/metube/data-stamps.json +/mnt/felhom-drives/adatlemez/backups/primary/metube/manifest.json +/mnt/felhom-drives/adatlemez/backups/primary/metube/compose +/mnt/felhom-drives/adatlemez/backups/primary/metube/compose/docker-compose.yml +/mnt/felhom-drives/adatlemez/backups/primary/metube/compose/.felhom.yml +/mnt/felhom-drives/adatlemez/backups/primary/metube/compose/app.yaml +/mnt/felhom-drives/adatlemez/backups/primary/metube/volume-dumps +/mnt/felhom-drives/adatlemez/backups/primary/metube/volume-dumps/metube_metube_state.tar +1.6G /mnt/felhom-drives/adatlemez/backups/primary/metube diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/04-gen2.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/04-gen2.txt new file mode 100644 index 00000000..aba33b6a --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/04-gen2.txt @@ -0,0 +1 @@ +gen2 — the household newest, written after the backup (2026-10-24T17:18:05Z) diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/05-restore-log-until-cut.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/05-restore-log-until-cut.txt new file mode 100644 index 00000000..c03575e4 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/05-restore-log-until-cut.txt @@ -0,0 +1,51 @@ +2026/10/24 17:18:28 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:18:28 [WARN] [offbox] jellyfin: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/24 17:18:28 [WARN] [offbox] plex: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/24 17:18:28 [WARN] [offbox] qbittorrent: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/24 17:18:28 [WARN] [offbox] radarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/24 17:18:28 [WARN] [offbox] sonarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/24 17:18:28 [WARN] [offbox] transmission: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/24 17:18:28 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/24 17:18:38 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:18:48 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:18:48 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/24 17:18:58 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:19:08 [INFO] [scheduler] Running job: backup-cache +2026/10/24 17:19:08 [INFO] [scheduler] Running job: docker-socket-users +2026/10/24 17:19:08 [INFO] [scheduler] Running job: system-health +2026/10/24 17:19:08 [INFO] [scheduler] Running job: offsite-credential-retry +2026/10/24 17:19:08 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/10/24 17:19:08 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:19:08 [INFO] [backup] Found 2 DB dump files across drives +2026/10/24 17:19:08 [INFO] [scheduler] Running job: agent-channel-health +2026/10/24 17:19:09 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:19:09 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:19:09 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:19:09 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:19:09 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:19:09 [INFO] [backup] Discovered 2 databases +2026/10/24 17:19:09 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:19:09 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/24 17:19:09 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:19:09 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:19:09 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:19:09 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:19:09 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:19:09 [INFO] [backup] Discovered app data: 4 apps +2026/10/24 17:19:09 [INFO] [backup] Backup status cache refreshed +2026/10/24 17:19:09 [INFO] [scheduler] Job backup-cache completed (took 345ms) +2026/10/24 17:19:09 [INFO] [scheduler] Job docker-socket-users completed (took 352ms) +2026/10/24 17:19:09 [INFO] [monitor] Health check: status=ok +2026/10/24 17:19:09 [INFO] [scheduler] Job system-health completed (took 503ms) +2026/10/24 17:19:09 [INFO] [scheduler] Job agent-channel-health completed (took 597ms) +2026/10/24 17:19:11 [WARN] [offbox] inventory: size of plex's newest snapshot unknown: offbox: a(z) 3af609d1 pillanatkép mérete ismeretlen (listing it anyway) +2026/10/24 17:19:18 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:19:27 [WARN] [web] Restore requested (async): stack=metube, snapshot=helyi from 172.18.0.7:35240 +2026/10/24 17:19:27 [INFO] [backup] Restoring metube from recovery unit /mnt/felhom-drives/adatlemez/backups/primary/metube: images=1, secrets recovered=0/0, data_keys=0 +2026/10/24 17:19:28 [INFO] [stacks] Stopping stack: metube +2026/10/24 17:19:28 [INFO] [stacks] Stack metube stopped successfully (took 0.3s) +2026/10/24 17:19:28 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:19:28 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:19:38 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:19:47 [INFO] [offbox] metube: the live volumes are copied before the restore (1 copied, 0 new from the snapshot) in 19.304s — /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z (R-893) +2026/10/24 17:19:47 [INFO] [backup] metube: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195) diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/05-restore-log-until-cut.txt.cut b/documentation/audits/restore-holes-2026-10-24/P1-tester1/05-restore-log-until-cut.txt.cut new file mode 100644 index 00000000..feea0712 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/05-restore-log-until-cut.txt.cut @@ -0,0 +1 @@ +CUT: line seen 2026-10-24T17:19:47.820Z, qm stop returned 2026-10-24T17:19:49.935Z diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/06-disk-off-and-boot.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/06-disk-off-and-boot.txt new file mode 100644 index 00000000..cbae8a59 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/06-disk-off-and-boot.txt @@ -0,0 +1,9 @@ +scsi0: nvme-scratch:341/vm-341-disk-1.raw,size=200G +scsi1: nvme-scratch:341/vm-341-disk-2.raw,size=50G +scsihw: virtio-scsi-single +update VM 341: -delete scsi1 +scsi0: nvme-scratch:341/vm-341-disk-1.raw,size=200G +scsihw: virtio-scsi-single +unused0: nvme-scratch:341/vm-341-disk-2.raw +status: running +17:20:05 diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/07-boot-without-drive.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/07-boot-without-drive.txt new file mode 100644 index 00000000..21f707fe --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/07-boot-without-drive.txt @@ -0,0 +1,16 @@ +2026/10/24 17:23:40 [WARN] [appstop] crash recovery: a local restore (op "unit-restore:metube") was interrupted and left 1 app(s) stopped — restarting them: [metube] +2026/10/24 17:23:40 [WARN] [appstop] crash recovery: metube's restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z cannot be reached yet (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z: no such file or directory) — NOT starting it; the put-back waits for its drive +2026/10/24 17:23:40 [WARN] [settings] restore hold SET for metube — the app stays stopped until it is cleared +2026/10/24 17:23:40 [WARN] [appstop] metube: the cut restore's copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z is on a drive that is not bound — the app is HELD until the copy is put back (R-1251) +2026/10/24 17:23:40 [WARN] [appstop] crash recovery: 1 app(s) were deliberately NOT restarted (drive absent: []; put-back waiting for its drive: [metube]) — KEEPING the marker; this is the gate working, not a fault +2026/10/24 17:23:40 [INFO] [appstop] metube: the restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z is still not reachable (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z: no such file or directory) — the app stays held until its drive is bound +2026/10/24 17:23:40 [WARN] [backup] restore of metube (restore, started 2026-10-24T17:19:27Z) was INTERRUPTED by a controller stop — recorded as failed; the household is told to run it again +2026/10/24 17:23:40 [WARN] [appstop] a local restore was interrupted by a controller restart — 1 app(s) are HELD until their data drive is bound: the copy taken before the change is on it and is put back first — not alarming: op=unit-restore:metube reason=unit_restore started_at=2026-10-24T17:19:28Z restarted=[] put_back_waiting_for_drive=[metube] +2026/10/24 17:23:40 [INFO] [report] hub wait channel active (hold ≤240s) +2026/10/24 17:23:40 [INFO] Event pushed: restore_interrupted (warning) [hu-only] — A(z) metube visszaállítása megszakadt, mert a doboz újraindult — indítsd el újra. +2026/10/24 17:23:40 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:24:00 [INFO] [bootrecon] "metube" is a boot orphan by intent but is HELD (an app-data operation is holding it — the app-stop guard restarts it when the operation ends) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:24:00 [INFO] [bootrecon] "paperless-ngx" is a boot orphan by intent but is HELD (drive /mnt/felhom-drives/adatlemez is not a live mountpoint) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:24:00 [INFO] [bootrecon] Boot reconciliation: nothing to start — 2 app(s) held (absent drive / quiesce / app-data operation): [metube paperless-ngx] +--- diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/08-start-refused.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/08-start-refused.txt new file mode 100644 index 00000000..1216431e --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/08-start-refused.txt @@ -0,0 +1,7 @@ +{"ok":false,"data":{"reason":"held"},"error":"A(z) metube visszaállítása megszakadt. A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el. Amikor a meghajtó újra elérhető, a doboz visszateszi ezeket az adatokat, és elindítja az alkalmazást. Addig az alkalmazás nem indul el."} + +409 +{"ok":false,"data":{"reason":"held"},"error":"A(z) metube visszaállítása megszakadt. A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el. Amikor a meghajtó újra elérhető, a doboz visszateszi ezeket az adatokat, és elindítja az alkalmazást. Addig az alkalmazás nem indul el."} + +409 +{'reason': 'putback_pending', 'at': '2026-10-24T17:23:40Z', 'safety_dump': '/mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z'} diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/09-controller-log-before-replug.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/09-controller-log-before-replug.txt new file mode 100644 index 00000000..73def089 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/09-controller-log-before-replug.txt @@ -0,0 +1,148 @@ +2026/10/24 17:23:39 [INFO] local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible +2026/10/24 17:23:39 [INFO] local-api: mount mp8 → /mnt/felhom-drives (storage=/mnt/felhom-drives, class=, backup=false) +2026/10/24 17:23:39 [INFO] local-api: mount mp9 → /etc/felhom-bootstrap (storage=/var/lib/felhom-agent/guests/9201/bootstrap, class=, backup=false) +2026/10/24 17:23:39 [INFO] local-api: mount mp0 → /var/lib/felhom (storage=local-lvm, class=, backup=true) +2026/10/24 17:23:39 [INFO] felhom-controller 0.343.0-dev.r1251a starting (customer: tester-1, domain: enkicsifelhom.hu) +2026/10/24 17:23:39 [INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json +2026/10/24 17:23:39 [INFO] Encryption key loaded from /opt/docker/felhom-controller/data/encryption.key +2026/10/24 17:23:40 [INFO] [stacks] Using compose command: docker compose +2026/10/24 17:23:40 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/24 17:23:40 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:23:40 [INFO] [stacks] InjectMissingFields: processed 4 stacks +2026/10/24 17:23:40 [INFO] [stacks] Encryption migration: no stacks needed migration +2026/10/24 17:23:40 [WARN] [appstop] crash recovery: a local restore (op "unit-restore:metube") was interrupted and left 1 app(s) stopped — restarting them: [metube] +2026/10/24 17:23:40 [WARN] [appstop] crash recovery: metube's restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z cannot be reached yet (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z: no such file or directory) — NOT starting it; the put-back waits for its drive +2026/10/24 17:23:40 [WARN] [settings] restore hold SET for metube — the app stays stopped until it is cleared +2026/10/24 17:23:40 [INFO] [quiesce] loop started (poll 5m0s, max-quiesce 30m0s) +2026/10/24 17:23:40 [INFO] [settings] Settings saved +2026/10/24 17:23:40 [WARN] [appstop] metube: the cut restore's copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z is on a drive that is not bound — the app is HELD until the copy is put back (R-1251) +2026/10/24 17:23:40 [WARN] [appstop] crash recovery: 1 app(s) were deliberately NOT restarted (drive absent: []; put-back waiting for its drive: [metube]) — KEEPING the marker; this is the gate working, not a fault +2026/10/24 17:23:40 [INFO] [appstop] metube: the restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z is still not reachable (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z: no such file or directory) — the app stays held until its drive is bound +2026/10/24 17:23:40 [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded (state ambiguous — legacy boot behaviour retained) +2026/10/24 17:23:40 [INFO] [stacks] installed-images backfill: 0 app(s) recorded, 4 already had a record, 0 left unrecorded (could not be observed completely — unknown, which renders nothing) +2026/10/24 17:23:40 [INFO] [stacks] pin adoption: 0 pinned, 4 already pinned, 0 left unpinned (0 not completely observed, 0 running something the template no longer offers) +2026/10/24 17:23:40 [INFO] [stacks] applied-meta backfill (R-646): recorded 0 []; skipped 0 [] — not current with the catalog, their pinned version's .felhom.yml is no longer on the box +2026/10/24 17:23:40 [INFO] [sync] Starting catalog sync (repo: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git, interval: 15m0s) +2026/10/24 17:23:40 [INFO] [sync] Starting catalog sync +2026/10/24 17:23:40 [INFO] Metrics store opened at /opt/docker/felhom-controller/data/metrics.db +2026/10/24 17:23:40 [INFO] Metrics collector started (60s interval) +2026/10/24 17:23:40 [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (branch: main) +2026/10/24 17:23:40 [INFO] [tier2] start-up: no leftover second-drive copy container +2026/10/24 17:23:40 [WARN] [backup] restore of metube (restore, started 2026-10-24T17:19:27Z) was INTERRUPTED by a controller stop — recorded as failed; the household is told to run it again +2026/10/24 17:23:40 [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: true) +2026/10/24 17:23:40 [INFO] [backup] app-data run record wired: /opt/docker/felhom-controller/data/appdata-run.json (an earlier cut still shown: false) +2026/10/24 17:23:40 [INFO] Notifier enabled (hub: https://hub.felhom.eu) +2026/10/24 17:23:40 [INFO] [upwatch] the host's weekly line test: 94.3 Mbit/s (measured 2026-10-24T08:00:58Z) +2026/10/24 17:23:40 [INFO] [upwatch] upload and CPU watch started (line 94 Mbit/s, qBittorrent share 50 %) +2026/10/24 17:23:40 [WARN] [appstop] a local restore was interrupted by a controller restart — 1 app(s) are HELD until their data drive is bound: the copy taken before the change is on it and is put back first — not alarming: op=unit-restore:metube reason=unit_restore started_at=2026-10-24T17:19:28Z restarted=[] put_back_waiting_for_drive=[metube] +2026/10/24 17:23:40 [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: status-refresh (every 10s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: docker-socket-watch (every 15s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: docker-socket-users (every 5m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: health-probes (every 10s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: system-health (every 5m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: after-unclean-boot (every 30s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: ring-spill (every 30s) +2026/10/24 17:23:40 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply +2026/10/24 17:23:40 [INFO] [scheduler] Daily job db-dump scheduled for 2026-10-25 02:30 CET +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: conversion-copy-release (every 1h0m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-10-25 03:30 CET +2026/10/24 17:23:40 [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-25 04:15 CET +2026/10/24 17:23:40 [INFO] [catch-up] controller start: no backup leg missed its last scheduled time — nothing to make up +2026/10/24 17:23:40 [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-10-25 05:10 CET +2026/10/24 17:23:40 [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-10-25 06:00 CET +2026/10/24 17:23:40 [INFO] [scheduler] Daily job offsite-proof scheduled for 2026-10-25 05:30 CET +2026/10/24 17:23:40 [INFO] [scheduler] Daily job metrics-prune scheduled for 2026-10-25 04:00 CET +2026/10/24 17:23:40 [INFO] [scheduler] Daily job fill-watch scheduled for 2026-10-25 03:30 CET +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: fill-watch-interval (every 10m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: hub-report (every 15m0s) +2026/10/24 17:23:40 [INFO] Hub reporting enabled (every 15m0s to https://hub.felhom.eu) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s) +2026/10/24 17:23:40 [INFO] ========== Startup Self-Test ========== +2026/10/24 17:23:40 [INFO] [PASS] Docker socket: reachable (v29.8.2) +2026/10/24 17:23:40 [INFO] [PASS] Stacks directory: /opt/docker/stacks +2026/10/24 17:23:40 [INFO] [PASS] Data directory: /opt/docker/felhom-controller/data (writable) +2026/10/24 17:23:40 [INFO] [PASS] System data path: /mnt/sys_drive +2026/10/24 17:23:40 [INFO] [WARN] Storage paths: 0 connected, 1 disconnected +2026/10/24 17:23:40 [INFO] [PASS] Git catalog: 65 app definitions found +2026/10/24 17:23:40 [INFO] [PASS] Hub connectivity: https://hub.felhom.eu reachable (HTTP 200) +2026/10/24 17:23:40 [INFO] [PASS] Metrics DB: 34.2 MB +2026/10/24 17:23:40 [INFO] ======================================== +2026/10/24 17:23:40 [INFO] Self-test complete: 7 passed, 1 warnings, 0 failed +2026/10/24 17:23:40 [INFO] [scheduler] Starting scheduler with 23 jobs +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: geo-verify (every 6h0m0s) +2026/10/24 17:23:40 [INFO] Geo-restriction support enabled (CF API token configured) +2026/10/24 17:23:40 [INFO] [infra] connected felhom-controller to traefik-public +2026/10/24 17:23:40 [INFO] [report] hub wait channel active (hold ≤240s) +2026/10/24 17:23:40 [INFO] Event pushed: restore_interrupted (warning) [hu-only] — A(z) metube visszaállítása megszakadt, mert a doboz újraindult — indítsd el újra. +2026/10/24 17:23:40 [INFO] [backup] Found 1 DB dump files across drives +2026/10/24 17:23:40 [INFO] [web] sessions: restored 5 dashboard session(s) from disk (dropped 0 expired, 0 malformed) +2026/10/24 17:23:40 [INFO] [web] Auth: using password from settings.json +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s) +2026/10/24 17:23:40 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/adatlemez not mounted — skipping userdata skeleton +2026/10/24 17:23:40 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_a not mounted — skipping userdata skeleton +2026/10/24 17:23:40 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_b not mounted — skipping userdata skeleton +2026/10/24 17:23:40 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_c not mounted — skipping userdata skeleton +2026/10/24 17:23:40 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_d not mounted — skipping userdata skeleton +2026/10/24 17:23:40 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:23:40 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: agent-channel-health (every 1m0s) +2026/10/24 17:23:40 [INFO] Web UI listening on :8080 +2026/10/24 17:23:41 [INFO] [backup] Discovered 2 databases +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:23:41 [INFO] [backup] Discovered app data: 4 apps +2026/10/24 17:23:41 [INFO] [infra] filebrowser compose exists — starting without regenerating +2026/10/24 17:23:41 [INFO] [backup] Backup status cache refreshed +2026/10/24 17:23:41 [ERROR] [stacks] Command failed: docker compose up -d (in /opt/docker/stacks/filebrowser) — exit code 1 (took 0.2s) +2026/10/24 17:23:41 [ERROR] [stacks] stderr (last part): ...emon: Conflict. The container name "/2efa44c97a56_filebrowser" is already in use by container "774b32721d18d4591e021f2b271a512ac523a6ad3faead89c971043242ffef41". You have to remove (or rename) that container to be able to reuse that name. +Error response from daemon: Conflict. The container name "/2efa44c97a56_filebrowser" is already in use by container "774b32721d18d4591e021f2b271a512ac523a6ad3faead89c971043242ffef41". You have to remove (or rename) that container to be able to reuse that name. +2026/10/24 17:23:41 [WARN] [infra] first-boot base-stack bring-up: base-infra bring-up: filebrowser: compose up: : exit code 1 +stderr: ...emon: Conflict. The container name "/2efa44c97a56_filebrowser" is already in use by container "774b32721d18d4591e021f2b271a512ac523a6ad3faead89c971043242ffef41". You have to remove (or rename) that container to be able to reuse that name. +Error response from daemon: Conflict. The container name "/2efa44c97a56_filebrowser" is already in use by container "774b32721d18d4591e021f2b271a512ac523a6ad3faead89c971043242ffef41". You have to remove (or rename) that container to be able to reuse that name. +2026/10/24 17:23:41 [INFO] [monitor] Health check: status=fail +2026/10/24 17:23:41 [INFO] [gate] boot 1792862417-19114: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/adatlemez] before recreating drive-backed apps +2026/10/24 17:23:42 [INFO] [sync] Catalog sync complete +2026/10/24 17:23:42 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +2026/10/24 17:23:42 [INFO] [web] FileBrowser mounts synced (recreated) — 5 storage path(s), config updated +2026/10/24 17:23:43 [INFO] [infra] connected felhom-controller to felhom-filebrowser (the FileBrowser default-login check, R-1126) +2026/10/24 17:23:45 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:23:45 [INFO] [report] Building system report +2026/10/24 17:23:45 [INFO] Event pushed: controller_started (info) [hu-only] — Controller elindult (0.343.0-dev.r1251a) +2026/10/24 17:23:46 [INFO] [monitor] Health check: status=warn +2026/10/24 17:23:47 [INFO] [infra] filebrowser: default login refused for this run — joined traefik-public, files. served (R-1126) +2026/10/24 17:23:48 [INFO] [report] Hub report pushed successfully (14617 bytes) +2026/10/24 17:23:48 [INFO] [settings] Settings saved +2026/10/24 17:23:48 [INFO] Startup hub report sent +2026/10/24 17:23:50 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:23:50 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.342.0 (we are 0.343.0-dev.r1251a), no managed update running +2026/10/24 17:23:50 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:23:50 [INFO] Health probes: 2 ok (of 2 probed) +2026/10/24 17:23:55 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:24:00 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:24:00 [INFO] [bootrecon] boot window: fleet settled after 15s (3 identical samples 5s apart) — sweeping +2026/10/24 17:24:00 [INFO] [bootrecon] "metube" is a boot orphan by intent but is HELD (an app-data operation is holding it — the app-stop guard restarts it when the operation ends) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:24:00 [INFO] [bootrecon] "paperless-ngx" is a boot orphan by intent but is HELD (drive /mnt/felhom-drives/adatlemez is not a live mountpoint) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:24:00 [INFO] [bootrecon] Boot reconciliation: nothing to start — 2 app(s) held (absent drive / quiesce / app-data operation): [metube paperless-ngx] +2026/10/24 17:24:00 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:24:10 [INFO] [deadapp] boot grace 15m0s: the host's last boot (2026-10-24T17:20:17Z) followed an UNCLEAN stop (R-856) +2026/10/24 17:24:10 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:24:20 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:24:25 [INFO] [web] Login from 192.168.0.180 +2026/10/24 17:24:25 [INFO] [api] start requested for stack: metube +2026/10/24 17:24:25 [INFO] [api] update requested for stack: metube +2026/10/24 17:24:30 [INFO] [stacks] Status refresh: 9 containers across 68 stacks diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/10-replug.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/10-replug.txt new file mode 100644 index 00000000..db14535c --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/10-replug.txt @@ -0,0 +1,5 @@ +update VM 341: -scsi1 nvme-scratch:341/vm-341-disk-2.raw +scsi0: nvme-scratch:341/vm-341-disk-1.raw,size=200G +scsi1: nvme-scratch:341/vm-341-disk-2.raw,size=50G +scsihw: virtio-scsi-single +17:24:42 diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/11-drive-returns.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/11-drive-returns.txt new file mode 100644 index 00000000..fba03acd --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/11-drive-returns.txt @@ -0,0 +1,30 @@ +2026/10/24 17:23:40 [WARN] [appstop] crash recovery: a local restore (op "unit-restore:metube") was interrupted and left 1 app(s) stopped — restarting them: [metube] +2026/10/24 17:23:40 [WARN] [appstop] crash recovery: metube's restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z cannot be reached yet (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z: no such file or directory) — NOT starting it; the put-back waits for its drive +2026/10/24 17:23:40 [WARN] [settings] restore hold SET for metube — the app stays stopped until it is cleared +2026/10/24 17:23:40 [WARN] [appstop] metube: the cut restore's copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z is on a drive that is not bound — the app is HELD until the copy is put back (R-1251) +2026/10/24 17:23:40 [WARN] [appstop] crash recovery: 1 app(s) were deliberately NOT restarted (drive absent: []; put-back waiting for its drive: [metube]) — KEEPING the marker; this is the gate working, not a fault +2026/10/24 17:23:40 [INFO] [appstop] metube: the restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z is still not reachable (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z: no such file or directory) — the app stays held until its drive is bound +2026/10/24 17:23:40 [WARN] [backup] restore of metube (restore, started 2026-10-24T17:19:27Z) was INTERRUPTED by a controller stop — recorded as failed; the household is told to run it again +2026/10/24 17:23:40 [WARN] [appstop] a local restore was interrupted by a controller restart — 1 app(s) are HELD until their data drive is bound: the copy taken before the change is on it and is put back first — not alarming: op=unit-restore:metube reason=unit_restore started_at=2026-10-24T17:19:28Z restarted=[] put_back_waiting_for_drive=[metube] +2026/10/24 17:23:40 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply +2026/10/24 17:23:40 [INFO] [report] hub wait channel active (hold ≤240s) +2026/10/24 17:23:40 [INFO] Event pushed: restore_interrupted (warning) [hu-only] — A(z) metube visszaállítása megszakadt, mert a doboz újraindult — indítsd el újra. +2026/10/24 17:23:40 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:23:41 [INFO] [gate] boot 1792862417-19114: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/adatlemez] before recreating drive-backed apps +2026/10/24 17:23:50 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.342.0 (we are 0.343.0-dev.r1251a), no managed update running +2026/10/24 17:24:00 [INFO] [bootrecon] "metube" is a boot orphan by intent but is HELD (an app-data operation is holding it — the app-stop guard restarts it when the operation ends) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:24:00 [INFO] [bootrecon] "paperless-ngx" is a boot orphan by intent but is HELD (drive /mnt/felhom-drives/adatlemez is not a live mountpoint) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:24:00 [INFO] [bootrecon] Boot reconciliation: nothing to start — 2 app(s) held (absent drive / quiesce / app-data operation): [metube paperless-ngx] +2026/10/24 17:24:25 [INFO] [api] start requested for stack: metube +2026/10/24 17:24:25 [INFO] [api] update requested for stack: metube +2026/10/24 17:25:01 [INFO] [gate] boot 1792862417-19114: live bind confirmed — recreating drive-backed app metube (state=stopped) onto /mnt/felhom-drives/adatlemez +2026/10/24 17:25:01 [WARN] [settings] restore hold SET for metube — the app stays stopped until it is cleared +2026/10/24 17:25:01 [ERROR] [appstop] metube: the interrupted restore could NOT be put back (reading the restore copy's journal: open /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z/undo.json: no such file or directory) — the app is HELD for support, the copy stays at /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z (R-1251 H2) +2026/10/24 17:25:01 [INFO] [gate] boot 1792862417-19114: live bind confirmed — recreating drive-backed app paperless-ngx (state=degraded) onto /mnt/felhom-drives/adatlemez +2026/10/24 17:25:14 [INFO] [gate] boot 1792862417-19114: re-syncing FileBrowser mounts against the live binds +2026/10/24 17:25:14 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +--- +TARGET SOURCE FSTYPE OPTIONS +/mnt/felhom-drives/adatlemez /dev/sdb[/felhom-data] ext4 rw,relatime +/mnt/felhom-drives/adatlemez /dev/sdb[/felhom-data] ext4 rw,relatime diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/12-after-replug-state.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/12-after-replug-state.txt new file mode 100644 index 00000000..5058c40f --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/12-after-replug-state.txt @@ -0,0 +1,15 @@ +/mnt/felhom-drives/adatlemez/backups/restore-undo/metube 4096 17:19:28.4662702010 +/mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z 4096 17:19:47.8045336330 +/mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z/undo.json.tmp 132 17:19:47.8045336330 +/mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z/definition 4096 17:19:47.8035336190 +/mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z/definition/docker-compose.yml 2244 17:19:47.7715331830 +/mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z/definition/.felhom.yml 12057 17:19:47.7805333060 +/mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z/definition/app.yaml 674 17:19:47.7885334150 +/mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z/metube_metube_state.tar 1610615808 17:19:44.0864829840 +total 8 +drwxr-xr-x 2 1000 1000 4096 Oct 24 17:19 . +drwx-----x 3 root root 4096 Oct 24 17:19 .. +-rw-r--r-- 1 root root 0 Oct 24 17:19 r1251-ballast.bin +-rw-r--r-- 1 root root 0 Oct 24 17:16 r1251-gen.txt +{'reason': 'restore_mixed', 'at': '2026-10-24T17:25:01Z', 'safety_dump': '/mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z', 'replay_error': "interrupted restore, put-back failed: reading the restore copy's journal: open /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z/undo.json: no such file or directory"} +ls: cannot access '/var/lib/docker/volumes/felhom-controller-data/_data/data/appstop-state.json': No such file or directory diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/13-controller-log-run1-end.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/13-controller-log-run1-end.txt new file mode 100644 index 00000000..845cce77 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/13-controller-log-run1-end.txt @@ -0,0 +1,191 @@ +2026/10/24 17:23:39 [INFO] local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible +2026/10/24 17:23:39 [INFO] local-api: mount mp8 → /mnt/felhom-drives (storage=/mnt/felhom-drives, class=, backup=false) +2026/10/24 17:23:39 [INFO] local-api: mount mp9 → /etc/felhom-bootstrap (storage=/var/lib/felhom-agent/guests/9201/bootstrap, class=, backup=false) +2026/10/24 17:23:39 [INFO] local-api: mount mp0 → /var/lib/felhom (storage=local-lvm, class=, backup=true) +2026/10/24 17:23:39 [INFO] felhom-controller 0.343.0-dev.r1251a starting (customer: tester-1, domain: enkicsifelhom.hu) +2026/10/24 17:23:39 [INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json +2026/10/24 17:23:39 [INFO] Encryption key loaded from /opt/docker/felhom-controller/data/encryption.key +2026/10/24 17:23:40 [INFO] [stacks] Using compose command: docker compose +2026/10/24 17:23:40 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/24 17:23:40 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:23:40 [INFO] [stacks] InjectMissingFields: processed 4 stacks +2026/10/24 17:23:40 [INFO] [stacks] Encryption migration: no stacks needed migration +2026/10/24 17:23:40 [WARN] [appstop] crash recovery: a local restore (op "unit-restore:metube") was interrupted and left 1 app(s) stopped — restarting them: [metube] +2026/10/24 17:23:40 [WARN] [appstop] crash recovery: metube's restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z cannot be reached yet (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z: no such file or directory) — NOT starting it; the put-back waits for its drive +2026/10/24 17:23:40 [WARN] [settings] restore hold SET for metube — the app stays stopped until it is cleared +2026/10/24 17:23:40 [INFO] [quiesce] loop started (poll 5m0s, max-quiesce 30m0s) +2026/10/24 17:23:40 [INFO] [settings] Settings saved +2026/10/24 17:23:40 [WARN] [appstop] metube: the cut restore's copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z is on a drive that is not bound — the app is HELD until the copy is put back (R-1251) +2026/10/24 17:23:40 [WARN] [appstop] crash recovery: 1 app(s) were deliberately NOT restarted (drive absent: []; put-back waiting for its drive: [metube]) — KEEPING the marker; this is the gate working, not a fault +2026/10/24 17:23:40 [INFO] [appstop] metube: the restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z is still not reachable (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z: no such file or directory) — the app stays held until its drive is bound +2026/10/24 17:23:40 [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded (state ambiguous — legacy boot behaviour retained) +2026/10/24 17:23:40 [INFO] [stacks] installed-images backfill: 0 app(s) recorded, 4 already had a record, 0 left unrecorded (could not be observed completely — unknown, which renders nothing) +2026/10/24 17:23:40 [INFO] [stacks] pin adoption: 0 pinned, 4 already pinned, 0 left unpinned (0 not completely observed, 0 running something the template no longer offers) +2026/10/24 17:23:40 [INFO] [stacks] applied-meta backfill (R-646): recorded 0 []; skipped 0 [] — not current with the catalog, their pinned version's .felhom.yml is no longer on the box +2026/10/24 17:23:40 [INFO] [sync] Starting catalog sync (repo: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git, interval: 15m0s) +2026/10/24 17:23:40 [INFO] [sync] Starting catalog sync +2026/10/24 17:23:40 [INFO] Metrics store opened at /opt/docker/felhom-controller/data/metrics.db +2026/10/24 17:23:40 [INFO] Metrics collector started (60s interval) +2026/10/24 17:23:40 [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (branch: main) +2026/10/24 17:23:40 [INFO] [tier2] start-up: no leftover second-drive copy container +2026/10/24 17:23:40 [WARN] [backup] restore of metube (restore, started 2026-10-24T17:19:27Z) was INTERRUPTED by a controller stop — recorded as failed; the household is told to run it again +2026/10/24 17:23:40 [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: true) +2026/10/24 17:23:40 [INFO] [backup] app-data run record wired: /opt/docker/felhom-controller/data/appdata-run.json (an earlier cut still shown: false) +2026/10/24 17:23:40 [INFO] Notifier enabled (hub: https://hub.felhom.eu) +2026/10/24 17:23:40 [INFO] [upwatch] the host's weekly line test: 94.3 Mbit/s (measured 2026-10-24T08:00:58Z) +2026/10/24 17:23:40 [INFO] [upwatch] upload and CPU watch started (line 94 Mbit/s, qBittorrent share 50 %) +2026/10/24 17:23:40 [WARN] [appstop] a local restore was interrupted by a controller restart — 1 app(s) are HELD until their data drive is bound: the copy taken before the change is on it and is put back first — not alarming: op=unit-restore:metube reason=unit_restore started_at=2026-10-24T17:19:28Z restarted=[] put_back_waiting_for_drive=[metube] +2026/10/24 17:23:40 [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: status-refresh (every 10s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: docker-socket-watch (every 15s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: docker-socket-users (every 5m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: health-probes (every 10s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: system-health (every 5m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: after-unclean-boot (every 30s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: ring-spill (every 30s) +2026/10/24 17:23:40 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply +2026/10/24 17:23:40 [INFO] [scheduler] Daily job db-dump scheduled for 2026-10-25 02:30 CET +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: conversion-copy-release (every 1h0m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-10-25 03:30 CET +2026/10/24 17:23:40 [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-25 04:15 CET +2026/10/24 17:23:40 [INFO] [catch-up] controller start: no backup leg missed its last scheduled time — nothing to make up +2026/10/24 17:23:40 [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-10-25 05:10 CET +2026/10/24 17:23:40 [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-10-25 06:00 CET +2026/10/24 17:23:40 [INFO] [scheduler] Daily job offsite-proof scheduled for 2026-10-25 05:30 CET +2026/10/24 17:23:40 [INFO] [scheduler] Daily job metrics-prune scheduled for 2026-10-25 04:00 CET +2026/10/24 17:23:40 [INFO] [scheduler] Daily job fill-watch scheduled for 2026-10-25 03:30 CET +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: fill-watch-interval (every 10m0s) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: hub-report (every 15m0s) +2026/10/24 17:23:40 [INFO] Hub reporting enabled (every 15m0s to https://hub.felhom.eu) +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s) +2026/10/24 17:23:40 [INFO] ========== Startup Self-Test ========== +2026/10/24 17:23:40 [INFO] [PASS] Docker socket: reachable (v29.8.2) +2026/10/24 17:23:40 [INFO] [PASS] Stacks directory: /opt/docker/stacks +2026/10/24 17:23:40 [INFO] [PASS] Data directory: /opt/docker/felhom-controller/data (writable) +2026/10/24 17:23:40 [INFO] [PASS] System data path: /mnt/sys_drive +2026/10/24 17:23:40 [INFO] [WARN] Storage paths: 0 connected, 1 disconnected +2026/10/24 17:23:40 [INFO] [PASS] Git catalog: 65 app definitions found +2026/10/24 17:23:40 [INFO] [PASS] Hub connectivity: https://hub.felhom.eu reachable (HTTP 200) +2026/10/24 17:23:40 [INFO] [PASS] Metrics DB: 34.2 MB +2026/10/24 17:23:40 [INFO] ======================================== +2026/10/24 17:23:40 [INFO] Self-test complete: 7 passed, 1 warnings, 0 failed +2026/10/24 17:23:40 [INFO] [scheduler] Starting scheduler with 23 jobs +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: geo-verify (every 6h0m0s) +2026/10/24 17:23:40 [INFO] Geo-restriction support enabled (CF API token configured) +2026/10/24 17:23:40 [INFO] [infra] connected felhom-controller to traefik-public +2026/10/24 17:23:40 [INFO] [report] hub wait channel active (hold ≤240s) +2026/10/24 17:23:40 [INFO] Event pushed: restore_interrupted (warning) [hu-only] — A(z) metube visszaállítása megszakadt, mert a doboz újraindult — indítsd el újra. +2026/10/24 17:23:40 [INFO] [backup] Found 1 DB dump files across drives +2026/10/24 17:23:40 [INFO] [web] sessions: restored 5 dashboard session(s) from disk (dropped 0 expired, 0 malformed) +2026/10/24 17:23:40 [INFO] [web] Auth: using password from settings.json +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s) +2026/10/24 17:23:40 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/adatlemez not mounted — skipping userdata skeleton +2026/10/24 17:23:40 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_a not mounted — skipping userdata skeleton +2026/10/24 17:23:40 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_b not mounted — skipping userdata skeleton +2026/10/24 17:23:40 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_c not mounted — skipping userdata skeleton +2026/10/24 17:23:40 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_d not mounted — skipping userdata skeleton +2026/10/24 17:23:40 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:23:40 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/24 17:23:40 [INFO] [scheduler] Registered periodic job: agent-channel-health (every 1m0s) +2026/10/24 17:23:40 [INFO] Web UI listening on :8080 +2026/10/24 17:23:41 [INFO] [backup] Discovered 2 databases +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:23:41 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:23:41 [INFO] [backup] Discovered app data: 4 apps +2026/10/24 17:23:41 [INFO] [infra] filebrowser compose exists — starting without regenerating +2026/10/24 17:23:41 [INFO] [backup] Backup status cache refreshed +2026/10/24 17:23:41 [ERROR] [stacks] Command failed: docker compose up -d (in /opt/docker/stacks/filebrowser) — exit code 1 (took 0.2s) +2026/10/24 17:23:41 [ERROR] [stacks] stderr (last part): ...emon: Conflict. The container name "/2efa44c97a56_filebrowser" is already in use by container "774b32721d18d4591e021f2b271a512ac523a6ad3faead89c971043242ffef41". You have to remove (or rename) that container to be able to reuse that name. +Error response from daemon: Conflict. The container name "/2efa44c97a56_filebrowser" is already in use by container "774b32721d18d4591e021f2b271a512ac523a6ad3faead89c971043242ffef41". You have to remove (or rename) that container to be able to reuse that name. +2026/10/24 17:23:41 [WARN] [infra] first-boot base-stack bring-up: base-infra bring-up: filebrowser: compose up: : exit code 1 +stderr: ...emon: Conflict. The container name "/2efa44c97a56_filebrowser" is already in use by container "774b32721d18d4591e021f2b271a512ac523a6ad3faead89c971043242ffef41". You have to remove (or rename) that container to be able to reuse that name. +Error response from daemon: Conflict. The container name "/2efa44c97a56_filebrowser" is already in use by container "774b32721d18d4591e021f2b271a512ac523a6ad3faead89c971043242ffef41". You have to remove (or rename) that container to be able to reuse that name. +2026/10/24 17:23:41 [INFO] [monitor] Health check: status=fail +2026/10/24 17:23:41 [INFO] [gate] boot 1792862417-19114: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/adatlemez] before recreating drive-backed apps +2026/10/24 17:23:42 [INFO] [sync] Catalog sync complete +2026/10/24 17:23:42 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +2026/10/24 17:23:42 [INFO] [web] FileBrowser mounts synced (recreated) — 5 storage path(s), config updated +2026/10/24 17:23:43 [INFO] [infra] connected felhom-controller to felhom-filebrowser (the FileBrowser default-login check, R-1126) +2026/10/24 17:23:45 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:23:45 [INFO] [report] Building system report +2026/10/24 17:23:45 [INFO] Event pushed: controller_started (info) [hu-only] — Controller elindult (0.343.0-dev.r1251a) +2026/10/24 17:23:46 [INFO] [monitor] Health check: status=warn +2026/10/24 17:23:47 [INFO] [infra] filebrowser: default login refused for this run — joined traefik-public, files. served (R-1126) +2026/10/24 17:23:48 [INFO] [report] Hub report pushed successfully (14617 bytes) +2026/10/24 17:23:48 [INFO] [settings] Settings saved +2026/10/24 17:23:48 [INFO] Startup hub report sent +2026/10/24 17:23:50 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:23:50 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.342.0 (we are 0.343.0-dev.r1251a), no managed update running +2026/10/24 17:23:50 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:23:50 [INFO] Health probes: 2 ok (of 2 probed) +2026/10/24 17:23:55 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:24:00 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:24:00 [INFO] [bootrecon] boot window: fleet settled after 15s (3 identical samples 5s apart) — sweeping +2026/10/24 17:24:00 [INFO] [bootrecon] "metube" is a boot orphan by intent but is HELD (an app-data operation is holding it — the app-stop guard restarts it when the operation ends) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:24:00 [INFO] [bootrecon] "paperless-ngx" is a boot orphan by intent but is HELD (drive /mnt/felhom-drives/adatlemez is not a live mountpoint) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:24:00 [INFO] [bootrecon] Boot reconciliation: nothing to start — 2 app(s) held (absent drive / quiesce / app-data operation): [metube paperless-ngx] +2026/10/24 17:24:00 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:24:10 [INFO] [deadapp] boot grace 15m0s: the host's last boot (2026-10-24T17:20:17Z) followed an UNCLEAN stop (R-856) +2026/10/24 17:24:10 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:24:20 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:24:25 [INFO] [web] Login from 192.168.0.180 +2026/10/24 17:24:25 [INFO] [api] start requested for stack: metube +2026/10/24 17:24:25 [INFO] [api] update requested for stack: metube +2026/10/24 17:24:30 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:24:40 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:24:40 [INFO] [scheduler] Running job: agent-channel-health +2026/10/24 17:24:41 [INFO] [scheduler] Job agent-channel-health completed (took 525ms) +2026/10/24 17:24:50 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:25:00 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:25:01 [INFO] [gate] boot 1792862417-19114: live bind confirmed — recreating drive-backed app metube (state=stopped) onto /mnt/felhom-drives/adatlemez +2026/10/24 17:25:01 [WARN] [settings] restore hold SET for metube — the app stays stopped until it is cleared +2026/10/24 17:25:01 [INFO] [settings] Settings saved +2026/10/24 17:25:01 [ERROR] [appstop] metube: the interrupted restore could NOT be put back (reading the restore copy's journal: open /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z/undo.json: no such file or directory) — the app is HELD for support, the copy stays at /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T171928Z (R-1251 H2) +2026/10/24 17:25:01 [INFO] [gate] boot 1792862417-19114: live bind confirmed — recreating drive-backed app paperless-ngx (state=degraded) onto /mnt/felhom-drives/adatlemez +2026/10/24 17:25:01 [INFO] [stacks] Stopping stack: paperless-ngx +2026/10/24 17:25:02 [INFO] [stacks] Stack paperless-ngx stopped successfully (took 0.5s) +2026/10/24 17:25:02 [INFO] [stacks] Status refresh: 6 containers across 68 stacks +2026/10/24 17:25:02 [INFO] [stacks] Starting stack: paperless-ngx +2026/10/24 17:25:10 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/10/24 17:25:10 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:25:13 [INFO] [stacks] Stack paperless-ngx started successfully (took 11.4s) +2026/10/24 17:25:14 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:25:14 [INFO] [gate] boot 1792862417-19114: re-syncing FileBrowser mounts against the live binds +2026/10/24 17:25:14 [INFO] [settings] Settings saved +2026/10/24 17:25:14 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_a not mounted — skipping userdata skeleton +2026/10/24 17:25:14 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_b not mounted — skipping userdata skeleton +2026/10/24 17:25:14 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_c not mounted — skipping userdata skeleton +2026/10/24 17:25:14 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_d not mounted — skipping userdata skeleton +2026/10/24 17:25:14 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:25:14 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/24 17:25:14 [INFO] [web] FileBrowser mounts synced (recreated) — 5 storage path(s), config updated +2026/10/24 17:25:17 [INFO] [stacks] Stack paperless-ngx post-start status: +2026/10/24 17:25:17 [INFO] [stacks] paperless-postgres postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 running Up 14 seconds (healthy) +2026/10/24 17:25:17 [INFO] [stacks] paperless-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy) +2026/10/24 17:25:17 [INFO] [stacks] paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15@sha256:6c86cad803970ea782683a8e80e7403444c5bf3cf70de63b4d3c8e87500db92f running Up 3 seconds (health: starting) +2026/10/24 17:25:17 [INFO] [infra] filebrowser: default login refused for this run — joined traefik-public, files. served (R-1126) +2026/10/24 17:25:20 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:25:30 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:25:40 [INFO] [scheduler] Running job: stack-scan +2026/10/24 17:25:40 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:25:40 [INFO] [scheduler] Running job: agent-channel-health +2026/10/24 17:25:41 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/24 17:25:41 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:25:41 [INFO] [scheduler] Job stack-scan completed (took 251ms) +2026/10/24 17:25:41 [INFO] [scheduler] Job agent-channel-health completed (took 594ms) +2026/10/24 17:25:50 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:26:00 [INFO] [stacks] Status refresh: 9 containers across 68 stacks diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt new file mode 100644 index 00000000..c2daa745 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt @@ -0,0 +1,6 @@ +{"ok":true,"data":{"removed":"metube","volumes_removed":["metube_metube_state"],"hdd_paths_removed":[],"hdd_paths_preserved":[],"userdata_kept":["/mnt/felhom-drives/adatlemez/userdata/media/metube (8.0K)"],"backup_paths_removed":["/mnt/felhom-drives/adatlemez/backups/primary/metube (1.6G)"],"verified":true},"message":"Stack metube removed"} + +200 +holds: {'metube': 'restore_mixed'} +/usr/local/bin/felhom-controller +/usr/local/bin/felhom-controller --config /opt/docker/felhom-controller/controller.yaml \ No newline at end of file diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/15-run2-setup.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/15-run2-setup.txt new file mode 100644 index 00000000..9ee9354e --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/15-run2-setup.txt @@ -0,0 +1,8 @@ + NOW RESTART THE CONTROLLER, or it will keep refusing to start the app: + systemctl restart felhom-controller-bootstrap.service + Check the app's data first — the undo copies are in its unit's db-dumps dir. +gitea.dooplex.hu/admin/felhom-controller:0.343.0-dev.r1251b Up 25 seconds (healthy) +holds: {} +offsite-proof +primary +secondary diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/20-run2-deploy.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/20-run2-deploy.txt new file mode 100644 index 00000000..21304c8f --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/20-run2-deploy.txt @@ -0,0 +1,17 @@ +{"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"} + +202 +metube Up 6 seconds (healthy) +3e1ec80f99e09173 +total 12 +drwxrwsr-x 2 1000 1000 4096 Oct 24 17:16 . +drwxrwsr-x 16 root 1000 4096 Oct 13 19:26 .. +-rw-r--r-- 1 1000 1000 53 Oct 24 17:16 r1251-household-video.txt +{"ok":true,"message":"Mentés elindítva"} + +200 +2026/10/24 17:41:33 [INFO] [stacks] Stack metube started successfully (took 0.7s) +2026/10/24 17:41:36 [INFO] [stacks] Stack metube post-start status: +2026/10/24 17:41:36 [INFO] [stacks] metube ghcr.io/alexta69/metube:2026.09.29@sha256:8e2fe9beeefc55a02f6e1d04cad0fc3c22e8f067d309188f016551dc99ec27b3 running Up 3 seconds (health: starting) +2026/10/24 17:41:59 [INFO] [backup] Recovery unit captured for metube → /mnt/felhom-drives/adatlemez/backups/primary/metube (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +gen2 — the household newest, after the backup (2026-10-24T17:42:05Z) diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/21-run2-log-until-cut.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/21-run2-log-until-cut.txt new file mode 100644 index 00000000..af92bada --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/21-run2-log-until-cut.txt @@ -0,0 +1,10 @@ +2026/10/24 17:42:16 [WARN] [web] Restore requested (async): stack=metube, snapshot=helyi from 172.18.0.4:34112 +2026/10/24 17:42:16 [INFO] [backup] Restoring metube from recovery unit /mnt/felhom-drives/adatlemez/backups/primary/metube: images=1, secrets recovered=0/0, data_keys=0 +2026/10/24 17:42:17 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:42:17 [INFO] [stacks] Stopping stack: metube +2026/10/24 17:42:18 [INFO] [stacks] Stack metube stopped successfully (took 0.6s) +2026/10/24 17:42:18 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:42:27 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:42:37 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:42:38 [INFO] [offbox] metube: the live volumes are copied before the restore (1 copied, 0 new from the snapshot) in 20.001s — /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z (R-893) +2026/10/24 17:42:38 [INFO] [backup] metube: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195) diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/21-run2-log-until-cut.txt.cut b/documentation/audits/restore-holes-2026-10-24/P1-tester1/21-run2-log-until-cut.txt.cut new file mode 100644 index 00000000..d0667a49 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/21-run2-log-until-cut.txt.cut @@ -0,0 +1 @@ +CUT: line seen 2026-10-24T17:42:38.379Z, qm stop returned 2026-10-24T17:42:40.608Z diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/22-run2-disk-off-and-boot.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/22-run2-disk-off-and-boot.txt new file mode 100644 index 00000000..ebb91fc2 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/22-run2-disk-off-and-boot.txt @@ -0,0 +1,6 @@ +status: stopped +update VM 341: -delete scsi1 +scsi0: nvme-scratch:341/vm-341-disk-1.raw,size=200G +scsihw: virtio-scsi-single +unused0: nvme-scratch:341/vm-341-disk-2.raw +17:42:47 diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/23-run2-boot-without-drive.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/23-run2-boot-without-drive.txt new file mode 100644 index 00000000..3fccd597 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/23-run2-boot-without-drive.txt @@ -0,0 +1,19 @@ +2026/10/24 17:47:24 [WARN] [appstop] crash recovery: a local restore (op "unit-restore:metube") was interrupted and left 1 app(s) stopped — restarting them: [metube] +2026/10/24 17:47:24 [WARN] [appstop] crash recovery: metube's restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z cannot be reached yet (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z: no such file or directory) — NOT starting it; the put-back waits for its drive +2026/10/24 17:47:24 [WARN] [settings] restore hold SET for metube — the app stays stopped until it is cleared +2026/10/24 17:47:24 [WARN] [appstop] metube: the cut restore's copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z is on a drive that is not bound — the app is HELD until the copy is put back (R-1251) +2026/10/24 17:47:24 [WARN] [appstop] crash recovery: 1 app(s) were deliberately NOT restarted (drive absent: []; put-back waiting for its drive: [metube]) — KEEPING the marker; this is the gate working, not a fault +2026/10/24 17:47:24 [INFO] [appstop] metube: the restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z is still not reachable (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z: no such file or directory) — the app stays held until its drive is bound +2026/10/24 17:47:24 [WARN] [backup] restore of metube (restore, started 2026-10-24T17:42:16Z) was INTERRUPTED by a controller stop — recorded as failed; the household is told to run it again +2026/10/24 17:47:24 [WARN] [appstop] a local restore was interrupted by a controller restart — 1 app(s) are HELD until their data drive is bound: the copy taken before the change is on it and is put back first — not alarming: op=unit-restore:metube reason=unit_restore started_at=2026-10-24T17:42:17Z restarted=[] put_back_waiting_for_drive=[metube] +2026/10/24 17:47:24 [INFO] [report] hub wait channel active (hold ≤240s) +2026/10/24 17:47:24 [INFO] Event pushed: restore_interrupted (warning) [hu-only] — metube: A visszaállítás félbeszakadt (áramszünet). Az alkalmazás nem indul el, amíg a meghajtója vissza nem tér: akkor a doboz visszateszi a visszaállítás előtti adatait, és elindítja. Ne indítsd újra a visszaállítást, mert az a mentést tölti vissza az adataid helyére. +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:47:39 [INFO] [bootrecon] boot window: fleet settled after 10s (3 identical samples 5s apart) — sweeping +2026/10/24 17:47:39 [INFO] [bootrecon] "metube" is a boot orphan by intent but is HELD (an app-data operation is holding it — the app-stop guard restarts it when the operation ends) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:47:39 [INFO] [bootrecon] "paperless-ngx" is a boot orphan by intent but is HELD (drive /mnt/felhom-drives/adatlemez is not a live mountpoint) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:47:39 [INFO] [bootrecon] Boot reconciliation: nothing to start — 2 app(s) held (absent drive / quiesce / app-data operation): [metube paperless-ngx] +--- +drive NOT bound +holds: {'metube': 'putback_pending'} diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/24-run2-start-refused.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/24-run2-start-refused.txt new file mode 100644 index 00000000..f224e298 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/24-run2-start-refused.txt @@ -0,0 +1,3 @@ +{"ok":false,"data":{"reason":"held"},"error":"A(z) metube visszaállítása megszakadt. A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el. Amikor a meghajtó újra elérhető, a doboz visszateszi ezeket az adatokat, és elindítja az alkalmazást. Addig az alkalmazás nem indul el."} + +409 diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/25-run2-log-before-replug.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/25-run2-log-before-replug.txt new file mode 100644 index 00000000..8a7c1b6b --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/25-run2-log-before-replug.txt @@ -0,0 +1,146 @@ +2026/10/24 17:47:23 [INFO] local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible +2026/10/24 17:47:23 [INFO] local-api: mount mp8 → /mnt/felhom-drives (storage=/mnt/felhom-drives, class=, backup=false) +2026/10/24 17:47:23 [INFO] local-api: mount mp9 → /etc/felhom-bootstrap (storage=/var/lib/felhom-agent/guests/9201/bootstrap, class=, backup=false) +2026/10/24 17:47:23 [INFO] local-api: mount mp0 → /var/lib/felhom (storage=local-lvm, class=, backup=true) +2026/10/24 17:47:23 [INFO] felhom-controller 0.343.0-dev.r1251b starting (customer: tester-1, domain: enkicsifelhom.hu) +2026/10/24 17:47:23 [INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json +2026/10/24 17:47:23 [INFO] Encryption key loaded from /opt/docker/felhom-controller/data/encryption.key +2026/10/24 17:47:23 [INFO] [stacks] Using compose command: docker compose +2026/10/24 17:47:24 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/24 17:47:24 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:47:24 [INFO] [stacks] InjectMissingFields: processed 4 stacks +2026/10/24 17:47:24 [INFO] [stacks] Encryption migration: no stacks needed migration +2026/10/24 17:47:24 [INFO] [quiesce] loop started (poll 5m0s, max-quiesce 30m0s) +2026/10/24 17:47:24 [WARN] [appstop] crash recovery: a local restore (op "unit-restore:metube") was interrupted and left 1 app(s) stopped — restarting them: [metube] +2026/10/24 17:47:24 [WARN] [appstop] crash recovery: metube's restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z cannot be reached yet (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z: no such file or directory) — NOT starting it; the put-back waits for its drive +2026/10/24 17:47:24 [WARN] [settings] restore hold SET for metube — the app stays stopped until it is cleared +2026/10/24 17:47:24 [INFO] [settings] Settings saved +2026/10/24 17:47:24 [WARN] [appstop] metube: the cut restore's copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z is on a drive that is not bound — the app is HELD until the copy is put back (R-1251) +2026/10/24 17:47:24 [WARN] [appstop] crash recovery: 1 app(s) were deliberately NOT restarted (drive absent: []; put-back waiting for its drive: [metube]) — KEEPING the marker; this is the gate working, not a fault +2026/10/24 17:47:24 [INFO] [appstop] metube: the restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z is still not reachable (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z: no such file or directory) — the app stays held until its drive is bound +2026/10/24 17:47:24 [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded (state ambiguous — legacy boot behaviour retained) +2026/10/24 17:47:24 [INFO] [stacks] installed-images backfill: 0 app(s) recorded, 4 already had a record, 0 left unrecorded (could not be observed completely — unknown, which renders nothing) +2026/10/24 17:47:24 [INFO] [stacks] pin adoption: 0 pinned, 4 already pinned, 0 left unpinned (0 not completely observed, 0 running something the template no longer offers) +2026/10/24 17:47:24 [INFO] [stacks] applied-meta backfill (R-646): recorded 0 []; skipped 0 [] — not current with the catalog, their pinned version's .felhom.yml is no longer on the box +2026/10/24 17:47:24 [INFO] [sync] Starting catalog sync (repo: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git, interval: 15m0s) +2026/10/24 17:47:24 [INFO] [sync] Starting catalog sync +2026/10/24 17:47:24 [INFO] Metrics store opened at /opt/docker/felhom-controller/data/metrics.db +2026/10/24 17:47:24 [INFO] Metrics collector started (60s interval) +2026/10/24 17:47:24 [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (branch: main) +2026/10/24 17:47:24 [INFO] [tier2] start-up: no leftover second-drive copy container +2026/10/24 17:47:24 [WARN] [backup] restore of metube (restore, started 2026-10-24T17:42:16Z) was INTERRUPTED by a controller stop — recorded as failed; the household is told to run it again +2026/10/24 17:47:24 [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: true) +2026/10/24 17:47:24 [INFO] [backup] app-data run record wired: /opt/docker/felhom-controller/data/appdata-run.json (an earlier cut still shown: false) +2026/10/24 17:47:24 [INFO] Notifier enabled (hub: https://hub.felhom.eu) +2026/10/24 17:47:24 [INFO] [upwatch] the host's weekly line test: 94.3 Mbit/s (measured 2026-10-24T08:00:58Z) +2026/10/24 17:47:24 [INFO] [upwatch] upload and CPU watch started (line 94 Mbit/s, qBittorrent share 50 %) +2026/10/24 17:47:24 [WARN] [appstop] a local restore was interrupted by a controller restart — 1 app(s) are HELD until their data drive is bound: the copy taken before the change is on it and is put back first — not alarming: op=unit-restore:metube reason=unit_restore started_at=2026-10-24T17:42:17Z restarted=[] put_back_waiting_for_drive=[metube] +2026/10/24 17:47:24 [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30) +2026/10/24 17:47:24 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: status-refresh (every 10s) +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s) +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: docker-socket-watch (every 15s) +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: docker-socket-users (every 5m0s) +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: health-probes (every 10s) +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: system-health (every 5m0s) +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: after-unclean-boot (every 30s) +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s) +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: ring-spill (every 30s) +2026/10/24 17:47:24 [INFO] [scheduler] Daily job db-dump scheduled for 2026-10-25 02:30 CET +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s) +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s) +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: conversion-copy-release (every 1h0m0s) +2026/10/24 17:47:24 [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-10-25 03:30 CET +2026/10/24 17:47:24 [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-25 04:15 CET +2026/10/24 17:47:24 [INFO] [catch-up] controller start: no backup leg missed its last scheduled time — nothing to make up +2026/10/24 17:47:24 [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-10-25 05:10 CET +2026/10/24 17:47:24 [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-10-25 06:00 CET +2026/10/24 17:47:24 [INFO] [scheduler] Daily job offsite-proof scheduled for 2026-10-25 05:30 CET +2026/10/24 17:47:24 [INFO] [scheduler] Daily job metrics-prune scheduled for 2026-10-25 04:00 CET +2026/10/24 17:47:24 [INFO] [scheduler] Daily job fill-watch scheduled for 2026-10-25 03:30 CET +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: fill-watch-interval (every 10m0s) +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: hub-report (every 15m0s) +2026/10/24 17:47:24 [INFO] Hub reporting enabled (every 15m0s to https://hub.felhom.eu) +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s) +2026/10/24 17:47:24 [INFO] ========== Startup Self-Test ========== +2026/10/24 17:47:24 [INFO] [PASS] Docker socket: reachable (v29.8.2) +2026/10/24 17:47:24 [INFO] [PASS] Stacks directory: /opt/docker/stacks +2026/10/24 17:47:24 [INFO] [PASS] Data directory: /opt/docker/felhom-controller/data (writable) +2026/10/24 17:47:24 [INFO] [PASS] System data path: /mnt/sys_drive +2026/10/24 17:47:24 [INFO] [WARN] Storage paths: 0 connected, 1 disconnected +2026/10/24 17:47:24 [INFO] [PASS] Git catalog: 65 app definitions found +2026/10/24 17:47:24 [INFO] [PASS] Hub connectivity: https://hub.felhom.eu reachable (HTTP 200) +2026/10/24 17:47:24 [INFO] [PASS] Metrics DB: 34.2 MB +2026/10/24 17:47:24 [INFO] ======================================== +2026/10/24 17:47:24 [INFO] Self-test complete: 7 passed, 1 warnings, 0 failed +2026/10/24 17:47:24 [INFO] [scheduler] Starting scheduler with 23 jobs +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: geo-verify (every 6h0m0s) +2026/10/24 17:47:24 [INFO] Geo-restriction support enabled (CF API token configured) +2026/10/24 17:47:24 [INFO] [report] hub wait channel active (hold ≤240s) +2026/10/24 17:47:24 [INFO] [infra] connected felhom-controller to traefik-public +2026/10/24 17:47:24 [INFO] [backup] Found 1 DB dump files across drives +2026/10/24 17:47:24 [INFO] Event pushed: restore_interrupted (warning) [hu-only] — metube: A visszaállítás félbeszakadt (áramszünet). Az alkalmazás nem indul el, amíg a meghajtója vissza nem tér: akkor a doboz visszateszi a visszaállítás előtti adatait, és elindítja. Ne indítsd újra a visszaállítást, mert az a mentést tölti vissza az adataid helyére. +2026/10/24 17:47:24 [INFO] [web] sessions: restored 8 dashboard session(s) from disk (dropped 0 expired, 0 malformed) +2026/10/24 17:47:24 [INFO] [web] Auth: using password from settings.json +2026/10/24 17:47:24 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/adatlemez not mounted — skipping userdata skeleton +2026/10/24 17:47:24 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_a not mounted — skipping userdata skeleton +2026/10/24 17:47:24 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_b not mounted — skipping userdata skeleton +2026/10/24 17:47:24 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_c not mounted — skipping userdata skeleton +2026/10/24 17:47:24 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_d not mounted — skipping userdata skeleton +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s) +2026/10/24 17:47:24 [INFO] [scheduler] Registered periodic job: agent-channel-health (every 1m0s) +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/24 17:47:24 [INFO] Web UI listening on :8080 +2026/10/24 17:47:24 [INFO] [backup] Discovered 2 databases +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:47:24 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:47:24 [INFO] [backup] Discovered app data: 4 apps +2026/10/24 17:47:24 [INFO] [backup] Backup status cache refreshed +2026/10/24 17:47:24 [INFO] [infra] filebrowser compose exists — starting without regenerating +2026/10/24 17:47:24 [INFO] [monitor] Health check: status=fail +2026/10/24 17:47:25 [ERROR] [stacks] Command failed: docker compose up -d (in /opt/docker/stacks/filebrowser) — exit code 1 (took 0.2s) +2026/10/24 17:47:25 [ERROR] [stacks] stderr (last part): ...emon: Conflict. The container name "/fb851f608269_filebrowser" is already in use by container "41f9f100523ff03118e131221dad8a11cab19f8fd4c4ba899e419d9ac9cf4722". You have to remove (or rename) that container to be able to reuse that name. +Error response from daemon: Conflict. The container name "/fb851f608269_filebrowser" is already in use by container "41f9f100523ff03118e131221dad8a11cab19f8fd4c4ba899e419d9ac9cf4722". You have to remove (or rename) that container to be able to reuse that name. +2026/10/24 17:47:25 [WARN] [infra] first-boot base-stack bring-up: base-infra bring-up: filebrowser: compose up: : exit code 1 +stderr: ...emon: Conflict. The container name "/fb851f608269_filebrowser" is already in use by container "41f9f100523ff03118e131221dad8a11cab19f8fd4c4ba899e419d9ac9cf4722". You have to remove (or rename) that container to be able to reuse that name. +Error response from daemon: Conflict. The container name "/fb851f608269_filebrowser" is already in use by container "41f9f100523ff03118e131221dad8a11cab19f8fd4c4ba899e419d9ac9cf4722". You have to remove (or rename) that container to be able to reuse that name. +2026/10/24 17:47:25 [INFO] [sync] Catalog sync complete +2026/10/24 17:47:25 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +2026/10/24 17:47:25 [INFO] [gate] boot 1792863842-19140: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/adatlemez] before recreating drive-backed apps +2026/10/24 17:47:25 [INFO] [web] FileBrowser mounts synced (recreated) — 5 storage path(s), config updated +2026/10/24 17:47:25 [INFO] [infra] connected felhom-controller to felhom-filebrowser (the FileBrowser default-login check, R-1126) +2026/10/24 17:47:29 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:47:29 [INFO] [report] Building system report +2026/10/24 17:47:29 [INFO] Event pushed: controller_started (info) [hu-only] — Controller elindult (0.343.0-dev.r1251b) +2026/10/24 17:47:29 [INFO] [infra] filebrowser: default login refused for this run — joined traefik-public, files. served (R-1126) +2026/10/24 17:47:30 [INFO] [monitor] Health check: status=warn +2026/10/24 17:47:31 [INFO] [report] Hub report pushed successfully (13681 bytes) +2026/10/24 17:47:31 [INFO] [settings] Settings saved +2026/10/24 17:47:31 [INFO] Startup hub report sent +2026/10/24 17:47:34 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:47:34 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.342.0 (we are 0.343.0-dev.r1251b), no managed update running +2026/10/24 17:47:34 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:47:34 [INFO] Health probes: 2 ok (of 2 probed) +2026/10/24 17:47:39 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:47:39 [INFO] [bootrecon] boot window: fleet settled after 10s (3 identical samples 5s apart) — sweeping +2026/10/24 17:47:39 [INFO] [bootrecon] "metube" is a boot orphan by intent but is HELD (an app-data operation is holding it — the app-stop guard restarts it when the operation ends) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:47:39 [INFO] [bootrecon] "paperless-ngx" is a boot orphan by intent but is HELD (drive /mnt/felhom-drives/adatlemez is not a live mountpoint) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:47:39 [INFO] [bootrecon] Boot reconciliation: nothing to start — 2 app(s) held (absent drive / quiesce / app-data operation): [metube paperless-ngx] +2026/10/24 17:47:44 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:47:54 [INFO] [deadapp] boot grace 15m0s: the host's last boot (2026-10-24T17:44:02Z) followed an UNCLEAN stop (R-856) +2026/10/24 17:47:54 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:48:04 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:48:14 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/24 17:48:14 [INFO] [web] Login from 192.168.0.180 +2026/10/24 17:48:15 [INFO] [api] start requested for stack: metube diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/26-run2-replug.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/26-run2-replug.txt new file mode 100644 index 00000000..800b664a --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/26-run2-replug.txt @@ -0,0 +1,5 @@ +update VM 341: -scsi1 nvme-scratch:341/vm-341-disk-2.raw +scsi0: nvme-scratch:341/vm-341-disk-1.raw,size=200G +scsi1: nvme-scratch:341/vm-341-disk-2.raw,size=50G +scsihw: virtio-scsi-single +17:48:30 diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/27-run2-drive-returns.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/27-run2-drive-returns.txt new file mode 100644 index 00000000..0a85bbad --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/27-run2-drive-returns.txt @@ -0,0 +1,49 @@ +2026/10/24 17:47:24 [WARN] [appstop] crash recovery: a local restore (op "unit-restore:metube") was interrupted and left 1 app(s) stopped — restarting them: [metube] +2026/10/24 17:47:24 [WARN] [appstop] crash recovery: metube's restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z cannot be reached yet (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z: no such file or directory) — NOT starting it; the put-back waits for its drive +2026/10/24 17:47:24 [WARN] [settings] restore hold SET for metube — the app stays stopped until it is cleared +2026/10/24 17:47:24 [WARN] [appstop] metube: the cut restore's copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z is on a drive that is not bound — the app is HELD until the copy is put back (R-1251) +2026/10/24 17:47:24 [WARN] [appstop] crash recovery: 1 app(s) were deliberately NOT restarted (drive absent: []; put-back waiting for its drive: [metube]) — KEEPING the marker; this is the gate working, not a fault +2026/10/24 17:47:24 [INFO] [appstop] metube: the restore copy /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z is still not reachable (stat /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z: no such file or directory) — the app stays held until its drive is bound +2026/10/24 17:47:24 [WARN] [backup] restore of metube (restore, started 2026-10-24T17:42:16Z) was INTERRUPTED by a controller stop — recorded as failed; the household is told to run it again +2026/10/24 17:47:24 [WARN] [appstop] a local restore was interrupted by a controller restart — 1 app(s) are HELD until their data drive is bound: the copy taken before the change is on it and is put back first — not alarming: op=unit-restore:metube reason=unit_restore started_at=2026-10-24T17:42:17Z restarted=[] put_back_waiting_for_drive=[metube] +2026/10/24 17:47:24 [INFO] [report] hub wait channel active (hold ≤240s) +2026/10/24 17:47:24 [INFO] Event pushed: restore_interrupted (warning) [hu-only] — metube: A visszaállítás félbeszakadt (áramszünet). Az alkalmazás nem indul el, amíg a meghajtója vissza nem tér: akkor a doboz visszateszi a visszaállítás előtti adatait, és elindítja. Ne indítsd újra a visszaállítást, mert az a mentést tölti vissza az adataid helyére. +2026/10/24 17:47:25 [INFO] [gate] boot 1792863842-19140: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/adatlemez] before recreating drive-backed apps +2026/10/24 17:47:39 [INFO] [bootrecon] "metube" is a boot orphan by intent but is HELD (an app-data operation is holding it — the app-stop guard restarts it when the operation ends) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:47:39 [INFO] [bootrecon] "paperless-ngx" is a boot orphan by intent but is HELD (drive /mnt/felhom-drives/adatlemez is not a live mountpoint) — NOT starting it; whatever is holding it owns its recovery +2026/10/24 17:47:39 [INFO] [bootrecon] Boot reconciliation: nothing to start — 2 app(s) held (absent drive / quiesce / app-data operation): [metube paperless-ngx] +2026/10/24 17:48:15 [INFO] [api] start requested for stack: metube +2026/10/24 17:48:47 [INFO] [gate] boot 1792863842-19140: live bind confirmed — recreating drive-backed app metube (state=stopped) onto /mnt/felhom-drives/adatlemez +2026/10/24 17:48:47 [INFO] [stacks] Stopping stack: metube +2026/10/24 17:48:47 [INFO] [stacks] Stack metube stopped successfully (took 0.1s) +2026/10/24 17:48:47 [INFO] [backup] Restoring Docker volume metube_metube_state for metube +2026/10/24 17:49:03 [INFO] [backup] Restored 1 Docker volume(s) for metube +2026/10/24 17:49:03 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/metube — 3 env vars, 0 encrypted, 0 sensitive fields +2026/10/24 17:49:03 [INFO] [stacks] Redeploying metube from recovery unit with 3 env vars +2026/10/24 17:49:03 [INFO] [stacks] SaveAppConfig: saved config for metube +2026/10/24 17:49:03 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/metube — 3 env vars, 0 encrypted, 0 sensitive fields +2026/10/24 17:49:03 [INFO] [stacks] pin metube: metube=ghcr.io/alexta69/metube:2026.09.29 +2026/10/24 17:49:03 [INFO] [stacks] SaveAppConfig: saved config for metube +2026/10/24 17:49:03 [WARN] [appstop] metube: the interrupted restore is put back from /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z — 1 volume(s) poured back, 0 removed, definition=true (R-1195) +2026/10/24 17:49:03 [INFO] [settings] restore hold CLEARED for metube +2026/10/24 17:49:03 [WARN] [appstop] metube: the drive is bound — the interrupted restore is put back from /mnt/felhom-drives/adatlemez/backups/restore-undo/metube/20261024T174218Z BEFORE the app starts; its data is as before the restore (R-1251) +2026/10/24 17:49:03 [INFO] [stacks] Stopping stack: metube +--- +TARGET SOURCE FSTYPE OPTIONS +/mnt/felhom-drives/adatlemez /dev/sdb[/felhom-data] ext4 rw,relatime +/mnt/felhom-drives/adatlemez /dev/sdb[/felhom-data] ext4 rw,relatime +metube Up 49 seconds (healthy) +gen2 — the household newest, after the backup (2026-10-24T17:42:05Z) +total 1572880 +drwxr-xr-x 2 1000 1000 4096 Oct 24 17:48 . +drwx-----x 3 root root 4096 Oct 24 17:48 .. +-rw-r--r-- 1 1000 1000 1610612736 Oct 24 17:40 r1251-ballast.bin +-rw-r--r-- 1 1000 1000 71 Oct 24 17:42 r1251-gen.txt +3e1ec80f99e09173 +total 12 +drwxrwsr-x 2 1000 1000 4096 Oct 24 17:16 . +drwxrwsr-x 16 root 1000 4096 Oct 13 19:26 .. +-rw-r--r-- 1 1000 1000 53 Oct 24 17:16 r1251-household-video.txt +ls: cannot access '/mnt/felhom-drives/adatlemez/backups/restore-undo/metube': No such file or directory +holds: {} +ls: cannot access '/var/lib/docker/volumes/felhom-controller-data/_data/data/appstop-state.json': No such file or directory diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/28-run2-later-restart.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/28-run2-later-restart.txt new file mode 100644 index 00000000..ddd2fd4d --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/28-run2-later-restart.txt @@ -0,0 +1,4 @@ +2026/10/24 17:50:11 [INFO] [report] hub wait channel active (hold ≤240s) +-- after restart: +gen3 — written after the put-back (2026-10-24T17:50:09Z) +metube Up About a minute (healthy) diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/29-run2-log-after-restart.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/29-run2-log-after-restart.txt new file mode 100644 index 00000000..9c1e8f27 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/29-run2-log-after-restart.txt @@ -0,0 +1,122 @@ +2026/10/24 17:50:11 [INFO] local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible +2026/10/24 17:50:11 [INFO] local-api: mount mp8 → /mnt/felhom-drives (storage=/mnt/felhom-drives, class=, backup=false) +2026/10/24 17:50:11 [INFO] local-api: mount mp9 → /etc/felhom-bootstrap (storage=/var/lib/felhom-agent/guests/9201/bootstrap, class=, backup=false) +2026/10/24 17:50:11 [INFO] local-api: mount mp0 → /var/lib/felhom (storage=local-lvm, class=, backup=true) +2026/10/24 17:50:11 [INFO] felhom-controller 0.343.0-dev.r1251b starting (customer: tester-1, domain: enkicsifelhom.hu) +2026/10/24 17:50:11 [INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json +2026/10/24 17:50:11 [INFO] Encryption key loaded from /opt/docker/felhom-controller/data/encryption.key +2026/10/24 17:50:11 [INFO] [stacks] Using compose command: docker compose +2026/10/24 17:50:11 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/24 17:50:11 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:50:11 [INFO] [stacks] InjectMissingFields: processed 4 stacks +2026/10/24 17:50:11 [INFO] [stacks] Encryption migration: no stacks needed migration +2026/10/24 17:50:11 [INFO] [quiesce] loop started (poll 5m0s, max-quiesce 30m0s) +2026/10/24 17:50:11 [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded (state ambiguous — legacy boot behaviour retained) +2026/10/24 17:50:11 [INFO] [stacks] installed-images backfill: 0 app(s) recorded, 4 already had a record, 0 left unrecorded (could not be observed completely — unknown, which renders nothing) +2026/10/24 17:50:11 [INFO] [stacks] pin adoption: 0 pinned, 4 already pinned, 0 left unpinned (0 not completely observed, 0 running something the template no longer offers) +2026/10/24 17:50:11 [INFO] [stacks] applied-meta backfill (R-646): recorded 0 []; skipped 0 [] — not current with the catalog, their pinned version's .felhom.yml is no longer on the box +2026/10/24 17:50:11 [INFO] [sync] Starting catalog sync (repo: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git, interval: 15m0s) +2026/10/24 17:50:11 [INFO] [sync] Starting catalog sync +2026/10/24 17:50:11 [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (branch: main) +2026/10/24 17:50:11 [INFO] Metrics store opened at /opt/docker/felhom-controller/data/metrics.db +2026/10/24 17:50:11 [INFO] Metrics collector started (60s interval) +2026/10/24 17:50:11 [INFO] [tier2] start-up: no leftover second-drive copy container +2026/10/24 17:50:11 [INFO] [backup] restore record loaded: 1 interrupted restore notice(s) still shown +2026/10/24 17:50:11 [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false) +2026/10/24 17:50:11 [INFO] [backup] app-data run record wired: /opt/docker/felhom-controller/data/appdata-run.json (an earlier cut still shown: false) +2026/10/24 17:50:11 [INFO] Notifier enabled (hub: https://hub.felhom.eu) +2026/10/24 17:50:11 [INFO] [upwatch] the host's weekly line test: 94.3 Mbit/s (measured 2026-10-24T08:00:58Z) +2026/10/24 17:50:11 [INFO] [upwatch] upload and CPU watch started (line 94 Mbit/s, qBittorrent share 50 %) +2026/10/24 17:50:11 [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: status-refresh (every 10s) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: docker-socket-watch (every 15s) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: docker-socket-users (every 5m0s) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: health-probes (every 10s) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: system-health (every 5m0s) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: after-unclean-boot (every 30s) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: ring-spill (every 30s) +2026/10/24 17:50:11 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply +2026/10/24 17:50:11 [INFO] [scheduler] Daily job db-dump scheduled for 2026-10-25 02:30 CET +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: conversion-copy-release (every 1h0m0s) +2026/10/24 17:50:11 [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-10-25 03:30 CET +2026/10/24 17:50:11 [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-25 04:15 CET +2026/10/24 17:50:11 [INFO] [catch-up] controller start: no backup leg missed its last scheduled time — nothing to make up +2026/10/24 17:50:11 [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-10-25 05:10 CET +2026/10/24 17:50:11 [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-10-25 06:00 CET +2026/10/24 17:50:11 [INFO] [scheduler] Daily job offsite-proof scheduled for 2026-10-25 05:30 CET +2026/10/24 17:50:11 [INFO] [scheduler] Daily job metrics-prune scheduled for 2026-10-25 04:00 CET +2026/10/24 17:50:11 [INFO] [scheduler] Daily job fill-watch scheduled for 2026-10-25 03:30 CET +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: fill-watch-interval (every 10m0s) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: hub-report (every 15m0s) +2026/10/24 17:50:11 [INFO] Hub reporting enabled (every 15m0s to https://hub.felhom.eu) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s) +2026/10/24 17:50:11 [INFO] ========== Startup Self-Test ========== +2026/10/24 17:50:11 [INFO] [PASS] Docker socket: reachable (v29.8.2) +2026/10/24 17:50:11 [INFO] [PASS] Stacks directory: /opt/docker/stacks +2026/10/24 17:50:11 [INFO] [PASS] Data directory: /opt/docker/felhom-controller/data (writable) +2026/10/24 17:50:11 [INFO] [PASS] System data path: /mnt/sys_drive +2026/10/24 17:50:11 [INFO] [PASS] Storage paths: 1 connected, 0 disconnected +2026/10/24 17:50:11 [INFO] [PASS] Git catalog: 65 app definitions found +2026/10/24 17:50:11 [INFO] [PASS] Hub connectivity: https://hub.felhom.eu reachable (HTTP 200) +2026/10/24 17:50:11 [INFO] [PASS] Metrics DB: 34.2 MB +2026/10/24 17:50:11 [INFO] ======================================== +2026/10/24 17:50:11 [INFO] Self-test complete: 8 passed, 0 warnings, 0 failed +2026/10/24 17:50:11 [INFO] [scheduler] Starting scheduler with 23 jobs +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: geo-verify (every 6h0m0s) +2026/10/24 17:50:11 [INFO] Geo-restriction support enabled (CF API token configured) +2026/10/24 17:50:11 [INFO] [report] hub wait channel active (hold ≤240s) +2026/10/24 17:50:11 [INFO] [backup] Found 2 DB dump files across drives +2026/10/24 17:50:11 [INFO] [infra] connected felhom-controller to traefik-public +2026/10/24 17:50:11 [INFO] [web] sessions: restored 9 dashboard session(s) from disk (dropped 0 expired, 0 malformed) +2026/10/24 17:50:11 [INFO] [web] Auth: using password from settings.json +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s) +2026/10/24 17:50:11 [INFO] [scheduler] Registered periodic job: agent-channel-health (every 1m0s) +2026/10/24 17:50:11 [INFO] Web UI listening on :8080 +2026/10/24 17:50:11 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_a not mounted — skipping userdata skeleton +2026/10/24 17:50:11 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_b not mounted — skipping userdata skeleton +2026/10/24 17:50:11 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_c not mounted — skipping userdata skeleton +2026/10/24 17:50:11 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_d not mounted — skipping userdata skeleton +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/24 17:50:11 [INFO] [backup] Discovered 2 databases +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/metube/docker-compose.yml +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:50:11 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/24 17:50:11 [INFO] [backup] Discovered app data: 4 apps +2026/10/24 17:50:11 [INFO] [backup] Backup status cache refreshed +2026/10/24 17:50:11 [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (5 storage path(s)) +2026/10/24 17:50:11 [INFO] [monitor] Health check: status=ok +2026/10/24 17:50:12 [INFO] [infra] connected felhom-controller to felhom-filebrowser (the FileBrowser default-login check, R-1126) +2026/10/24 17:50:12 [INFO] [sync] Catalog sync complete +2026/10/24 17:50:12 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +2026/10/24 17:50:16 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:50:16 [INFO] [report] Building system report +2026/10/24 17:50:16 [INFO] Event pushed: controller_started (info) [hu-only] — Controller elindult (0.343.0-dev.r1251b) +2026/10/24 17:50:17 [INFO] [monitor] Health check: status=ok +2026/10/24 17:50:18 [INFO] [report] Hub report pushed successfully (7568 bytes) +2026/10/24 17:50:18 [INFO] [settings] Settings saved +2026/10/24 17:50:18 [INFO] Startup hub report sent +2026/10/24 17:50:21 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.342.0 (we are 0.343.0-dev.r1251b), no managed update running +2026/10/24 17:50:21 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:50:21 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:50:21 [INFO] Health probes: 4 ok (of 4 probed) +2026/10/24 17:50:26 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:50:26 [INFO] [bootrecon] boot window: fleet settled after 10s (3 identical samples 5s apart) — sweeping +2026/10/24 17:50:26 [INFO] [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start) +2026/10/24 17:50:31 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:50:41 [INFO] [deadapp] boot grace 15m0s: the host's last boot (2026-10-24T17:44:02Z) followed an UNCLEAN stop (R-856) +2026/10/24 17:50:41 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/24 17:50:51 [INFO] [stacks] Status refresh: 10 containers across 68 stacks diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/30-run2-agent-journal.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/30-run2-agent-journal.txt new file mode 100644 index 00000000..9ff9bc02 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/30-run2-agent-journal.txt @@ -0,0 +1,20 @@ +Oct 24 19:46:44 felhom felhom-agent[1146]: time=2026-10-24T19:46:44.625+02:00 level=INFO msg="storage: enrolled drive absent by UUID — not re-asserting (will retry when it enumerates)" name=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 where=/mnt/adatlemez uuid=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 24 19:47:04 felhom felhom-agent[1146]: time=2026-10-24T19:47:04.625+02:00 level=INFO msg="storage: enrolled drive absent by UUID — not re-asserting (will retry when it enumerates)" name=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 where=/mnt/adatlemez uuid=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 24 19:47:22 felhom felhom-agent[1146]: time=2026-10-24T19:47:22.539+02:00 level=WARN msg="local-api: storage view unavailable for class hints" err="storage: NodeStorage: proxmox: GET /nodes/felhom/storage: Get \"https://127.0.0.1:8006/api2/json/nodes/felhom/storage\": context canceled" +Oct 24 19:47:24 felhom felhom-agent[1146]: time=2026-10-24T19:47:24.626+02:00 level=INFO msg="storage: enrolled drive absent by UUID — not re-asserting (will retry when it enumerates)" name=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 where=/mnt/adatlemez uuid=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 24 19:47:44 felhom felhom-agent[1146]: time=2026-10-24T19:47:44.625+02:00 level=INFO msg="storage: enrolled drive absent by UUID — not re-asserting (will retry when it enumerates)" name=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 where=/mnt/adatlemez uuid=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 24 19:48:04 felhom felhom-agent[1146]: time=2026-10-24T19:48:04.625+02:00 level=INFO msg="storage: enrolled drive absent by UUID — not re-asserting (will retry when it enumerates)" name=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 where=/mnt/adatlemez uuid=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 24 19:48:24 felhom felhom-agent[1146]: time=2026-10-24T19:48:24.625+02:00 level=INFO msg="storage: enrolled drive absent by UUID — not re-asserting (will retry when it enumerates)" name=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 where=/mnt/adatlemez uuid=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 24 19:48:33 felhom felhom-agent[1146]: time=2026-10-24T19:48:33.979+02:00 level=INFO msg="storage: watchdog self-heal — re-mounting enrolled drive (device returned, no eject intent)" target=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 where=/mnt/adatlemez durable_id=uuid:863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 24 19:48:33 felhom felhom-agent[1146]: time=2026-10-24T19:48:33.979+02:00 level=INFO msg="audit: gate decision" class=storage_mount host=tester-1-d70be4 guest=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 source=desired_delta disposition=benign allowed=true reason=benign key_id="" nonce="" durable_id="" +Oct 24 19:48:33 felhom felhom-agent[1146]: time=2026-10-24T19:48:33.979+02:00 level=INFO msg="gate decision" class=storage_mount guest=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 source=desired_delta disposition=benign allowed=true reason=benign +Oct 24 19:48:33 felhom sudo[6340]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-priv-apply unit mnt-adatlemez.mount +Oct 24 19:48:34 felhom felhom-priv-apply[6343]: felhom-priv-apply: SAME unit /etc/systemd/system/mnt-adatlemez.mount +Oct 24 19:48:34 felhom sudo[6383]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/systemctl enable --now -- mnt-adatlemez.mount +Oct 24 19:48:34 felhom felhom-agent[1146]: time=2026-10-24T19:48:34.896+02:00 level=INFO msg="storage: ensured mount" name=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 where=/mnt/adatlemez unit=mnt-adatlemez.mount +Oct 24 19:48:34 felhom felhom-agent[1146]: time=2026-10-24T19:48:34.896+02:00 level=INFO msg="storage: re-mounted returned target" target=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 where=/mnt/adatlemez +Oct 24 19:48:38 felhom felhom-agent[1146]: time=2026-10-24T19:48:38.978+02:00 level=WARN msg="storage: watchdog detected target state change" target=863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 from=disconnected to=attached +Oct 24 19:48:38 felhom felhom-agent[1146]: time=2026-10-24T19:48:38.979+02:00 level=INFO msg="storage: watchdog triggering out-of-band host-report" transitions=1 +Oct 24 19:48:39 felhom sudo[6514]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +Oct 24 19:48:45 felhom sudo[6810]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mount --bind /mnt/adatlemez/felhom-data /mnt/felhom-drives/adatlemez +Oct 24 19:48:45 felhom felhom-agent[1146]: time=2026-10-24T19:48:45.447+02:00 level=INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/adatlemez stable=/mnt/felhom-drives/adatlemez prior_binds=0 diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/40-teardown.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/40-teardown.txt new file mode 100644 index 00000000..ac3d16d6 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/40-teardown.txt @@ -0,0 +1,36 @@ +{"ok":false,"error":"stack \"metube\" is still running — stop it first before removing"} + +409 +audiobooks +books +comics +grimmory +movies +music +nzbget +photos +podcasts +torrents +offsite-proof +primary +restore-undo +secondary +1 +app.yaml +applied-compose.yml +applied-meta +docker-compose.yml +gitea.dooplex.hu/admin/felhom-controller:0.342.0 +gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 30 seconds (healthy) +Deleted: sha256:59d3821e5f5eed4430cd649b853ca587ced89cfef1dad88466007c9b5efeac04 +Deleted: sha256:c854f867806d050e014d0f82ca46a1f12f775b083dd31b7c9e058083c721886e +200 +{"ok":true,"data":{"removed":"metube","volumes_removed":["metube_metube_state"],"hdd_paths_removed":[],"hdd_paths_preserved":[],"userdata_kept":["/mnt/felhom-drives/adatlemez/userdata/media/metube (4.0K)"],"backup_paths_removed":["/mnt/felhom-drives/adatlemez/backups/primary/metube (1.6G)"],"verified":true},"message":"Stack metube removed"} + +200 +offsite-proof +primary +secondary +0 +docker-compose.yml +0 diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/41-at-end.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/41-at-end.txt new file mode 100644 index 00000000..d90ceae2 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/41-at-end.txt @@ -0,0 +1,45 @@ +# Tester 1 at end — 2026-10-24T17:52:52Z +## host +felhom +felhom-agent 0.164.0 +│ └─/mnt/felhom-drives/adatlemez /dev/sdb[/felhom-data] +└─/mnt/adatlemez /dev/sdb +/dev/sdb 49G 31G 17G 66% /mnt/adatlemez +scsi0: nvme-scratch:341/vm-341-disk-1.raw,size=200G +scsi1: nvme-scratch:341/vm-341-disk-2.raw,size=50G +scsihw: virtio-scsi-single +## guest +gitea.dooplex.hu/admin/felhom-controller:0.342.0 +bookstack lscr.io/linuxserver/bookstack:26.09.1 Up 5 minutes (healthy) +bookstack-db mariadb:12.3 Up 5 minutes (healthy) +cloudflared cloudflare/cloudflared:2026.9.3 Up 5 minutes (healthy) +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up About a minute (healthy) +filebrowser gtstef/filebrowser:1.5.6-stable Up 3 minutes (healthy) +paperless-postgres postgres:18-alpine Up 3 minutes (healthy) +paperless-redis redis:7-alpine Up 3 minutes (healthy) +paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 3 minutes (healthy) +privatebin privatebin/pdo:2.0.6 Up 5 minutes (healthy) +traefik traefik:v3.7.13 Up 5 minutes +-- named volumes +bookstack_bookstack_config +bookstack_bookstack_db_data +felhom-controller-data +filebrowser_filebrowser_data +paperless-ngx_paperless_data +paperless-ngx_paperless_postgres_data +paperless-ngx_paperless_redis_data +privatebin_privatebin_data +-- anonymous volumes: 202 +/mnt/felhom-drives/adatlemez/backups/: +offsite-proof +primary +secondary + +/mnt/felhom-drives/adatlemez/backups/primary: +paperless-ngx +holds: [] +ls: cannot access '/var/lib/docker/volumes/felhom-controller-data/_data/data/appstop-state.json': No such file or directory +docker-compose.yml +0.342.0 +0.341.0 +0.318.0-rc.2 diff --git a/documentation/audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt b/documentation/audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt new file mode 100644 index 00000000..1ec9cdef --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt @@ -0,0 +1,14 @@ +2a78047f78ed975d6988b9a7c1a20e2d8996beb4d761c0658ef6139f7a1e501b 2026-10-24T17:17:50Z 72K +b9682d7c22236517f96a418e7788abd5ebb950bba1ade0ff8e12a63ff867b6e6 2026-10-24T17:17:50Z 4.0K +d2f28971e52b7f500ea3273d6eed43ab0036f2548f1e5bcbd40e7970a0e9c660 2026-10-24T17:17:50Z 24K + +/privatebin 78a9fff9b159c6b367934dbc8b9ffd0b393521c44a05c52b122dd5e81d1ae152 365e817db1f058ff2ec9adb1a90e7a865b2fc1c8907 +2a78047f78ed975d6988b9a7c1a20e2d8996beb4d761c0658ef6139f7a1e501b +b9682d7c22236517f96a418e7788abd5ebb950bba1ade0ff8e12a63ff867b6e6 +d2f28971e52b7f500ea3273d6eed43ab0036f2548f1e5bcbd40e7970a0e9c660 +anonymous now: 199 +78a9fff9b159c6b367934dbc8b9ffd0b393521c44a05c52b122dd5e81d1ae152 -> /run +365e817db1f058ff2ec9adb1a90e7a865b2fc1c8907fa73a410d4300278229e3 -> /tmp +privatebin_privatebin_data -> /srv/data +3878ff257f55ead8bd4a48864fd4d0bb694e971bc4e5e111c293d1445e0c6641 -> /var/lib/nginx/tmp + diff --git a/documentation/audits/restore-holes-2026-10-24/P2-9202/00-as-found.txt b/documentation/audits/restore-holes-2026-10-24/P2-9202/00-as-found.txt new file mode 100644 index 00000000..a0d42ee5 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P2-9202/00-as-found.txt @@ -0,0 +1,43 @@ +# 9202 as found — 2026-10-24T18:08:53Z +gitea.dooplex.hu/admin/felhom-controller:0.342.0 +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 10 hours (healthy) +filebrowser gtstef/filebrowser:1.5.6-stable Up 27 hours (healthy) +immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 5 hours (healthy) +immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 5 hours (healthy) +immich-redis redis:7-alpine Up 5 hours (healthy) +immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 5 hours (healthy) +paperless-postgres postgres:18-alpine Up 5 hours (healthy) +paperless-redis redis:7-alpine Up 5 hours (healthy) +paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 5 hours (healthy) +traefik traefik:v3.7.13 Up 27 hours +-- named volumes +felhom-controller-data +filebrowser_filebrowser_data +immich_immich_ml_cache +immich_immich_postgres_data +immich_immich_redis_data +paperless-ngx_paperless_data +paperless-ngx_paperless_postgres_data +paperless-ngx_paperless_redis_data +privatebin_privatebin_data +-- anonymous: 294 +/mnt/felhom-drives/scratch_hdd/backups/primary: + +/mnt/sizing-bench/backups/secondary: +paperless-ngx +audiobooks +books +comics +grimmory +metube +movies +music +nzbget +photos +podcasts +torrents +transmission +tv +usenet +holds: ['chaoscrash', 'chaosoom', 'chaosoomb', 'gokapi'] kept: 0 +docker-compose.yml diff --git a/documentation/audits/restore-holes-2026-10-24/P2-9202/01-switch.txt b/documentation/audits/restore-holes-2026-10-24/P2-9202/01-switch.txt new file mode 100644 index 00000000..cb25c96c --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P2-9202/01-switch.txt @@ -0,0 +1,6 @@ +gitea.dooplex.hu/admin/felhom-controller:0.343.0-dev.r1252a Up 25 seconds (healthy) +{"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"} + +202 +audiobookshelf Up 21 seconds (healthy) +-rw-r--r-- 1 root 1000 58 Oct 24 18:10 /mnt/felhom-drives/scratch_hdd/userdata/media/audiobooks/r1252-marker.txt diff --git a/documentation/audits/restore-holes-2026-10-24/P2-9202/02-control-runs.txt b/documentation/audits/restore-holes-2026-10-24/P2-9202/02-control-runs.txt new file mode 100644 index 00000000..d20e93ea --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P2-9202/02-control-runs.txt @@ -0,0 +1,52 @@ +== control run 1 +{"data":{"legs":["db-dump","tier2","update-leg"]},"message":"started","ok":true} + +202 + +2026/10/24 18:10:41 night_chain.go:108: [INFO] [night-chain] manual run started from 172.18.0.4:37144: [db-dump tier2 update-leg] +2026/10/24 18:10:41 night_chain.go:84: [INFO] [night-chain] manual run of tonight's chain: dump → second drive → off-site → update leg +2026/10/24 18:10:41 night_chain.go:77: [INFO] [night-chain] db-dump: started +2026/10/24 18:11:36 night_chain.go:82: [INFO] [night-chain] db-dump: done in 55s +2026/10/24 18:11:36 night_chain.go:77: [INFO] [night-chain] tier2: started +2026/10/24 18:11:37 tier2.go:521: [INFO] [backup] Tier 2 copied audiobookshelf → /mnt/sizing-bench/backups/secondary/audiobookshelf (484.0 KB, 1 leg(s), 2s) +2026/10/24 18:11:51 tier2.go:521: [INFO] [backup] Tier 2 copied immich → /mnt/felhom-drives/scratch_hdd/backups/secondary/immich (1.1 GB, 2 leg(s), 13s) +2026/10/24 18:11:53 tier2.go:521: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sizing-bench/backups/secondary/paperless-ngx (128.3 MB, 1 leg(s), 2s) +2026/10/24 18:11:53 tier2.go:521: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/scratch_hdd/backups/secondary/privatebin (8.4 KB, 0 leg(s), 0s) +2026/10/24 18:11:53 tier2.go:583: [INFO] [backup] Tier 2 run complete: 4 app(s) processed (incl. volume-only — F6) +2026/10/24 18:11:53 night_chain.go:82: [INFO] [night-chain] tier2: done in 17s +2026/10/24 18:11:53 night_chain.go:90: [INFO] [night-chain] offsite: no off-site target on this box — skipped, as at night +2026/10/24 18:11:53 night_chain.go:77: [INFO] [night-chain] update-leg: started +2026/10/24 18:11:53 night_chain.go:82: [INFO] [night-chain] update-leg: done in 0s +2026/10/24 18:11:53 night_chain.go:93: [INFO] [night-chain] finished in 1m12s +-- rejected-block lines: 0 +-- the copy of the marker: +total 12 +drwxrwsr-x 2 root 1000 4096 Oct 24 18:10 . +drwxr-xr-x 3 root root 4096 Oct 24 18:11 .. +-rw-r--r-- 1 root 1000 58 Oct 24 18:10 r1252-marker.txt +== control run 2 +{"data":{"legs":["db-dump","tier2","update-leg"]},"message":"started","ok":true} + +202 + +2026/10/24 18:11:59 night_chain.go:108: [INFO] [night-chain] manual run started from 172.18.0.4:37144: [db-dump tier2 update-leg] +2026/10/24 18:11:59 night_chain.go:84: [INFO] [night-chain] manual run of tonight's chain: dump → second drive → off-site → update leg +2026/10/24 18:11:59 night_chain.go:77: [INFO] [night-chain] db-dump: started +2026/10/24 18:12:50 night_chain.go:82: [INFO] [night-chain] db-dump: done in 52s +2026/10/24 18:12:50 night_chain.go:77: [INFO] [night-chain] tier2: started +2026/10/24 18:12:51 tier2.go:521: [INFO] [backup] Tier 2 copied audiobookshelf → /mnt/sizing-bench/backups/secondary/audiobookshelf (487.0 KB, 1 leg(s), 1s) +2026/10/24 18:13:07 tier2.go:521: [INFO] [backup] Tier 2 copied immich → /mnt/felhom-drives/scratch_hdd/backups/secondary/immich (1.1 GB, 2 leg(s), 16s) +2026/10/24 18:13:13 tier2.go:521: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sizing-bench/backups/secondary/paperless-ngx (128.3 MB, 1 leg(s), 6s) +2026/10/24 18:13:13 tier2.go:521: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/scratch_hdd/backups/secondary/privatebin (8.4 KB, 0 leg(s), 0s) +2026/10/24 18:13:13 tier2.go:583: [INFO] [backup] Tier 2 run complete: 4 app(s) processed (incl. volume-only — F6) +2026/10/24 18:13:13 night_chain.go:82: [INFO] [night-chain] tier2: done in 23s +2026/10/24 18:13:13 night_chain.go:90: [INFO] [night-chain] offsite: no off-site target on this box — skipped, as at night +2026/10/24 18:13:13 night_chain.go:77: [INFO] [night-chain] update-leg: started +2026/10/24 18:13:13 night_chain.go:82: [INFO] [night-chain] update-leg: done in 0s +2026/10/24 18:13:13 night_chain.go:93: [INFO] [night-chain] finished in 1m15s +-- rejected-block lines: 0 +-- the copy of the marker: +total 12 +drwxrwsr-x 2 root 1000 4096 Oct 24 18:10 . +drwxr-xr-x 3 root root 4096 Oct 24 18:11 .. +-rw-r--r-- 1 root 1000 58 Oct 24 18:10 r1252-marker.txt diff --git a/documentation/audits/restore-holes-2026-10-24/P2-9202/03-break.txt b/documentation/audits/restore-holes-2026-10-24/P2-9202/03-break.txt new file mode 100644 index 00000000..30f5fd89 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P2-9202/03-break.txt @@ -0,0 +1,3 @@ +28- # If Viktor rules excluded, flip THIS LINE ONLY before committing. +29: - path: media/r1252-typo +30- class: optional # R-1252 LIVE PROOF: a path no compose bind has diff --git a/documentation/audits/restore-holes-2026-10-24/P2-9202/04-fault-run.txt b/documentation/audits/restore-holes-2026-10-24/P2-9202/04-fault-run.txt new file mode 100644 index 00000000..262cd3c0 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P2-9202/04-fault-run.txt @@ -0,0 +1,29 @@ +{"data":{"legs":["db-dump","tier2","update-leg"]},"message":"started","ok":true} + +202 + +2026/10/24 18:13:38 night_chain.go:108: [INFO] [night-chain] manual run started from 172.18.0.4:49314: [db-dump tier2 update-leg] +2026/10/24 18:13:38 night_chain.go:84: [INFO] [night-chain] manual run of tonight's chain: dump → second drive → off-site → update leg +2026/10/24 18:13:38 night_chain.go:77: [INFO] [night-chain] db-dump: started +2026/10/24 18:14:30 night_chain.go:82: [INFO] [night-chain] db-dump: done in 51s +2026/10/24 18:14:30 night_chain.go:77: [INFO] [night-chain] tier2: started +2026/10/24 18:14:30 tier2.go:501: [WARN] [backup] Tier 2 audiobookshelf: the backup block is REJECTED (userdata["media/r1252-typo"]: matches no compose bind (stale or typoed path)) — nothing reconciled, every existing copy on the second drive is KEPT (R-1252) +2026/10/24 18:14:30 tier2.go:521: [INFO] [backup] Tier 2 copied audiobookshelf → /mnt/sizing-bench/backups/secondary/audiobookshelf (489.9 KB, 0 leg(s), 1s) +2026/10/24 18:14:46 tier2.go:521: [INFO] [backup] Tier 2 copied immich → /mnt/felhom-drives/scratch_hdd/backups/secondary/immich (1.1 GB, 2 leg(s), 15s) +2026/10/24 18:14:48 tier2.go:521: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sizing-bench/backups/secondary/paperless-ngx (128.3 MB, 1 leg(s), 3s) +2026/10/24 18:14:49 tier2.go:521: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/scratch_hdd/backups/secondary/privatebin (8.4 KB, 0 leg(s), 1s) +2026/10/24 18:14:49 tier2.go:583: [INFO] [backup] Tier 2 run complete: 4 app(s) processed (incl. volume-only — F6) +2026/10/24 18:14:49 night_chain.go:82: [INFO] [night-chain] tier2: done in 19s +2026/10/24 18:14:49 night_chain.go:90: [INFO] [night-chain] offsite: no off-site target on this box — skipped, as at night +2026/10/24 18:14:49 night_chain.go:77: [INFO] [night-chain] update-leg: started +2026/10/24 18:14:49 night_chain.go:82: [INFO] [night-chain] update-leg: done in 0s +2026/10/24 18:14:49 night_chain.go:93: [INFO] [night-chain] finished in 1m11s +-- rejected-block lines: 12 +-- the copy of the marker: +total 12 +drwxrwsr-x 2 root 1000 4096 Oct 24 18:10 . +drwxr-xr-x 3 root root 4096 Oct 24 18:11 .. +-rw-r--r-- 1 root 1000 58 Oct 24 18:10 r1252-marker.txt +last_warning: None +2026/10/24 18:14:30 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event backup_run_failures (severity error) — further backup_run_failures events are logged at DEBUG only +app_backup {"enabled": false, "cross_drive": {"enabled": true, "method": "rsync", "destination_path": "/mnt/sizing-bench", "schedule": "daily", "last_run": "2026-10-24T18:14:30Z", "last_status": "ok", "last_success": "2026-10-24T18:14:30Z", "success_tracked": true, "last_warning": "Az alkalmazás mentési beállításai most nem olvashatók, ezért a második meghajtón egyetlen korábbi másolat sem törlődött. Az új mappák másolása addig szünetel. A Felhom ügyfélszolgálata értesítést kapott.", "unit_package_date": "2026-10-24T18:14:14Z", "last_duration": "1s", "last_size_human": "489.9 KB"}} diff --git a/documentation/audits/restore-holes-2026-10-24/P2-9202/05-unbreak-run.txt b/documentation/audits/restore-holes-2026-10-24/P2-9202/05-unbreak-run.txt new file mode 100644 index 00000000..85b430e0 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P2-9202/05-unbreak-run.txt @@ -0,0 +1,27 @@ +0 +restored +{"data":{"legs":["db-dump","tier2","update-leg"]},"message":"started","ok":true} + +202 + +2026/10/24 18:15:22 night_chain.go:108: [INFO] [night-chain] manual run started from 172.18.0.4:49314: [db-dump tier2 update-leg] +2026/10/24 18:15:22 night_chain.go:84: [INFO] [night-chain] manual run of tonight's chain: dump → second drive → off-site → update leg +2026/10/24 18:15:22 night_chain.go:77: [INFO] [night-chain] db-dump: started +2026/10/24 18:16:21 night_chain.go:82: [INFO] [night-chain] db-dump: done in 59s +2026/10/24 18:16:21 night_chain.go:77: [INFO] [night-chain] tier2: started +2026/10/24 18:16:22 tier2.go:521: [INFO] [backup] Tier 2 copied audiobookshelf → /mnt/sizing-bench/backups/secondary/audiobookshelf (493.0 KB, 1 leg(s), 1s) +2026/10/24 18:16:46 tier2.go:521: [INFO] [backup] Tier 2 copied immich → /mnt/felhom-drives/scratch_hdd/backups/secondary/immich (1.1 GB, 2 leg(s), 23s) +2026/10/24 18:16:49 tier2.go:521: [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sizing-bench/backups/secondary/paperless-ngx (128.3 MB, 1 leg(s), 3s) +2026/10/24 18:16:50 tier2.go:521: [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/scratch_hdd/backups/secondary/privatebin (8.4 KB, 0 leg(s), 1s) +2026/10/24 18:16:50 tier2.go:583: [INFO] [backup] Tier 2 run complete: 4 app(s) processed (incl. volume-only — F6) +2026/10/24 18:16:50 night_chain.go:82: [INFO] [night-chain] tier2: done in 29s +2026/10/24 18:16:50 night_chain.go:90: [INFO] [night-chain] offsite: no off-site target on this box — skipped, as at night +2026/10/24 18:16:50 night_chain.go:77: [INFO] [night-chain] update-leg: started +2026/10/24 18:16:50 night_chain.go:82: [INFO] [night-chain] update-leg: done in 0s +2026/10/24 18:16:50 night_chain.go:93: [INFO] [night-chain] finished in 1m28s +-- rejected-block lines: 0 +-- the copy of the marker: +total 12 +drwxrwsr-x 2 root 1000 4096 Oct 24 18:10 . +drwxr-xr-x 3 root root 4096 Oct 24 18:11 .. +-rw-r--r-- 1 root 1000 58 Oct 24 18:10 r1252-marker.txt diff --git a/documentation/audits/restore-holes-2026-10-24/P2-9202/06-teardown.txt b/documentation/audits/restore-holes-2026-10-24/P2-9202/06-teardown.txt new file mode 100644 index 00000000..5a6ba58c --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P2-9202/06-teardown.txt @@ -0,0 +1,5 @@ +{"ok":true,"data":{"removed":"audiobookshelf","volumes_removed":["audiobookshelf_audiobookshelf_config","audiobookshelf_audiobookshelf_metadata"],"hdd_paths_removed":[],"hdd_paths_preserved":[],"userdata_kept":["/mnt/felhom-drives/scratch_hdd/userdata/media/audiobooks (8.0K)","/mnt/felhom-drives/scratch_hdd/userdata/media/podcasts (4.0K)"],"backup_paths_removed":["/mnt/felhom-drives/scratch_hdd/backups/primary/audiobookshelf (524K)","/mnt/sizing-bench/backups/secondary/audiobookshelf (493.0 KB)"],"verified":true},"message":"Stack audiobookshelf removed"} + +200 +0 +rmi=0 diff --git a/documentation/audits/restore-holes-2026-10-24/P2-9202/07-at-end.txt b/documentation/audits/restore-holes-2026-10-24/P2-9202/07-at-end.txt new file mode 100644 index 00000000..c9d8ab8a --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P2-9202/07-at-end.txt @@ -0,0 +1,43 @@ +# 9202 at end — 2026-10-24T18:18:14Z +gitea.dooplex.hu/admin/felhom-controller:0.342.0 +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 29 seconds (healthy) +filebrowser gtstef/filebrowser:1.5.6-stable Up 27 hours (healthy) +immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 2 minutes (healthy) +immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 2 minutes (healthy) +immich-redis redis:7-alpine Up 2 minutes (healthy) +immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 2 minutes (healthy) +paperless-postgres postgres:18-alpine Up 2 minutes (healthy) +paperless-redis redis:7-alpine Up 2 minutes (healthy) +paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up About a minute (healthy) +traefik traefik:v3.7.13 Up 27 hours +-- named volumes +felhom-controller-data +filebrowser_filebrowser_data +immich_immich_ml_cache +immich_immich_postgres_data +immich_immich_redis_data +paperless-ngx_paperless_data +paperless-ngx_paperless_postgres_data +paperless-ngx_paperless_redis_data +privatebin_privatebin_data +-- anonymous: 298 +/mnt/felhom-drives/scratch_hdd/backups/primary: + +/mnt/sizing-bench/backups/secondary: +paperless-ngx +audiobooks +books +comics +grimmory +metube +movies +music +nzbget +photos +podcasts +torrents +transmission +tv +usenet +holds: ['chaoscrash', 'chaosoom', 'chaosoomb', 'gokapi'] kept: 0 +docker-compose.yml diff --git a/documentation/audits/restore-holes-2026-10-24/P2-9202/08-anon.txt b/documentation/audits/restore-holes-2026-10-24/P2-9202/08-anon.txt new file mode 100644 index 00000000..977e1f09 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P2-9202/08-anon.txt @@ -0,0 +1,8 @@ +0aa8e2d4562f2a2de55adb01e10c04f28b4309ea4944c3ac28bac4c17c41edbf 2026-10-24T18:12:16Z 4.0K +4af480448ba89ec358af3f5f9ba57b97013b4b8167b831a19174f4a80ff1f6c9 2026-10-24T18:13:55Z 4.0K +2447807a3f1f9a61b8f0dfb8e0c5d9a2fe990d404aeafa39c00c2112dd7ff215 2026-10-24T18:10:59Z 4.0K +removed 0aa8e2d4562f2a2de55adb01e10c04f28b4309ea4944c3ac28bac4c17c41edbf +removed 4af480448ba89ec358af3f5f9ba57b97013b4b8167b831a19174f4a80ff1f6c9 +removed 2447807a3f1f9a61b8f0dfb8e0c5d9a2fe990d404aeafa39c00c2112dd7ff215 +anonymous now: 295 +1 diff --git a/documentation/audits/restore-holes-2026-10-24/P3-9202/00-as-found.txt b/documentation/audits/restore-holes-2026-10-24/P3-9202/00-as-found.txt new file mode 100644 index 00000000..6312b828 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P3-9202/00-as-found.txt @@ -0,0 +1,25 @@ +# 9202 as found (R-1253 run) — 2026-10-24T18:49:05Z +gitea.dooplex.hu/admin/felhom-controller:0.342.0 +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 +filebrowser gtstef/filebrowser:1.5.6-stable +immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 +immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 +immich-redis redis:7-alpine +immich-server ghcr.io/immich-app/immich-server:v3.2.4 +paperless-postgres postgres:18-alpine +paperless-redis redis:7-alpine +paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +traefik traefik:v3.7.13 +felhom-controller-data +filebrowser_filebrowser_data +immich_immich_ml_cache +immich_immich_postgres_data +immich_immich_redis_data +paperless-ngx_paperless_data +paperless-ngx_paperless_postgres_data +paperless-ngx_paperless_redis_data +privatebin_privatebin_data +anonymous: 295 +/mnt/felhom-drives/scratch_hdd/backups/primary: + +/mnt/felhom-drives/scratch_hdd/exports: diff --git a/documentation/audits/restore-holes-2026-10-24/P3-9202/01-setup.txt b/documentation/audits/restore-holes-2026-10-24/P3-9202/01-setup.txt new file mode 100644 index 00000000..5f052a27 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P3-9202/01-setup.txt @@ -0,0 +1,16 @@ +gitea.dooplex.hu/admin/felhom-controller:0.343.0-dev.r1253a Up 25 seconds (healthy) +{"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"} + +202 +{"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"} + +202 +uptime-kuma Up 6 seconds (healthy) +metube Up 35 seconds (healthy) +drwx-----x 3 root root 4096 Oct 24 18:50 .. +-rw-r--r-- 1 root root 1610612736 Oct 24 18:51 r1253-ballast.bin +{"ok":true,"message":"Mentés elindítva"} + +200 +2026/10/24 18:52:34 recovery_unit.go:289: [INFO] [backup] Recovery unit captured for metube → /mnt/felhom-drives/scratch_hdd/backups/primary/metube (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/24 18:52:34 recovery_unit.go:289: [INFO] [backup] Recovery unit captured for uptime-kuma → /mnt/sys_drive/felhom-data/backups/primary/uptime-kuma (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) diff --git a/documentation/audits/restore-holes-2026-10-24/P3-9202/02-race.txt b/documentation/audits/restore-holes-2026-10-24/P3-9202/02-race.txt new file mode 100644 index 00000000..359f1d93 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P3-9202/02-race.txt @@ -0,0 +1,26 @@ +302 +-- marker during the restore: +{ + "active": true, + "op_id": "unit-restore:metube", + "reason": "unit_restore", + "stacks": [ + "metube" + ], + "started_at": "2026-10-24T18:52:57.639136768Z" +}-- export stop-first of uptime-kuma during the restore: +{"error":"Egy másik művelet (például egy visszaállítás) éppen leállítva tart egy alkalmazást, ezért most nem indítható olyan exportálás, amely leállítja az alkalmazást. Próbáld újra, amikor az a művelet befejeződött.","ok":false} + +409 +-- marker after: +cat: /var/lib/docker/volumes/felhom-controller-data/_data/data/appstop-state.json: No such file or directory +metube Up 24 seconds (healthy) +uptime-kuma Up About a minute (healthy) +2026/10/24 18:52:57 handlers.go:1888: [WARN] [web] Restore requested (async): stack=metube, snapshot=helyi from 172.18.0.4:37066 +2026/10/24 18:53:08 handler_export.go:161: [DEBUG] [web] apiExportStart: stack="uptime-kuma" drive="/mnt/felhom-drives/scratch_hdd" encrypted=false stopApp=true +2026/10/24 18:53:08 export.go:192: [WARN] Export of uptime-kuma refused: unit-restore:metube holds the app-stop marker — an export that stops the app would replace its crash record (R-1253) +2026/10/24 18:53:08 handler_export.go:201: [ERROR] [web] Export start failed for uptime-kuma: Egy másik művelet (például egy visszaállítás) éppen leállítva tart egy alkalmazást, ezért most nem indítható olyan exportálás, amely leállítja az al +2026/10/24 18:53:08 handler_export.go:202: [DEBUG] [web] apiExportStart error: Egy másik művelet (például egy visszaállítás) éppen leállítva tart egy alkalmazást, ezért most nem indítható olyan exportálás, amely leállítja az alkalmazást. Pr� +2026/10/24 18:53:12 restore_crash_putback.go:89: [INFO] [backup] metube: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195) +2026/10/24 18:53:32 restore_unit.go:571: [INFO] [backup] Restore-from-unit completed: metube — 1 volume(s) of 1 listed, 0 database(s) of 0 listed +2026/10/24 18:53:32 handlers.go:1900: [INFO] [web] Restore completed (async): stack=metube in 35.522574346s (volumes 1/1, dbs 0/0) diff --git a/documentation/audits/restore-holes-2026-10-24/P3-9202/03-control-and-teardown.txt b/documentation/audits/restore-holes-2026-10-24/P3-9202/03-control-and-teardown.txt new file mode 100644 index 00000000..fe98844a --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P3-9202/03-control-and-teardown.txt @@ -0,0 +1,38 @@ +-- control: the same export after the restore +{"ok":true} + +200 +2026/10/24 18:53:08 export.go:192: [WARN] Export of uptime-kuma refused: unit-restore:metube holds the app-stop marker — an export that stops the app would replace its crash record (R-1253) +2026/10/24 18:53:08 handler_export.go:201: [ERROR] [web] Export start failed for uptime-kuma: Egy másik művelet (például egy visszaállítás) éppen leállítva tart egy alkalmazást, ezért most +2026/10/24 18:53:58 handler_export.go:207: [INFO] [web] Export started for uptime-kuma to /mnt/felhom-drives/scratch_hdd +2026/10/24 18:54:01 export.go:282: [INFO] Export: stopping uptime-kuma +2026/10/24 18:54:06 export.go:516: [INFO] Export completed: uptime-kuma → /mnt/felhom-drives/scratch_hdd/exports/uptime-kuma_20261024-185406.fab (22.0 KB) in 8.35546611s +2026/10/24 18:54:06 export.go:295: [INFO] Export: restarting uptime-kuma +/mnt/felhom-drives/scratch_hdd/exports/uptime-kuma_20261024-185406.fab +200 +200 +{"ok":true,"data":{"removed":"metube","volumes_removed":["metube_metube_state"],"hdd_paths_removed":[],"hdd_paths_preserved":[],"userdata_kept":["/mnt/felhom-drives/scratch_hdd/userdata/media/metube (4.0K)"],"backup_paths_removed":["/mnt/felhom-drives/scratch_hdd/backups/primary/metube (1.6G)"],"verified":true},"message":"Stack metube removed"} + +200 +{"ok":true,"data":{"removed":"uptime-kuma","volumes_removed":["uptime-kuma_uptime_kuma_data"],"hdd_paths_removed":[],"hdd_paths_preserved":[],"hdd_note":"Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott nem volt mit törölni.","userdata_kept":[],"backup_paths_removed":["/mnt/sys_drive/felhom-data/backups/primary/uptime-kuma (328K)"],"verified":true},"message":"Stack uptime-kuma removed"} + +200 +drwxrwsr-x root:UNKNOWN movies +drwxrwsr-x root:UNKNOWN music +drwxrwsr-x root:UNKNOWN nzbget +drwxrwsr-x UNKNOWN:UNKNOWN transmission +drwxrwsr-x root:UNKNOWN metube +audiobooks +books +comics +grimmory +metube +movies +music +nzbget +photos +podcasts +torrents +transmission +tv +usenet diff --git a/documentation/audits/restore-holes-2026-10-24/P3-9202/04-at-end.txt b/documentation/audits/restore-holes-2026-10-24/P3-9202/04-at-end.txt new file mode 100644 index 00000000..c0342b82 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/P3-9202/04-at-end.txt @@ -0,0 +1,25 @@ +# 9202 at end (R-1253 run) — 2026-10-24T18:56:32Z +gitea.dooplex.hu/admin/felhom-controller:0.342.0 +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 +filebrowser gtstef/filebrowser:1.5.6-stable +immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 +immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 +immich-redis redis:7-alpine +immich-server ghcr.io/immich-app/immich-server:v3.2.4 +paperless-postgres postgres:18-alpine +paperless-redis redis:7-alpine +paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +traefik traefik:v3.7.13 +felhom-controller-data +filebrowser_filebrowser_data +immich_immich_ml_cache +immich_immich_postgres_data +immich_immich_redis_data +paperless-ngx_paperless_data +paperless-ngx_paperless_postgres_data +paperless-ngx_paperless_redis_data +privatebin_privatebin_data +anonymous: 296 +/mnt/felhom-drives/scratch_hdd/backups/primary: + +/mnt/felhom-drives/scratch_hdd/exports: diff --git a/documentation/audits/restore-holes-2026-10-24/README.md b/documentation/audits/restore-holes-2026-10-24/README.md new file mode 100644 index 00000000..c9c674a9 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/README.md @@ -0,0 +1,65 @@ +# Restore and backup holes — fixed on main, proven on test boxes (2026-10-24 night, unattended) + +Brief: „fix the restore and backup holes the review found, keep BYO hosts untouched, and pre-answer the review's +notes". **Operator ruling recorded:** D43 = A (by starting the run) — `CONTEXT.md`, `03`, `11` §1. + +**Baselines (read at start, 16:47 UTC):** felhom.eu `80f1ab31`, felhom-controller `2449223` (v0.342.0 released), felhom-agent +`f5c3d77` (v0.164.0), app-catalog `227a544`. Register 207 rows. Architecture read first: `07` §6.3 / §8 / §8.2a, `03`, +`05`, `11` §1. + +## What was fixed (commit only — nothing released, deployed, vouched or published) + +| Row | Fix | Commit | CI | +|---|---|---|---| +| R-1251 (P1) | a pending put-back blocks every start: a copy on an unbound drive WAITS (restore hold `putback_pending`); the put-back runs first when the drive binds; a failed put-back holds (H2); Stop during a restore still gets the put-back (H7) | controller `bb2809d` | 2301 / #482 success | +| R-1251 (found by the live proof) | the copy's folders are synced before the marker names the copy; the household is not told to run the restore again while the put-back waits | controller `062f9b4` | 2302 / #483 success | +| R-1252 (P2) | the second-drive reconcile runs only under a valid classification; a rejected block is said on the status and once a day to the operator; kept copies stay kept (H2); Remove drops kept records it deleted (H9) | controller `518a135` | 2303 / #484 success | +| R-1253 (P2) | a stop-first export refuses while another operation holds the app-stop marker | controller `c9b4b82` | 2304 / #485 success | +| R-1254 (P2, hub half) | key, hash and identity blob in ONE transaction; a bad identity refused before any write | felhom.eu `dcf4f84c` | 2305 / #1256 success | +| R-1257 (P2, D43 = A) | the sshd drop-in only on an appliance; BYO: nothing written, our copy removed; X3 LAN addresses; X4 only the SSH step undone | felhom-agent `c8d3c17` | (final report) | + +Every rule was red-proved: the rule taken out, the test seen failing, the rule put back (the commands are in the session +log; each test file names its red-proof). Images: controller dev images `0.343.0-dev.r1251a/b`, `r1252a`, `r1253a` +were built locally and loaded straight into the test guests (never pushed); hub `0.170.0-rc.r1254` (`67db584a45d7`) +built locally from a clean clone at `dcf4f84c`, NOT pushed, NOT deployed. + +## The proofs — before (today's code) and after + +| Proof | Before | After | Evidence | +|---|---|---|---| +| R-1251, Tester 1 (VM 341) | a cut restore + a late drive: the app ran on the cut data, a later start poured the old copy over new writes (review proof H1, lean clone) | run 2: held while the drive was away (Start/Update 409 with the sentence); the disk back → the put-back BEFORE the start; „gen2" (the newest) and the household's download there; a later restart poured nothing | `P1-tester1/20…30` | +| R-1251 run 1 (a finding) | — | the cut left the journal as `undo.json.tmp`: the H2 hold kept the app stopped for support (never started on the cut data) → the folder-sync fix → run 2 | `P1-tester1/05…13` | +| R-1252, 9202 | one bogus path → „removed stale dest dir userdata/media" (review, 0.342.0, same box, same day) | two control runs kept the copy; the bogus path → „REJECTED … every existing copy … KEPT", the copy there, the status sentence set, the digest sent | `P2-9202/` | +| R-1253, 9202 | a stop-first export during a restore replaced the restore's marker (static, review HUNT-1 H3) | the export refused (409) during a MeTube restore, uptime-kuma never stopped, the restore recorded its copy and finished; the same export after it ran (control) | `P3-9202/` | +| R-1254 | — (static) | store and endpoint tests only (the hub is not deployed) | the test files | +| R-1257 | — (static) | the bundle harness (7 tests) | the test file | + +How the late drive was made on Tester 1: the restore was cut by `qm stop 341` 2.1 s after the controller logged „the +restore copy is recorded" (`tools/cut_on_line.sh`); the data disk (`nvme-scratch:341/vm-341-disk-2.raw`, not the thin +pool) was detached with `qm set 341 --delete scsi1`, the VM started, and the disk re-attached with `qm set 341 --scsi1 …`; +the agent re-mounted it by itself („watchdog self-heal") and the boot recreate ran the put-back. + +## Teardown (three layers) + +- **Tester 1:** MeTube removed through the product (Stop, then Remove with data and backups); my household file and the + `media/metube` folder I made removed; my run-1 copy folder removed; the controller back on 0.342.0 and the two dev + images removed; the data disk back on `scsi1`; no hold, no marker. One difference: privatebin's recreates left 3 empty + anonymous volumes I cannot tell apart safely (196 → 199 after removing the 3 dangling ones my backup run made) — + R-1267. (`P1-tester1/00-as-found.txt`, `41-at-end.txt`, `42-anon-volumes.txt`.) +- **9202:** the throwaway apps (audiobookshelf; MeTube and uptime-kuma) removed through the product, my marker and my + `.fab` removed, the controller back on 0.342.0, the dev images removed; the dangling anonymous volumes my runs made + removed by name. One difference: Immich's running container uses a new empty anonymous volume (R-1267). +- **Host (demo-hp):** nothing provisioned; VM 341 config as found (`scsi1` back). The catalog sweep's VM 9360 not touched. +- **Hub:** nothing changed. The boxes pushed their normal events (Tester 1: the drive away and back, the interrupted + restore). + +## Observations + +- `FILED: R-1268` — Remove with „remove backups" leaves a restore's undo copy (`backups/restore-undo//`) on the drive. +- `FILED: R-1267` (extended) — privatebin's image volumes leak like Immich's. +- `NOT-A-FINDING:` a support hold (`restore_mixed`) stays after the app is removed — by design (R-379: only a person + clears it, `-clear-restore-hold`). +- `NOT-A-FINDING:` the boot sweep logged MeTube as held „by an app-data operation" while the marker waited — true (the + kept marker), and the hold refused it as well. +- `NOT-A-FINDING:` a restore or the night dump replacing an EXPORT's marker — the replaced record only owed the exported + app a restart, which the boot sweep gives every app the household wants running. diff --git a/documentation/audits/restore-holes-2026-10-24/tools/chain9202.sh b/documentation/audits/restore-holes-2026-10-24/tools/chain9202.sh new file mode 100755 index 00000000..bb0fb0fb --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/tools/chain9202.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +# Run 9202's night chain through the product's endpoint and print its Tier 2 lines + the copy of the marker (R-1252). +cd "$(dirname "$0")/.." +since=$(python3 tools/t1.py 9202 guest 'date -u +%Y-%m-%dT%H:%M:%SZ' | tr -d '\n') +python3 tools/t1.py 9202 api POST /api/debug/backup/night-chain | head -c 200; echo +python3 tools/t1.py 9202 guest "for i in \$(seq 1 120); do docker logs --since $since felhom-controller 2>&1 | grep -q 'night-chain\] finished' && break; sleep 10; done +docker logs --since $since felhom-controller 2>&1 | grep -E 'night-chain\]|Tier 2|backup block|reconcile|R-1252' | grep -v DEBUG | grep -v 'block rejected in' | cut -c1-300 +echo '-- rejected-block lines:' \$(docker logs --since $since felhom-controller 2>&1 | grep -c 'block rejected in') +echo '-- the copy of the marker:'; ls -la /mnt/sizing-bench/backups/secondary/audiobookshelf/userdata/media/audiobooks/ 2>&1 | head -5" diff --git a/documentation/audits/restore-holes-2026-10-24/tools/cut_on_line.sh b/documentation/audits/restore-holes-2026-10-24/tools/cut_on_line.sh new file mode 100755 index 00000000..ebde5943 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/tools/cut_on_line.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# Power-cut VM 341 (Tester 1) the moment the controller logs that the restore copy is recorded (R-1251 live proof). +# Streams the controller log to $1 (the evidence file) — the container is recreated at boot, so this stream IS the log. +out="$1"; pat='restore copy is recorded' +ssh -o BatchMode=yes root@192.168.0.154 'pct exec 9201 -- docker logs -f --since 1s felhom-controller 2>&1' | while IFS= read -r line; do + printf '%s\n' "$line" >> "$out" + case "$line" in *"$pat"*) + t0=$(date -u +%FT%T.%3NZ) + ssh -o BatchMode=yes demo-hp 'qm stop 341 --skiplock 1' >> "$out.cut" 2>&1 + echo "CUT: line seen $t0, qm stop returned $(date -u +%FT%T.%3NZ)" >> "$out.cut" + exit 0;; + esac +done diff --git a/documentation/audits/restore-holes-2026-10-24/tools/t1.py b/documentation/audits/restore-holes-2026-10-24/tools/t1.py new file mode 100644 index 00000000..691725d5 --- /dev/null +++ b/documentation/audits/restore-holes-2026-10-24/tools/t1.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python3 +"""Drive one box's controller the way the dashboard does (restore-holes night, 2026-10-24). + +Usage (the password comes ONLY from the environment, through `sec run`, and is never printed): + sec run TESTER1_PASSWORD -- python3 t1.py tester-1 login + python3 t1.py tester-1 api GET /api/stacks + python3 t1.py tester-1 form POST /backups/restore stack_name=metube snapshot_id=unit + python3 t1.py tester-1 guest '' # inside the customer guest + python3 t1.py tester-1 host '' # on the box's Proxmox host +Session cookie + CSRF token live in the session scratchpad (0600), never in the repo. +""" +import os, re, subprocess, sys, secrets as _s + +SC = os.environ.get("T1_SCRATCH", "/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/63b6c84e-0c09-4657-a160-855d2e4fe326/scratchpad/t1") +os.makedirs(SC, mode=0o700, exist_ok=True) +BOXES = { + "tester-1": {"ssh": ["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=20", "root@192.168.0.154"], "guest": "9201", + "domain": "enkicsifelhom.hu", "pwenv": "TESTER1_PASSWORD"}, + "9202": {"ssh": ["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=20", "demo-hp"], "guest": "9202", + "domain": "enkisfelhom.hu", "pwenv": "REHEARSAL_2026_10_20_9202_PW"}, +} +box = BOXES[sys.argv[1]] +cmd = sys.argv[2] +HOST = f"Host: felhom.{box['domain']}" +pre = f"{SC}/{sys.argv[1]}" + + +def run(args, inp=None, timeout=900): + return subprocess.run(args, input=inp, capture_output=True, text=True, errors="replace", timeout=timeout) + + +def host(script, timeout=900): + return run(box["ssh"] + ["bash -s"], inp=script, timeout=timeout) + + +def guest(script, timeout=900): + t = f"/tmp/rh-{os.getpid()}-{_s.token_hex(4)}.sh" + g = box["guest"] + return run(box["ssh"] + [f"cat > {t}; pct push {g} {t} {t}; pct exec {g} -- bash {t}; pct exec {g} -- rm -f {t}; rm -f {t}"], + inp=script, timeout=timeout) + + +def base(): + r = guest("ip -4 -o addr show eth0") + m = re.search(r"inet (\d+\.\d+\.\d+\.\d+)/", r.stdout) + if not m: + sys.exit("no guest address: " + (r.stderr or r.stdout)[-200:]) + return "https://" + m.group(1) + + +def login(): + pw = os.environ.get(box["pwenv"]) + if not pw: + sys.exit(f"{box['pwenv']} is not in the environment — run through `sec run {box['pwenv']} --`") + b = base() + hdr = f"{pre}.hdr" + run(["curl", "-sk", "-D", hdr, "-o", "/dev/null", "-H", HOST, "-X", "POST", "--data-urlencode", f"password={pw}", f"{b}/login"]) + h = open(hdr).read() + os.remove(hdr) + m = re.search(r"felhom_session=[A-Za-z0-9._-]+", h, re.I) + if not m: + sys.exit("login failed: no session cookie (" + h.splitlines()[0] + ")") + r = run(["curl", "-sk", "-L", "-H", HOST, "-H", f"Cookie: {m.group(0)}", f"{b}/"]) + c = re.search(r' 5 else None) +elif cmd == "form": + call(sys.argv[3], sys.argv[4], form=sys.argv[5:]) +elif cmd in ("guest", "host"): + r = (guest if cmd == "guest" else host)(sys.argv[3]) + sys.stdout.write(r.stdout) + sys.stderr.write(r.stderr[-2000:]) + sys.exit(r.returncode) diff --git a/documentation/audits/review-round-2-2026-10-24/D44-D45-picks.md b/documentation/audits/review-round-2-2026-10-24/D44-D45-picks.md new file mode 100644 index 00000000..36d0e696 --- /dev/null +++ b/documentation/audits/review-round-2-2026-10-24/D44-D45-picks.md @@ -0,0 +1,59 @@ +# D44 and D45 — one line per note, with CC's pick (prepared 2026-10-24 night) + +**Answer „all picks" or name the ones you disagree with.** Rows: D44 = R-1265 (decisions the code takes that no +document named), D45 = R-1266 (rulings or promises the code does not do as written). Count: **25 + 7 = 32** (the review +said „about 30" for D44; 25 is the number of notes actually written). Each line: what the code does · the risk if it is +wrong · **the pick** · the cost. „Confirm" = the note loses its „undocumented" tag and stays as written. + +## D44 — safe to confirm (21) + +| # | Where | What the code does | Risk if wrong | Pick | Cost | +|---|---|---|---|---|---| +| 1 | `03` local API | 41 routes; 24 are not listed in `03` (one starts a host action: the 55 s line test) | a review that reads `03` misses what a guest can ask the host | **change the document** (list every route) | docs only | +| 2 | `07` §8.2c | a second-drive copy stops at 60 min; a stray copy blocks restores; one copy per destination at a time | a first copy of a very large library takes several nights | **confirm** | none (rsync resumes each night) | +| 3 | `07` §8.2c | „Eltávolítás" of a kept copy is refused while a backup, restore or copy runs | none — a refusal, said on the page | **confirm** | none | +| 4 | `07` §8.2c | a start that fails AFTER a good put-back leaves the app stopped, not held | none — the data is as before; Start is safe | **confirm** | none | +| 5 | `07` §8.2c | the hub's off-site delivery check skips a box that waits for the recovery code | a box stuck there is not alarmed as „stuck" | **confirm** | none (the box says it waits) | +| 6 | `07` §8.2c | the nightly release copy to ep0 (D33) has no row in the recovery matrix | a reader of §8 does not find it | **change the document** (add the row) | docs only | +| 7 | `09` §6.4.x | `build.sh --push` also refuses a dirty tree, a HEAD not at `origin/main`, a non-git directory | none | **confirm** | none | +| 8 | `09` §6.4.x | the night leg retries a „busy" refusal once in the same night | none | **confirm** | none | +| 9 | `09` §6.4.x | the night leg skips an app the household stopped (an update would start it) | none — R-721's own rule | **confirm** | none | +| 10 | `09` §6.4.x | a manual chain run gives the leg its normal length as the deadline, not W+5 h | none | **confirm** | none | +| 11 | `09` §6.4.x | `already_current` and `engine_no_test` are plain skips | none | **confirm** | none | +| 12 | `09` §6.4.x | an unreadable catalog template leaves the app's compose alone that cycle | none — the safe side | **confirm** | none | +| 13 | `09` §6.4.x | the R-971 image allow-list runs before ANY compose write | none — the safe side | **confirm** | none | +| 14 | `14` | a re-enroll during the power-off hold counts as „gone" (a reinstall) | none known | **confirm** | none | +| 15 | `14` | setting and clearing the off time each make an info event, not mailed | none | **confirm** | none | +| 16 | `14` | the confirmation word is accepted as „KIKAPCSOLÁS", „KIKAPCSOLAS" and „TURN OFF" | none | **confirm** | none | +| 17 | `15` | the Sonarr/Radarr API key is read from `/config/config.xml` | an upstream format change breaks the wiring (loud, not silent) | **confirm** | none | +| 18 | `16` | `systemd-detect-virt` is read once per agent process; empty → „unknown" | none — fail-closed | **confirm** | none | +| 19 | `16` | a path the controller cannot read counts as the SAME disk (no second copy claimed) | none — fail-closed | **confirm** | none | +| 20 | `16` | the internal place refuses eject, decommission and wipe; moving its data is allowed | none | **confirm** | none | +| 21 | `16` | the internal place's size subtracts every thin volume already promised | none — the safe side | **confirm** | none | + +## D44 — the few that matter (4) + +| # | Where | What the code does | Risk if wrong | Pick | Cost | +|---|---|---|---|---|---| +| 22 | `04` §2 | a signed operation's window has no maximum length (`felhom-opsign` defaults to 30 min) | a signed job made with a long window by mistake stays usable for that whole time | **change the code**: the agent and the root wrapper refuse a window longer than 24 h | one small agent change + test; rides an agent release | +| 23 | `13` (file manager sign-in) | FileBrowser trusts its proxy header; the only guard is that the header NAME is a per-box secret, and every app container can reach `filebrowser:80` | an app that learns the name (a hacked app on the box) reads and writes every household file | **change the code**: FileBrowser joins only the network traefik uses, so no app container reaches it directly | a compose/network change in the controller + a live check; one release | +| 24 | `15` | the „Később" 03:30 media restart is an in-memory timer — a controller restart forgets it (only the notice stays) | media apps do not see the new drives until the household presses „now" | **change the code**: store the due time and run it after a restart | small controller change | +| 25 | catalog `c50adfc` (R-1019) | MeTube, NZBGet and SABnzbd downloads are excluded from every backup, like qBittorrent and Transmission | MeTube downloads may be gone from their source site — on a drive failure they cannot be fetched again | **change the catalog for MeTube only**: `optional` (the second-drive copy carries it; off-site does not); NZBGet/SABnzbd stay excluded (re-downloadable) | second-drive space for MeTube's folder | + +## D45 — the text or the code (7) + +The default pick is **„the text follows the code"** wherever the code is the safer or the decided behaviour; CC +then rewrites each passage and you read the diff. + +| # | Where | The document says | The code does | Risk | Pick | Cost | +|---|---|---|---|---|---|---| +| 1 | `09` decision 13 | the tag rule moves to the catalog gate | it stays in the controller (`updateorder.go:134`) | none — the controller check is the safe one | **text follows code** | docs only | +| 2 | `09` decision 14 | a box „never jumps" versions | a PERSON's press on an app older than the ladder takes the catalog's current definition; the night leg never jumps | a big jump can fail — the guarded update undoes it | **text follows code** (say: the night never jumps; a person's press may) | docs only | +| 3 | `09` decision 250 item 5 | the golden gate judges by AGE while the floor is above it | the gate cannot read the floor; it uses „release ahead" | none — the gate is stricter | **text follows code** | docs only | +| 4 | `01` §1/§4 | „the hub never connects into a box", „no inbound control endpoint", „no standing operator backdoor" | the hub sends a UDP poke (no command) through ep0; the operator's WireGuard peer reaches the operator door (felhom-sshd) | a household reads a promise that is not exact | **text follows code**, with the exact words: a poke carries no command, the box still pulls; the operator door is the documented break-glass | docs + the website check (R-985 style) | +| 5 | `03` §5, `06` §6 | break-glass „via the Cloudflare Tunnel behind Access"; „no inbound listener" | old text — the break-glass is the operator door over WireGuard | none in code; the text misleads | **text follows code** | docs only | +| 6 | `01` §8 | the operator „cannot open / read" the off-site backup | the operator holds the retained escrow key and can open an old store (R-985 already softened the website) | a promise to the household that is not exact | **text follows code**, in the words the website already uses | docs only | +| 7 | `06` §8 | the agent will manage cloudflared | `03` §2: it will not | none | **text follows code** (`03` wins) | docs only | + +**What happens if you do nothing:** the 25 notes stay tagged „undocumented" and the 7 passages keep saying something +the code does not do. Nothing in a box changes either way until you answer. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index d8b00692..538b1e17 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -142,7 +142,8 @@ stopping line that lies. | **R-1043** | Apps & catalog | P4 | **Sonarr, Radarr and Prowlarr build their login redirect as an absolute `http://` URL.** They trust forwarded headers only from their „Trusted networks” (the household set `192.168.0.0/24`; traefik is `172.18.x`), so they ignore `X-Forwarded-Proto: https` and answer `Location: http:///login` to an https visitor. Since controller v0.312.0 that hop is sent back to https by traefik (a GET, no password), so it is cosmetic; a template could seed traefik's network as trusted. (`audits/https-only-2026-10-14/``1A-3-4.txt`) | **OPEN** — filed 2026-10-14 | — | Seed the docker network as a trusted proxy in the *arr templates (with the R-1030 seed), measure the redirect | CC | | **R-1099** | Apps & catalog | P4 | **The N100 may hold RomM games in the old upload place, now shown under a fake „roms” platform.** Until 2026-10-16 RomM's own upload wrote to the drive's `roms/roms//`; after the mapping fix (catalog `fb18471`, R-1089) those files are at RomM's `library/roms/roms/…`. Measured on 9201: such a game showed as missing in its platform and again under a platform „roms” (`audits/romm-ghost-2026-10-16/B/b-after.txt`). Files put in with the file manager are not affected — they are now found. The N100 runs RomM (hub `/apps/romm`); CC does not log in to the lived-in box. | OPEN — filed 2026-10-16 | Operator: look at `roms/roms/` on the N100's drive in the file manager | If it holds platform folders: move them up one level into `roms/`, then run „Scan” in RomM; if it is empty, close | Operator | | **R-1215** | Apps & catalog | P4 | **One Immich photo upload hung 10 minutes with no answer and was not stored, while the box was healthy.** Night soak 2026-10-21 (`audits/night-2026-10-21/P1-soak/08-load/proof.log`): cycle 36, `IMG_20120021.jpg` sent 19:38:30 UTC, the uploader gave up at its 600 s read limit; Immich's database holds 302 photos for that user, the other cycles 303 (`P1-soak/09-immich-cycle36.txt`). At that moment memory „full" pressure was 3.9 % and `/api/health` answered in 0.26 s. 1 of about 26 000 uploads that night. Not diagnosed: Immich restarted at 02:16 for the off-site dump, so its log of 19:38 is gone. The Immich phone app retries a failed upload. | WATCHING — **2026-10-22 night:** not reproducible tonight; the Immich server log of 19:38 is gone (Immich restarted at 02:16 for the off-site dump). What the household sees: the upload fails after 10 minutes and the Immich phone app sends it again. | — | Watch the next long upload run with the Immich server log saved; file upstream only with a log | CC | -| **R-1267** | Apps & catalog | P4 | **Every recreate of Immich leaves one empty anonymous Docker volume behind: the image declares `VOLUME /data`, the template does not map it.** Found 2026-10-24 (review round 2, while putting 9202 back): 9202 holds 296 empty anonymous volumes; each night chain's recreate of `immich-server` made a new one (12:58, 13:00, 13:01 — the three test runs; the two dangling ones this session made were removed by name, nothing pruned). The photos are NOT affected: Immich picks `/usr/src/app/upload` whenever both folders exist (`detectMediaLocation` in the image's `server/dist/services/storage.service.js:28-39`), and the template binds that folder (`templates/immich/docker-compose.yml:49`) — NOT-A-FINDING for data placement. | OPEN | — | map `/data` explicitly (or set `IMMICH_MEDIA_LOCATION=/usr/src/app/upload`) through a tested catalog step; clean the old volumes through the product on each box | CC | +| **R-1267** | Apps & catalog | P4 | **Every recreate of Immich leaves one empty anonymous Docker volume behind: the image declares `VOLUME /data`, the template does not map it.** Found 2026-10-24 (review round 2, while putting 9202 back): 9202 holds 296 empty anonymous volumes; each night chain's recreate of `immich-server` made a new one (12:58, 13:00, 13:01 — the three test runs; the two dangling ones this session made were removed by name, nothing pruned). The photos are NOT affected: Immich picks `/usr/src/app/upload` whenever both folders exist (`detectMediaLocation` in the image's `server/dist/services/storage.service.js:28-39`), and the template binds that folder (`templates/immich/docker-compose.yml:49`) — NOT-A-FINDING for data placement. **Seen again 2026-10-24 night on Tester 1:** privatebin's image declares `/run`, `/tmp` and `/var/lib/nginx/tmp` as volumes — each recreate (the night dump's stop/start) leaves three empty anonymous volumes; the same class as Immich's `/data` (`audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt`). | OPEN | — | declare the image's VOLUME paths as tmpfs or named volumes in the catalog (Immich, privatebin), or remove the dangling ones a recreate leaves, by name | CC | +| **R-1268** | Backup & restore | P3 | **Remove with „remove backups” leaves a restore's undo copy on the drive (`backups/restore-undo//`).** Seen 2026-10-24 night on Tester 1 (restore-holes Part 1, run 1): MeTube was held for support after a failed crash put-back, then removed with `remove_backups=true` — the 1.5 GB copy stayed (the removal deletes `backups/primary` and the second-drive mirror, not `restore-undo`), and nothing lists it (`audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt`; removed by hand, CC's own test data). The copy is kept on purpose while a hold exists; after the app is gone it is orphaned. | OPEN | — | Remove with `remove_backups` also deletes the app's `restore-undo` folder (under the same path guard), or lists it as a kept copy | CC | ## App updates — 5 rows (P2 1, P3 4) @@ -159,13 +160,13 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-1251** | Backup & restore | P1 | **A power cut during a restore of a DRIVE-backed app: the app then runs on the cut restore's data, and a later controller restart pours the pre-restore copy back over everything written since — the D22 promise is false for every drive-backed app.** Found 2026-10-24 (review round 2, HUNT-1 H1). Read at `file:line` (controller `321df1f`): the undo copy lives on the app's drive (`internal/backup/offbox_restore_undo.go:47-48`); `AppStopGuard.Recover` runs at start BEFORE the drive is bound, finds the copy „gone” and restarts „as before” (`appstop_marker.go:362-367`); the gated start refuses (drive absent) → marker and Undo KEPT (`:383-387,408-411`); the drive binds and the boot recreate starts the app checking only „held” (`internal/web/intermediary.go:470-476`); the household writes; the next controller start (e.g. the nightly self-update) finds the copy present and pours it back. **Proven with the real `Recover` code** in a lean clone (`audits/review-round-2-2026-10-24/``proofs/H1-drive-cut-restore/`); NOT reproduced on 9202 (its drive is mounted at boot and its agent API is off, so the recreate never runs there). Same family: H2 — a FAILED crash put-back leaves the app stopped and unheld, so Start or the recreate runs it and a later restart pours an old copy back (`appstop_marker.go:368-371`; `api/router.go:690-709`); H7 — Stop pressed during a restore skips the put-back after a cut (`appstop_marker.go:356-361`). | **OPEN** | recovery code — not fixed in an unattended review | an attended fix: an active marker holding an Undo for an app blocks EVERY start path (Recover's, the boot recreate, Start, the update) until the put-back ran; a copy on an UNBOUND drive waits, it is not „gone”; the drill on Tester 1 (a real bind at boot) | CC | +| **R-1251** | Backup & restore | P1 | **A power cut during a restore of a DRIVE-backed app: the app then runs on the cut restore's data, and a later controller restart pours the pre-restore copy back over everything written since — the D22 promise is false for every drive-backed app.** Found 2026-10-24 (review round 2, HUNT-1 H1). Read at `file:line` (controller `321df1f`): the undo copy lives on the app's drive (`internal/backup/offbox_restore_undo.go:47-48`); `AppStopGuard.Recover` runs at start BEFORE the drive is bound, finds the copy „gone” and restarts „as before” (`appstop_marker.go:362-367`); the gated start refuses (drive absent) → marker and Undo KEPT (`:383-387,408-411`); the drive binds and the boot recreate starts the app checking only „held” (`internal/web/intermediary.go:470-476`); the household writes; the next controller start (e.g. the nightly self-update) finds the copy present and pours it back. **Proven with the real `Recover` code** in a lean clone (`audits/review-round-2-2026-10-24/``proofs/H1-drive-cut-restore/`); NOT reproduced on 9202 (its drive is mounted at boot and its agent API is off, so the recreate never runs there). Same family: H2 — a FAILED crash put-back leaves the app stopped and unheld, so Start or the recreate runs it and a later restart pours an old copy back (`appstop_marker.go:368-371`; `api/router.go:690-709`); H7 — Stop pressed during a restore skips the put-back after a cut (`appstop_marker.go:356-361`). **Fixed on main 2026-10-24 night** (controller `bb2809d` + `062f9b4`; restore-holes brief Part 1): a copy on an unbound drive WAITS (restore hold `putback_pending`), every start path honours the hold, the put-back runs first when the drive binds; H2 a failed put-back holds (D20); H7 Stop still gets the put-back; the copy's folders are synced before the marker names it (run 1 of the live proof found `undo.json.tmp` after the cut). **Proven live on Tester 1** (`audits/restore-holes-2026-10-24/P1-tester1/`). | **VERIFY** — fixed on main, waits for the release | the next controller release | release the controller (Monday list); re-read P1's two observables on the released image (a waiting hold at start, the put-back before the start) | CC | | **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | NARROWED — **2026-10-23 (operator yes D33, D34):** the hub DB now leaves every 6 hours (hub 0.167.0 + `scripts/hub-db-backup/`, alarm at 13 h; restored back byte-identical) and the released images and agent packages leave nightly (`scripts/release-copy/`, R-1227 closed). (c)–(g) unchanged. **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/` **LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest **2026-10-24 (attended): `DooplexGiteaOffsiteStale` fired from 00:57Z — not this path's fault.** The 00:20 push refused, as designed, a 12 h old Gitea dump: the PostgreSQL dump CronJob had failed every run since 2026-10-23 16:00Z (`28P01`) because the window reboot restarted the database and CNPG put the bootstrap secret's password in force (homelab `531023d`, operator's pick: the CronJob only). Fixed; dump, push and restore test run by hand (26 456 files, 12 repos `git fsck`, `gitea.dump` 1114 entries); cleared 10:51Z (`audits/dooplex-alarms-2026-10-24/`). Backup alarms now arrive as ONE mail every 4 h (homelab `22d7697`). **2026-10-23 night — survey again, read only (`audits/dooplex-survival-2026-10-22/PLAN.md`):** both nightly off-site jobs and both weekly restore tests green (copies 2026-10-22, tests 2026-10-18); the local tree is still the same box (2026-10-22 03:16 ok). New in the survey: the hub DB copy is up to 24 h old (G5: every 6 h — the operator's yes; a DooPlex change). | — | — | operator | | **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):** `_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the first verdict lands at 07:10 hub time (05:10 UTC), after the night session that reached it ended; that brief did not include this read. Nothing is lost: the hub keeps its listing; read the 2026-10-20 07:10 lines on 10-21 (or later on 10-20). **FIRST VERDICT READ 2026-10-21 (rehearsal session, read-only):** the 07:10 CEST run judged three tiers against yesterday — demo-felhom 5, demo-hp 14, tester-1 4 new snapshot files, „each inside a reported box run"; nothing vanished, nothing unexplained; Tester4 stored its first baseline (44 files); Tester-2's own listing failed (no `snapshots/` dir — the box has never pushed). No `offsite_snapshots_vanished` / `_unexplained` mail in the operator mailbox (second channel). Evidence `audits/due-checks-2026-10-21/R-895-hub-0710.txt`. The positive alarm path is still proven only by the unit red-proofs. | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC | | **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | NARROWED — **2026-10-23 (D32):** the sheet is current — names and places only, one A4, HU + EN (`runbooks/break-glass-sheet.md`); every place checked to exist. LEFT: the operator prints it, fills it by hand, keeps it outside the home. **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G1/G3/G10):** unchanged; the sheet is still the one act that breaks the circle, and S5's offline form is unknown to CC — on the sheet too (decide list in STATUS). | operator: print the sheet | Operator: print + store the sheet; then close | operator | -| **R-1252** | Backup & restore | P2 | **One rejected backup block in an app's `.felhom.yml` makes that night's second-drive copy DELETE the app's copies of the household's files.** REPRODUCED on 9202 2026-10-24 (`audits/review-round-2-2026-10-24/``proofs/H2-bad-backup-block/`): audiobookshelf with a marker file in `media/audiobooks`; two unchanged runs kept the copy (the positive control); one bogus path in the block → „backup block rejected … matches no compose bind” → the legacy legs (appdata only) → „Tier 2 reconcile: removed stale dest dir userdata/media”; the original on the drive untouched. Mechanism: `internal/stacks/metadata.go:550-557` (any defect rejects the WHOLE block) → `appbackup/classify.go:217-221` → `backup/tier2_capture.go:84-100` → `tier2.go:492` `tier2Reconcile` (`:325-329`), against D25's own header (`tier2_kept.go:17-18`). Same family (static): a D25 kept copy is reconciled away when its leg leaves the declared set and its record stays (HUNT-2 H2, `tier2.go:485-492`); Remove with `remove_backups` deletes kept copies and leaves their records (HUNT-2 H9, `api/router.go:1086-1087`). | OPEN | — | an attended fix in the backup code: a rejected or absent classification never reconciles (keep every dest dir that night, alarm instead); a kept copy stays kept; Remove drops the kept records it deletes; the red-proof is this run's 9202 steps | CC | -| **R-1253** | Backup & restore | P2 | **An export with „stop the app first” during another app's restore overwrites the ONE app-stop marker file — the restore's crash record and its Undo are lost; a power cut then leaves the restored app on mixed data with no put-back.** Found 2026-10-24 (HUNT-1 H3): `AppStopGuard.Begin` writes unconditionally (`internal/backup/appstop_marker.go:279-292`) and `End()` removes the file (`:306-320`); the exporter takes the guard (`internal/appexport/export.go:139-143,262-266`) and neither `StartExport` (`:173-178`) nor the web handlers (`internal/web/handler_export*.go`, grep: no restore check) look for a running restore. Static. | OPEN | — | refuse an export while a restore or an app-stop marker is active (or one marker per operation); a test with two Begins | CC | -| **R-1254** | Backup & restore | P2 | **The hub saves a household's new escrow package in TWO writes; a failure between them leaves the new repository hash beside the OLD identity package, and the box then marks itself escrowed — on a total loss the written code opens nothing.** Found 2026-10-24 (HUNT-3 H1), read at `file:line`: `hub/internal/api/handler.go:1415` `SaveHostEscrow` commits blob + hash, `:1445` `SaveHostDRBundle` sets `identity_blob` in a second statement (`internal/store/store.go:3788-3796`; the upsert `:3490-3511` never touches it); the controller's confirm reads the hash (`felhom-controller/controller/internal/report/escrow_confirm.go:195-217`); the agent sends once. Same family (H2): a lost or late answer (30 s) tells the household „the previous code still works” when the hub may already hold the new package (`felhom-agent/internal/localapi/escrow_ceremony.go:79,532-534,639-641`). Static. | OPEN | — | one transaction for both writes (a store test: the second write fails → nothing stored); the ceremony asks the hub what it holds before it says „the previous code still works\” | CC | +| **R-1252** | Backup & restore | P2 | **One rejected backup block in an app's `.felhom.yml` makes that night's second-drive copy DELETE the app's copies of the household's files.** REPRODUCED on 9202 2026-10-24 (`audits/review-round-2-2026-10-24/``proofs/H2-bad-backup-block/`): audiobookshelf with a marker file in `media/audiobooks`; two unchanged runs kept the copy (the positive control); one bogus path in the block → „backup block rejected … matches no compose bind” → the legacy legs (appdata only) → „Tier 2 reconcile: removed stale dest dir userdata/media”; the original on the drive untouched. Mechanism: `internal/stacks/metadata.go:550-557` (any defect rejects the WHOLE block) → `appbackup/classify.go:217-221` → `backup/tier2_capture.go:84-100` → `tier2.go:492` `tier2Reconcile` (`:325-329`), against D25's own header (`tier2_kept.go:17-18`). Same family (static): a D25 kept copy is reconciled away when its leg leaves the declared set and its record stays (HUNT-2 H2, `tier2.go:485-492`); Remove with `remove_backups` deletes kept copies and leaves their records (HUNT-2 H9, `api/router.go:1086-1087`). **Fixed on main 2026-10-24 night** (controller `518a135`; Part 2): reconcile only under a valid classification; a rejected block is said on the app's status and once a day to the operator (`backup_run_failures`, leg `tier2-settings-unreadable`); kept copies stay in the keep set (H2); Remove drops the kept records it deleted (H9). **Proven live on 9202** (`audits/restore-holes-2026-10-24/P2-9202/`: two control runs, then the bogus path → „REJECTED … every existing copy … KEPT”). | **VERIFY** — fixed on main, waits for the release | the next controller release | release the controller (Monday list) | CC | +| **R-1253** | Backup & restore | P2 | **An export with „stop the app first” during another app's restore overwrites the ONE app-stop marker file — the restore's crash record and its Undo are lost; a power cut then leaves the restored app on mixed data with no put-back.** Found 2026-10-24 (HUNT-1 H3): `AppStopGuard.Begin` writes unconditionally (`internal/backup/appstop_marker.go:279-292`) and `End()` removes the file (`:306-320`); the exporter takes the guard (`internal/appexport/export.go:139-143,262-266`) and neither `StartExport` (`:173-178`) nor the web handlers (`internal/web/handler_export*.go`, grep: no restore check) look for a running restore. Static. **Fixed on main 2026-10-24 night** (controller `c9b4b82`; Part 3): an export that stops its app refuses (409, „err.appexport.another_operation_running”) while another operation of this process holds the marker (`BusyWith`, `BeginExclusive`); picked over one marker per app (smaller: the exporter and one guard method). **Proven live on 9202** (`audits/restore-holes-2026-10-24/P3-9202/`: refused during a MeTube restore, uptime-kuma never stopped; the same export after it ran). | **VERIFY** — fixed on main, waits for the release | the next controller release | release the controller (Monday list) | CC | +| **R-1254** | Backup & restore | P2 | **The hub saves a household's new escrow package in TWO writes; a failure between them leaves the new repository hash beside the OLD identity package, and the box then marks itself escrowed — on a total loss the written code opens nothing.** Found 2026-10-24 (HUNT-3 H1), read at `file:line`: `hub/internal/api/handler.go:1415` `SaveHostEscrow` commits blob + hash, `:1445` `SaveHostDRBundle` sets `identity_blob` in a second statement (`internal/store/store.go:3788-3796`; the upsert `:3490-3511` never touches it); the controller's confirm reads the hash (`felhom-controller/controller/internal/report/escrow_confirm.go:195-217`); the agent sends once. Same family (H2): a lost or late answer (30 s) tells the household „the previous code still works” when the hub may already hold the new package (`felhom-agent/internal/localapi/escrow_ceremony.go:79,532-534,639-641`). Static. **The hub half is fixed on main 2026-10-24 night** (felhom.eu `dcf4f84c`; Part 3): key, hash and identity blob in ONE transaction (`SaveHostEscrowWithIdentity`), a bad identity refused before any write; hub image `0.170.0-rc.r1254` built locally (`67db584a45d7`), NOT pushed, NOT deployed. The agent ceremony half (H2: „the previous code still works” after a lost answer) is still open. | **NARROWED** — the hub half fixed on main (waits for the hub release); the ceremony half open | the hub release (attended); the agent half | deploy hub v0.170.0 (Monday list); then the ceremony asks the hub what it holds before it says „the previous code still works” | CC | | **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC | | **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC | | **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.** **What is NOT claimed here:** `restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC | @@ -234,7 +235,7 @@ stopping line that lies. |---|---|---|---|---|---|---|---| | **R-1241** | Security & access | P1 | **2026-10-24: the Felhom half is DONE** — hub 0.168.0 deployed (the tokens sealed; every copy leaving the volume is clean, the live file's old bytes accepted by the operator, R-975 closed), the hub volume out of the Longhorn backup (D40, R-1243 VERIFY), the two old plain copies deleted by the operator. **What stays here is the homelab half:** the open NFS export and Longhorn's login (the homelab session). **A hacked app pod on DooPlex can read the hub's nightly database copies — with every customer's unsealed Cloudflare, tunnel and git token (R-975) — through the Longhorn NFS export, and by a published file-handle escape the operator signing keys on the same disk: whoever breaks one homelab app can take over the boxes' tunnels and sign agent updates.** MEASURED 2026-10-23 (Plex2Shell impact audit, `audits/plex2shell-felhom-impact-2026-10-23/`): export `/mnt/5_hdd/backup/longhorn-pvc` rw, `no_root_squash`, `no_subtree_check`, a sub-directory of sda1, which also holds `/mnt/5_hdd/felhom.eu/felhom-op-*` (no passphrase, R-924); `secure` is void in pods (containerd sets `ip_unprivileged_port_start=0`; a uid-1000 probe with no capabilities bound port 700); the four `felhom-system` volumes have `encrypted=false`; Longhorn's API on :9500 answers any pod with no login (backups, targets, volumes). In the Plex2Shell window neither door was used — verdict NOT REACHED (nfsd accepted 0 new connections 10-10 → 10-23, positive control; no off-schedule Longhorn backup or volume). 2026-10-23 ~15:18Z, operator yes: the export was narrowed from `*` to `10.42.0.0/16` — the home network and the tailnet are closed, pods are not (Longhorn connects from its pod address). Not exercised: the escape itself (Tier-2 machine). | OPEN | the homelab session (DooPlex host + cluster); the operator (R-924) | Homelab: a NetworkPolicy so only `longhorn-system` reaches :2049/:111 and :9500, or the backup store on its own filesystem. Felhom: R-975 sealed in hub 0.168.0 (deploy pending); the hub's plain Longhorn copies → R-1243; a passphrase on the signing keys (R-924) | operator | | **R-1139** | Security & access | P2 | **Tester 2's break-glass root password was not rotated with the others (R-954, operator decision D1 = A, 2026-10-17): the box has been offline since 2026-10-04.** demo-hp, Tester 1 and demo-felhom were rotated and proven on 2026-10-17 (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | OPEN — filed 2026-10-17 **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a fresh install sets and vaults a new console password).** | Tester 2 reporting again | When it reports: the installer's step 4b alone (`P3/rotate-step4b.sh` shape), then the vaulted value logs in (PVE API 200) and the old one is refused | CC | -| **R-1257** | Security & access | P2 | **D4's root-key-only sshd drop-in reaches BYO hosts too, while `11` §1 says a BYO host gets guest and Docker updates only — never a host change — and the installer's BYO screen lists what it leaves alone.** Found 2026-10-24 (SEC-2 X1), read at `file:line`: `felhom-agent/configs/felhom-os-apply:230` writes the drop-in in every mode (`plan_writes` holds back only policy `oob`, `:1985-2000`; `check_appliance` guards only the OS layer, `:590,1070`); `felhom.eu/scripts/felhom-host-install.sh` (the BYO screen). A BYO owner who logs in to root by password over SSH loses that at the next bundle. Same area: the „in force” check probes only `addr=192.0.2.1`, so a `Match Address ` block reads as in force (X3, `:2101-2110`); a failing `sshd -t` refuses and undoes the WHOLE bundle (X4, `:2102,2112-2115`). | **WAITING-ON-OPERATOR** | the operator's choice (D43) | rule D43; then CC changes the bundle policy and the check | operator | +| **R-1257** | Security & access | P2 | **D4's root-key-only sshd drop-in reaches BYO hosts too, while `11` §1 says a BYO host gets guest and Docker updates only — never a host change — and the installer's BYO screen lists what it leaves alone.** Found 2026-10-24 (SEC-2 X1), read at `file:line`: `felhom-agent/configs/felhom-os-apply:230` writes the drop-in in every mode (`plan_writes` holds back only policy `oob`, `:1985-2000`; `check_appliance` guards only the OS layer, `:590,1070`); `felhom.eu/scripts/felhom-host-install.sh` (the BYO screen). A BYO owner who logs in to root by password over SSH loses that at the next bundle. Same area: the „in force” check probes only `addr=192.0.2.1`, so a `Match Address ` block reads as in force (X3, `:2101-2110`); a failing `sshd -t` refuses and undoes the WHOLE bundle (X4, `:2102,2112-2115`). **Operator ruling D43 = A (2026-10-24 evening, by starting the restore-holes run)**: the drop-in applies only in appliance mode. **Fixed on main** (felhom-agent `c8d3c17`; Part 4): policy `appliance`; BYO → nothing written and our copy removed; no proof → left as it is; X3 the in-force check reads every LAN address; X4 a failing `sshd -t` undoes only the SSH step. Bundle harness `ByoKeepsItsSSH` (7 tests, each rule seen failing). | **VERIFY** — ruled D43 = A; fixed on main, waits for the agent release + the operator's vouch | the agent release + vouch (operator) | release the agent, the operator vouches (Monday list); then a signed bundle on a BYO test host shows `removed`/`skipped` | CC | | **R-1042** | Security & access | P3 | **The boxes' Cloudflare keys cannot change zone settings, so Cloudflare's own „Always Use HTTPS” cannot be switched on.** Measured 2026-10-14 on all three zones (`audits/https-only-2026-10-14/``1A-1-zones-and-keys.txt`): each `cf_api_token` is valid and sees only its own zone, but GET/PATCH `settings/always_use_https`, `ssl`, `security_header` answer 403; the redirect and transform rule phases too. The https-only fix therefore lives on the box (decision 246). A second layer at Cloudflare needs „Zone Settings: Edit” (and, for headers, „Transform Rules”) added to each customer's key — the hub's zone-reach check stays. | **WAITING-ON-OPERATOR** — filed 2026-10-14 | a key permission only the operator can grant | Optional: widen each customer key, then switch „Always Use HTTPS” on per zone | operator | | **R-1242** | Security & access | P3 | **The demo NAS export `/mnt/5_hdd/felhom-data` → 192.168.0.162 (demo-felhom) is a sub-directory of DooPlex's sda1 with `no_subtree_check`, so a compromised demo-felhom host could walk the whole disk as uid 101000 and read every world-readable file there (the GPG secrets exports are 0644, every repo, the public keys).** Measured 2026-10-23 (`audits/plex2shell-felhom-impact-2026-10-23/` Part 1; `exportfs -v`: `all_squash,anonuid=101000`). 0600 files (the signing keys) stay out of reach. Not exercised. | OPEN | a DooPlex host change (fence) | Give the demo share its own filesystem, or `subtree_check`; or remove other-read from `/mnt/5_hdd/backup/secrets/exports` | operator | | **R-1104** | Security & access | P3 | **Check https-only on Tester 2 when it reports again.** R-1033 (plain http on every public name) is closed on the three boxes that report (35 names → 301, 2026-10-16). Tester 2 (`sajatfelhom.hu`) has been off since 2026-10-04 on controller 0.292.0; its addresses answer Cloudflare 530 (tunnel down), so nothing is served in plain http while it is off. When it comes back it takes the floor (≥ 0.312.0 carries the redirect). | OPEN — filed 2026-10-16 | Tester 2 reporting again | Run `scripts/https_only_check.py` with Tester 2 in it; close | CC | @@ -346,8 +347,8 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-1083** | Process & tooling | P3 | **The scratch box 9202 cannot use any feature that needs the agent: its `bootstrap.json` has no `local_api` block, so its controller never gets an agent client.** MEASURED 2026-10-15 (storage-identity build A): 9202 put on controller 0.317.0 by hand to show that a NAS share no longer hides the one-copy notice (R-1075); its `bootstrap.json` (read on the box) holds no `local_api`, so the controller has no agent to ask and keeps the old `st_dev` rule — the check could not tell old from new — 9202 was put back on 0.314.0. Every agent-coupled feature (drives, same-disk, format, network storage) is untestable on the scratch box today, and the NAS leg of build A is proven by a red-proofed unit test only. Evidence: `audits/storage-identity-buildA-2026-10-15/9202-upgrade.txt`. | **OPEN — the exact gap, read 2026-10-19 (no agent change tonight):** the agent mints a per-guest local-API token in ONE place, `--selftest=provision`'s back half (`cmd/felhom-agent/main.go` `bh.Provision` after `engine.RunBringUp`), which only runs for a guest it brings up from a golden archive in the same call. There is no product way to mint for an EXISTING guest (9202) — no subcommand, no local-API route; the token store `/var/lib/felhom-agent/local-tokens.log` is written only by that path. Options: an agent subcommand `--mint-guest-token ` (owner-only, writes bootstrap's `local_api`), or re-provision 9202 through `selftest=provision` (destroys and rebuilds the scratch guest — not done; rule 6). Until then 9202 is controller-only. OPEN **2026-10-20 — left out of agent 0.161.0 on purpose:** the code would be small (a root-only subcommand that mints the token and adds `local_api` to an existing guest's `bootstrap.json`), but an agent key makes 9202's controller book the agent's whole-guest backups, one tier of which writes to ep0 (the demo-hp namespace) — and 9202 is outside the `felhom` pool. Needs, before it ships: a way to give a scratch guest a key with NO whole-guest tiers (or the operator's word that 9202 may write to ep0), and 9202 in the pool. | — | Give 9202 a local-API token the normal way (the agent mints it for its guest), or record in `runbooks/target-selection.md` that 9202 is controller-only; then show the NAS notice live. | CC | -| **R-1265** | Process & tooling | P3 | **Decisions found in code with no document — written as `[DESIGN — undocumented, found 2026-10-24]` notes; the operator confirms them (review round 2).** The notes: `04` §2 (a signed op's window has no maximum length; opsign's default 30 min); `07` §8.2c (the 60-min copy limit and stray handling; Eltávolítás refused during a backup; a start that fails after a put-back is not held; the hub's off-site delivery checker and its R-1230 skip; the release copy has no matrix row); `09` §6.4.x (build.sh's three refusals; the once-retry; stopped_by_household; the manual leg's deadline; two preflight refusals; the unreadable-template row and the R-971 admission); `14` (a re-enroll during the hold is „gone”; two info events; three confirmation words); `15` (the „Később” timer is in memory; the Sonarr/Radarr key read from config.xml); `16` (detect-virt read once; an unreadable path = the same disk; the internal place's lifecycle refusals and its size rule); and R-1019's extension to MeTube, NZBGet and SABnzbd (catalog `c50adfc`: the brief named only qBittorrent and Transmission; MeTube's downloads may be gone from their source site — HUNT-2 H3). | **WAITING-ON-OPERATOR** | the operator's word (D44) | confirm all, or name the ones to change; a confirmed note loses its „undocumented\” tag | operator | -| **R-1266** | Process & tooling | P3 | **Rulings and promises in the documents that the code does not do as written — the operator rules whether the text or the code moves (review round 2).** (1) `09` decision 13: „the tag rule (`CompareImageRefs`) moves to the catalog gate” — it did not (`felhom-controller/controller/internal/stacks/updateorder.go:134`; `09` §6.4 part 4 says so); (2) decision 14: a box „never jumps” — a person's press on an app older than the ladder takes the catalog's current definition (`ladder.go:34-36,175-176`; the night leg does not jump); (3) decision 250 item 5: the golden gate judges „by its AGE … while the floor is above it” — the gate cannot read the floor and uses „release ahead” (`scripts/golden_currency_gate.py:8-12`); (4) `01` §1/§4 „the hub never connects into a box”, „no inbound control endpoint”, „no standing operator backdoor” — the hub pokes a box (UDP to its WG /32:51822 via ep0, `hub/internal/poke/client.go`; agent `internal/poke/poke.go:35`) and the operator's WG peer reaches felhom-sshd (`06` §4.5); `03` §5's break-glass „via the Cloudflare Tunnel behind Access” and `06` §6's „no inbound listener” are the same old text; (5) `01` §8: the operator „cannot open / read” the off-site backup — R-985 already softened the website; (6) `06` §8's plan that the agent manages cloudflared — `03` §2 says it will not. | **WAITING-ON-OPERATOR** | the operator's choice (D45) | „the text follows the code\” → CC rewrites each and the operator reads the diff; „the code follows the text\” → a row per item | operator | +| **R-1265** | Process & tooling | P3 | **Decisions found in code with no document — written as `[DESIGN — undocumented, found 2026-10-24]` notes; the operator confirms them (review round 2).** The notes: `04` §2 (a signed op's window has no maximum length; opsign's default 30 min); `07` §8.2c (the 60-min copy limit and stray handling; Eltávolítás refused during a backup; a start that fails after a put-back is not held; the hub's off-site delivery checker and its R-1230 skip; the release copy has no matrix row); `09` §6.4.x (build.sh's three refusals; the once-retry; stopped_by_household; the manual leg's deadline; two preflight refusals; the unreadable-template row and the R-971 admission); `14` (a re-enroll during the hold is „gone”; two info events; three confirmation words); `15` (the „Később” timer is in memory; the Sonarr/Radarr key read from config.xml); `16` (detect-virt read once; an unreadable path = the same disk; the internal place's lifecycle refusals and its size rule); and R-1019's extension to MeTube, NZBGet and SABnzbd (catalog `c50adfc`: the brief named only qBittorrent and Transmission; MeTube's downloads may be gone from their source site — HUNT-2 H3). **Picks prepared 2026-10-24 night:** `audits/review-round-2-2026-10-24/D44-D45-picks.md` — 25 notes (21 „safe to confirm”, 4 that matter, each with a pick and its cost). | **WAITING-ON-OPERATOR** | the operator's word (D44) | answer „all picks” or name the ones to change | operator | +| **R-1266** | Process & tooling | P3 | **Rulings and promises in the documents that the code does not do as written — the operator rules whether the text or the code moves (review round 2).** (1) `09` decision 13: „the tag rule (`CompareImageRefs`) moves to the catalog gate” — it did not (`felhom-controller/controller/internal/stacks/updateorder.go:134`; `09` §6.4 part 4 says so); (2) decision 14: a box „never jumps” — a person's press on an app older than the ladder takes the catalog's current definition (`ladder.go:34-36,175-176`; the night leg does not jump); (3) decision 250 item 5: the golden gate judges „by its AGE … while the floor is above it” — the gate cannot read the floor and uses „release ahead” (`scripts/golden_currency_gate.py:8-12`); (4) `01` §1/§4 „the hub never connects into a box”, „no inbound control endpoint”, „no standing operator backdoor” — the hub pokes a box (UDP to its WG /32:51822 via ep0, `hub/internal/poke/client.go`; agent `internal/poke/poke.go:35`) and the operator's WG peer reaches felhom-sshd (`06` §4.5); `03` §5's break-glass „via the Cloudflare Tunnel behind Access” and `06` §6's „no inbound listener” are the same old text; (5) `01` §8: the operator „cannot open / read” the off-site backup — R-985 already softened the website; (6) `06` §8's plan that the agent manages cloudflared — `03` §2 says it will not. **Picks prepared 2026-10-24 night:** `audits/review-round-2-2026-10-24/D44-D45-picks.md` — 7 passages, each „text follows code” with its reason. | **WAITING-ON-OPERATOR** | the operator's choice (D45) | answer „all picks” or name the ones to change | operator | | **R-1240** | Process & tooling | P4 | **Scratch guest 9202 holds 34 saved copies of its `controller.yaml`, each carrying the catalog's git token** (`controller.yaml.pre-`, 2026-09-21 → 2026-10-17, root 0600, inside the guest's controller volume). Every drill session that repointed 9202 saved one and none removed it after its byte-identical restore. Found 2026-10-23 (the D39 session removed its own after `cmp` proved it identical). Not read: whether the token in them is still live (R-925 rotated tokens). | **OPEN** | — | read which token the copies carry (by hash, never printed); if the live one, delete the copies; add „remove the save after an identical restore" to `repoint.py restore` | CC | | **R-1081** | Process & tooling | P4 | **Two published controller tags hold content that is not what their number says: `0.315.0` and `0.316.3`.** Both were caused by CC on 2026-10-15, and both by the same shape — a guard that was not on the path the work takes. **`0.316.3`:** the build directory's `build.sh` is a symlink into the SHARED clone's working copy, which predated the `FELHOM_REPO_DIR` override, so the override was silently ignored and the build copied that tree — which held 31 uncommitted files of another session's physical-disk storage work. The banner printed `Commit: 0225255` with no `Repo:` line. **Nobody deployed it; that was luck, not a control.** Fixed at the root in controller v0.316.4: `build.sh` now refuses `--push` from a tree that is dirty, not at `origin/main`, or not a git checkout, with four test cases red-proved. **Neither tag is re-pushed or deleted** — a version number is pushed once, and that applies to numbers CC spoiled. **No box runs either:** demo-hp 9201 is on 0.317.0, the floor is 0.314.0, and the button offers the highest tag. **`0.315.0`:** Caused by CC on 2026-10-15 while proving R-1080's own fix: the guard was added to `controller/build.sh` in the repo, but the documented release command runs `$FELHOM_ROOT/build/felhom-controller/build.sh`, which was a **separate copy taken 2026-07-18** and had no guard. Running the refusal against a real existing tag therefore rebuilt and re-pushed 0.315.0 — a fourth time — with v0.316.0's code under it (digest `8887343011d6` → `b809201841f7`). **Fixed at the root in the same session:** the build directory's script is now a symlink to the repo's, the old copy kept beside it as `build.sh.stale-copy-2026-07-18`, and the refusal was then proven to fire from the documented path. **No box is affected:** nothing runs 0.315.0 (demo-hp 9201 is on 0.316.2), the floor is 0.314.0, and the household button offers the highest tag. What remains is two tags that must never be deployed and are no longer reproducible from the registry. | **OPEN** — filed 2026-10-15; nothing to do unless someone needs one of those two images, and then the answer is the current release | — | Leave them. If an audit ever needs the real 0.315.0 or 0.316.3, build it from the matching commit under a NEW number and say so | CC | | **R-1034** | Process & tooling | P3 | **Gate 17 checks the FAQ JSON-LD on two pages only, by a hard-coded list — a FAQPage anywhere else is unchecked.** `scripts/site_gates.py` gate 17 loops `for p in ("gyik.html", os.path.join("en", "faq.html"))`, so its „JSON-LD equals the visible questions and answers” guarantee stops at those two files. **Since 2026-10-14 two more pages carry a FAQPage block the gate never reads:** `/sajat-felho-tarhely` and `/en/own-cloud-storage`. Their JSON-LD was *derived* from the visible headings and paragraphs by a script and then verified verbatim against the rendered text, so it is correct today — but nothing stops the next copy edit moving the visible text and leaving the structured data behind, which is exactly the failure gate 17 exists to catch, and the one Google penalises. **Why it is not a one-line fix:** gate 17's extractor is written for the FAQ pages' accordion markup (`