From 836b47f1ecd15bb3eca612c4f4a50fda01d7cc99 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 20 Oct 2026 20:07:30 +0200 Subject: [PATCH] =?UTF-8?q?Rehearsal:=20B6=20#16=20English=20pass,=20#9=20?= =?UTF-8?q?two-at-once=20(as=20expected),=20#13=20second=20drive=20(not=20?= =?UTF-8?q?offered=20as=20backup=20target),=20#4=20drive=20pulled=20(handl?= =?UTF-8?q?ed=20well;=20last=20uploads=200=20bytes=20=E2=86=92=20R-1196=20?= =?UTF-8?q?P3);=20B9=20W24-W29?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../rehearsal-2026-10-20/A6-plan/order.md | 9 ++ .../B4-use/health-log.txt | 23 +++++ .../rehearsal-2026-10-20/B4-use/memwatch.txt | 49 +++++++++- .../s13-second-drive/00-expected.txt | 1 + .../B6-failures/s13-second-drive/01-plug.txt | 3 + .../s13-second-drive/02-decommission.txt | 8 ++ .../B6-failures/s13-second-drive/README.md | 12 +++ .../B6-failures/s16-README.md | 8 ++ .../B6-failures/s16-english-texts.txt | 15 ++++ .../B6-failures/s16-hungarian-left.txt | 15 ++++ .../s4-drive-pulled/00-expected.txt | 1 + .../s4-drive-pulled/01-upload-trickle.txt | 90 +++++++++++++++++++ .../B6-failures/s4-drive-pulled/02-pull.txt | 6 ++ .../s4-drive-pulled/03-after-pull.txt | 13 +++ .../s4-drive-pulled/04-replug-same.txt | 6 ++ .../s4-drive-pulled/05-after-replug-same.txt | 8 ++ .../s4-drive-pulled/06-uploads-after.txt | 7 ++ .../s4-drive-pulled/06b-uploads-sizes.txt | 4 + .../s4-drive-pulled/07-pull2-other-slot.txt | 17 ++++ .../B6-failures/s4-drive-pulled/README.md | 15 ++++ .../s9-two-at-once/00-expected.txt | 1 + .../B6-failures/s9-two-at-once/01-run.txt | 8 ++ .../B6-failures/s9-two-at-once/README.md | 7 ++ .../rehearsal-2026-10-20/B9-words/words.md | 6 ++ .../tools/nc_trick_check.sh | 5 ++ .../tools/nc_trick_check2.sh | 4 + .../rehearsal-2026-10-20/tools/nc_trickle.sh | 3 + .../rehearsal-2026-10-20/tools/twoatonce.py | 29 ++++++ documentation/backlog/OPEN-ITEMS.md | 3 +- 29 files changed, 374 insertions(+), 2 deletions(-) create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/00-expected.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/01-plug.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/02-decommission.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/README.md create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s16-README.md create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s16-english-texts.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s16-hungarian-left.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/00-expected.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/01-upload-trickle.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/02-pull.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/03-after-pull.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/04-replug-same.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/05-after-replug-same.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/06-uploads-after.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/06b-uploads-sizes.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/07-pull2-other-slot.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/README.md create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s9-two-at-once/00-expected.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s9-two-at-once/01-run.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s9-two-at-once/README.md create mode 100644 documentation/audits/rehearsal-2026-10-20/tools/nc_trick_check.sh create mode 100644 documentation/audits/rehearsal-2026-10-20/tools/nc_trick_check2.sh create mode 100755 documentation/audits/rehearsal-2026-10-20/tools/nc_trickle.sh create mode 100644 documentation/audits/rehearsal-2026-10-20/tools/twoatonce.py diff --git a/documentation/audits/rehearsal-2026-10-20/A6-plan/order.md b/documentation/audits/rehearsal-2026-10-20/A6-plan/order.md index fa7a9dcb..821e1f25 100644 --- a/documentation/audits/rehearsal-2026-10-20/A6-plan/order.md +++ b/documentation/audits/rehearsal-2026-10-20/A6-plan/order.md @@ -40,3 +40,12 @@ Method: `random.Random(20261020)`; shuffle the ★ list [1..10], then shuffle th page and the hub say while slow, and whether the run finishes or resumes after. - The box froze twice under ordinary uploads (R-1192). Every heavy scenario now runs ALONE, with the health watch beside it; a freeze during a scenario is recorded as such, not as the scenario's own result. + +## Amendment 2026-10-20 ~17:50 UTC — before any of these ran + +Run tonight, before the night chain (B5 must stay „no hands"): #16 English pass, #9 two-at-once, #13 second drive later, +#4 drive pulled, #3 disk full (the „nightly backup" stood in by „Mentés most" — the one real night is reserved for B5; +the filler goes to a share with Felhőmentés OFF and is deleted before the night), #12 DNS dead, #6 off-site blocked +(shortened to one manual run + the catch-up, not a night — same reason). After the night: #7 DST, #8 wrong clock, +#10 three days off (+#15 inside it), #1 power cut mid install, #14 kept data, #11 if an update exists (none expected — +the catalog is frozen). Reason: the clock scenarios move the box's time and would spoil the first night's record. diff --git a/documentation/audits/rehearsal-2026-10-20/B4-use/health-log.txt b/documentation/audits/rehearsal-2026-10-20/B4-use/health-log.txt index ee61e51f..19a40539 100644 --- a/documentation/audits/rehearsal-2026-10-20/B4-use/health-log.txt +++ b/documentation/audits/rehearsal-2026-10-20/B4-use/health-log.txt @@ -93,3 +93,26 @@ 2026-10-20T17:40:52 health=200 0.011215 diskread=5GB 2026-10-20T17:41:53 health=200 0.011258 diskread=5GB 2026-10-20T17:42:54 health=200 0.012194 diskread=5GB +2026-10-20T17:43:56 health=200 0.010113 diskread=5GB +2026-10-20T17:44:58 health=200 0.010286 diskread=5GB +2026-10-20T17:45:59 health=200 0.010282 diskread=5GB +2026-10-20T17:47:01 health=200 0.010560 diskread=5GB +2026-10-20T17:48:02 health=200 0.009732 diskread=5GB +2026-10-20T17:49:03 health=200 0.010065 diskread=5GB +2026-10-20T17:50:05 health=200 0.010006 diskread=5GB +2026-10-20T17:51:06 health=200 0.010605 diskread=5GB +2026-10-20T17:52:08 health=200 0.012332 diskread=5GB +2026-10-20T17:53:09 health=200 0.010066 diskread=5GB +2026-10-20T17:54:11 health=200 0.011118 diskread=5GB +2026-10-20T17:55:12 health=200 0.010705 diskread=5GB +2026-10-20T17:56:13 health=200 0.011581 diskread=5GB +2026-10-20T17:57:14 health=200 0.009849 diskread=6GB +2026-10-20T17:58:16 health=200 0.010308 diskread=6GB +2026-10-20T17:59:17 health=200 0.009395 diskread=6GB +2026-10-20T18:00:19 health=200 0.010613 diskread=6GB +2026-10-20T18:01:20 health=200 0.010129 diskread=7GB +2026-10-20T18:02:22 health=200 0.010121 diskread=7GB +2026-10-20T18:03:24 health=200 0.012405 diskread=7GB +2026-10-20T18:04:25 health=200 0.012071 diskread=7GB +2026-10-20T18:05:27 health=200 0.012165 diskread=7GB +2026-10-20T18:06:29 health=200 0.021087 diskread=8GB diff --git a/documentation/audits/rehearsal-2026-10-20/B4-use/memwatch.txt b/documentation/audits/rehearsal-2026-10-20/B4-use/memwatch.txt index 120feabd..e0452d5f 100644 --- a/documentation/audits/rehearsal-2026-10-20/B4-use/memwatch.txt +++ b/documentation/audits/rehearsal-2026-10-20/B4-use/memwatch.txt @@ -168,4 +168,51 @@ 17:41:56Z used=3638MB avail=458MB load1=0.8 cpu=29% immich-server:584/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:250/1000MB 0% | felhom-controller:152/4096MB 17% | immich-machine-learning:129/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% 17:42:26Z used=3638MB avail=458MB load1=0.84 cpu=29% immich-server:585/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:206/1000MB 0% | felhom-controller:153/4096MB 20% | immich-machine-learning:129/2048MB 0% | nextcloud:119/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% 17:42:56Z used=3638MB avail=458MB load1=0.58 cpu=9% immich-server:585/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:206/1000MB 0% | felhom-controller:153/4096MB 20% | immich-machine-learning:129/2048MB 0% | nextcloud:119/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% -17:43:27Z used=3638MB avail=458MB load1=0.43 cpu=30% immich-server:585/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:154/4096MB 1% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | book \ No newline at end of file +17:43:27Z used=3638MB avail=458MB load1=0.43 cpu=30% immich-server:585/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:154/4096MB 1% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 5% +17:43:57Z used=3638MB avail=458MB load1=0.48 cpu=9% immich-server:585/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:154/4096MB 1% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 5% +17:44:27Z used=3638MB avail=458MB load1=0.65 cpu=31% immich-server:586/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:152/4096MB 1% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 5% +17:44:57Z used=3638MB avail=458MB load1=0.45 cpu=9% immich-server:586/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:152/4096MB 1% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 5% +17:45:27Z used=3638MB avail=458MB load1=0.35 cpu=30% immich-server:586/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:152/4096MB 0% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% +17:45:57Z used=3638MB avail=458MB load1=0.29 cpu=20% immich-server:586/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:152/4096MB 0% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% +17:46:27Z used=3638MB avail=458MB load1=0.5 cpu=30% immich-server:592/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:154/4096MB 3% | immich-machine-learning:129/2048MB 0% | nextcloud:119/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% +17:46:57Z used=3638MB avail=458MB load1=0.43 cpu=10% immich-server:592/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:154/4096MB 3% | immich-machine-learning:129/2048MB 0% | nextcloud:119/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% +17:47:28Z used=3638MB avail=458MB load1=0.58 cpu=30% immich-server:598/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:154/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 6% | nextcloud-db:89/512MB 0% +17:47:58Z used=3638MB avail=458MB load1=0.48 cpu=15% immich-server:598/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:154/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 6% | nextcloud-db:89/512MB 0% +17:48:28Z used=3638MB avail=458MB load1=1.9 cpu=33% immich-server:599/3072MB 0% | paperless-webserver:427/1280MB 0% | felhom-controller:316/4096MB 154% | mealie:225/1000MB 0% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 6% | nextcloud-db:89/512MB 0% +17:48:58Z used=3638MB avail=458MB load1=1.44 cpu=9% immich-server:599/3072MB 0% | paperless-webserver:427/1280MB 0% | felhom-controller:316/4096MB 154% | mealie:225/1000MB 0% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 6% | nextcloud-db:89/512MB 0% +17:49:28Z used=3638MB avail=458MB load1=1.4 cpu=45% immich-server:599/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:156/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 5% | nextcloud-db:89/512MB 0% +17:49:58Z used=3638MB avail=458MB load1=1.13 cpu=9% immich-server:599/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:156/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 5% | nextcloud-db:89/512MB 0% +17:50:28Z used=3638MB avail=458MB load1=0.93 cpu=33% immich-server:600/3072MB 1% | paperless-webserver:427/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:196/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% +17:50:58Z used=3638MB avail=458MB load1=0.7 cpu=20% immich-server:600/3072MB 1% | paperless-webserver:427/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:196/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% +17:51:28Z used=3638MB avail=458MB load1=0.56 cpu=32% immich-server:673/3072MB 63% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:192/4096MB 77% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% +17:51:59Z used=3638MB avail=458MB load1=0.48 cpu=10% immich-server:673/3072MB 63% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:192/4096MB 77% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% +17:52:29Z used=3638MB avail=458MB load1=0.48 cpu=34% immich-server:641/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:89/512MB 0% +17:52:59Z used=3638MB avail=458MB load1=0.55 cpu=9% immich-server:641/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:89/512MB 0% +17:53:29Z used=3638MB avail=458MB load1=0.47 cpu=31% immich-server:641/3072MB 1% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 0% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:89/512MB 0% +17:53:59Z used=3638MB avail=458MB load1=0.67 cpu=20% immich-server:641/3072MB 1% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 0% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:89/512MB 0% +17:54:29Z used=3638MB avail=458MB load1=0.92 cpu=32% immich-server:641/3072MB 0% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:90/512MB 0% +17:54:59Z used=3638MB avail=458MB load1=0.67 cpu=10% immich-server:641/3072MB 0% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:90/512MB 0% +17:55:29Z used=3638MB avail=458MB load1=0.49 cpu=49% immich-server:641/3072MB 0% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:90/512MB 0% +17:55:59Z used=3638MB avail=458MB load1=0.5 cpu=22% immich-server:641/3072MB 0% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:90/512MB 0% +17:56:30Z used=3638MB avail=458MB load1=0.77 cpu=37% immich-server:641/3072MB 0% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 22% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:90/512MB 0% +17:57:00Z used=1240MB avail=2856MB load1=0.65 cpu=11% immich-server:641/3072MB 0% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 22% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:90/512MB 0% +17:57:30Z used=640MB avail=3456MB load1=0.67 cpu=29% mealie:225/1000MB 0% | felhom-controller:160/4096MB 1% | bookstack-db:93/256MB 0% | traefik:71/4096MB 0% | actualbudget:65/256MB 9% | bookstack:50/512MB 0% | felhom-samba:36/4096MB 0% | filebrowser:35/256MB 0% +17:58:00Z used=640MB avail=3456MB load1=0.59 cpu=8% mealie:225/1000MB 0% | felhom-controller:160/4096MB 1% | bookstack-db:93/256MB 0% | traefik:71/4096MB 0% | actualbudget:65/256MB 9% | bookstack:50/512MB 0% | felhom-samba:36/4096MB 0% | filebrowser:35/256MB 0% +17:58:30Z used=640MB avail=3456MB load1=0.5 cpu=29% mealie:225/1000MB 0% | felhom-controller:161/4096MB 1% | bookstack-db:93/256MB 0% | traefik:72/4096MB 0% | actualbudget:65/256MB 8% | bookstack:50/512MB 0% | felhom-samba:36/4096MB 0% | filebrowser:36/256MB 0% +17:59:00Z used=640MB avail=3456MB load1=0.36 cpu=8% mealie:225/1000MB 0% | felhom-controller:161/4096MB 1% | bookstack-db:93/256MB 0% | traefik:72/4096MB 0% | actualbudget:65/256MB 8% | bookstack:50/512MB 0% | felhom-samba:36/4096MB 0% | filebrowser:36/256MB 0% +17:59:30Z used=640MB avail=3456MB load1=0.35 cpu=30% mealie:225/1000MB 0% | felhom-controller:161/4096MB 0% | bookstack-db:93/256MB 0% | traefik:71/4096MB 1% | actualbudget:70/256MB 8% | bookstack:50/512MB 0% | filebrowser:36/256MB 0% | felhom-samba:36/4096MB 0% +18:00:00Z used=640MB avail=3456MB load1=0.52 cpu=8% mealie:225/1000MB 0% | felhom-controller:161/4096MB 0% | bookstack-db:93/256MB 0% | traefik:71/4096MB 1% | actualbudget:70/256MB 8% | bookstack:50/512MB 0% | filebrowser:36/256MB 0% | felhom-samba:36/4096MB 0% +18:00:30Z used=3638MB avail=458MB load1=1.51 cpu=65% immich-machine-learning:407/2048MB 0% | mealie:225/1000MB 0% | felhom-controller:183/4096MB 16% | immich-server:174/3072MB 97% | nextcloud-db:111/512MB 18% | bookstack-db:93/256MB 0% | actualbudget:80/256MB 8% | traefik:71/4096MB 0% +18:01:01Z used=3638MB avail=458MB load1=2.16 cpu=79% immich-machine-learning:407/2048MB 0% | mealie:225/1000MB 0% | felhom-controller:183/4096MB 16% | immich-server:174/3072MB 97% | nextcloud-db:111/512MB 18% | bookstack-db:93/256MB 0% | actualbudget:80/256MB 8% | traefik:71/4096MB 0% +18:01:31Z used=3638MB avail=458MB load1=1.66 cpu=9% immich-server:936/3072MB 0% | paperless-webserver:532/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:172/2048MB 0% | nextcloud-db:106/512MB 0% | actualbudget:103/256MB 0% | immich-postgres:82/768MB 0% | traefik:79/4096MB 3% +18:02:01Z used=3638MB avail=458MB load1=1.3 cpu=7% immich-server:936/3072MB 0% | paperless-webserver:532/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:172/2048MB 0% | nextcloud-db:106/512MB 0% | actualbudget:103/256MB 0% | immich-postgres:82/768MB 0% | traefik:79/4096MB 3% +18:02:31Z used=3638MB avail=458MB load1=0.93 cpu=10% immich-server:750/3072MB 0% | paperless-webserver:536/1280MB 0% | mealie:211/1000MB 0% | immich-machine-learning:172/2048MB 0% | nextcloud-db:104/512MB 0% | felhom-controller:86/4096MB 0% | immich-postgres:84/768MB 0% | traefik:79/4096MB 0% +18:03:01Z used=3638MB avail=458MB load1=0.76 cpu=22% immich-server:750/3072MB 0% | paperless-webserver:536/1280MB 0% | mealie:211/1000MB 0% | immich-machine-learning:172/2048MB 0% | nextcloud-db:104/512MB 0% | felhom-controller:86/4096MB 0% | immich-postgres:84/768MB 0% | traefik:79/4096MB 0% +18:03:31Z used=3638MB avail=458MB load1=0.74 cpu=12% immich-server:752/3072MB 0% | paperless-webserver:536/1280MB 0% | mealie:211/1000MB 0% | immich-machine-learning:172/2048MB 0% | nextcloud:113/1024MB 2% | nextcloud-db:110/512MB 0% | felhom-controller:89/4096MB 0% | immich-postgres:84/768MB 0% +18:04:01Z used=1240MB avail=2856MB load1=0.74 cpu=24% immich-server:752/3072MB 0% | paperless-webserver:536/1280MB 0% | mealie:211/1000MB 0% | immich-machine-learning:172/2048MB 0% | nextcloud:113/1024MB 2% | nextcloud-db:110/512MB 0% | felhom-controller:89/4096MB 0% | immich-postgres:84/768MB 0% +18:04:31Z used=640MB avail=3456MB load1=0.66 cpu=13% mealie:212/1000MB 0% | traefik:96/4096MB 0% | felhom-controller:83/4096MB 1% | bookstack-db:63/256MB 0% | bookstack:53/512MB 0% | vaultwarden:44/256MB 0% | filebrowser:44/256MB 0% | actualbudget:41/256MB 0% +18:05:02Z used=640MB avail=3456MB load1=0.45 cpu=4% mealie:212/1000MB 0% | traefik:96/4096MB 0% | felhom-controller:83/4096MB 1% | bookstack-db:63/256MB 0% | bookstack:53/512MB 0% | vaultwarden:44/256MB 0% | filebrowser:44/256MB 0% | actualbudget:41/256MB 0% +18:05:32Z used=2688MB avail=1408MB load1=0.64 cpu=19% mealie:212/1000MB 0% | traefik:95/4096MB 0% | felhom-controller:83/4096MB 0% | bookstack-db:63/256MB 0% | bookstack:53/512MB 0% | vaultwarden:44/256MB 0% | filebrowser:44/256MB 0% | actualbudget:41/256MB 0% +18:06:02Z used=3638MB avail=458MB load1=1.53 cpu=31% mealie:212/1000MB 0% | traefik:95/4096MB 0% | felhom-controller:83/4096MB 0% | bookstack-db:63/256MB 0% | bookstack:53/512MB 0% | vaultwarden:44/256MB 0% | filebrowser:44/256MB 0% | actualbudget:41/256MB 0% +18:06:32Z used=3638MB avail=458MB load1=2.27 cpu=82% immich-server:1089/3072MB 0% | immich-machine-learning:271/2048MB 0% | mealie:212/1000MB 0% | paperless-webserver:143/1280MB 102% | nextcloud-db:119/512MB 0% | felhom-controller:112/4096MB 85% | immich-postgres:97/768MB 0% | traefik:95/4096MB 3% +18:07:02Z used=3638MB avail=458MB load1=1.75 cpu=9% immich-server:1089/3072MB 0% | immich-machine-learning:271/2048MB 0% | mealie:212/1000MB 0% | paperless-webserver:143/1280MB 102% | nextcloud-db:119/512MB 0% | felhom-controller:112/4096MB 85% | imm \ No newline at end of file diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/00-expected.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/00-expected.txt new file mode 100644 index 00000000..b78df2a4 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/00-expected.txt @@ -0,0 +1 @@ +Expected: the new disk appears as formattable; after init it is offered as a backup target; choosing it moves the target; removing it safely is possible and the target falls back with a clear message. diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/01-plug.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/01-plug.txt new file mode 100644 index 00000000..9057cde4 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/01-plug.txt @@ -0,0 +1,3 @@ +Formatting '/mnt/hdd_1/images/9320/vm-9320-disk-2.raw', fmt=raw size=53687091200 preallocation=off +scsi2: successfully created disk 'nvme-scratch:9320/vm-9320-disk-2.raw,size=50G' +17:52:46 diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/02-decommission.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/02-decommission.txt new file mode 100644 index 00000000..e8aa67b4 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/02-decommission.txt @@ -0,0 +1,8 @@ +17:55:14Z decommission ('200', {'data': {'decommissioned': True, 'stopped_apps': None, 'where': '/mnt/felhom-drives/hdd_2'}, 'ok': True}) +update VM 9320: -delete scsi2 -force 0 +scsi0: nvme-scratch:9320/vm-9320-disk-0.raw,size=256G +scsi1: nvme-scratch:9320/vm-9320-disk-1.raw,size=200G +scsihw: virtio-scsi-single +unused0: nvme-scratch:9320/vm-9320-disk-2.raw +17:55:21 +storage page: helyezése Végleges leszerelés Mentolemez /mnt/felhom-drives/hdd_2 Kiváltva Adatok átköltöztetve ide: . () Időpont: 2026. 10. 20. 19:55 Visszacsatlakoztatás Eltávolítás a rendszerből Adatok áthelyezése … function migFmtGB(b){ return fhNum(Number(b||0)/1e9, 1)+' GB'; } function migRender(job){ var names={stop:'Alkalmazások leállít diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/README.md b/documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/README.md new file mode 100644 index 00000000..3ba3e01a --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/README.md @@ -0,0 +1,12 @@ +# B6 #13 — a second drive added later (2026-10-20 17:52–17:56 UTC) + +Expected: offered as a backup target; accepting it moves the target; safe removal possible. + +- Hot-plugged 50 GB disk → the init page lists „QEMU HARDDISK /dev/sdc · 50 GB Üres — formázható"; initialised as + `hdd_2` „Mentolemez" (not default) → „sikeresen inicializálva és regisztrálva". +- **Not offered:** the backups page keeps offering only „Családi adatlemez — Kijelölöm" (the disk holding the apps' data): + `internal/web/backup_target_offer.go:101` `firstOfferableDrive` offers the FIRST eligible drive, never a choice; the + drive's own card has no „use for backups" either. A family cannot point the whole-box backup at its new backup disk. + NOT accepted (the only offer was the data disk; tonight's B5 keeps the as-installed target). +- Safe removal: `POST /api/storage/decommission` (mode anyway) → „decommissioned"; the card then reads „Kiváltva — + Adatok átköltöztetve ide: . ()" (empty blanks, W29); disk unplugged (`qm unlink`, left as `unused0`). diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s16-README.md b/documentation/audits/rehearsal-2026-10-20/B6-failures/s16-README.md new file mode 100644 index 00000000..f34e63df --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s16-README.md @@ -0,0 +1,8 @@ +# B6 #16 — the dashboard in English (2026-10-20 ~17:55 UTC) + +1. An `en-US` browser alone: every page stays Hungarian — the dashboard follows the household's language setting, not + the browser (by design; W27). +2. The household switched the dashboard to English (`POST /settings/language lang=en`, the globe), 15 pages read in real + Chromium (`screens/b6-s16-en/`, texts `s16-english-texts.txt`), then switched back to Hungarian. + Hungarian left: the monitoring page's storage descriptions (W26), „Megosztások" on the restore page, a stored restore + message; the drive names are the household's own labels (data, not copy). No broken English seen in the scan. diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s16-english-texts.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s16-english-texts.txt new file mode 100644 index 00000000..27284da3 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s16-english-texts.txt @@ -0,0 +1,15 @@ +Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug SETTINGS System Notifications Security and access 0.336.0 Sign out ↗ Launcher tester4.test Share the launcher ActualBudget BookStack Filebrowser Immich Mealie Nextcloud Paperless-ngx Radicale Vaultwarden +Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug SETTINGS System Notifications Security and access 0.336.0 Sign out ↗ Dashboard tester4.test 11 Running apps 0 Stopped 68 All apps MEMORY 1.9 GB / 4.0 GB (47%) CPU 12% Load: 1.32 / 0.96 / 0.73 SYSTEM 12.9 GB / 49.0 GB (28%) BELSŐ TÁRHELY 4.21 GB / 54.8 GB (8%) CSALÁDI ADATLEMEZ 4.21 GB / 196 GB (2%) Disk health QEMU QEMU HARDDISK Nincs adat 0 °C QEMU QEMU HARDDISK Nincs adat 0 °C QEMU QEMU HARDDISK Nincs adat 0 °C Backup Last backup: 2026-10-20 19:21 Databases: 4 backed up Installed apps ActualBudget Personal finances and budgeting Running Open Log BookStack A simple, book-shaped wiki and documentation platform Running Open Log FileBrowser File manager — browse the storage files from your browser. Running Protected Open Immich Photos and videos in one place (a Google Photos alternative) Running Open Log Mealie A recipe manager and meal planner Running Open Log Nextcloud Your own cloud storage - a Google Drive/Dropbox alternative Running Open Log Paperless-ngx Digitise your documents and keep them in order Running Open Log Radicale Calendar and contacts on your own server (CalDAV / CardDAV) Running Open Log Network sharing File sharing on the local network — the folders appear in Windows Explorer and the Mac Finder. Running Protected Traefik Traffic router (reverse proxy) — sends each request to the right app. Running Protected Vaultward +404 page not found +Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug SETTINGS System Notifications Security and access 0.336.0 Sign out ↗ ← Apps Immich A Google Photos alternative - your photos back themselves up and sort themselves ● Running Up to date photos.tester4.test Open ↗ Settings Data and backup Guide Logs Running Running Restart Stop Signing in to the app You create the admin account on the first visit. The box does not know the password — keep it in your password manager. Address and keys These were created at install time and cannot be changed. Address photos.tester4.test Storage Családi adatlemez Database password Show +Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug SETTINGS System Notifications Security and access 0.336.0 Sign out ↗ ← Apps Paperless-ngx A digital filing cabinet - scanning, OCR and automatic sorting ● Running Up to date paperless.tester4.test Open ↗ Files ↗ Settings Data and backup Guide Logs Running Running Restart Stop Signing in to the app For the first sign-in. If you change the password in the app, the old one stays here. Admin user name Admin password Show Address and keys These were created at install time and cannot be changed. Address paperless.tester4.test Storage Családi adatlemez Database password Show Encryption key Show OCR language Hungarian +Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug SETTINGS System Notifications Security and access 0.336.0 Sign out ↗ Storage — Drives Data storage Manage external drives for storing app data. Kept data Data that removed apps left on the drives Belső tárhely /mnt/felhom-drives/belso Active 4.2 GB / 54.8 GB It is on the machine's own disk, but it is not part of the full system backup. A different disk or the off-site backup protects against a disk failure. ext4 · /dev/mapper/pve-felhom--belso[/felhom-data] No app on this storage User data uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d Make default Block new installs Családi adatlemez /mnt/felhom-drives/hdd_1 Default Active 4.2 GB / 195.8 GB ext4 · /dev/sdb[/felhom-data] · QEMU HARDDISK Used by 3 apps User data uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692 Disconnect Format… Block new installs Move to… Belső tárhely (/mnt/felhom-drives/belso) Move all data Decommission for good Initialise a new drive Attach an existing drive System drives protected — can only be changed with an operator signature local local · — System — protected Host storage — system templates, ISOs, host-level backups. It holds no app data. path:/var/lib/vz 5.4 GB / 72 GB (8%) local-lvm lvmthin · — System — protected App system Internal SSD — the server system, Docker and the databases of the installed apps are here. pve/data 25 GB / 154 GB (16%) felhom-pbs pbs · — Backup — +Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug SETTINGS System Notifications Security and access 0.336.0 Sign out ↗ Backup tester4.test Felhom cannot check that the second drive is really a different disk (inside a virtual machine and behind some USB enclosures this is not visible), so it does not count it as a second copy. The system backup is currently on the same disk as the system — so it protects against corrupted files, but not against a disk failure. Attach a second drive for full protection. You can choose this drive as the place for the system backup. Felhom cannot check that this drive is really a different disk (inside a virtual machine and behind some USB enclosures this is not visible), so we cannot promise that you can restore the system after a disk failure. Családi adatlemez Use this drive Storage overview System 12.9 GB / 49.0 GB (28%) Belső tárhely 4.21 GB / 54.8 GB (8%) Családi adatlemez 4.21 GB / 196 GB (2%) Database backups 7 files System backup (full backup) A regular backup of the whole server — apps, settings and databases together — that can bring back the entire device. The host agent makes and manages it. ✓ Last full backup 2026-10-20 14:09 (5 hours ago) – Local storage (local) Up to date Next backup the last one was 5 hours ago — within the backup window – Restore checked Not run yet Local storage (local) ✓ Last successful backup: 2026-10-20 14:09 (5 hou +Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug SETTINGS System Notifications Security and access 0.336.0 Sign out ↗ Backup — Apps tester4.test A(z) nextcloud hiányzó fájljai helyreállítva az élő adatok közé. App backups (database + configuration) A detailed backup of each app — database dumps, settings and app files. Independent of the full backup above, it can be restored per app. Schedule Database backup 02:30 Next: tomorrow 02:30 Last database backup: 2026-10-20 19:21 (27 minutes ago) Back up now Databases APP TYPE SIZE LAST VALIDATION STATUS bookstack MariaDB 71.9 KB 19:21 41 tables OK immich PostgreSQL 51.1 MB 19:21 71 tables OK nextcloud MariaDB 3.4 MB 19:21 131 tables OK paperless-ngx PostgreSQL 462.7 KB 19:21 72 tables OK Backup status of the apps ActualBudget Config + Data ▶ BookStack Config + DB + Data ▶ Immich Családi adatlemez 56.8 MB ▶ Mealie Config + Data ▶ Nextcloud Családi adatlemez 64.9 MB ▶ Paperless-ngx Családi adatlemez 2.2 MB ▶ Radicale Config + Data ▶ Vaultwarden Config + Data ▶ +Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug SETTINGS System Notifications Security and access 0.336.0 Sign out ↗ Backup — Remote backup tester4.test Remote backup (backup 3) — encrypted, offsite An encrypted copy of the app backups on a remote store — your own NAS or Felhom offsite storage — over restic + SFTP. The store sees only encrypted data. This is the „1 off-site" leg of the 3-2-1 rule — independent of the local second copy and the full system backup. ✓ OK Last remote backup 25 minutes ago 1.1 GB Store size · 26 snapshots u629488-sub5@u629488-sub5.your-storagebox.de /home/felhom-repo The recovery code is in escrow. Make a new recovery code Remote backup now Which apps are backed up to the remote store? ActualBudget Turn off remote backup BookStack Turn off remote backup Immich Turn off remote backup Mealie Turn off remote backup Nextcloud Turn off remote backup Paperless-ngx Turn off remote backup Radicale Turn off remote backup Vaultwarden Turn off remote backup Set up a remote backup target +Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug SETTINGS System Notifications Security and access 0.336.0 Sign out ↗ Backup — Restore tester4.test Restore App: — Choose — ActualBudget BookStack Immich Mealie Nextcloud Paperless-ngx Radicale Vaultwarden Snapshot: — Choose an app — The restore overwrites the app’s current data with the state of the chosen backup. The app stops and restarts automatically during the process. I understand, start the restore. Start restore Import from a saved package (.fab) Restore from the remote store Choose the app, and on the next page decide what you want: a check into a separate folder, bringing back only the missing files, or a full restore. None of them starts without asking. ActualBudget Restore… BookStack Restore… Immich Restore… Mealie Restore… Nextcloud Restore… Paperless-ngx Restore… Radicale Restore… Vaultwarden Restore… Megosztások Restore shares Existing check copies These restored copies take space on the drive. They do not touch your real data, and you can delete them any time. bookstack 153.7 MB · 2026-10-20 19:38 · /mnt/felhom-drives/belso/backups/offsite-restore/bookstack Delete copy radicale 34.1 KB · 2026-10-20 19:26 · /mnt/felhom-drives/belso/backups/offsite-restore/radicale Delete copy Portable backup package (.fab) A portable snapshot — keep it anywhere, and load it back from a drive any time. Continuous protection comes from bac +Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug SETTINGS System Notifications Security and access 0.336.0 Sign out ↗ Network sharing Shared folders appear in Windows Explorer and the Mac Finder as if they were a network drive. Sharing works only on the home network — not from the internet. Allow sharing Server name The machine appears on the network under this name (\\FELHOM). At most 15 characters: letters, digits, hyphen, underscore. Save Status: running On some networks the machine appears late in the device list. Typing the \\FELHOM address into the address bar works then too. Connect to the share Windows: in the File Explorer address bar: \\FELHOM Mac: Finder → Go → Connect to Server: smb://FELHOM.local If the name does not work, use the direct address: smb://192.168.0.133 The direct address can change with the network — always try the name first. Set the sharing password The household has one shared password. When connecting, use this as the username: felhom. New password Confirm password Save password Shared folders NAME FOLDER READ-ONLY CLOUD BACKUP DELETE Filmek /mnt/felhom-drives/hdd_1/shares/Filmek no on Delete beolvasas /mnt/sys_drive/felhom-data/userdata/import no off system When you delete a share, the folder and its files stay — only the network access ends. Backing up the shares Shared folders are backed up together with the other data. „Cloud backup” on means the fo +Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug SETTINGS System Notifications Security and access 0.336.0 Sign out ↗ Settings — Security and access Change password Current password New password Confirm new password Change password Country restriction Access to the web apps can be limited by country with the Cloudflare WAF. Access from the local network is always allowed (it does not pass through Cloudflare). The Cloudflare API token is not configured. Ask the operator to set it up. The token needs the Zone WAF:Edit permission. Family Family members sign in to the family-gated apps with their own name and password. It does not open this dashboard. irisz New passwordRemove kata New passwordRemove odon New passwordRemove Add a family member +Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug SETTINGS System Notifications Security and access 0.336.0 Sign out ↗ Settings — Notifications Notifications E-mail address What should we e-mail you about? My machine and drives You get an e-mail when a drive is unplugged or back, filling up, full or failing, or the off-site backup is short of space. These need you. My apps custom You get an e-mail when an app is not running or we stopped it because it kept crashing, and when an update was undone or failed. This group has your own selection. Switching it on turns on all of its notifications. Details What we handle You get an e-mail when a backup failed or did not run, or the machine cannot be reached. We see these too and deal with them; you need not do anything. Details Save Send test e-mail App e-mail Apps can send e-mail through Felhom (for example password resets, invitations), without setting up a separate e-mail provider. Each app’s sender address is its own @felhom.eu address. Allow app e-mail Sender name (optional) This appears as the sender of outgoing e-mails, next to the app name. Save After turning it on, allow sending e-mail for each app too (on the app’s page). +Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug SETTINGS System Notifications Security and access 0.336.0 Sign out ↗ System monitor Server health (host) Updated: 19:50:26 CPU usage 0.0% Load 0.93 0.96 0.75 Memory 3.7 GB / 7.8 GB (48%) CPU temperature n/a Uptime 16 min Storage capacity Belső SSD – rendszer és alkalmazások (local-lvm) · thin-pool: 16% 25.1 GB / 154 GB (16%) Az operációs rendszer és a telepített alkalmazások tárhelye. Belső lemez – sablonok és helyi mentések (local) · 0 °C 5.4 GB / 72.1 GB (8%) Rendszersablonok és helyi biztonsági mentések. Távoli biztonsági mentés (felhom-pbs) 0 GB / 0 GB (0%) Titkosított, telephelyen kívüli biztonsági mentések. System overview Hostname Tester4 Operating system Debian GNU/Linux 13 (trixie) Kernel 7.0.2-6-pve Processor AMD Ryzen Embedded V1756B with Radeon Vega Gfx (4 mag) Uptime 16 min Boot time 2026-10-20 19:34 Storage System 12.9 GB / 49.0 GB (28%) Belső tárhely 4.21 GB / 54.8 GB (8%) External storage — the large files of your installed apps (media, documents) go here; the databases stay on the internal SSD. Családi adatlemez 4.21 GB / 196 GB (2%) External storage — the large files of your installed apps (media, documents) go here; the databases stay on the internal SSD. Hub connection Connected — the central system is actively watching the server. Hub URL https://hub.felhom.eu Customer ID Tester4 Last successful report 1 minute ago +404 page not found diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s16-hungarian-left.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s16-hungarian-left.txt new file mode 100644 index 00000000..1ef16d8c --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s16-hungarian-left.txt @@ -0,0 +1,15 @@ +/ 0 [] +/dashboard 3 ['BELSŐ', 'CSALÁDI', 'TÁRHELY'] +/apps 0 [] +/apps/immich 1 ['Családi'] +/apps/paperless-ngx 1 ['Családi'] +/storage 3 ['Belső', 'Családi', 'tárhely'] +/backups 3 ['Belső', 'Családi', 'tárhely'] +/backups/apps 6 ['Családi', 'fájljai', 'helyreállítva', 'hiányzó', 'közé', 'élő'] +/backups/remote 0 [] +/backups/restore 1 ['Megosztások'] +/sharing 0 [] +/settings/security 0 [] +/settings/notifications 0 [] +/monitoring 14 ['Belső', 'Családi', 'Titkosított', 'Távoli', 'alkalmazások', 'biztonsági', 'kívüli', 'mentés', 'mentések', 'operációs', 'telepített', 'tárhely', 'tárhelye', 'és'] +/system 0 [] diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/00-expected.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/00-expected.txt new file mode 100644 index 00000000..8cc4788a --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/00-expected.txt @@ -0,0 +1 @@ +Expected (before): the apps on the drive stop or are held with a clear message; the upload fails honestly; nothing is written to the empty mount point; after the drive returns (same slot or another) the apps come back by themselves or with one press; data = before. diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/01-upload-trickle.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/01-upload-trickle.txt new file mode 100644 index 00000000..ab82ef3f --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/01-upload-trickle.txt @@ -0,0 +1,90 @@ +17:56:14 put 1 201 +17:56:17 put 2 201 +17:56:19 put 3 201 +17:56:22 put 4 201 +17:56:24 put 5 201 +17:56:27 put 6 201 +17:56:30 put 7 503 +17:56:32 put 8 503 +17:56:34 put 9 503 +17:56:36 put 10 503 +17:56:38 put 11 503 +17:56:40 put 12 503 +17:56:42 put 13 503 +17:56:44 put 14 503 +17:56:46 put 15 503 +17:56:49 put 16 503 +17:56:51 put 17 503 +17:56:53 put 18 503 +17:56:55 put 19 503 +17:56:57 put 20 503 +17:57:19 put 21 000 +17:57:21 put 22 404 +17:57:23 put 23 404 +17:57:25 put 24 404 +17:57:27 put 25 404 +17:57:29 put 26 404 +17:57:31 put 27 404 +17:57:33 put 28 404 +17:57:35 put 29 404 +17:57:37 put 30 404 +17:57:39 put 31 404 +17:57:41 put 32 404 +17:57:43 put 33 404 +17:57:45 put 34 404 +17:57:47 put 35 404 +17:57:49 put 36 404 +17:57:51 put 37 404 +17:57:53 put 38 404 +17:57:55 put 39 404 +17:57:57 put 40 404 +17:58:00 put 41 404 +17:58:02 put 42 404 +17:58:04 put 43 404 +17:58:06 put 44 404 +17:58:08 put 45 404 +17:58:10 put 46 404 +17:58:12 put 47 404 +17:58:14 put 48 404 +17:58:16 put 49 404 +17:58:18 put 50 404 +17:58:20 put 51 404 +17:58:22 put 52 404 +17:58:24 put 53 404 +17:58:26 put 54 404 +17:58:28 put 55 404 +17:58:30 put 56 404 +17:58:32 put 57 404 +17:58:34 put 58 404 +17:58:36 put 59 404 +17:58:38 put 60 404 +17:58:40 put 61 404 +17:58:42 put 62 404 +17:58:44 put 63 404 +17:58:46 put 64 404 +17:58:48 put 65 404 +17:58:50 put 66 404 +17:58:52 put 67 404 +17:58:54 put 68 404 +17:58:56 put 69 404 +17:58:58 put 70 404 +17:59:00 put 71 404 +17:59:03 put 72 404 +17:59:05 put 73 404 +17:59:07 put 74 404 +17:59:09 put 75 404 +17:59:11 put 76 404 +17:59:13 put 77 404 +17:59:15 put 78 404 +17:59:17 put 79 404 +17:59:19 put 80 404 +17:59:21 put 81 404 +17:59:23 put 82 404 +17:59:25 put 83 404 +17:59:27 put 84 404 +17:59:29 put 85 404 +17:59:31 put 86 404 +17:59:33 put 87 404 +17:59:35 put 88 404 +17:59:37 put 89 404 +17:59:39 put 90 404 diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/02-pull.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/02-pull.txt new file mode 100644 index 00000000..9330c036 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/02-pull.txt @@ -0,0 +1,6 @@ +PULL 17:56:26 +update VM 9320: -delete scsi1 +scsi0: nvme-scratch:9320/vm-9320-disk-0.raw,size=256G +scsihw: virtio-scsi-single +unused0: nvme-scratch:9320/vm-9320-disk-2.raw +unused1: nvme-scratch:9320/vm-9320-disk-1.raw diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/03-after-pull.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/03-after-pull.txt new file mode 100644 index 00000000..f8212133 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/03-after-pull.txt @@ -0,0 +1,13 @@ +immich stopped None +nextcloud stopped None +paperless-ngx stopped None +mealie running None +/storage Családi adatlemez -> n a tárolón Legyen alapértelmezett Új telepítések letiltása Családi adatlemez /mnt/felhom-drives/hdd_1 Leválasztva Leválasztva: 2026. 10. 20. 19:57 Leállított alkalmazások: immich, nextcloud, paperless-ngx Csatlakoztatás Mentolemez /mnt/felhom-drives/hdd_2 Kiváltva Adatok átköltö +/dashboard Családi adatlemez -> 2,9 GB / 49,0 GB (28%) Belső tárhely 4,21 GB / 54,8 GB (8%) Családi adatlemez — Leválasztva Lemezek állapota QEMU QEMU HARDDISK Nincs adat 0 °C QEMU QEMU HARDDISK Nincs adat 0 °C Biztonsági mentés Utolsó mentés: 2026. 10. 20. 19:21 Adatbázisok: 4 mentve Telepített alkalmazások Ac +17:56:11 INFO [backup] Found 7 DB dump files across drives +17:57:08 INFO [settings] Storage path /mnt/felhom-drives/hdd_1 disconnected=true +17:57:08 WARN [gate] drive ABSENT /mnt/felhom-drives/hdd_1 — stopped+blocked 3 app(s): [immich nextcloud paperless-ngx] +17:57:08 INFO [web] FileBrowser: drive /mnt/felhom-drives/hdd_1 is removed or dead — not bound this pass +17:57:08 INFO [web] FileBrowser: drive /mnt/felhom-drives/hdd_2 not mounted — skipping userdata skeleton +17:57:08 INFO Event pushed: storage_disconnected (error) [hu-only] — Meghajtó váratlanul leválasztva: Családi adatlemez +17:57:08 INFO [web] FileBrowser mounts synced (recreated) — 3 storage path(s), config updated diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/04-replug-same.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/04-replug-same.txt new file mode 100644 index 00000000..029fbcf5 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/04-replug-same.txt @@ -0,0 +1,6 @@ +PLUG BACK same slot 17:59:25 +update VM 9320: -scsi1 nvme-scratch:9320/vm-9320-disk-1.raw +scsi0: nvme-scratch:9320/vm-9320-disk-0.raw,size=256G +scsi1: nvme-scratch:9320/vm-9320-disk-1.raw,size=200G +scsihw: virtio-scsi-single +unused0: nvme-scratch:9320/vm-9320-disk-2.raw diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/05-after-replug-same.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/05-after-replug-same.txt new file mode 100644 index 00000000..8c9b8aca --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/05-after-replug-same.txt @@ -0,0 +1,8 @@ +immich running None +nextcloud running None +paperless-ngx running None +storage: Családi adatlemez /mnt/felhom-drives/hdd_1 Alapértelmezett Aktív 4,2 GB / 195,8 GB ext4 · /dev/sdc[/felhom-data] · QEMU HARDDISK 3 alkalmazás használja Immich 56,8 MB Mozgatás Nextcloud 64,9 MB Mozgatás Paperless-ngx 2,2 MB Mozgatás Új telepítések letiltása Át +18:00:27 INFO [gate] drive RETURNED /mnt/felhom-drives/hdd_1 — re-attached + restarted gate-stopped apps +18:00:27 INFO [web] FileBrowser: drive /mnt/felhom-drives/hdd_2 not mounted — skipping userdata skeleton +18:00:27 INFO Event pushed: storage_reconnected (info) [hu-only] — Meghajtó újra csatlakoztatva: Családi adatlemez +18:01:11 INFO [backup] Found 7 DB dump files across drives diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/06-uploads-after.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/06-uploads-after.txt new file mode 100644 index 00000000..b388b323 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/06-uploads-after.txt @@ -0,0 +1,7 @@ +feltoltes_1: 200 sha-differs +feltoltes_2: 200 sha-differs +feltoltes_3: 200 sha-differs +feltoltes_4: 200 sha-differs +feltoltes_5: 404 sha-differs +feltoltes_6: 404 sha-differs +feltoltes_7: 404 sha-differs diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/06b-uploads-sizes.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/06b-uploads-sizes.txt new file mode 100644 index 00000000..65001619 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/06b-uploads-sizes.txt @@ -0,0 +1,4 @@ +feltoltes_1: got 0 B e3b0c44298fc | src IMG_20150024.jpg 108354 B 65f0e857b355 | head: +feltoltes_2: got 0 B e3b0c44298fc | src IMG_20150042.jpg 110472 B 3149e1a352cd | head: +feltoltes_3: got 0 B e3b0c44298fc | src IMG_20150060.jpg 109211 B 3fd3b7609dd7 | head: +feltoltes_4: got 0 B e3b0c44298fc | src IMG_20150078.jpg 110528 B 5b0a124ec005 | head: diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/07-pull2-other-slot.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/07-pull2-other-slot.txt new file mode 100644 index 00000000..086275b6 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/07-pull2-other-slot.txt @@ -0,0 +1,17 @@ +PULL 2 18:03:33 +PLUG into scsi3 18:05:05 +update VM 9320: -scsi3 nvme-scratch:9320/vm-9320-disk-1.raw +scsi0: nvme-scratch:9320/vm-9320-disk-0.raw,size=256G +scsi3: nvme-scratch:9320/vm-9320-disk-1.raw,size=200G +scsihw: virtio-scsi-single +unused0: nvme-scratch:9320/vm-9320-disk-2.raw +immich running +nextcloud running +paperless-ngx running +18:00:27 INFO [gate] drive RETURNED /mnt/felhom-drives/hdd_1 — re-attached + restarted gate-stopped apps +18:00:27 INFO Event pushed: storage_reconnected (info) [hu-only] — Meghajtó újra csatlakoztatva: Családi adatlemez +18:04:07 INFO [settings] Storage path /mnt/felhom-drives/hdd_1 disconnected=true +18:04:07 WARN [gate] drive ABSENT /mnt/felhom-drives/hdd_1 — stopped+blocked 3 app(s): [immich nextcloud paperless-ngx] +18:04:07 INFO Event pushed: storage_disconnected (error) [hu-only] — Meghajtó váratlanul leválasztva: Családi adatlemez +18:05:59 INFO [gate] drive RETURNED /mnt/felhom-drives/hdd_1 — re-attached + restarted gate-stopped apps +18:05:59 INFO Event pushed: storage_reconnected (info) [hu-only] — Meghajtó újra csatlakoztatva: Családi adatlemez diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/README.md b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/README.md new file mode 100644 index 00000000..cc356b15 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/README.md @@ -0,0 +1,15 @@ +# B6 #4 — the data drive pulled during uploads, put back (same slot, then another) — 2026-10-20 + +Expected (`00-expected.txt`): the apps on it stop with a clear message; the upload fails honestly; nothing written to the +empty mount point; after the return the apps come back; data = before. + +| | pull | detected | apps | household sees | put back | back by itself | +|---|---|---|---|---|---|---| +| 1 (same slot) | 17:56:26, during a Nextcloud upload every 2 s | 17:57:08 (42 s): `drive ABSENT … stopped+blocked 3 app(s)` | Immich, Nextcloud, Paperless stopped | storage card „Leválasztva … Leállított alkalmazások: immich, nextcloud, paperless-ngx"; dashboard „Családi adatlemez — Leválasztva"; event `storage_disconnected` „Meghajtó váratlanul leválasztva" | 17:59:25 (came back as /dev/sdc) | 18:00:27 (62 s): `drive RETURNED … restarted gate-stopped apps`, event `storage_reconnected` | +| 2 (other slot, scsi3) | 18:03:33 | 18:04:07 (34 s) | same | same | 18:05:05 | 18:05:59 (54 s) | + +**Uploads around pull 1 (`01-upload-trickle.txt`, `06*`):** files 1–6 answered **201** (the last one 1 s AFTER the pull); +then 503 while Nextcloud failed, then 404 once the gate stopped it. Afterwards: files 1–4 are listed by Nextcloud but are +**0 bytes**; files 5–6 are gone. The household was told each upload succeeded and nothing later says the last ones broke +(the ext4 delayed-write window: data not yet on disk is lost when a disk is yanked). → R-1196 (P3). +Data otherwise: the apps' other files and the films unchanged (spot checks); no writes to the empty mount point seen. diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s9-two-at-once/00-expected.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s9-two-at-once/00-expected.txt new file mode 100644 index 00000000..9e593be0 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s9-two-at-once/00-expected.txt @@ -0,0 +1 @@ +Expected (written before): the second act is refused before it changes anything; the restore finishes; the page says the truth; Radicale keeps its 2 events. diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s9-two-at-once/01-run.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s9-two-at-once/01-run.txt new file mode 100644 index 00000000..cbef71e7 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s9-two-at-once/01-run.txt @@ -0,0 +1,8 @@ +17:51:39Z (a) restore + remove: A: HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] | B: ('409', {'ok': False, 'error': 'Az alkalmazáson mentés vagy visszaállítás fut. Várd meg, amíg befejeződik.'}) + (0.1, (True, True, 'A(z) nextcloud hiányzó fájljai helyreállítva az élő adatok közé.')) + (10.2, (False, True, 'A(z) radicale: 1 adatkötet visszaállítva — az alkalmazás újraindult.')) + app: running deployed: True +17:51:59Z (b) double-click restore: A: HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] | B: HTTP/2 302 ['location: /backups/restore?flash_error=Egy+vissza%C3%A1ll%C3%ADt%C3%A1si+m%C5%B1velet+%28radicale%29+m%C3%A1r+fut%2C+ez%C3%A9rt+most+nem+ind%C3%ADthat%C3%B3+%C3%BAjabb.+Az+%C3%A1llapot%C3%A1t+ezen+az+oldalon+k%C3%B6vetheted%3B+amint+befejez%C5%91dik%2C+%C3%BAjra+ind%C3%ADthatsz+vissza%C3%A1ll%C3%ADt%C3%A1st.'] + (0.1, (True, True, 'A(z) radicale: 1 adatkötet visszaállítva — az alkalmazás újraindult.')) + (10.2, (False, True, 'A(z) radicale: 1 adatkötet visszaállítva — az alkalmazás újraindult.')) + app: running deployed: True diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s9-two-at-once/README.md b/documentation/audits/rehearsal-2026-10-20/B6-failures/s9-two-at-once/README.md new file mode 100644 index 00000000..91cc7e1f --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s9-two-at-once/README.md @@ -0,0 +1,7 @@ +# B6 #9 — two actions at once (2026-10-20 17:51 UTC) — AS EXPECTED + +(a) „Visszaállítás" (Radicale) and „Eltávolítás" 50 ms apart from two sessions: the remove → 409 „Az alkalmazáson mentés vagy +visszaállítás fut. Várd meg, amíg befejeződik."; the restore finished („1 adatkötet visszaállítva"); the app deployed and running. +(b) a double click on „Visszaállítás": the second press → „Egy visszaállítási művelet (radicale) már fut, ezért most nem +indítható újabb…"; the status stayed truthful (ok). Reviewer candidate T13 (the page shows failed while the restore runs) +did NOT reproduce at this spacing — it stays a candidate in R-1189 (i), marked „not reproduced at 50 ms". diff --git a/documentation/audits/rehearsal-2026-10-20/B9-words/words.md b/documentation/audits/rehearsal-2026-10-20/B9-words/words.md index 3236b73f..af9cd3c8 100644 --- a/documentation/audits/rehearsal-2026-10-20/B9-words/words.md +++ b/documentation/audits/rehearsal-2026-10-20/B9-words/words.md @@ -25,3 +25,9 @@ | W21 | sharing page after „switch on" | „Beállítás mentve. A megosztási szolgáltatás előkészítése folyamatban…" | the service waits for the share PASSWORD, which the page does not say (log: „deploy skipped — household SMB password not set yet"); stays „Állapot: áll" for ever | „Állíts be egy megosztási jelszót — enélkül nem indul el." | | W22 | a failed local restore | „Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1 stderr: ...twork rallly_rallly-internal Creating …" (9202, R-1177) | raw English Docker output | a Hungarian sentence + what to do | | W23 | Actual Budget | the whole app is English (v26.9.0) and shows „A new version of Actual is available!" | English; an upgrade nag the household cannot act on | catalog note | +| W24 | Távoli mentés page | an SSH key / known_hosts form with „no blind TOFU", „ssh-keyscan -p " on the household's page | operator detail, English | hide behind „Haladó" | +| W25 | failed off-site run | „✗ Hiba … Utolsó hiba: offbox backup immich: exit status 1 … a pillanatképek száma még ismeretlen"; operator mail „A távoli mentés ismeretlen okból nem…"; household mail „A biztonsági mentés sikertelen! Kérjük, ellenőrizd a rendszert." | English and technical; says nothing of the apps that DID copy; the household mail gives no action | name the app in Hungarian and say „a többi alkalmazás rendben" + „holnap éjjel újra próbálja" | +| W26 | English dashboard (B6 #16) | monitoring page: „Belső SSD – rendszer és alkalmazások", „Az operációs rendszer és a telepített alkalmazások tárhelye.", „Belső lemez – sablonok és helyi mentések", „Távoli biztonsági mentés", „Titkosított, telephelyen kívüli biztonsági mentések." (`internal/web/agent_host_metrics_handler.go:91-95`); restore page „Megosztások" (`internal/backup/shares_payload.go:42`); a stored restore message stays Hungarian | untranslated in English mode | msg keys (the localisation slice) | +| W27 | an English BROWSER | the whole dashboard stays Hungarian with `en-US` — it follows the household's setting, not the browser | by design (a visitor switch exists for signed-out pages) | — (NOT-A-FINDING: the household's setting wins, `10-localisation.md`) | +| W28 | share restore | „Nem állítható vissza (a mappa nincs élő adatmeghajtón): beolvasas." | names a share („beolvasas") the household never created | say what it is (Paperless's scan folder?) or skip it silently | +| W29 | storage card after a plain decommission | „Kiváltva — Adatok átköltöztetve ide: . () Időpont: …" | the migration sentence shown with empty blanks when nothing was moved | „Leszerelve, adat nem volt rajta" | diff --git a/documentation/audits/rehearsal-2026-10-20/tools/nc_trick_check.sh b/documentation/audits/rehearsal-2026-10-20/tools/nc_trick_check.sh new file mode 100644 index 00000000..54dedff2 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/tools/nc_trick_check.sh @@ -0,0 +1,5 @@ +#!/bin/bash +for i in 1 2 3 4 5 6 7; do src=/root/data/immich/2015/$(ls /root/data/immich/2015 | sed -n "$((i % 15 + 1))p") + got=$(curl -sk -u "odon:$PW" "https://cloud.tester4.test/remote.php/dav/files/odon/Csalad/feltoltes_$i.jpg" -o - | sha256sum | cut -c1-12) + code=$(curl -sk -u "odon:$PW" -o /dev/null -w %{http_code} "https://cloud.tester4.test/remote.php/dav/files/odon/Csalad/feltoltes_$i.jpg") + echo "feltoltes_$i: $code $( [ "$got" = "$(sha256sum "$src" | cut -c1-12)" ] && echo SHA-MATCH || echo sha-differs)"; done diff --git a/documentation/audits/rehearsal-2026-10-20/tools/nc_trick_check2.sh b/documentation/audits/rehearsal-2026-10-20/tools/nc_trick_check2.sh new file mode 100644 index 00000000..2cf72eb6 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/tools/nc_trick_check2.sh @@ -0,0 +1,4 @@ +#!/bin/bash +for i in 1 2 3 4; do src=/root/data/immich/2015/$(ls /root/data/immich/2015 | sed -n "$((i % 15 + 1))p") + curl -sk -u "odon:$PW" "https://cloud.tester4.test/remote.php/dav/files/odon/Csalad/feltoltes_$i.jpg" -o /tmp/f$i + echo "feltoltes_$i: got $(stat -c %s /tmp/f$i) B $(sha256sum /tmp/f$i | cut -c1-12) | src $(basename $src) $(stat -c %s $src) B $(sha256sum $src | cut -c1-12) | head: $(head -c 4 /tmp/f$i | od -An -tx1 | tr -d ' ')"; rm -f /tmp/f$i; done diff --git a/documentation/audits/rehearsal-2026-10-20/tools/nc_trickle.sh b/documentation/audits/rehearsal-2026-10-20/tools/nc_trickle.sh new file mode 100755 index 00000000..82093eac --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/tools/nc_trickle.sh @@ -0,0 +1,3 @@ +#!/bin/bash +# nc_trickle.sh N — the family keeps uploading small photos to Nextcloud (one every 2 s), printing each status code +for i in $(seq 1 $1); do c=$(curl -sk -m 20 -u "odon:$PW" -T /root/data/immich/2015/$(ls /root/data/immich/2015 | sed -n "$((i % 15 + 1))p") -o /dev/null -w %{http_code} "https://cloud.tester4.test/remote.php/dav/files/odon/Csalad/feltoltes_$i.jpg"); echo "$(date -u +%T) put $i $c"; sleep 2; done diff --git a/documentation/audits/rehearsal-2026-10-20/tools/twoatonce.py b/documentation/audits/rehearsal-2026-10-20/tools/twoatonce.py new file mode 100644 index 00000000..c9539dcf --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/tools/twoatonce.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python3 +"""twoatonce.py — B6 #9: (a) restore + remove on the same app from two sessions at once; (b) a double-click restore. +Each act is the POST the page sends; the two of a pair start within ~50 ms (threads). Prints answers and the status.""" +import json, os, subprocess, sys, threading, time +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))); import ctl +APP = sys.argv[1] +def restore_post(): + out = subprocess.run(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", ctl.HOST, "-H", "Cookie: " + ctl._SESS, "-X", "POST", + "--data-urlencode", "_csrf=" + ctl._CSRF, "--data-urlencode", "stack_name=" + APP, + "--data-urlencode", "snapshot_id=helyi", ctl.BASE + "/backup/restore"], capture_output=True, text=True).stdout + return (out.splitlines()[0] if out else "?") + " " + str([l for l in out.splitlines() if l.lower().startswith("location")][:1]) +def run_pair(a, b, label): + res = {} + ta = threading.Thread(target=lambda: res.__setitem__("A", a())); tb = threading.Thread(target=lambda: res.__setitem__("B", b())) + t0 = time.time(); ta.start(); time.sleep(0.05); tb.start(); ta.join(); tb.join() + print(time.strftime("%H:%M:%SZ", time.gmtime()), label, "A:", res["A"], "| B:", res["B"], flush=True) + seen = [] + for i in range(40): + c, d = ctl.call("GET", "/api/backup/restore-status", timeout=30); dd = d.get("data") or {} + s = (dd.get("running"), (dd.get("last") or {}).get("ok"), (dd.get("last") or {}).get("message", "")[:140]) + if not seen or seen[-1][1] != s: seen.append((round(time.time() - t0, 1), s)) + if not dd.get("running") and time.time() - t0 > 10: break + time.sleep(2) + for x in seen: print(" ", x, flush=True) + c, d = ctl.call("GET", f"/api/stacks/{APP}"); print(" app:", (d.get("data") or {}).get("state"), "deployed:", (d.get("data") or {}).get("deployed"), flush=True) +ctl.login() +run_pair(restore_post, lambda: str(ctl.call("POST", f"/api/stacks/{APP}/remove", {"remove_hdd_data": False, "remove_backups": False})), "(a) restore + remove:") +time.sleep(10) +run_pair(restore_post, restore_post, "(b) double-click restore:") diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 79fc9231..501f531f 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -198,7 +198,7 @@ stopping line that lies. | **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt, not yet proven:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it, and the hub reads the same clock (agent `localapi/server.go:1178-1192`, `backup/runner.go:607-629`, hub `monitor/deadline.go:243`; R-973 cap counts future runs `backup_cap.go:61`); (b) the whole-guest start creeps ~5 min/night until a night is skipped silently (`server.go:1184`, `quiesce.go:150, 807-818`); (c) the whole-guest quiesce stops apps mid-restore or mid-update (`quiesce.go:213-275, 565`); (d) CONFIRMED in code: the off-site proof and integrity check are plain daily jobs outside the catch-up (`nightchain.go` header: the catch-up runs only the three backup legs), so they never run on a box that is off at 05:30–06:00, while the comment at `cmd/controller/main.go:1575-1577` says the opposite; (e) CONFIRMED in code: the hub's 05:00 check wants a dump „since Budapest midnight" (`hub/internal/monitor/deadline.go:416-419, 556-565`), so an evening backup time (which the banner suggests) raises a false `expected_dbdump_missed` error every morning — and on 2026-10-25 the check also runs at 04:00 (R-1183); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — design question; (g) spring forward 2027-03-28 fires dump and second copy together; (h) self-update busy check before the pull only (`updater.go:532-545, 764`); (i) a double-clicked restore shows failed while it runs (`web/handlers.go:1853-1874`). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`. | VERIFY | — | (a) is B6 #8's live test; (b) read the agent journal on demo-hp; the rest by unit tests | CC | | **R-1190** | Backup & restore | P3 | **Key and retention candidates from the data-loss hunt, not yet proven:** (a) a recovery ceremony while off-site is not configured seals a package with no repo password and supersedes the good one (`escrow_handlers.go:217`, agent `main.go:2975-2981`, hub `store.go:3392`); (b) the global „create your recovery code" bar links to the ceremony the backups page hides during a recovery offer (`escrow_banner.go:23-33` vs `backups_remote.html:144-160`); (c) after a reuse-path reinstall the new PBS key K is never escrowed and nothing compares fingerprints (`felhom-pbs-apply:99`, `escrow_confirm.go:182-205`); (d) whole-guest off-site copies run before any escrow exists (design question vs 07 §6.1's Tier-3 gate); (e) the escrow PUT is two statements; (f) the operator's off-site re-issue resets the provider password before saving it (hub `offsite.go:217-245`); (g) „Megőrzött adatok" can list a running app's live folder when its app.yaml is unreadable (`kept.go:220-235, 330-402`); (h) a unit restore can point an app back at an old drive and F5 then prunes the current drive's unit (`backup.go:1517-1574`). Evidence `audits/dataloss-hunt-2026-10-20/C1-keys-candidates.md`, `C1-delete-retention-candidates.md`. | VERIFY | — | Unit tests first for (a), (e), (g) | CC | -## Storage & devices — 9 rows (P3 3, P4 6) +## Storage & devices — 10 rows (P3 4, P4 6) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -207,6 +207,7 @@ stopping line that lies. | **R-1064** | Storage & devices | P4 | **Removing a storage path does not mention the cross-drive backup copies it holds.** demo-hp 2026-10-15: `/mnt/sizing-bench` removed through `/settings/storage/remove` (302, „Adattároló eltávolítva") while calibre-web's and paperless-ngx's `cross_drive.destination_path` pointed at it; the copies stay on disk, the records still name the removed path. The in-use check counts HDD_PATH only (`appsUsingPath`). | **OPEN** — filed 2026-10-15 | — | Read what tonight's tier-2 run (01:30) does with the two records; then: name the copies in the removal's confirmation (or count them as in use) | CC | | **R-330** | Storage & devices | P3 | **Disk health Phase 2 — the three SMART attributes the wire does not carry.** The failing drive's most telling counter was **187 `Reported_Uncorrect`**, sitting at normalized **1** against threshold **0** with a raw count of **1001** — one point from failing and structurally unable to get there. Also wanted: **199 `UDMA_CRC_Error_Count`** (cabling) and **188 `Command_Timeout`**. None are on the agent→controller wire today, so v0.215.0's ladder could not use them. Phase 2 also persists periodic SMART **samples** (the right home is `metrics.MetricsStore`, NOT the Phase-1 state file, which is one record per disk and must stay that way). **This is a declared WIRE change, so under the G-1 gate the hub must model the new fields in the SAME session** — that is precisely why it was kept out of Phase 1, where it would have turned a one-word severity fix into a three-repo change | **READY** — **2026-10-12: the wire half is CLOSED in this row** — released in agent v0.150.0 (boxes run 0.154.0), controller and hub decode 187/188/199. **Only open work:** the verdict slice (use the three counters in the disk verdict and the household's words) and persisting samples. **READY (M) — NEW 2026-08-14** **2026-10-06 night: the wire half BUILT on main (unreleased):** the agent sends SMART 187 `reported_uncorrect`, 188 `command_timeout` (raw, vendor-packed on some drives — carried as reported), 199 `udma_crc_errors` (pointer + omitempty: unknown is absent, never 0); the controller decodes them and the hub's host-report mirror models them (G-1 wire gate green; it convicted the agent half alone). **Carried only** — no verdict, banner, mail or alarm reads them (`TestR330_CountersChangeNoVerdictYet`); using them changes what a household is told and is the next slice, with persisting samples. Red-proofs `audits/night-burndown-2026-10-06/r330/`. Ships with agent v0.150.0 and the next controller and hub releases. | R-328 (closed) | The verdict slice: 187/188/199 into the disk verdict with tests; then persist samples | CC | | **R-332** | Storage & devices | P3 | **The new Hiba-from-counters path has never fired on real hardware.** v0.215.0's whole point is a verdict the product could not previously reach, and it is proven only against the committed fixture's values in unit tests (12 scenario groups, 11 of 12 red-proofs failing as required). The live validation on demo-hp proved the **negative** — three healthy disks still read Rendben across the deploy, no false alert — and the **severity wire** end to end, but no live disk has actually reached Hiba. **This is the honest gap and it must not be closed by pointing at the fixture tests**: the drive that produced the fixture is in DooPlex, which is Tier 2 and never a drill target, and the demo boxes are all-flash and healthy | **WATCHING — NEW 2026-08-14, NARROWED same day.** One item originally in this gap is now PROVEN LIVE: the **persisted state surviving a controller restart**. The v0.215.0→v0.216.0 redeploy destroyed and rebuilt the container, and the new one read back a `changed_at` written by the PREVIOUS version (`2026-08-14T07:23:14.640216851Z`, still intact at 09:31:35Z) instead of re-baselining — Scenario L on real hardware, not just the production-path unit test. **What remains unproven is the verdict itself, plus the stronger restart half: an already-ALERTED disk not re-alerting** | a real degrading disk, or an injection harness | **Closing condition:** a live disk reaching Hiba from counters, OR a deliberate injection through the REAL pipeline (agent `/disks` → controller check → hub event), not a hand-set verdict | CC | +| **R-1196** | Storage & devices | P3 | **When a data drive is pulled, the files the household uploaded in the last seconds before it are broken (0 bytes) or gone, although each upload was confirmed, and nothing tells the household.** Rehearsal 2026-10-20 (`audits/rehearsal-2026-10-20/B6-failures/s4-drive-pulled/`): uploads every 2 s to Nextcloud, the drive pulled at 17:56:26 → the box handled the drive itself well (absent in 42 s, apps stopped with a clear message, back by itself 62 s after the re-plug) — but uploads 1–6 (201 at 17:56:14–27) came back as four 0-byte files and two missing ones. The page's „Leválasztva" message and the „újra csatlakoztatva" event do not say that the last uploads may be damaged. The cause is the filesystem's delayed write (not a Felhom defect as such); the gap is the silence. | OPEN | — | Add one sentence to the drive-disconnected message („az utolsó percben mentett fájlok sérülhettek — ellenőrizd őket"); consider `commit=`/`sync` mount options for data drives | CC | | **R-331** | Storage & devices | P4 | **Disk health Phase 3 — growth-rate detection, and retiring the static 64.** The v0.215.0 count backstop (64 unreadable sectors → Hiba) is **a judgement from ONE drive**: the observed benign excursion peaked at 16 and cleared inside an hour, and the terminal run passed 64 at 13 Aug 11:28 and never came back. It is deliberately a backstop BEHIND the sustain rule, not the primary signal, but it is still a magic number tuned on a single sample and it will be wrong for some drive. With Phase 2's history the box can ask the question that actually matters — *is this count climbing, and how fast* — which distinguishes a drive with eight stable aging sectors from one adding forty a day, something no static threshold can do. Revisit 64 when that exists | **READY (M) — NEW 2026-08-14** | R-330 | Growth-rate rule over persisted samples; re-derive or delete the static 64 | CC | | **R-1076** | Storage & devices | P4 | **A cheap SSD reports a vendor-less WWN, so a WWN-first disk key may not be unique.** MEASURED 2026-10-15 on the N100's system SSD (AirDisk 512 GB): `ID_WWN=0x5000000123456f1f` — NAA-5 with OUI `000000`, a pattern that looks like a placeholder; its serial `QDF922W009654S30EX` looks real. The agent's wipe binding prefers `wwn-` links (A/internal/storage/durable_device.go:32). Not known whether two units share the WWN. Evidence: `audits/storage-identity-2026-10-15/B-n100.txt`. | OPEN | — | Check a second unit of the same model (or vendor data); if shared, make the agent skip OUI-000000 WWNs (the design already keys on model+serial, `architecture/16-storage-physical-identity.md` §3). | CC | | **R-1077** | Storage & devices | P4 | **No USB enclosure that hides or fakes a disk's serial has been measured.** The storage-identity spike (2026-10-15) had none at hand and found none in earlier evidence; the design's `usb_bridge` unknown rule (`architecture/16-storage-physical-identity.md` §3) is therefore an assumption. Cheap single- and dual-bay bridges are known to report one bridge serial for any disk. Evidence: `audits/storage-identity-2026-10-15/README.md` case 4b. | OPEN | — | Buy or borrow one cheap USB enclosure (single and dual bay), run `tools/physdisk_probe.py` against it, and set the rule from what it reports. | Operator (a device) then CC |