admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Rehearsal: B6 #12 DNS dead an hour (as expected; household told nothing); R-1199 (P2 VERIFY) the household's ticked alarms reached only the operator (1 of 9 mails); R-1189 (b) demo-hp journal: three skipped nights
gates / gates (push) Successful in 13m31s
Details

Browse Source
This commit is contained in:
authentik Default Admin
2026-10-20 21:50:47 +02:00
parent 12090df2d9
commit 92de2d324e
6 changed files with 220 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/rehearsal-2026-10-20/B4-use/health-log.txt
+65
View File
@@ -152,3 +152,68 @@
2026-10-20T18:41:23 health=200 0.012298 diskread=30GB
2026-10-20T18:42:25 health=200 0.011961 diskread=34GB
2026-10-20T18:43:27 health=200 0.011279 diskread=35GB
2026-10-20T18:44:28 health=200 0.011642 diskread=35GB
2026-10-20T18:45:30 health=200 0.009993 diskread=35GB
2026-10-20T18:46:31 health=200 0.009978 diskread=35GB
2026-10-20T18:47:33 health=200 0.010907 diskread=35GB
2026-10-20T18:48:34 health=200 0.011536 diskread=35GB
2026-10-20T18:49:35 health=200 0.010283 diskread=35GB
2026-10-20T18:50:37 health=200 0.010350 diskread=35GB
2026-10-20T18:51:38 health=200 0.013428 diskread=35GB
2026-10-20T18:52:40 health=200 0.010571 diskread=35GB
2026-10-20T18:53:42 health=200 0.010260 diskread=35GB
2026-10-20T18:54:43 health=200 0.010452 diskread=35GB
2026-10-20T18:55:44 health=200 0.011272 diskread=35GB
2026-10-20T18:56:46 health=200 0.013617 diskread=35GB
2026-10-20T18:57:47 health=200 0.011142 diskread=35GB
2026-10-20T18:58:49 health=200 0.009910 diskread=35GB
2026-10-20T18:59:50 health=200 0.009594 diskread=35GB
2026-10-20T19:00:52 health=200 0.010854 diskread=35GB
2026-10-20T19:01:54 health=200 0.010374 diskread=35GB
2026-10-20T19:02:55 health=200 0.012541 diskread=35GB
2026-10-20T19:03:57 health=200 0.010744 diskread=35GB
2026-10-20T19:04:59 health=200 0.010581 diskread=35GB
2026-10-20T19:06:00 health=200 0.012816 diskread=35GB
2026-10-20T19:07:02 health=200 0.009981 diskread=35GB
2026-10-20T19:08:03 health=200 0.010464 diskread=35GB
2026-10-20T19:09:05 health=200 0.011231 diskread=35GB
2026-10-20T19:10:07 health=200 0.009675 diskread=35GB
2026-10-20T19:11:08 health=200 0.011965 diskread=35GB
2026-10-20T19:12:10 health=200 0.010637 diskread=35GB
2026-10-20T19:13:11 health=200 0.010037 diskread=35GB
2026-10-20T19:14:13 health=200 0.014316 diskread=35GB
2026-10-20T19:15:14 health=200 0.010666 diskread=35GB
2026-10-20T19:16:16 health=200 0.010275 diskread=35GB
2026-10-20T19:17:18 health=200 0.010881 diskread=35GB
2026-10-20T19:18:19 health=200 0.011001 diskread=35GB
2026-10-20T19:19:21 health=200 0.011536 diskread=35GB
2026-10-20T19:20:23 health=200 0.010414 diskread=35GB
2026-10-20T19:21:24 health=200 0.013076 diskread=35GB
2026-10-20T19:22:26 health=200 0.010698 diskread=35GB
2026-10-20T19:23:28 health=200 0.017508 diskread=35GB
2026-10-20T19:24:29 health=200 0.010675 diskread=36GB
2026-10-20T19:25:31 health=200 0.010865 diskread=36GB
2026-10-20T19:26:32 health=200 0.010450 diskread=36GB
2026-10-20T19:27:34 health=200 0.011980 diskread=36GB
2026-10-20T19:28:36 health=200 0.010846 diskread=36GB
2026-10-20T19:29:37 health=200 0.010569 diskread=36GB
2026-10-20T19:30:39 health=200 0.009678 diskread=36GB
2026-10-20T19:31:40 health=200 0.009699 diskread=36GB
2026-10-20T19:32:42 health=200 0.010025 diskread=36GB
2026-10-20T19:33:43 health=200 0.011595 diskread=36GB
2026-10-20T19:34:45 health=200 0.010119 diskread=36GB
2026-10-20T19:35:47 health=200 0.010214 diskread=36GB
2026-10-20T19:36:48 health=200 0.018419 diskread=36GB
2026-10-20T19:37:50 health=200 0.010102 diskread=36GB
2026-10-20T19:38:51 health=200 0.009866 diskread=36GB
2026-10-20T19:39:53 health=200 0.010483 diskread=36GB
2026-10-20T19:40:55 health=200 0.012985 diskread=36GB
2026-10-20T19:41:57 health=200 0.011569 diskread=36GB
2026-10-20T19:42:58 health=200 0.009786 diskread=36GB
2026-10-20T19:44:00 health=200 0.010759 diskread=36GB
2026-10-20T19:45:01 health=200 0.009923 diskread=36GB
2026-10-20T19:46:03 health=200 0.010965 diskread=36GB
2026-10-20T19:47:05 health=200 0.012170 diskread=36GB
2026-10-20T19:48:06 health=200 0.010029 diskread=36GB
2026-10-20T19:49:08 health=200 0.009547 diskread=36GB
2026-10-20T19:50:09 health=200 0.011250 diskread=36GB
documentation/audits/rehearsal-2026-10-20/B4-use/memwatch.txt
+136 -1
View File
@@ -285,4 +285,139 @@
18:41:30Z used=1590MB avail=2506MB load1=2.74 cpu=27% immich-server:584/3072MB 0% | paperless-webserver:458/1280MB 0% | mealie:319/1000MB 19% | immich-postgres:156/768MB 0% | traefik:151/4096MB 0% | immich-machine-learning:122/2048MB 0% | bookstack-db:121/256MB 0% | felhom-controller:117/4096MB 42%
18:42:00Z used=3288MB avail=808MB load1=2.93 cpu=70% immich-server:584/3072MB 0% | paperless-webserver:458/1280MB 0% | mealie:319/1000MB 19% | immich-postgres:156/768MB 0% | traefik:151/4096MB 0% | immich-machine-learning:122/2048MB 0% | bookstack-db:121/256MB 0% | felhom-controller:117/4096MB 42%
18:42:31Z used=3638MB avail=458MB load1=2.55 cpu=35% immich-server:871/3072MB 7% | paperless-webserver:316/1280MB 0% | mealie:195/1000MB 0% | immich-machine-learning:167/2048MB 15% | immich-postgres:91/768MB 0% | bookstack-db:81/256MB 4% | felhom-controller:77/4096MB 0% | bookstack:61/512MB 0%
18:43:01Z used=3638MB avail=458MB load1=2.65 cpu=38% immich-server:871/3072MB 7% | paperless-webserver:316/1280MB 0% | mealie:195/1000MB 0% | immich-machine-learning:167/2048MB 15% | immich-postgres:91/768MB 0% | bookstack-db:81/256MB 4% | felhom-
18:43:01Z used=3638MB avail=458MB load1=2.65 cpu=38% immich-server:871/3072MB 7% | paperless-webserver:316/1280MB 0% | mealie:195/1000MB 0% | immich-machine-learning:167/2048MB 15% | immich-postgres:91/768MB 0% | bookstack-db:81/256MB 4% | felhom-controller:77/4096MB 0% | bookstack:61/512MB 0%
18:43:31Z used=3638MB avail=458MB load1=2.71 cpu=18% immich-server:913/3072MB 0% | paperless-webserver:322/1280MB 273% | mealie:198/1000MB 0% | immich-machine-learning:172/2048MB 4% | nextcloud:171/1024MB 0% | nextcloud-db:129/512MB 4% | immich-postgres:93/768MB 6% | felhom-controller:91/4096MB 1%
18:44:01Z used=3638MB avail=458MB load1=2.57 cpu=7% immich-server:913/3072MB 0% | paperless-webserver:322/1280MB 273% | mealie:198/1000MB 0% | immich-machine-learning:172/2048MB 4% | nextcloud:171/1024MB 0% | nextcloud-db:129/512MB 4% | immich-postgres:93/768MB 6% | felhom-controller:91/4096MB 1%
18:44:31Z used=3638MB avail=458MB load1=1.68 cpu=18% immich-server:703/3072MB 0% | paperless-webserver:553/1280MB 0% | mealie:189/1000MB 0% | immich-machine-learning:168/2048MB 0% | nextcloud-db:117/512MB 5% | immich-postgres:93/768MB 0% | felhom-controller:93/4096MB 0% | bookstack-db:84/256MB 0%
18:45:01Z used=3638MB avail=458MB load1=1.17 cpu=7% immich-server:703/3072MB 0% | paperless-webserver:553/1280MB 0% | mealie:189/1000MB 0% | immich-machine-learning:168/2048MB 0% | nextcloud-db:117/512MB 5% | immich-postgres:93/768MB 0% | felhom-controller:93/4096MB 0% | bookstack-db:84/256MB 0%
18:45:31Z used=3638MB avail=458MB load1=0.95 cpu=20% immich-server:704/3072MB 0% | paperless-webserver:553/1280MB 0% | mealie:189/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | immich-postgres:93/768MB 0% | felhom-controller:93/4096MB 0% | bookstack-db:84/256MB 0%
18:46:01Z used=3638MB avail=458MB load1=0.57 cpu=7% immich-server:704/3072MB 0% | paperless-webserver:553/1280MB 0% | mealie:189/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | immich-postgres:93/768MB 0% | felhom-controller:93/4096MB 0% | bookstack-db:84/256MB 0%
18:46:31Z used=3638MB avail=458MB load1=0.48 cpu=18% immich-server:698/3072MB 0% | paperless-webserver:553/1280MB 0% | mealie:189/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | felhom-controller:95/4096MB 27% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:47:02Z used=3638MB avail=458MB load1=0.45 cpu=8% immich-server:698/3072MB 0% | paperless-webserver:553/1280MB 0% | mealie:189/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | felhom-controller:95/4096MB 27% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:47:32Z used=3638MB avail=458MB load1=0.4 cpu=19% immich-server:699/3072MB 0% | paperless-webserver:553/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | felhom-controller:96/4096MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:48:02Z used=3638MB avail=458MB load1=0.3 cpu=9% immich-server:699/3072MB 0% | paperless-webserver:553/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | felhom-controller:96/4096MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:48:32Z used=3638MB avail=458MB load1=0.31 cpu=19% immich-server:700/3072MB 0% | paperless-webserver:553/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | immich-postgres:93/768MB 0% | felhom-controller:93/4096MB 1% | bookstack-db:84/256MB 0%
18:49:02Z used=3638MB avail=458MB load1=0.34 cpu=8% immich-server:700/3072MB 0% | paperless-webserver:553/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | immich-postgres:93/768MB 0% | felhom-controller:93/4096MB 1% | bookstack-db:84/256MB 0%
18:49:32Z used=3638MB avail=458MB load1=0.35 cpu=44% immich-server:700/3072MB 0% | paperless-webserver:553/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | felhom-controller:94/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:50:02Z used=3638MB avail=458MB load1=0.29 cpu=9% immich-server:700/3072MB 0% | paperless-webserver:553/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | felhom-controller:94/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:50:32Z used=3638MB avail=458MB load1=0.38 cpu=17% immich-server:701/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | felhom-controller:94/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:51:02Z used=3638MB avail=458MB load1=0.49 cpu=9% immich-server:701/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | felhom-controller:94/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:51:33Z used=3638MB avail=458MB load1=0.42 cpu=18% immich-server:702/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | felhom-controller:119/4096MB 70% | nextcloud-db:116/512MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:52:03Z used=3638MB avail=458MB load1=0.26 cpu=9% immich-server:702/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | felhom-controller:119/4096MB 70% | nextcloud-db:116/512MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:52:33Z used=3638MB avail=458MB load1=0.23 cpu=18% immich-server:705/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | felhom-controller:106/4096MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:53:03Z used=3638MB avail=458MB load1=0.4 cpu=8% immich-server:705/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | felhom-controller:106/4096MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:53:33Z used=3638MB avail=458MB load1=0.44 cpu=17% immich-server:706/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | felhom-controller:106/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:54:03Z used=3638MB avail=458MB load1=0.41 cpu=8% immich-server:706/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:116/512MB 0% | felhom-controller:106/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:54:33Z used=3638MB avail=458MB load1=0.32 cpu=20% immich-server:707/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 16% | immich-machine-learning:168/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:55:03Z used=3638MB avail=458MB load1=0.38 cpu=9% immich-server:707/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 16% | immich-machine-learning:168/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:55:33Z used=3638MB avail=458MB load1=0.3 cpu=20% immich-server:703/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 18% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:56:04Z used=3638MB avail=458MB load1=0.24 cpu=8% immich-server:703/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 18% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:56:34Z used=3638MB avail=458MB load1=0.35 cpu=17% immich-server:702/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 16% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 22% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:57:04Z used=3638MB avail=458MB load1=0.39 cpu=7% immich-server:702/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 16% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 22% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:57:34Z used=3638MB avail=458MB load1=0.39 cpu=16% immich-server:702/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:58:04Z used=3638MB avail=458MB load1=0.23 cpu=7% immich-server:702/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:58:34Z used=3638MB avail=458MB load1=0.3 cpu=17% immich-server:705/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:59:04Z used=3638MB avail=458MB load1=0.74 cpu=8% immich-server:705/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
18:59:34Z used=3638MB avail=458MB load1=0.58 cpu=17% immich-server:706/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 1% | immich-postgres:94/768MB 8% | bookstack-db:84/256MB 8%
19:00:04Z used=3638MB avail=458MB load1=0.49 cpu=11% immich-server:706/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 1% | immich-postgres:94/768MB 8% | bookstack-db:84/256MB 8%
19:00:35Z used=3638MB avail=458MB load1=0.6 cpu=19% immich-server:706/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | immich-postgres:94/768MB 10% | bookstack-db:85/256MB 6%
19:01:05Z used=3638MB avail=458MB load1=0.41 cpu=10% immich-server:706/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | immich-postgres:94/768MB 10% | bookstack-db:85/256MB 6%
19:01:35Z used=3638MB avail=458MB load1=0.86 cpu=5% immich-server:704/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 11% | immich-postgres:93/768MB 0% | bookstack-db:85/256MB 5%
19:02:05Z used=3638MB avail=458MB load1=0.52 cpu=7% immich-server:704/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 11% | immich-postgres:93/768MB 0% | bookstack-db:85/256MB 5%
19:02:35Z used=3638MB avail=458MB load1=0.61 cpu=5% immich-server:704/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:03:05Z used=3638MB avail=458MB load1=0.55 cpu=7% immich-server:704/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:03:35Z used=3638MB avail=458MB load1=0.4 cpu=4% immich-server:704/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:105/4096MB 15% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:04:05Z used=3638MB avail=458MB load1=0.36 cpu=7% immich-server:704/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:105/4096MB 15% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:04:35Z used=3638MB avail=458MB load1=0.54 cpu=4% immich-server:706/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 5% | felhom-controller:107/4096MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:05:06Z used=3638MB avail=458MB load1=0.52 cpu=11% immich-server:706/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 5% | felhom-controller:107/4096MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:05:36Z used=3638MB avail=458MB load1=0.45 cpu=4% immich-server:706/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 6% | felhom-controller:107/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:06:06Z used=3638MB avail=458MB load1=0.27 cpu=11% immich-server:706/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 6% | felhom-controller:107/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:06:36Z used=3638MB avail=458MB load1=0.35 cpu=5% immich-server:706/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 6% | felhom-controller:115/4096MB 56% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:07:06Z used=3638MB avail=458MB load1=0.37 cpu=7% immich-server:706/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 6% | felhom-controller:115/4096MB 56% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:07:36Z used=3638MB avail=458MB load1=0.44 cpu=6% immich-server:706/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:08:06Z used=3638MB avail=458MB load1=0.44 cpu=8% immich-server:706/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 0% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:08:36Z used=3638MB avail=458MB load1=0.51 cpu=5% immich-server:707/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:85/256MB 0%
19:09:06Z used=3638MB avail=458MB load1=0.68 cpu=8% immich-server:707/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:85/256MB 0%
19:09:37Z used=3638MB avail=458MB load1=0.48 cpu=6% immich-server:707/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:85/256MB 0%
19:10:07Z used=3638MB avail=458MB load1=0.37 cpu=8% immich-server:707/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:85/256MB 0%
19:10:37Z used=3638MB avail=458MB load1=0.4 cpu=5% immich-server:707/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:105/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:85/256MB 0%
19:11:07Z used=3638MB avail=458MB load1=0.31 cpu=12% immich-server:707/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:105/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:85/256MB 0%
19:11:37Z used=3638MB avail=458MB load1=0.31 cpu=5% immich-server:707/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 17% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:12:07Z used=3638MB avail=458MB load1=0.31 cpu=7% immich-server:707/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 17% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:12:37Z used=3638MB avail=458MB load1=0.46 cpu=5% immich-server:708/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:85/256MB 0%
19:13:07Z used=3638MB avail=458MB load1=0.33 cpu=7% immich-server:708/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:85/256MB 0%
19:13:37Z used=3638MB avail=458MB load1=0.4 cpu=6% immich-server:708/3072MB 1% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:14:08Z used=3638MB avail=458MB load1=0.66 cpu=7% immich-server:708/3072MB 1% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 1% | immich-postgres:93/768MB 0% | bookstack-db:84/256MB 0%
19:14:38Z used=3638MB avail=458MB load1=0.6 cpu=5% immich-server:708/3072MB 1% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 0% | immich-postgres:87/768MB 0% | bookstack-db:85/256MB 0%
19:15:08Z used=3638MB avail=458MB load1=0.36 cpu=6% immich-server:708/3072MB 1% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 0% | immich-postgres:87/768MB 0% | bookstack-db:85/256MB 0%
19:15:38Z used=3638MB avail=458MB load1=0.39 cpu=5% immich-server:708/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:105/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:84/768MB 0%
19:16:08Z used=3638MB avail=458MB load1=0.44 cpu=11% immich-server:708/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:105/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:84/768MB 0%
19:16:38Z used=3638MB avail=458MB load1=0.38 cpu=5% immich-server:711/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:105/4096MB 11% | immich-postgres:85/768MB 7% | bookstack-db:84/256MB 0%
19:17:08Z used=3638MB avail=458MB load1=0.29 cpu=6% immich-server:711/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:105/4096MB 11% | immich-postgres:85/768MB 7% | bookstack-db:84/256MB 0%
19:17:38Z used=3638MB avail=458MB load1=0.3 cpu=5% immich-server:709/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | immich-postgres:84/768MB 8% | bookstack-db:84/256MB 0%
19:18:08Z used=3638MB avail=458MB load1=0.3 cpu=6% immich-server:709/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | immich-postgres:84/768MB 8% | bookstack-db:84/256MB 0%
19:18:39Z used=3638MB avail=458MB load1=0.32 cpu=4% immich-server:710/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:84/768MB 0%
19:19:09Z used=3638MB avail=458MB load1=0.43 cpu=7% immich-server:710/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:84/768MB 0%
19:19:39Z used=3638MB avail=458MB load1=0.4 cpu=5% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:84/768MB 0%
19:20:09Z used=3638MB avail=458MB load1=0.42 cpu=6% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:84/768MB 0%
19:20:39Z used=3638MB avail=458MB load1=0.39 cpu=5% immich-server:709/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:81/768MB 0%
19:21:09Z used=3638MB avail=458MB load1=0.34 cpu=11% immich-server:709/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:81/768MB 0%
19:21:39Z used=3638MB avail=458MB load1=0.47 cpu=5% immich-server:709/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | felhom-controller:118/4096MB 69% | nextcloud-db:117/512MB 0% | bookstack-db:84/256MB 5% | immich-postgres:81/768MB 0%
19:22:09Z used=3638MB avail=458MB load1=0.49 cpu=6% immich-server:709/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | felhom-controller:118/4096MB 69% | nextcloud-db:117/512MB 0% | bookstack-db:84/256MB 5% | immich-postgres:81/768MB 0%
19:22:39Z used=3638MB avail=458MB load1=0.36 cpu=5% immich-server:709/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:168/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:85/256MB 6% | immich-postgres:81/768MB 0%
19:23:10Z used=3638MB avail=458MB load1=0.35 cpu=6% immich-server:709/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:168/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:85/256MB 6% | immich-postgres:81/768MB 0%
19:23:40Z used=3638MB avail=458MB load1=0.41 cpu=4% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 35% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | bookstack-db:85/256MB 6% | immich-postgres:81/768MB 0%
19:24:10Z used=3638MB avail=458MB load1=0.32 cpu=7% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 35% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | bookstack-db:85/256MB 6% | immich-postgres:81/768MB 0%
19:24:40Z used=3638MB avail=458MB load1=0.26 cpu=6% immich-server:710/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:81/768MB 0%
19:25:10Z used=3638MB avail=458MB load1=0.21 cpu=6% immich-server:710/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:81/768MB 0%
19:25:40Z used=3638MB avail=458MB load1=0.34 cpu=5% immich-server:710/3072MB 1% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:81/768MB 0%
19:26:10Z used=3638MB avail=458MB load1=0.27 cpu=11% immich-server:710/3072MB 1% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:81/768MB 0%
19:26:40Z used=3638MB avail=458MB load1=0.23 cpu=5% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 2% | felhom-controller:106/4096MB 6% | bookstack-db:84/256MB 0% | immich-postgres:81/768MB 0%
19:27:10Z used=3638MB avail=458MB load1=0.17 cpu=3% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 2% | felhom-controller:106/4096MB 6% | bookstack-db:84/256MB 0% | immich-postgres:81/768MB 0%
19:27:41Z used=3638MB avail=458MB load1=0.3 cpu=3% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 6% | felhom-controller:106/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:28:11Z used=3638MB avail=458MB load1=0.18 cpu=4% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 6% | felhom-controller:106/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:28:41Z used=3638MB avail=458MB load1=0.35 cpu=3% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 5% | felhom-controller:107/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:29:11Z used=3638MB avail=458MB load1=0.34 cpu=4% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 5% | felhom-controller:107/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:29:41Z used=3638MB avail=458MB load1=0.39 cpu=3% immich-server:710/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:30:11Z used=3638MB avail=458MB load1=0.36 cpu=5% immich-server:710/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:30:41Z used=3638MB avail=458MB load1=0.28 cpu=4% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:31:11Z used=3638MB avail=458MB load1=0.25 cpu=5% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:31:42Z used=3638MB avail=458MB load1=0.46 cpu=3% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 16% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:32:12Z used=3638MB avail=458MB load1=0.27 cpu=4% immich-server:710/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 16% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:32:42Z used=3638MB avail=458MB load1=0.35 cpu=5% immich-server:711/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 12% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:33:12Z used=3638MB avail=458MB load1=0.29 cpu=4% immich-server:711/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 12% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:33:42Z used=3638MB avail=458MB load1=0.4 cpu=4% immich-server:711/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 10% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 9%
19:34:12Z used=3638MB avail=458MB load1=0.57 cpu=4% immich-server:711/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 10% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 9%
19:34:42Z used=3638MB avail=458MB load1=0.47 cpu=4% immich-server:711/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 11% | bookstack-db:84/256MB 0% | immich-postgres:78/768MB 9%
19:35:12Z used=3638MB avail=458MB load1=0.49 cpu=4% immich-server:711/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 11% | bookstack-db:84/256MB 0% | immich-postgres:78/768MB 9%
19:35:42Z used=3638MB avail=458MB load1=0.55 cpu=4% immich-server:711/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 11% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:36:13Z used=3638MB avail=458MB load1=0.33 cpu=5% immich-server:711/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | felhom-controller:120/4096MB 56% | nextcloud-db:117/512MB 0% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:36:43Z used=3638MB avail=458MB load1=0.5 cpu=4% immich-server:711/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | felhom-controller:120/4096MB 56% | nextcloud-db:117/512MB 0% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:37:13Z used=3638MB avail=458MB load1=0.51 cpu=5% immich-server:711/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 5% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:37:43Z used=3638MB avail=458MB load1=0.68 cpu=3% immich-server:711/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 5% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:38:13Z used=3638MB avail=458MB load1=0.55 cpu=4% immich-server:711/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 5% | bookstack-db:84/256MB 0% | immich-postgres:75/768MB 0%
19:38:43Z used=3638MB avail=458MB load1=0.46 cpu=4% immich-server:711/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 5% | bookstack-db:84/256MB 0% | immich-postgres:75/768MB 0%
19:39:13Z used=3638MB avail=458MB load1=0.47 cpu=5% immich-server:711/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 5% | bookstack-db:84/256MB 0% | immich-postgres:75/768MB 0%
19:39:43Z used=3638MB avail=458MB load1=0.54 cpu=4% immich-server:711/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 5% | bookstack-db:84/256MB 0% | immich-postgres:75/768MB 0%
19:40:13Z used=3638MB avail=458MB load1=0.46 cpu=4% immich-server:717/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:40:44Z used=3638MB avail=458MB load1=0.55 cpu=4% immich-server:717/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:41:14Z used=3638MB avail=458MB load1=0.67 cpu=4% immich-server:717/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 17% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:41:44Z used=3638MB avail=458MB load1=0.59 cpu=4% immich-server:717/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:108/4096MB 17% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:42:14Z used=3638MB avail=458MB load1=0.62 cpu=5% immich-server:717/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:42:44Z used=3638MB avail=458MB load1=0.64 cpu=4% immich-server:717/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | bookstack-db:84/256MB 0% | immich-postgres:77/768MB 0%
19:43:14Z used=3638MB avail=458MB load1=0.7 cpu=4% immich-server:717/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:85/256MB 6% | immich-postgres:77/768MB 0%
19:43:44Z used=3638MB avail=458MB load1=0.48 cpu=4% immich-server:717/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:85/256MB 6% | immich-postgres:77/768MB 0%
19:44:14Z used=3638MB avail=458MB load1=0.49 cpu=5% immich-server:717/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:85/256MB 6% | immich-postgres:77/768MB 0%
19:44:44Z used=3638MB avail=458MB load1=0.36 cpu=3% immich-server:717/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:85/256MB 6% | immich-postgres:77/768MB 0%
19:45:14Z used=3638MB avail=458MB load1=0.5 cpu=4% immich-server:717/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 0% | bookstack-db:85/256MB 6% | immich-postgres:77/768MB 0%
19:45:45Z used=3638MB avail=458MB load1=0.49 cpu=4% immich-server:717/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 0% | bookstack-db:85/256MB 6% | immich-postgres:77/768MB 0%
19:46:15Z used=3638MB avail=458MB load1=0.3 cpu=5% immich-server:717/3072MB 1% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 15% | bookstack-db:85/256MB 0% | immich-postgres:77/768MB 0%
19:46:45Z used=3638MB avail=458MB load1=0.55 cpu=3% immich-server:717/3072MB 1% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 15% | bookstack-db:85/256MB 0% | immich-postgres:77/768MB 0%
19:47:15Z used=3638MB avail=458MB load1=0.7 cpu=4% immich-server:718/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | bookstack-db:85/256MB 0% | immich-postgres:77/768MB 0%
19:47:45Z used=3638MB avail=458MB load1=0.56 cpu=3% immich-server:718/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:106/4096MB 1% | bookstack-db:85/256MB 0% | immich-postgres:77/768MB 0%
19:48:15Z used=3638MB avail=458MB load1=0.59 cpu=4% immich-server:717/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:85/256MB 0% | immich-postgres:77/768MB 0%
19:48:45Z used=3638MB avail=458MB load1=1.23 cpu=3% immich-server:717/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:199/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 0% | felhom-controller:107/4096MB 1% | bookstack-db:85/256MB 0% | immich-postgres:77/768MB 0%
19:49:15Z used=3638MB avail=458MB load1=1.09 cpu=4% immich-server:718/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 6% | felhom-controller:108/4096MB 1% | bookstack-db:85/256MB 0% | immich-postgres:77/768MB 0%
19:49:45Z used=3638MB avail=458MB load1=1.02 cpu=4% immich-server:718/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 6% | felhom-controller:108/4096MB 1% | bookstack-db:85/256MB 0% | immich-postgres:77/768MB 0%
19:50:16Z used=3638MB avail=458MB load1=0.62 cpu=3% immich-server:717/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 6% | felhom-controller:107/4096MB 1% | bookstack-db:85/256MB 0% | immich-postgres:77/768MB 8%
19:50:46Z used=3638MB avail=458MB load1=0.63 cpu=3% immich-server:717/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:167/2048MB 0% | nextcloud-db:117/512MB 6% | felhom-controller:107/4096MB 1% | booksta
documentation/audits/rehearsal-2026-10-20/B6-failures/notifications-defaults.txt
+2
View File
@@ -0,0 +1,2 @@
checked boxes: ['group_machine', 'group_handled', 'event_storage_disconnected', 'event_storage_reconnected', 'event_disk_warning', 'event_disk_critical', 'event_offbox_enlarge_blocked', 'event_app_stopped_unhealthy', 'event_app_update_held', 'event_app_update_undone', 'event_backup_failed', 'event_db_dump_failed', 'event_backup_integrity_failed', 'event_crossdrive_failed', 'event_expected_backup_missed', 'event_expected_dbdump_missed', 'event_node_down', 'event_health_critical', 'event_health_recovered']
documentation/audits/rehearsal-2026-10-20/B6-failures/s12-dns-dead/02-dns-on.txt
+3
View File
@@ -0,0 +1,3 @@
packets 216
packets 0
DNS back 19:43:07
documentation/audits/rehearsal-2026-10-20/B6-failures/s12-dns-dead/README.md
+11
View File
@@ -0,0 +1,11 @@
# B6 #12 — DNS answers nothing for an hour (2026-10-20 18:43:21–19:43:07 UTC) — AS EXPECTED
Method: nftables on demo-hp dropped UDP/TCP 53 coming from the VM's tap (positive control: the drop counter rose — 216
lookups dropped in the hour; the box caches names, so lookups are rare).
- The dashboard over the LAN kept working (health 200 throughout).
- The hub: `host_stale` 19:20 and `node_stale` 19:22 to the operator (≈ 45 min after the last report); hub pages
STALE / WARN.
- DNS back 19:43:07 → the host ONLINE at 19:49:35 (6 min); the controller line followed later.
- The household was told nothing (R-873's „at most once per 7 days" liveness rule and R-1199).
- No data harm seen.
documentation/backlog/OPEN-ITEMS.md
+3 -2
View File
@@ -196,7 +196,7 @@ stopping line that lies.
| **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC |
| **R-1187** | Backup & restore | P3 | **`.fab` export/import candidates from the data-loss hunt, not yet proven:** (a) an export silently drops a drive folder that holds a symlink (`appexport/fabplan.go:292`, `export.go:714`); (b) the import never sets file owners and drops symlinks (`restore.go:1157-1176`); (c) import into a STOPPED app merges into the existing volumes, and the default export (stop ticked) carries no SQL dump (`restore.go:361-374`, `export.go:262,366`, `appbackup/dbdump.go:142`); (d) no concurrency guard, no undo, no record (`restore.go:181-186`, `web/handler_export.go:312-353`); (e) PROVEN and split out as R-1191 (the import drops the family gate); (f) it trusts the bundle's HDD_PATH (`restore.go:589-631`); (g) the DB import runs psql without ON_ERROR_STOP and misses postgis (`restore.go:1052-1053, 928-937`). Read by reviewer 1; main session spot-checked (a); (e) proven live → R-1191. Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Prove (c) first on 9202 with a throwaway app; split into rows only what reproduces | CC |
| **R-1188** | Backup & restore | P3 | **Restore-path candidates from the data-loss hunt, not yet proven (one confirmed in code):** (a) PROVEN LIVE → R-1195 (a power cut mid-restore restarts the app on the backup's data; the undo copy is not used); (b) a failed `volume rm` is ignored before the untar (`restore.go:218`); (c) PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-R12-leftover-temp/`): an interrupted second-drive whole restore leaves `<file>.felhom-restoring`, and EVERY retry fails „fájlmásolás sikertelen: … file exists" with the file never back (`tier2_whole.go:86-87`, O_EXCL); (d) the full off-site download has a 30-min cap per press and the household is not told to retry (`web/offbox_handlers.go:455`); (e) place and full restore re-read the LATEST snapshot instead of the downloaded one (`offbox_restore.go:675-679`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Fix (c) now (remove a stale temp of the same name before the copy); prove (a) with the B6 power-cut-mid-restore scenario | CC |
| **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt, not yet proven:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it, and the hub reads the same clock (agent `localapi/server.go:1178-1192`, `backup/runner.go:607-629`, hub `monitor/deadline.go:243`; R-973 cap counts future runs `backup_cap.go:61`); (b) the whole-guest start creeps ~5 min/night until a night is skipped silently (`server.go:1184`, `quiesce.go:150, 807-818`); (c) the whole-guest quiesce stops apps mid-restore or mid-update (`quiesce.go:213-275, 565`); (d) CONFIRMED in code: the off-site proof and integrity check are plain daily jobs outside the catch-up (`nightchain.go` header: the catch-up runs only the three backup legs), so they never run on a box that is off at 05:30–06:00, while the comment at `cmd/controller/main.go:1575-1577` says the opposite; (e) CONFIRMED in code: the hub's 05:00 check wants a dump „since Budapest midnight" (`hub/internal/monitor/deadline.go:416-419, 556-565`), so an evening backup time (which the banner suggests) raises a false `expected_dbdump_missed` error every morning — and on 2026-10-25 the check also runs at 04:00 (R-1183); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — design question; (g) spring forward 2027-03-28 fires dump and second copy together; (h) self-update busy check before the pull only (`updater.go:532-545, 764`); (i) a double-clicked restore shows failed while it runs (`web/handlers.go:1853-1874`). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`. | VERIFY | — | (a) is B6 #8's live test; (b) read the agent journal on demo-hp; the rest by unit tests | CC |
| **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt, not yet proven:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it, and the hub reads the same clock (agent `localapi/server.go:1178-1192`, `backup/runner.go:607-629`, hub `monitor/deadline.go:243`; R-973 cap counts future runs `backup_cap.go:61`); (b) the whole-guest start creeps ~5 min/night until a night is skipped silently (`server.go:1184`, `quiesce.go:150, 807-818`) — PARTLY SUPPORTED by demo-hp's agent journal 2026-09-30…10-20: completions drift 04:36→04:50 between resets, and THREE nights had no local whole-box backup (Oct 2, 6, 9), each right after an off-cycle daytime/evening run (Oct 1 22:15, Oct 5 11:24, Oct 8 22:25) — the 24-hour „not due yet" rule skips the next night; nothing tells anyone; (c) the whole-guest quiesce stops apps mid-restore or mid-update (`quiesce.go:213-275, 565`); (d) CONFIRMED in code: the off-site proof and integrity check are plain daily jobs outside the catch-up (`nightchain.go` header: the catch-up runs only the three backup legs), so they never run on a box that is off at 05:30–06:00, while the comment at `cmd/controller/main.go:1575-1577` says the opposite; (e) CONFIRMED in code: the hub's 05:00 check wants a dump „since Budapest midnight" (`hub/internal/monitor/deadline.go:416-419, 556-565`), so an evening backup time (which the banner suggests) raises a false `expected_dbdump_missed` error every morning — and on 2026-10-25 the check also runs at 04:00 (R-1183); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — design question; (g) spring forward 2027-03-28 fires dump and second copy together; (h) self-update busy check before the pull only (`updater.go:532-545, 764`); (i) a double-clicked restore shows failed while it runs (`web/handlers.go:1853-1874`). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`. | VERIFY | — | (a) is B6 #8's live test; (b) read the agent journal on demo-hp; the rest by unit tests | CC |
| **R-1190** | Backup & restore | P3 | **Key and retention candidates from the data-loss hunt, not yet proven:** (a) a recovery ceremony while off-site is not configured seals a package with no repo password and supersedes the good one (`escrow_handlers.go:217`, agent `main.go:2975-2981`, hub `store.go:3392`); (b) the global „create your recovery code" bar links to the ceremony the backups page hides during a recovery offer (`escrow_banner.go:23-33` vs `backups_remote.html:144-160`); (c) after a reuse-path reinstall the new PBS key K is never escrowed and nothing compares fingerprints (`felhom-pbs-apply:99`, `escrow_confirm.go:182-205`); (d) whole-guest off-site copies run before any escrow exists (design question vs 07 §6.1's Tier-3 gate); (e) the escrow PUT is two statements; (f) the operator's off-site re-issue resets the provider password before saving it (hub `offsite.go:217-245`); (g) „Megőrzött adatok" can list a running app's live folder when its app.yaml is unreadable (`kept.go:220-235, 330-402`); (h) a unit restore can point an app back at an old drive and F5 then prunes the current drive's unit (`backup.go:1517-1574`). Evidence `audits/dataloss-hunt-2026-10-20/C1-keys-candidates.md`, `C1-delete-retention-candidates.md`. | VERIFY | — | Unit tests first for (a), (e), (g) | CC |
| **R-1198** | Backup & restore | P3 | **A backup disk the family adds later is never offered as the place for the whole-box backup; the page keeps offering the disk that holds the apps' own data.** Rehearsal 2026-10-20 B6 #13 (`audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/`): a 50 GB disk plugged in and initialised as „Mentolemez" → the backups page still offers only „Családi adatlemez — Kijelölöm"; `internal/web/backup_target_offer.go:101` `firstOfferableDrive` offers the FIRST eligible drive, never a choice, and the drive's card has no „use for backups". | OPEN | — | Offer every eligible drive (largest empty first) and let the household pick | CC |
@@ -258,12 +258,13 @@ stopping line that lies.
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
## Monitoring & notifications — 11 rows (P2 1, P3 7, P4 3)
## Monitoring & notifications — 12 rows (P2 2, P3 7, P4 3)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC |
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
| **R-1199** | Monitoring & notifications | P2 | **The household's notification page shows disk-warning, drive-disconnected and backup-failed mails switched ON, but on the rehearsal day only the OPERATOR got them — the household's mailbox received one alarm (a failed off-site run) out of nine.** Rehearsal 2026-10-20, read from the Gmail catch-all (the second channel): to `tester4@` only the setup code and „A biztonsági mentés sikertelen!" (17:01); to `admin@` also `disk_warning` 89 % (18:36 — the box logged „notifying the customer"), `storage_disconnected` ×2, `storage_reconnected` ×2, `backup_run_failures` (3 of 8 apps), `app_oom`, `host/node_stale`. The box's page (`/settings/notifications`) has `event_disk_warning`, `event_storage_disconnected`, `event_backup_failed`, `event_crossdrive_failed` … ticked; the hub sends household mail only by ITS OWN stored prefs (`hub/internal/notify/dispatcher.go` `processCustomer`: `isEventEnabled(prefs.EnabledEvents, …)`) — whether the two lists disagree is not read (the token cannot read the customer record). Evidence `audits/rehearsal-2026-10-20/B6-failures/notifications-defaults.txt`, `journal.md`. | VERIFY | an operator read of Tester4's hub notification prefs and notification log | Compare the hub's `notification_prefs` / `notification_log` for Tester4 with the box's page; if they differ, make one the source of truth | CC |
| **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC |
| **R-340** | Monitoring & notifications | P3 | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice** **REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc/<pid>/fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC |
| **R-388** | Monitoring & notifications | P3 | **PRODUCT DECISION (not a defect): the customer notification model is the wrong shape, and the settings page grows by one toggle per detector.** The operator's framing, recorded verbatim 2026-08-23: *"A customer should be notified only about things they can act on or are responsible for — the drive they unplugged, the storage they filled. **A failed backup is our incident, not theirs.** The intended shape is that we detect it, we tell them we noticed and are dealing with it, and they are not handed an error they cannot solve. The subscription should feel like being looked after, not like being on call."* Today's page is the opposite shape — one switch per detector, and it **grew from 12 to 15 in a single session** (one new alarm plus two compound toggles split into four). That growth is the argument, not an aside: a page that grows per detector keeps asking a household to make engineering decisions. | **READY** — **2026-10-12 evening: slice 1 delivered** (operator Q1/Q2 „as picked", `09` §3 decisions 234–235): the page shows three group switches; the per-detector switches and the cooldown sit under „Részletek"; an „egyedi" group is saved back unchanged; the default list fills groups 1 and 3 (households with no saved list), group 2 stays „egyedi" (`app_start_failed` off, R-329). Tests red-proved. controller 0.309.0, delivered 2026-10-12 18:08 to demo-hp, demo-felhom and Tester 1 (`controller-0.309.0-delivery.txt`). **READY** — **2026-10-12: the design is written** (`08` §8.1): three group switches („A gépem és a meghajtóim", „Az alkalmazásaim", „Amit mi intézünk") mapped onto today's 18 detectors; option A picked; slice 1 changes no wire. Two questions for the operator (group 3 default; the cooldown field). **READY** — operator ruling 2026-10-12 (S5, `09` §3 decision 200): design now (one page: options, a pick, the first slice), build in slices later. **OPEN — DIRECTION, operator's call** | a decision on scope; nothing here is a bug | Slice 2 (hub): group 3's mails say „we noticed X and are handling it" instead of the raw error | CC (+ operator: two answers) |