admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e78726314ea43aecaea37a2fc081edef922ea607
felhom.eu/documentation/audits/night-2026-09-24/E/round-12-controller.log
T

432 lines
56 KiB
Plaintext
Raw Blame History

2026/09/24 12:37:58 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.2:51556 (X-Forwarded-For: 192.168.0.180)
2026/09/24 12:37:58 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session
2026/09/24 12:37:58 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T12:37:58Z, active_sessions=4
2026/09/24 12:37:58 auth.go:222: [INFO] [web] Login from 172.18.0.2:51556
2026/09/24 12:37:58 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.2:51556
2026/09/24 12:37:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.2:51556
2026/09/24 12:37:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556
2026/09/24 12:37:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556
2026/09/24 12:37:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556
2026/09/24 12:37:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556
2026/09/24 12:37:58 router.go:591: [INFO] [api] start requested for stack: chaosoomb
2026/09/24 12:37:58 settings.go:2677: [INFO] [settings] unhealthy-stop hold LIFTED for chaosoomb (Start)
2026/09/24 12:37:58 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18389 bytes)
2026/09/24 12:37:58 settings.go:847: [INFO] [settings] Settings saved
2026/09/24 12:37:58 router.go:615: [INFO] [api] chaosoomb: the unhealthy stop is LIFTED by Start — one more try (decision 28)
2026/09/24 12:37:58 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5507MB avail≈20391MB
2026/09/24 12:37:58 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24182664 → total=25898MB avail=20391MB used=5507MB (21.3%)
2026/09/24 12:37:58 manager.go:1194: [DEBUG] [stacks] StartStack chaosoomb: current state=stopped deployed=true
2026/09/24 12:37:58 manager.go:1197: [INFO] [stacks] Starting stack: chaosoomb
2026/09/24 12:37:58 manager.go:1204: [DEBUG] [stacks] StartStack chaosoomb: prepared 8 env vars for compose
2026/09/24 12:37:58 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/09/24 12:37:58 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/chaosoomb)
2026/09/24 12:37:58 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 0.4s)
2026/09/24 12:37:58 manager.go:1212: [INFO] [stacks] Stack chaosoomb started successfully (took 0.4s)
2026/09/24 12:37:59 installed.go:409: [DEBUG] [stacks] installed-images chaosoomb: unchanged (1 service(s)) — app.yaml not rewritten
2026/09/24 12:37:59 lifecycle.go:70: [DEBUG] [integrations] OnStackStart: stack=chaosoomb, waiting 5s for state refresh
2026/09/24 12:38:01 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 4m24s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 4m20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:01 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:01 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container)
2026/09/24 12:38:01 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused
2026/09/24 12:38:01 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed)
2026/09/24 12:38:02 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/09/24 12:38:02 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/chaosoomb)
2026/09/24 12:38:02 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s)
2026/09/24 12:38:02 manager.go:1543: [INFO] [stacks] Stack chaosoomb post-start status:
2026/09/24 12:38:02 manager.go:1546: [INFO] [stacks] chaosoomb alpine:3.20 running Up 3 seconds
2026/09/24 12:38:04 lifecycle.go:86: [DEBUG] [integrations] OnStackStart: stack=chaosoomb integrationsFound=0
2026/09/24 12:38:11 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:38:11 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:38:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 4m34s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 4m30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:11 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container)
2026/09/24 12:38:11 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused
2026/09/24 12:38:11 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused
2026/09/24 12:38:11 healthprobe.go:159: [WARN] Health probes: 0 ok, 2 unhealthy (of 2 probed)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5431MB avail≈20467MB
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24273396 → total=25898MB avail=20467MB used=5431MB (21.0%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=27.8GB avail=37.0GB (40.7%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readLoadAvg: raw="2.69 3.49 3.43 3/1532 2820" → 1m=2.69 5m=3.49 15m=3.43
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 68.5°C (hwmon2)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetInfo done in 67ms — mem=5431MB/25898MB (21.0%), rootDisk=27.8GB/68.4GB (40.7%), load=2.69/3.49/3.43, temp=68.5°C (hwmon2), cpu=30.3%
2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:38:21 scheduler.go:346: [INFO] [scheduler] Running job: offsite-credential-retry
2026/09/24 12:38:21 scheduler.go:346: [INFO] [scheduler] Running job: backup-cache
2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job offsite-credential-retry: execution starting
2026/09/24 12:38:21 scheduler.go:363: [INFO] [scheduler] Job offsite-credential-retry completed (took 0s)
2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job backup-cache: execution starting
2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:38:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 4m44s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 4m40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:21 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container)
2026/09/24 12:38:21 scheduler.go:346: [INFO] [scheduler] Running job: system-health
2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job system-health: execution starting
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true)
2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/09/24 12:38:21 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused
2026/09/24 12:38:21 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused
2026/09/24 12:38:21 healthprobe.go:159: [WARN] Health probes: 0 ok, 2 unhealthy (of 2 probed)
2026/09/24 12:38:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/nextcloud → [nextcloud]
2026/09/24 12:38:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx → [paperless-ngx]
2026/09/24 12:38:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [privatebin, vikunja, wishlist, chaosoom, chaosoomb, n8n]
2026/09/24 12:38:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/romm → [romm]
2026/09/24 12:38:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/navidrome → [navidrome]
2026/09/24 12:38:21 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18389 bytes)
2026/09/24 12:38:21 settings.go:847: [INFO] [settings] Settings saved
2026/09/24 12:38:21 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18617 bytes)
2026/09/24 12:38:21 settings.go:847: [INFO] [settings] Settings saved
2026/09/24 12:38:21 backup.go:1119: [INFO] [backup] Found 10 DB dump files across drives
2026/09/24 12:38:21 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5439MB avail≈20459MB
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24263608 → total=25898MB avail=20459MB used=5439MB (21.0%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=27.8GB avail=37.0GB (40.7%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readLoadAvg: raw="2.69 3.49 3.43 5/1557 2877" → 1m=2.69 5m=3.49 15m=3.43
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 68.0°C (hwmon2)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetInfo done in 78ms — mem=5439MB/25898MB (21.0%), rootDisk=27.8GB/68.4GB (40.7%), load=2.69/3.49/3.43, temp=68.0°C (hwmon2), cpu=31.6%
2026/09/24 12:38:21 healthcheck.go:86: [DEBUG] [monitor] Raw values: disk=40.7%, hdd=7.8% (configured=true), mem=21.0% (5439MB/25898MB), cpu=31.6%, temp=68.0°C (hwmon2)
2026/09/24 12:38:21 healthcheck.go:118: [DEBUG] [monitor] SSD disk: OK (41%)
2026/09/24 12:38:21 healthcheck.go:151: [DEBUG] [monitor] Memory: OK (21%)
2026/09/24 12:38:21 healthcheck.go:169: [DEBUG] [monitor] CPU: OK (32%)
2026/09/24 12:38:21 healthcheck.go:187: [DEBUG] [monitor] Temperature: OK (68°C)
2026/09/24 12:38:21 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 231d20f4db9e chaosoomb chaosoomb alpine:3.20
ec2856735161 nextcloud nextcloud nextcloud:34.0.4-apache
560874757b4e nextcloud-redis nextcloud redis:7-alpine
f990f2f4a0e6 nextcloud-db nextcloud mariadb:12.3
517b7ba21f80 n8n n8n n8nio/n8n:2.40.5
a0b33d0b9958 wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1
9f47a1f5273b vikunja vikunja vikunja/vikunja:2.3.0
1adf3ec73c9b romm romm rommapp/romm:5.3.1
e059bbc146b0 romm-redis romm redis:7-alpine
fc5ed5428e75 romm-db romm mariadb:11.8
361a05346765 priv...
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container chaosoomb (image=alpine:3.20, not a database)
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database)
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database)
2026/09/24 12:38:21 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=f990f2f4a0e6)
2026/09/24 12:38:21 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.40.5, not a database)
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database)
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database)
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database)
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database)
2026/09/24 12:38:21 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=fc5ed5428e75)
2026/09/24 12:38:21 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database)
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
2026/09/24 12:38:21 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5e5c371b82fa)
2026/09/24 12:38:21 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
2026/09/24 12:38:21 healthcheck.go:201: [DEBUG] [monitor] Docker daemon: OK
2026/09/24 12:38:21 healthcheck.go:209: [DEBUG] [monitor] Checking 1 protected containers: [traefik]
2026/09/24 12:38:21 main.go:872: [WARN] [deadapp] chaosoomb: container chaosoomb was OOM-killed (started 2026-09-24T12:37:58.784734494Z; kills 18, limit 32M, peak 32M)
2026/09/24 12:38:21 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_oom (severity warning) — further app_oom events are logged at DEBUG only
2026/09/24 12:38:21 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database)
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database)
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database)
2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database)
2026/09/24 12:38:21 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 15 non-DB container(s)
2026/09/24 12:38:21 dbdump.go:208: [INFO] [backup] Discovered 3 databases
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/navidrome/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/09/24 12:38:21 healthcheck.go:219: [DEBUG] [monitor] All protected containers running
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 12:38:21 healthcheck.go:238: [INFO] [monitor] Health check: status=warn
2026/09/24 12:38:21 healthcheck.go:242: [DEBUG] [monitor] Final status: warn (issues=0, warnings=4, info=5)
2026/09/24 12:38:21 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event health_change (severity warning) — further health_change events are logged at DEBUG only
2026/09/24 12:38:21 scheduler.go:363: [INFO] [scheduler] Job system-health completed (took 192ms)
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml
2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml
2026/09/24 12:38:21 [INFO] [backup] Discovered app data: 10 apps
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 12:38:21 offbox_reconstitute.go:319: [INFO] [backup] nextcloud: pruned old undo copy pre-restore-20260924T105053Z-nextcloud-mariadb.sql (keeping the newest 3)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%)
2026/09/24 12:38:21 recovery_unit.go:224: [INFO] [backup] Recovery unit captured for chaosoomb → /mnt/sys_drive/felhom-data/backups/primary/chaosoomb (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%)
2026/09/24 12:38:21 recovery_unit.go:224: [INFO] [backup] Recovery unit captured for n8n → /mnt/sys_drive/felhom-data/backups/primary/n8n (images=1, secrets-referenced=1, data_keys=0, portable-carried=1/1, withheld=0)
2026/09/24 12:38:21 backup.go:1189: [INFO] [backup] Backup status cache refreshed
2026/09/24 12:38:21 scheduler.go:363: [INFO] [scheduler] Job backup-cache completed (took 222ms)
2026/09/24 12:38:21 infra.go:67: [INFO] [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node)
2026/09/24 12:38:31 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:38:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 4m54s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:38:31 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:38:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 4m50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:38:31 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container)
2026/09/24 12:38:31 healthprobe.go:181: [DEBUG] Health probe navidrome: API GET :4533/ping → 200 (2ms)
2026/09/24 12:38:31 healthprobe.go:181: [DEBUG] Health probe privatebin: HTTP GET :8080/ → 200 (5ms)
2026/09/24 12:38:31 healthprobe.go:161: [INFO] Health probes: 2 ok (of 2 probed)
2026/09/24 12:38:41 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:38:41 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:38:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:41 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 4 targets (4 skipped not due, 0 skipped no container)
2026/09/24 12:38:41 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused
2026/09/24 12:38:41 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused
2026/09/24 12:38:41 healthprobe.go:181: [DEBUG] Health probe n8n: API GET :5678/healthz → 200 (1ms)
2026/09/24 12:38:41 healthprobe.go:181: [DEBUG] Health probe wishlist: HTTP GET :3000/ → 307 (4ms)
2026/09/24 12:38:41 healthprobe.go:159: [WARN] Health probes: 2 ok, 2 unhealthy (of 4 probed)
2026/09/24 12:38:51 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:38:51 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/09/24 12:38:51 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:38:51 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container)
2026/09/24 12:38:51 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:38:51 main.go:872: [WARN] [deadapp] chaosoomb: container chaosoomb was OOM-killed (started 2026-09-24T12:38:41.277810467Z; kills 8, limit 32M, peak 32M)
2026/09/24 12:38:51 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_oom (severity warning)
2026/09/24 12:39:01 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:39:01 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:39:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:01 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container)
2026/09/24 12:39:01 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused
2026/09/24 12:39:01 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused
2026/09/24 12:39:01 healthprobe.go:159: [WARN] Health probes: 0 ok, 2 unhealthy (of 2 probed)
2026/09/24 12:39:11 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 5m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 5m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:11 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:11 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container)
2026/09/24 12:39:21 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true)
2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5509MB avail≈20389MB
2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24179764 → total=25898MB avail=20389MB used=5509MB (21.3%)
2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=27.8GB avail=37.0GB (40.7%)
2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%)
2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readLoadAvg: raw="3.02 3.42 3.41 7/1539 3107" → 1m=3.02 5m=3.42 15m=3.41
2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 69.4°C (hwmon2)
2026/09/24 12:39:21 info.go:13: [DEBUG] [system] GetInfo done in 80ms — mem=5509MB/25898MB (21.3%), rootDisk=27.8GB/68.4GB (40.7%), load=3.02/3.42/3.41, temp=69.4°C (hwmon2), cpu=34.7%
2026/09/24 12:39:21 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/09/24 12:39:21 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:39:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:21 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:39:21 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/09/24 12:39:21 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan
2026/09/24 12:39:21 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting
2026/09/24 12:39:21 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 4 targets (4 skipped not due, 0 skipped no container)
2026/09/24 12:39:21 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused
2026/09/24 12:39:21 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused
2026/09/24 12:39:21 healthprobe.go:181: [DEBUG] Health probe romm: HTTP GET :8080/ → 200 (2ms)
2026/09/24 12:39:21 healthprobe.go:181: [DEBUG] Health probe paperless-ngx: HTTP GET :8000/ → 302 (6ms)
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml
2026/09/24 12:39:21 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates
2026/09/24 12:39:21 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection
2026/09/24 12:39:21 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available)
2026/09/24 12:39:21 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 108ms)
2026/09/24 12:39:21 healthprobe.go:159: [WARN] Health probes: 2 ok, 2 unhealthy (of 4 probed)
2026/09/24 12:39:21 main.go:872: [WARN] [deadapp] chaosoomb: container chaosoomb was OOM-killed (started 2026-09-24T12:39:20.327438775Z; kills 1, limit 32M, peak 32M)
2026/09/24 12:39:21 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_oom (severity warning)
2026/09/24 12:39:31 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:39:31 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:39:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:31 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container)
2026/09/24 12:39:31 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused
2026/09/24 12:39:31 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused
2026/09/24 12:39:31 healthprobe.go:159: [WARN] Health probes: 0 ok, 2 unhealthy (of 2 probed)
2026/09/24 12:39:41 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:39:41 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:39:41 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container)
2026/09/24 12:39:51 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:39:51 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/09/24 12:39:51 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/09/24 12:39:51 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:39:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:39:51 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container)
2026/09/24 12:39:51 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused
2026/09/24 12:39:51 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused
2026/09/24 12:39:51 healthprobe.go:159: [WARN] Health probes: 0 ok, 2 unhealthy (of 2 probed)
2026/09/24 12:39:51 main.go:872: [WARN] [deadapp] chaosoomb: container chaosoomb was OOM-killed (started 2026-09-24T12:39:20.327438775Z; kills 22, limit 32M, peak 32M)
2026/09/24 12:39:51 notifier.go:738: [ERROR] [notify] chaosoomb: container chaosoomb OOM STORM — 22 kills in 30 min (limit 32M, peak 32M)
2026/09/24 12:39:51 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_oom_storm (severity error) — further app_oom_storm events are logged at DEBUG only
2026/09/24 12:39:51 main.go:3671: [WARN] [deadapp] chaosoomb: oom_storm — 21 in 30m0s; STOPPING it (decision 28)
2026/09/24 12:39:51 manager.go:1283: [DEBUG] [stacks] StopStack chaosoomb: current state=running deployed=true containers=1
2026/09/24 12:39:51 manager.go:1286: [INFO] [stacks] Stopping stack: chaosoomb
2026/09/24 12:39:51 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/09/24 12:39:51 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/chaosoomb)
2026/09/24 12:40:01 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:40:01 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:40:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:40:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:40:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:40:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:40:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:40:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:40:01 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container)
2026/09/24 12:40:01 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused
2026/09/24 12:40:01 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused
2026/09/24 12:40:01 healthprobe.go:159: [WARN] Health probes: 0 ok, 2 unhealthy (of 2 probed)
2026/09/24 12:40:01 manager.go:1483: [DEBUG] Command completed: docker compose down (took 10.3s)
2026/09/24 12:40:01 manager.go:1295: [INFO] [stacks] Stack chaosoomb stopped successfully (took 10.3s)
2026/09/24 12:40:01 settings.go:1814: [WARN] [settings] restore hold SET for chaosoomb — the app stays stopped until it is cleared
2026/09/24 12:40:01 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18794 bytes)
2026/09/24 12:40:01 settings.go:847: [INFO] [settings] Settings saved
2026/09/24 12:40:01 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18794 bytes)
2026/09/24 12:40:01 settings.go:847: [INFO] [settings] Settings saved
2026/09/24 12:40:01 update_guard.go:776: [WARN] [backup] chaosoomb is STOPPED by the box: oom_storm (trip 2 within 24h0m0s) — Start gives it one more try (decision 28)
2026/09/24 12:40:01 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_stopped_unhealthy (severity warning) — further app_stopped_unhealthy events are logged at DEBUG only
2026/09/24 12:40:03 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556
2026/09/24 12:40:03 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556
2026/09/24 12:40:03 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556
2026/09/24 12:40:03 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556
Reference in New Issue View Git Blame Copy Permalink