admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e78726314ea43aecaea37a2fc081edef922ea607
felhom.eu/documentation/audits/night-2026-09-24/E/round-01-controller.log
T
admin 4502af6bb1
gates / gates (push) Successful in 24s
Details
night 2026-09-24: chaos rounds 1-8 evidence; 07/08/09/capability map for decisions 26-28 + digests; R-681 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-24 14:21:14 +02:00

555 lines
74 KiB
Plaintext
Raw Blame History

2026/09/24 11:45:59 main.go:335: [INFO] felhom-controller 0.269.1 starting (customer: demo-hp, domain: enkisfelhom.hu)
2026/09/24 11:45:59 settings.go:709: [INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json
2026/09/24 11:45:59 settings.go:1671: [DEBUG] [settings] AutoDiscoverStoragePaths discovered=[/mnt/felhom-drives/scratch_hdd/userdata/navidrome /mnt/felhom-drives/scratch_hdd/userdata/nextcloud /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx /mnt/felhom-drives/scratch_hdd/userdata/romm] fallback="" existing=5
2026/09/24 11:45:59 main.go:361: [INFO] Encryption key loaded from /opt/docker/felhom-controller/data/encryption.key
2026/09/24 11:46:00 manager.go:328: [INFO] [stacks] Using compose command: docker compose
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=false composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 11:46:00 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml
2026/09/24 11:46:00 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates
2026/09/24 11:46:00 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection
2026/09/24 11:46:00 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (9 deployed, 49 available)
2026/09/24 11:46:00 deploy.go:1104: [DEBUG] [stacks] InjectMissingFields: checking 9 stacks
2026/09/24 11:46:00 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack wishlist — 2 deploy fields, 2 existing env vars
2026/09/24 11:46:00 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack privatebin — 2 deploy fields, 2 existing env vars
2026/09/24 11:46:00 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack paperless-ngx — 8 deploy fields, 8 existing env vars
2026/09/24 11:46:00 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack nextcloud — 7 deploy fields, 7 existing env vars
2026/09/24 11:46:00 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack romm — 6 deploy fields, 6 existing env vars
2026/09/24 11:46:00 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack navidrome — 3 deploy fields, 3 existing env vars
2026/09/24 11:46:00 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack gokapi — 3 deploy fields, 3 existing env vars
2026/09/24 11:46:00 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack chaosoom — 2 deploy fields, 2 existing env vars
2026/09/24 11:46:00 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack vikunja — 3 deploy fields, 3 existing env vars
2026/09/24 11:46:00 deploy.go:1189: [INFO] [stacks] InjectMissingFields: processed 9 stacks
2026/09/24 11:46:00 manager.go:467: [DEBUG] [stacks] MigrateEncryption: checking 9 deployed stacks for plaintext sensitive values
2026/09/24 11:46:00 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "paperless-ngx" (3 sensitive fields)
2026/09/24 11:46:00 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "nextcloud" (3 sensitive fields)
2026/09/24 11:46:00 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "romm" (3 sensitive fields)
2026/09/24 11:46:00 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "gokapi" (1 sensitive fields)
2026/09/24 11:46:00 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "vikunja" (1 sensitive fields)
2026/09/24 11:46:00 manager.go:511: [INFO] [stacks] Encryption migration: no stacks needed migration
2026/09/24 11:46:00 desiredstate.go:137: [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded (state ambiguous — legacy boot behaviour retained)
2026/09/24 11:46:00 installed.go:537: [INFO] [stacks] installed-images backfill: 0 app(s) recorded, 9 already had a record, 0 left unrecorded (could not be observed completely — unknown, which renders nothing)
2026/09/24 11:46:00 pin.go:290: [INFO] [stacks] pin adoption: 0 pinned, 9 already pinned, 0 left unpinned (0 not completely observed, 0 running something the template no longer offers)
2026/09/24 11:46:00 undo.go:656: [INFO] [stacks] applied-meta backfill (R-646): recorded 0 []; skipped 0 [] — not current with the catalog, their pinned version's .felhom.yml is no longer on the box
2026/09/24 11:46:00 sync.go:117: [INFO] [sync] Starting catalog sync (repo: https://gitea.dooplex.hu/admin/app-catalog-drill.git, interval: 15m0s)
2026/09/24 11:46:00 info.go:13: [DEBUG] [system] CPUCollector.Start: sampleRate=5s
2026/09/24 11:46:00 sync.go:208: [INFO] [sync] Starting catalog sync
2026/09/24 11:46:00 sync.go:296: [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-drill.git (branch: main)
2026/09/24 11:46:00 sync.go:298: [DEBUG] [sync] git fetch --depth 1 origin main in /opt/docker/felhom-controller/data/catalog-cache
2026/09/24 11:46:00 sync.go:627: [DEBUG] [sync] Running: git fetch --depth 1 origin main
2026/09/24 11:46:00 main.go:515: [INFO] Metrics store opened at /opt/docker/felhom-controller/data/metrics.db
2026/09/24 11:46:00 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true)
2026/09/24 11:46:00 main.go:527: [INFO] Metrics collector started (60s interval)
2026/09/24 11:46:00 restore_record_wiring.go:35: [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false)
2026/09/24 11:46:00 notifier.go:81: [INFO] Notifier disabled (hub not configured)
2026/09/24 11:46:00 main.go:692: [INFO] [mailrelay] app-email shim unavailable (no hub configured or disabled in config)
2026/09/24 11:46:00 scheduler.go:102: [INFO] [scheduler] Registered periodic job: status-refresh (every 10s)
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="status-refresh" interval=10s totalJobs=1
2026/09/24 11:46:00 scheduler.go:102: [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s)
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="stack-scan" interval=2m0s totalJobs=2
2026/09/24 11:46:00 scheduler.go:102: [INFO] [scheduler] Registered periodic job: health-probes (every 10s)
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="health-probes" interval=10s totalJobs=3
2026/09/24 11:46:00 scheduler.go:102: [INFO] [scheduler] Registered periodic job: system-health (every 5m0s)
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="system-health" interval=5m0s totalJobs=4
2026/09/24 11:46:00 scheduler.go:102: [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s)
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="deadapp-check" interval=30s totalJobs=5
2026/09/24 11:46:00 scheduler.go:102: [INFO] [scheduler] Registered periodic job: ring-spill (every 30s)
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="ring-spill" interval=30s totalJobs=6
2026/09/24 11:46:00 scheduler.go:132: [INFO] [scheduler] Daily job db-dump scheduled for 2026-09-25 13:20 CEST
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="db-dump" schedule="13:20" nextRun=2026-09-25T13:20:00+02:00 totalJobs=7
2026/09/24 11:46:00 scheduler.go:102: [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s)
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="offsite-credential-retry" interval=5m0s totalJobs=8
2026/09/24 11:46:00 scheduler.go:102: [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s)
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="backup-cache" interval=5m0s totalJobs=9
2026/09/24 11:46:00 scheduler.go:132: [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-09-24 14:20 CEST
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="tier2-backup" schedule="14:20" nextRun=2026-09-24T14:20:00+02:00 totalJobs=10
2026/09/24 11:46:00 scheduler.go:132: [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-09-24 15:05 CEST
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offbox-backup" schedule="15:05" nextRun=2026-09-24T15:05:00+02:00 totalJobs=11
2026/09/24 11:46:00 scheduler.go:132: [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-09-25 05:10 CEST
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offsite-abandon-sweep" schedule="05:10" nextRun=2026-09-25T05:10:00+02:00 totalJobs=12
2026/09/24 11:46:00 scheduler.go:132: [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-09-25 06:00 CEST
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offsite-integrity" schedule="06:00" nextRun=2026-09-25T06:00:00+02:00 totalJobs=13
2026/09/24 11:46:00 scheduler.go:132: [INFO] [scheduler] Daily job offsite-proof scheduled for 2026-09-25 05:30 CEST
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offsite-proof" schedule="05:30" nextRun=2026-09-25T05:30:00+02:00 totalJobs=14
2026/09/24 11:46:00 scheduler.go:132: [INFO] [scheduler] Daily job metrics-prune scheduled for 2026-09-25 04:00 CEST
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="metrics-prune" schedule="04:00" nextRun=2026-09-25T04:00:00+02:00 totalJobs=15
2026/09/24 11:46:00 scheduler.go:132: [INFO] [scheduler] Daily job fill-watch scheduled for 2026-09-25 03:30 CEST
2026/09/24 11:46:00 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="fill-watch" schedule="03:30" nextRun=2026-09-25T03:30:00+02:00 totalJobs=16
2026/09/24 11:46:00 selftest.go:50: [INFO] ========== Startup Self-Test ==========
2026/09/24 11:46:00 sync.go:212: [ERROR] [sync] Catalog sync failed: git fetch: git fetch --depth 1 origin main: exit status 128
stderr: fatal: unable to access 'https://gitea.dooplex.hu/admin/app-catalog-drill.git/': Could not resolve host: gitea.dooplex.hu
2026/09/24 11:46:00 sync.go:122: [INFO] [sync] Initial sync: Git hiba: git fetch: git fetch --depth 1 origin main: exit status 128
stderr: fatal: unable to access 'https://gitea.dooplex.hu/admin/app-catalog-drill.git/': Could not resolve host: gitea.dooplex.hu
2026/09/24 11:46:01 selftest.go:68: [INFO] [PASS] Docker socket: reachable (v29.8.0)
2026/09/24 11:46:01 selftest.go:68: [INFO] [PASS] Stacks directory: /opt/docker/stacks
2026/09/24 11:46:01 selftest.go:68: [INFO] [PASS] Data directory: /opt/docker/felhom-controller/data (writable)
2026/09/24 11:46:01 selftest.go:68: [INFO] [PASS] System data path: /mnt/sys_drive
2026/09/24 11:46:01 selftest.go:68: [INFO] [PASS] Storage paths: 1 connected, 4 disconnected
2026/09/24 11:46:01 selftest.go:68: [INFO] [PASS] Git catalog: 56 app definitions found
2026/09/24 11:46:01 selftest.go:68: [INFO] [PASS] Hub connectivity: hub disabled, skipped
2026/09/24 11:46:01 selftest.go:68: [INFO] [PASS] Metrics DB: 8.1 MB
2026/09/24 11:46:01 selftest.go:70: [INFO] ========================================
2026/09/24 11:46:01 selftest.go:71: [INFO] Self-test complete: 8 passed, 0 warnings, 0 failed
2026/09/24 11:46:01 scheduler.go:206: [INFO] [scheduler] Starting scheduler with 16 jobs
2026/09/24 11:46:01 scheduler.go:67: [DEBUG] [scheduler] scheduler started: periodic=8 daily=8
2026/09/24 11:46:01 scheduler.go:67: [DEBUG] [scheduler] daily job db-dump: next run at 2026-09-25 13:20:00 CEST (waiting 23h33m59s)
2026/09/24 11:46:01 scheduler.go:67: [DEBUG] [scheduler] daily job offsite-abandon-sweep: next run at 2026-09-25 05:10:00 CEST (waiting 15h23m59s)
2026/09/24 11:46:01 scheduler.go:67: [DEBUG] [scheduler] daily job tier2-backup: next run at 2026-09-24 14:20:00 CEST (waiting 33m59s)
2026/09/24 11:46:01 scheduler.go:67: [DEBUG] [scheduler] daily job offbox-backup: next run at 2026-09-24 15:05:00 CEST (waiting 1h18m59s)
2026/09/24 11:46:01 scheduler.go:67: [DEBUG] [scheduler] daily job offsite-proof: next run at 2026-09-25 05:30:00 CEST (waiting 15h43m59s)
2026/09/24 11:46:01 scheduler.go:67: [DEBUG] [scheduler] daily job offsite-integrity: next run at 2026-09-25 06:00:00 CEST (waiting 16h13m59s)
2026/09/24 11:46:01 scheduler.go:67: [DEBUG] [scheduler] daily job metrics-prune: next run at 2026-09-25 04:00:00 CEST (waiting 14h13m59s)
2026/09/24 11:46:01 scheduler.go:67: [DEBUG] [scheduler] daily job fill-watch: next run at 2026-09-25 03:30:00 CEST (waiting 13h43m59s)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈4187MB avail≈21711MB
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=25748124 → total=25898MB avail=21711MB used=4187MB (16.2%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=22.5GB avail=42.4GB (32.8%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readLoadAvg: raw="5.31 2.81 2.70 19/1482 125" → 1m=5.31 5m=2.81 15m=2.70
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 80.8°C (hwmon2)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetInfo done in 421ms — mem=4187MB/25898MB (16.2%), rootDisk=22.5GB/68.4GB (32.8%), load=5.31/2.81/2.70, temp=80.8°C (hwmon2), cpu=0.0%
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true)
2026/09/24 11:46:01 server.go:250: [DEBUG] [web] NewServer: initializing web server v0.269.1
2026/09/24 11:46:01 server.go:251: [DEBUG] [web] NewServer: backup=true scheduler=true alertMgr=true notifier=true updater=false
2026/09/24 11:46:01 infra.go:228: [INFO] [infra] connected felhom-controller to traefik-public
2026/09/24 11:46:01 infra.go:67: [INFO] [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node)
2026/09/24 11:46:01 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/navidrome → [navidrome]
2026/09/24 11:46:01 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/nextcloud → [nextcloud]
2026/09/24 11:46:01 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx → [paperless-ngx]
2026/09/24 11:46:01 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/romm → [romm]
2026/09/24 11:46:01 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [gokapi, privatebin, vikunja, wishlist, chaosoom]
2026/09/24 11:46:01 backup.go:1119: [INFO] [backup] Found 9 DB dump files across drives
2026/09/24 11:46:01 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
2026/09/24 11:46:01 server.go:327: [DEBUG] [web] loadTemplates: lang=hu loaded 81 templates, 1886 markers expanded in 117.796502ms
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈4223MB avail≈21675MB
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=25706264 → total=25898MB avail=21675MB used=4223MB (16.3%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=22.5GB avail=42.4GB (32.8%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readLoadAvg: raw="5.31 2.81 2.70 16/1476 174" → 1m=5.31 5m=2.81 15m=2.70
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 80.9°C (hwmon2)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetInfo done in 260ms — mem=4223MB/25898MB (16.3%), rootDisk=22.5GB/68.4GB (32.8%), load=5.31/2.81/2.70, temp=80.9°C (hwmon2), cpu=0.0%
2026/09/24 11:46:01 healthcheck.go:86: [DEBUG] [monitor] Raw values: disk=32.8%, hdd=7.8% (configured=true), mem=16.3% (4223MB/25898MB), cpu=0.0%, temp=80.9°C (hwmon2)
2026/09/24 11:46:01 healthcheck.go:118: [DEBUG] [monitor] SSD disk: OK (33%)
2026/09/24 11:46:01 healthcheck.go:151: [DEBUG] [monitor] Memory: OK (16%)
2026/09/24 11:46:01 healthcheck.go:179: [WARN] [monitor] Temperature threshold breached: 81°C (limit: 75°C)
2026/09/24 11:46:01 healthcheck.go:182: [DEBUG] [monitor] Temperature: WARN (81°C >= 75°C)
2026/09/24 11:46:01 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 86ba63e28bd7 felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.269.1
557bdb05d3bd vikunja vikunja vikunja/vikunja:2.3.0
8ed284296786 romm-db romm mariadb:11.8
0a82332f8e1c romm romm rommapp/romm:5.3.1
9ec77cdb422a romm-redis romm redis:7-alpine
d9846695ae0d wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1
8867e567d339 privatebin privatebin privatebin/pdo:2.0.6
0732d8573acd paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
3df88f0fdd1d paperless-redis pap...
2026/09/24 11:46:01 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database)
2026/09/24 11:46:01 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database)
2026/09/24 11:46:01 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=8ed284296786)
2026/09/24 11:46:01 server.go:327: [DEBUG] [web] loadTemplates: lang=en loaded 81 templates, 1886 markers expanded in 172.029097ms
2026/09/24 11:46:01 server.go:270: [INFO] [web] Auth: using password from settings.json
2026/09/24 11:46:01 server.go:814: [DEBUG] [web] CatchAllMiddleware: controller host=felhom.enkisfelhom.hu
2026/09/24 11:46:01 main.go:1950: [INFO] Web UI listening on :8080
2026/09/24 11:46:01 handlers.go:3492: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping userdata skeleton
2026/09/24 11:46:01 handlers.go:3492: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping userdata skeleton
2026/09/24 11:46:01 handlers.go:3492: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping userdata skeleton
2026/09/24 11:46:01 handlers.go:3492: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping userdata skeleton
2026/09/24 11:46:01 manager.go:258: [DEBUG] [integrations] ReapplyConfigForTarget: target=filebrowser integrations=0
2026/09/24 11:46:01 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm
2026/09/24 11:46:01 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database)
2026/09/24 11:46:01 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database)
2026/09/24 11:46:01 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database)
2026/09/24 11:46:01 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database)
2026/09/24 11:46:01 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
2026/09/24 11:46:01 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
2026/09/24 11:46:01 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5da8830bc973)
2026/09/24 11:46:01 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
2026/09/24 11:46:01 auth.go:123: [DEBUG] [web] auth: rejected GET /api/stacks/chaosoom from 172.18.0.6:51320 (invalid/expired session)
2026/09/24 11:46:01 auth.go:126: [WARN] [api] Unauthorized request to /api/stacks/chaosoom from 172.18.0.6:51320
2026/09/24 11:46:01 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.6:51320 (X-Forwarded-For: 192.168.0.180)
2026/09/24 11:46:01 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
2026/09/24 11:46:01 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database)
2026/09/24 11:46:01 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=9837f4d677cb)
2026/09/24 11:46:01 healthcheck.go:201: [DEBUG] [monitor] Docker daemon: OK
2026/09/24 11:46:01 healthcheck.go:209: [DEBUG] [monitor] Checking 1 protected containers: [traefik]
2026/09/24 11:46:01 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud
2026/09/24 11:46:01 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database)
2026/09/24 11:46:01 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database)
2026/09/24 11:46:01 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database)
2026/09/24 11:46:01 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database)
2026/09/24 11:46:01 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 13 non-DB container(s)
2026/09/24 11:46:01 dbdump.go:208: [INFO] [backup] Discovered 3 databases
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/navidrome/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 11:46:01 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml
2026/09/24 11:46:01 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T11:46:01Z, active_sessions=1
2026/09/24 11:46:01 auth.go:222: [INFO] [web] Login from 172.18.0.6:51320
2026/09/24 11:46:01 healthcheck.go:219: [DEBUG] [monitor] All protected containers running
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:46:01 healthcheck.go:238: [INFO] [monitor] Health check: status=warn
2026/09/24 11:46:01 healthcheck.go:242: [DEBUG] [monitor] Final status: warn (issues=0, warnings=5, info=3)
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 11:46:01 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 11:46:01 [INFO] [backup] Discovered app data: 9 apps
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:46:01 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.6:51320
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:46:01 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.6:51320
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=22.5 GB avail=42.4GB (32.8%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=22.5 GB avail=42.4GB (32.8%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=22.5 GB avail=42.4GB (32.8%)
2026/09/24 11:46:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=22.5 GB avail=42.4GB (32.8%)
2026/09/24 11:46:01 recovery_unit.go:224: [INFO] [backup] Recovery unit captured for chaosoom → /mnt/sys_drive/felhom-data/backups/primary/chaosoom (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/09/24 11:46:01 backup.go:1189: [INFO] [backup] Backup status cache refreshed
2026/09/24 11:46:01 handlers.go:3432: [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (5 storage path(s))
2026/09/24 11:46:05 info.go:13: [DEBUG] [system] CPUCollector: first sample — cpu=97.1% (idle=116 total=3976)
2026/09/24 11:46:06 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event controller_started (severity info) — further controller_started events are logged at DEBUG only
2026/09/24 11:46:10 main.go:2356: [DEBUG] [bootrecon] boot window: fleet still changing (9 app(s)) — resampling
2026/09/24 11:46:11 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 11:46:11 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 11:46:11 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 4 targets (0 skipped not due, 0 skipped no container)
2026/09/24 11:46:11 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.7:8999: connect: connection refused
2026/09/24 11:46:11 healthprobe.go:181: [DEBUG] Health probe privatebin: HTTP GET :8080/ → 200 (12ms)
2026/09/24 11:46:11 healthprobe.go:181: [DEBUG] Health probe navidrome: API GET :4533/ping → 200 (15ms)
2026/09/24 11:46:11 healthprobe.go:181: [DEBUG] Health probe nextcloud: API GET :80/status.php → 200 (67ms)
2026/09/24 11:46:11 healthprobe.go:159: [WARN] Health probes: 3 ok, 1 unhealthy (of 4 probed)
2026/09/24 11:46:15 main.go:2356: [DEBUG] [bootrecon] boot window: fleet still changing (9 app(s)) — resampling
2026/09/24 11:46:21 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 11:46:21 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 11:46:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:21 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (3 skipped not due, 0 skipped no container)
2026/09/24 11:46:21 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.7:8999: connect: connection refused
2026/09/24 11:46:21 healthprobe.go:181: [DEBUG] Health probe wishlist: HTTP GET :3000/ → 307 (7ms)
2026/09/24 11:46:21 healthprobe.go:159: [WARN] Health probes: 1 ok, 1 unhealthy (of 2 probed)
2026/09/24 11:46:25 main.go:2367: [INFO] [bootrecon] boot window: fleet settled after 20s (3 identical samples 5s apart) — sweeping
2026/09/24 11:46:25 bootrecon.go:236: [INFO] [bootrecon] "gokapi" is a boot orphan by intent but is HELD (held after a failed restore whose rollback also failed (2026-09-24T10:37:48Z) — clear the hold to start it) — NOT starting it; whatever is holding it owns its recovery
2026/09/24 11:46:25 bootrecon.go:251: [INFO] [bootrecon] Boot reconciliation: nothing to start — 1 app(s) held (absent drive / quiesce / app-data operation): [gokapi]
2026/09/24 11:46:31 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 11:46:31 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 11:46:31 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/09/24 11:46:31 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/09/24 11:46:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:31 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (4 skipped not due, 0 skipped no container)
2026/09/24 11:46:31 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.7:8999: connect: connection refused
2026/09/24 11:46:31 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed)
2026/09/24 11:46:41 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 11:46:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 11:46:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:41 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 0 skipped no container)
2026/09/24 11:46:41 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 11:46:51 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 11:46:51 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 11:46:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 11:46:51 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 3 targets (4 skipped not due, 0 skipped no container)
2026/09/24 11:46:51 healthprobe.go:181: [DEBUG] Health probe romm: HTTP GET :8080/ → 200 (1ms)
2026/09/24 11:46:51 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.7:8999: connect: connection refused
2026/09/24 11:46:51 healthprobe.go:181: [DEBUG] Health probe paperless-ngx: HTTP GET :8000/ → 302 (6ms)
2026/09/24 11:46:51 healthprobe.go:159: [WARN] Health probes: 2 ok, 1 unhealthy (of 3 probed)
2026/09/24 11:47:00 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true)
2026/09/24 11:47:00 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5236MB avail≈20662MB
2026/09/24 11:47:00 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24504156 → total=25898MB avail=20662MB used=5236MB (20.2%)
2026/09/24 11:47:00 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=22.5GB avail=42.4GB (32.8%)
2026/09/24 11:47:00 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%)
2026/09/24 11:47:00 info.go:13: [DEBUG] [system] readLoadAvg: raw="6.18 3.55 2.97 8/1495 392" → 1m=6.18 5m=3.55 15m=2.97
2026/09/24 11:47:00 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
2026/09/24 11:47:00 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 76.9°C (hwmon2)
2026/09/24 11:47:00 info.go:13: [DEBUG] [system] GetInfo done in 74ms — mem=5236MB/25898MB (20.2%), rootDisk=22.5GB/68.4GB (32.8%), load=6.18/3.55/2.97, temp=76.9°C (hwmon2), cpu=53.9%
2026/09/24 11:47:01 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/09/24 11:47:01 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 11:47:01 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 11:47:01 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/09/24 11:47:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 11:47:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:01 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container)
2026/09/24 11:47:11 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 11:47:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:11 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 11:47:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:11 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container)
2026/09/24 11:47:11 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.7:8999: connect: connection refused
2026/09/24 11:47:11 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed)
2026/09/24 11:47:21 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 11:47:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:21 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 11:47:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:21 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container)
2026/09/24 11:47:21 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.7:8999: connect: connection refused
2026/09/24 11:47:21 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed)
2026/09/24 11:47:30 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=22.5 GB avail=42.4GB (32.8%)
2026/09/24 11:47:30 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:47:30 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:47:30 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:47:30 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:47:30 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 11:47:30 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=22.5 GB avail=42.4GB (32.8%)
2026/09/24 11:47:30 fillwatch.go:256: [INFO] [fillwatch] checked 7 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
2026/09/24 11:47:31 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 11:47:31 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 11:47:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 11:47:31 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/09/24 11:47:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:31 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/09/24 11:47:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:31 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container)
2026/09/24 11:47:31 main.go:872: [WARN] [deadapp] chaosoom: container chaosoom was OOM-killed (started 2026-09-24T11:47:10.100857309Z; kills 15, limit 32M, peak 32M)
2026/09/24 11:47:31 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_oom (severity warning) — further app_oom events are logged at DEBUG only
2026/09/24 11:47:41 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 11:47:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:41 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 11:47:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:41 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container)
2026/09/24 11:47:41 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.7:8999: connect: connection refused
2026/09/24 11:47:41 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed)
2026/09/24 11:47:51 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 11:47:51 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 11:47:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m40s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m40s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m40s ago, effective interval 5m0s, healthy=true
2026/09/24 11:47:51 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container)
2026/09/24 11:47:51 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.7:8999: connect: connection refused
2026/09/24 11:47:51 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed)
2026/09/24 11:48:00 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true)
2026/09/24 11:48:00 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5188MB avail≈20710MB
2026/09/24 11:48:00 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24561624 → total=25898MB avail=20710MB used=5188MB (20.0%)
2026/09/24 11:48:00 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=22.5GB avail=42.4GB (32.8%)
2026/09/24 11:48:00 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%)
2026/09/24 11:48:00 info.go:13: [DEBUG] [system] readLoadAvg: raw="4.01 3.41 2.95 4/1501 529" → 1m=4.01 5m=3.41 15m=2.95
2026/09/24 11:48:00 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
2026/09/24 11:48:00 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 69.5°C (hwmon2)
2026/09/24 11:48:00 info.go:13: [DEBUG] [system] GetInfo done in 61ms — mem=5188MB/25898MB (20.0%), rootDisk=22.5GB/68.4GB (32.8%), load=4.01/3.41/2.95, temp=69.5°C (hwmon2), cpu=32.9%
2026/09/24 11:48:01 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 11:48:01 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/09/24 11:48:01 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/09/24 11:48:01 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan
2026/09/24 11:48:01 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting
2026/09/24 11:48:01 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 11:48:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/09/24 11:48:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m40s ago, effective interval 5m0s, healthy=true
2026/09/24 11:48:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/09/24 11:48:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 11:48:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/09/24 11:48:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 11:48:01 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container)
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml
2026/09/24 11:48:01 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.7:8999: connect: connection refused
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=false composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 11:48:01 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml
2026/09/24 11:48:01 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates
2026/09/24 11:48:01 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection
2026/09/24 11:48:01 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (9 deployed, 49 available)
2026/09/24 11:48:01 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 91ms)
2026/09/24 11:48:01 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed)
2026/09/24 11:48:01 main.go:872: [WARN] [deadapp] chaosoom: container chaosoom was OOM-killed (started 2026-09-24T11:47:10.100857309Z; kills 38, limit 32M, peak 32M)
2026/09/24 11:48:01 notifier.go:738: [ERROR] [notify] chaosoom: container chaosoom OOM STORM — 38 kills in 30 min (limit 32M, peak 32M)
2026/09/24 11:48:01 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_oom_storm (severity error) — further app_oom_storm events are logged at DEBUG only
2026/09/24 11:48:01 main.go:3671: [WARN] [deadapp] chaosoom: oom_storm — 23 in 30m0s; STOPPING it (decision 28)
2026/09/24 11:48:01 manager.go:1283: [DEBUG] [stacks] StopStack chaosoom: current state=running deployed=true containers=1
2026/09/24 11:48:01 manager.go:1286: [INFO] [stacks] Stopping stack: chaosoom
2026/09/24 11:48:01 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/09/24 11:48:01 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/chaosoom)
2026/09/24 11:48:11 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 11:48:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 11:48:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/09/24 11:48:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m0s ago, effective interval 5m0s, healthy=true
2026/09/24 11:48:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 11:48:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m0s ago, effective interval 5m0s, healthy=true
2026/09/24 11:48:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 11:48:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m0s ago, effective interval 5m0s, healthy=true
2026/09/24 11:48:11 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container)
2026/09/24 11:48:11 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 11:48:11 manager.go:1483: [DEBUG] Command completed: docker compose down (took 10.3s)
2026/09/24 11:48:11 manager.go:1295: [INFO] [stacks] Stack chaosoom stopped successfully (took 10.3s)
2026/09/24 11:48:11 settings.go:1814: [WARN] [settings] restore hold SET for chaosoom — the app stays stopped until it is cleared
2026/09/24 11:48:11 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (17886 bytes)
2026/09/24 11:48:11 settings.go:847: [INFO] [settings] Settings saved
2026/09/24 11:48:11 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (17926 bytes)
2026/09/24 11:48:11 settings.go:847: [INFO] [settings] Settings saved
2026/09/24 11:48:11 update_guard.go:776: [WARN] [backup] chaosoom is STOPPED by the box: oom_storm (trip 1 within 24h0m0s) — Start gives it one more try (decision 28)
2026/09/24 11:48:11 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_stopped_unhealthy (severity warning) — further app_stopped_unhealthy events are logged at DEBUG only
2026/09/24 11:48:13 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.6:51320 (X-Forwarded-For: 192.168.0.180)
2026/09/24 11:48:13 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session
2026/09/24 11:48:13 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T11:48:13Z, active_sessions=2
2026/09/24 11:48:13 auth.go:222: [INFO] [web] Login from 172.18.0.6:51320
2026/09/24 11:48:13 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.6:51320
2026/09/24 11:48:13 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.6:51320
2026/09/24 11:48:13 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoom from 172.18.0.6:51320
2026/09/24 11:48:13 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoom from 172.18.0.6:51320
2026/09/24 11:48:13 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoom from 172.18.0.6:51320
2026/09/24 11:48:13 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoom from 172.18.0.6:51320
Reference in New Issue View Git Blame Copy Permalink