Files
felhom.eu/documentation/audits/night-2026-09-24/E/round-07-controller.log
T
2026-09-24 14:21:14 +02:00

623 lines
82 KiB
Plaintext

2026/09/24 12:14:30 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.10:35772 (X-Forwarded-For: 192.168.0.180)
2026/09/24 12:14:30 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session
2026/09/24 12:14:30 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T12:14:30Z, active_sessions=5
2026/09/24 12:14:30 auth.go:222: [INFO] [web] Login from 172.18.0.10:35772
2026/09/24 12:14:30 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.10:35772
2026/09/24 12:14:30 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.10:35772
2026/09/24 12:14:30 router.go:1065: [INFO] [api] Manual catalog sync requested
2026/09/24 12:14:32 router.go:362: [INFO] [api] Manual stack rescan requested
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=false composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 12:14:32 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml
2026/09/24 12:14:32 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates
2026/09/24 12:14:32 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection
2026/09/24 12:14:32 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (9 deployed, 49 available)
2026/09/24 12:14:32 router.go:369: [INFO] [api] Stack rescan completed: 58 stacks found
2026/09/24 12:14:34 router.go:431: [INFO] [api] Deploy requested for stack: chaoscrash
2026/09/24 12:14:34 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%)
2026/09/24 12:14:34 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:14:34 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:14:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:14:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 25s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:34 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container)
2026/09/24 12:14:34 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5475MB avail≈20423MB
2026/09/24 12:14:34 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24220444 → total=25898MB avail=20423MB used=5475MB (21.1%)
2026/09/24 12:14:35 deploy.go:1250: [INFO] [stacks] Memory check: total=25898MB, reserved=384MB, usable=25514MB, committed_used=1818MB, new_req=8MB, remaining=23688MB
2026/09/24 12:14:35 deploy.go:253: [DEBUG] Deploy chaoscrash: received 2 user values
2026/09/24 12:14:35 deploy.go:258: [DEBUG] SUBDOMAIN = "chaoscrash"
2026/09/24 12:14:35 deploy.go:258: [DEBUG] DOMAIN = "enkisfelhom.hu"
2026/09/24 12:14:35 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/chaoscrash — 2 env vars, 0 encrypted, 0 sensitive fields
2026/09/24 12:14:35 [INFO] [stacks] SaveAppConfig: saved config for chaoscrash
2026/09/24 12:14:35 deploy.go:383: [INFO] [stacks] Deploying stack chaoscrash with 2 env vars: [DOMAIN, SUBDOMAIN]
2026/09/24 12:14:35 manager.go:1582: [INFO] [stacks] Deploying stack chaoscrash — checking 1 images...
2026/09/24 12:14:35 manager.go:1588: [DEBUG] alpine:3.20 — found locally
2026/09/24 12:14:35 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_deploy_started (severity info) — further app_deploy_started events are logged at DEBUG only
2026/09/24 12:14:35 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/09/24 12:14:35 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/chaoscrash)
2026/09/24 12:14:35 lifecycle.go:70: [DEBUG] [integrations] OnStackStart: stack=chaoscrash, waiting 5s for state refresh
2026/09/24 12:14:35 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/chaoscrash — 2 env vars, 0 encrypted, 0 sensitive fields
2026/09/24 12:14:35 [INFO] [stacks] SaveAppConfig: saved config for chaoscrash
2026/09/24 12:14:35 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 0.4s)
2026/09/24 12:14:35 deploy.go:462: [INFO] [stacks] Stack chaoscrash deployed successfully (took 0.4s)
2026/09/24 12:14:35 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/chaoscrash — 2 env vars, 0 encrypted, 0 sensitive fields
2026/09/24 12:14:35 [INFO] [stacks] SaveAppConfig: saved config for chaoscrash
2026/09/24 12:14:35 installed.go:420: [INFO] [stacks] installed-images chaoscrash: recorded 1 service(s) (chaoscrash=alpine:3.20 (sha256:d9e853e87e55…))
2026/09/24 12:14:35 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/chaoscrash — 2 env vars, 0 encrypted, 0 sensitive fields
2026/09/24 12:14:35 [INFO] [stacks] SaveAppConfig: saved config for chaoscrash
2026/09/24 12:14:35 pin.go:93: [INFO] [stacks] pin chaoscrash: chaoscrash=alpine:3.20
2026/09/24 12:14:35 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_deployed (severity info) — further app_deployed events are logged at DEBUG only
2026/09/24 12:14:38 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/09/24 12:14:38 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/chaoscrash)
2026/09/24 12:14:38 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.2s)
2026/09/24 12:14:38 manager.go:1543: [INFO] [stacks] Stack chaoscrash post-start status:
2026/09/24 12:14:38 manager.go:1546: [INFO] [stacks] chaoscrash alpine:3.20 restarting Restarting (1) Less than a second ago
2026/09/24 12:14:40 lifecycle.go:86: [DEBUG] [integrations] OnStackStart: stack=chaoscrash integrationsFound=0
2026/09/24 12:14:44 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:14:44 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:14:44 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:44 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:44 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:44 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:44 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 35s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:44 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:44 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:44 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container)
2026/09/24 12:14:44 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused
2026/09/24 12:14:44 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed)
2026/09/24 12:14:54 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:14:54 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/09/24 12:14:54 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/09/24 12:14:54 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:14:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 45s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:14:54 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container)
2026/09/24 12:14:54 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused
2026/09/24 12:14:54 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed)
2026/09/24 12:14:55 router.go:1201: [INFO] [api] Manual app-data backup (DB dump) triggered
2026/09/24 12:14:55 backup.go:522: [INFO] [backup] Starting database dump run
2026/09/24 12:14:55 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
2026/09/24 12:14:55 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: f8196d9721e6 chaoscrash chaoscrash alpine:3.20
642ad1cc5a7c navidrome navidrome deluan/navidrome:0.64.1
124648859ae4 n8n n8n n8nio/n8n:2.31.3
422cba16c96c wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1
22b2c4451d37 vikunja vikunja vikunja/vikunja:2.3.0
6d01f8e3b5ca romm romm rommapp/romm:5.3.1
b613ce74ba5d romm-db romm mariadb:11.8
aecb75c1bdbf romm-redis romm redis:7-alpine
3e311b8f4fdc privatebin privatebin privatebin/pdo:2.0.6
e70d5bc9c661 paperless-webserver paperless-ngx ghcr.io/paperle...
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container chaoscrash (image=alpine:3.20, not a database)
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database)
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.31.3, not a database)
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database)
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database)
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database)
2026/09/24 12:14:55 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=b613ce74ba5d)
2026/09/24 12:14:55 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database)
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database)
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
2026/09/24 12:14:55 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=41af1c5c28cc)
2026/09/24 12:14:55 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
2026/09/24 12:14:55 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database)
2026/09/24 12:14:55 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=55ce027b0fae)
2026/09/24 12:14:55 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database)
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database)
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database)
2026/09/24 12:14:55 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database)
2026/09/24 12:14:55 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 15 non-DB container(s)
2026/09/24 12:14:55 dbdump.go:208: [INFO] [backup] Discovered 3 databases
2026/09/24 12:14:55 backup.go:550: [INFO] [backup] Discovered 3 database(s): romm-db(mariadb), paperless-postgres(postgres), nextcloud-db(mariadb)
2026/09/24 12:14:55 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 12:14:55 dbdump.go:266: [DEBUG] DumpOne: starting dump for container=romm-db, stack=romm, dbType=mariadb, dest=/mnt/felhom-drives/scratch_hdd/userdata/romm/backups/primary/romm/db-dumps/romm-mariadb.sql
2026/09/24 12:14:55 dbdump.go:321: [DEBUG] DumpOne: mariadb-dump command: docker exec b613ce74ba5d mariadb-dump -u root -p*** --single-transaction --routines --triggers romm
2026/09/24 12:14:55 dbdump.go:405: [DEBUG] DumpOne: completed romm-db → romm-mariadb.sql (size=112.9 KB, valid=true, tables=36, duration=313ms)
2026/09/24 12:14:55 dbdump.go:410: [INFO] [backup] DB dump: romm-db → romm-mariadb.sql (112.9 KB, 313ms, 36 tables)
2026/09/24 12:14:55 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (17657 bytes)
2026/09/24 12:14:55 settings.go:847: [INFO] [settings] Settings saved
2026/09/24 12:14:55 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 12:14:55 dbdump.go:266: [DEBUG] DumpOne: starting dump for container=paperless-postgres, stack=paperless-ngx, dbType=postgres, dest=/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx/backups/primary/paperless-ngx/db-dumps/paperless-ngx-postgres.sql
2026/09/24 12:14:55 dbdump.go:303: [DEBUG] DumpOne: pg_dump command: docker exec 41af1c5c28cc pg_dump -U paperless -d paperless --clean --if-exists --no-owner --no-privileges
2026/09/24 12:14:55 dbdump.go:405: [DEBUG] DumpOne: completed paperless-postgres → paperless-ngx-postgres.sql (size=364.1 KB, valid=true, tables=72, duration=388ms)
2026/09/24 12:14:55 dbdump.go:410: [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (364.1 KB, 388ms, 72 tables)
2026/09/24 12:14:55 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (17657 bytes)
2026/09/24 12:14:55 settings.go:847: [INFO] [settings] Settings saved
2026/09/24 12:14:55 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 12:14:55 dbdump.go:266: [DEBUG] DumpOne: starting dump for container=nextcloud-db, stack=nextcloud, dbType=mariadb, dest=/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/nextcloud-mariadb.sql
2026/09/24 12:14:55 dbdump.go:321: [DEBUG] DumpOne: mariadb-dump command: docker exec 55ce027b0fae mariadb-dump -u root -p*** --single-transaction --routines --triggers nextcloud
2026/09/24 12:14:56 dbdump.go:405: [DEBUG] DumpOne: completed nextcloud-db → nextcloud-mariadb.sql (size=855.4 KB, valid=true, tables=131, duration=634ms)
2026/09/24 12:14:56 dbdump.go:410: [INFO] [backup] DB dump: nextcloud-db → nextcloud-mariadb.sql (855.4 KB, 634ms, 131 tables)
2026/09/24 12:14:56 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (17657 bytes)
2026/09/24 12:14:56 settings.go:847: [INFO] [settings] Settings saved
2026/09/24 12:14:56 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%)
2026/09/24 12:14:56 backup.go:920: [INFO] [backup] Stopping navidrome for safe volume dump
2026/09/24 12:14:56 manager.go:1283: [DEBUG] [stacks] StopStack navidrome: current state=running deployed=true containers=1
2026/09/24 12:14:56 manager.go:1286: [INFO] [stacks] Stopping stack: navidrome
2026/09/24 12:14:56 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, HDD_PATH, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (10 app + 0 system)
2026/09/24 12:14:56 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/navidrome)
2026/09/24 12:14:56 manager.go:1483: [DEBUG] Command completed: docker compose down (took 0.3s)
2026/09/24 12:14:56 manager.go:1295: [INFO] [stacks] Stack navidrome stopped successfully (took 0.3s)
2026/09/24 12:14:56 backup.go:796: [DEBUG] [backup] Dumping volume navidrome_navidrome_data for navidrome
2026/09/24 12:14:57 backup.go:821: [INFO] [backup] Volume dump: navidrome/navidrome_navidrome_data → 707.0 KB
2026/09/24 12:14:57 backup.go:930: [INFO] [backup] Restarting navidrome after volume dump
2026/09/24 12:14:57 manager.go:1194: [DEBUG] [stacks] StartStack navidrome: current state=stopped deployed=true
2026/09/24 12:14:57 manager.go:1197: [INFO] [stacks] Starting stack: navidrome
2026/09/24 12:14:57 manager.go:1204: [DEBUG] [stacks] StartStack navidrome: prepared 10 env vars for compose
2026/09/24 12:14:57 manager.go:410: [INFO] [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping ensure (held by drive gate)
2026/09/24 12:14:57 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, HDD_PATH, SUBDOMAIN, DOMAIN, USERDATA_PATH, IMPORT_PATH] (10 app + 0 system)
2026/09/24 12:14:57 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/navidrome)
2026/09/24 12:14:57 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 0.4s)
2026/09/24 12:14:57 manager.go:1212: [INFO] [stacks] Stack navidrome started successfully (took 0.4s)
2026/09/24 12:14:57 installed.go:409: [DEBUG] [stacks] installed-images navidrome: unchanged (1 service(s)) — app.yaml not rewritten
2026/09/24 12:14:57 backup.go:920: [INFO] [backup] Stopping nextcloud for safe volume dump
2026/09/24 12:14:57 manager.go:1283: [DEBUG] [stacks] StopStack nextcloud: current state=running deployed=true containers=3
2026/09/24 12:14:57 manager.go:1286: [INFO] [stacks] Stopping stack: nextcloud
2026/09/24 12:14:57 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system)
2026/09/24 12:14:57 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/nextcloud)
2026/09/24 12:14:59 manager.go:1483: [DEBUG] Command completed: docker compose down (took 1.8s)
2026/09/24 12:14:59 manager.go:1295: [INFO] [stacks] Stack nextcloud stopped successfully (took 1.8s)
2026/09/24 12:14:59 backup.go:796: [DEBUG] [backup] Dumping volume nextcloud_nextcloud_html for nextcloud
2026/09/24 12:15:00 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, HDD_PATH, SUBDOMAIN, DOMAIN, USERDATA_PATH, IMPORT_PATH] (10 app + 0 system)
2026/09/24 12:15:00 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/navidrome)
2026/09/24 12:15:00 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s)
2026/09/24 12:15:00 manager.go:1543: [INFO] [stacks] Stack navidrome post-start status:
2026/09/24 12:15:00 manager.go:1546: [INFO] [stacks] navidrome deluan/navidrome:0.64.1@sha256:df22d661b8d0322c33754d999b567150df9e131a47ae53be71fed0154bf8cef5 running Up 3 seconds (health: starting)
2026/09/24 12:15:02 backup.go:821: [INFO] [backup] Volume dump: nextcloud/nextcloud_nextcloud_html → 761.6 MB
2026/09/24 12:15:02 backup.go:796: [DEBUG] [backup] Dumping volume nextcloud_nextcloud_redis_data for nextcloud
2026/09/24 12:15:03 backup.go:821: [INFO] [backup] Volume dump: nextcloud/nextcloud_nextcloud_redis_data → 281.0 KB
2026/09/24 12:15:03 backup.go:796: [DEBUG] [backup] Dumping volume nextcloud_nextcloud_db_data for nextcloud
2026/09/24 12:15:04 backup.go:821: [INFO] [backup] Volume dump: nextcloud/nextcloud_nextcloud_db_data → 164.9 MB
2026/09/24 12:15:04 backup.go:930: [INFO] [backup] Restarting nextcloud after volume dump
2026/09/24 12:15:04 manager.go:1194: [DEBUG] [stacks] StartStack nextcloud: current state=stopped deployed=true
2026/09/24 12:15:04 manager.go:1197: [INFO] [stacks] Starting stack: nextcloud
2026/09/24 12:15:04 manager.go:1204: [DEBUG] [stacks] StartStack nextcloud: prepared 14 env vars for compose
2026/09/24 12:15:04 manager.go:410: [INFO] [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping ensure (held by drive gate)
2026/09/24 12:15:04 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system)
2026/09/24 12:15:04 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/nextcloud)
2026/09/24 12:15:04 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:15:04 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:15:04 manager.go:851: [DEBUG] [stacks] restart-policy of down member "nextcloud" = "unless-stopped"
2026/09/24 12:15:04 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:04 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:04 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:04 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:04 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:04 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:04 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container)
2026/09/24 12:15:04 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused
2026/09/24 12:15:04 healthprobe.go:181: [DEBUG] Health probe navidrome: API GET :4533/ping → 200 (2ms)
2026/09/24 12:15:04 healthprobe.go:159: [WARN] Health probes: 1 ok, 1 unhealthy (of 2 probed)
2026/09/24 12:15:10 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 6.5s)
2026/09/24 12:15:10 manager.go:1212: [INFO] [stacks] Stack nextcloud started successfully (took 6.5s)
2026/09/24 12:15:10 installed.go:409: [DEBUG] [stacks] installed-images nextcloud: unchanged (3 service(s)) — app.yaml not rewritten
2026/09/24 12:15:10 backup.go:920: [INFO] [backup] Stopping paperless-ngx for safe volume dump
2026/09/24 12:15:10 manager.go:1283: [DEBUG] [stacks] StopStack paperless-ngx: current state=running deployed=true containers=3
2026/09/24 12:15:10 manager.go:1286: [INFO] [stacks] Stopping stack: paperless-ngx
2026/09/24 12:15:10 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, PAPERLESS_SECRET_KEY, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, PAPERLESS_ADMIN_PASSWORD, PAPERLESS_ADMIN_USER, PAPERLESS_OCR_LANGUAGE, USERDATA_PATH, IMPORT_PATH] (15 app + 0 system)
2026/09/24 12:15:10 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/paperless-ngx)
2026/09/24 12:15:13 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system)
2026/09/24 12:15:13 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/nextcloud)
2026/09/24 12:15:14 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s)
2026/09/24 12:15:14 manager.go:1543: [INFO] [stacks] Stack nextcloud post-start status:
2026/09/24 12:15:14 manager.go:1546: [INFO] [stacks] nextcloud nextcloud:34.0.4-apache@sha256:a5ace30c695afe48c2c406e940ee7886a81e13fa382e57cd68b2416d1a66914c running Up 3 seconds (health: starting)
2026/09/24 12:15:14 manager.go:1546: [INFO] [stacks] nextcloud-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy)
2026/09/24 12:15:14 manager.go:1546: [INFO] [stacks] nextcloud-redis redis:7-alpine@sha256:c35af3bbcef51a62c8bae5a9a563c6f1b60d7ebaea4cb5a3ccbcc157580ae098 running Up 9 seconds (healthy)
2026/09/24 12:15:14 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:15:14 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:15:14 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:14 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:14 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:14 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:14 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:14 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:14 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:15:14 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container)
2026/09/24 12:15:18 manager.go:1483: [DEBUG] Command completed: docker compose down (took 7.3s)
2026/09/24 12:15:18 manager.go:1295: [INFO] [stacks] Stack paperless-ngx stopped successfully (took 7.3s)
2026/09/24 12:15:18 backup.go:796: [DEBUG] [backup] Dumping volume paperless-ngx_paperless_data for paperless-ngx
2026/09/24 12:15:18 backup.go:821: [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 1.0 MB
2026/09/24 12:15:18 backup.go:796: [DEBUG] [backup] Dumping volume paperless-ngx_paperless_postgres_data for paperless-ngx
2026/09/24 12:15:19 backup.go:821: [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 67.9 MB
2026/09/24 12:15:19 backup.go:796: [DEBUG] [backup] Dumping volume paperless-ngx_paperless_redis_data for paperless-ngx
2026/09/24 12:15:19 backup.go:821: [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_redis_data → 1.0 MB
2026/09/24 12:15:19 backup.go:930: [INFO] [backup] Restarting paperless-ngx after volume dump
2026/09/24 12:15:19 manager.go:1194: [DEBUG] [stacks] StartStack paperless-ngx: current state=stopped deployed=true
2026/09/24 12:15:19 manager.go:1197: [INFO] [stacks] Starting stack: paperless-ngx
2026/09/24 12:15:19 manager.go:1204: [DEBUG] [stacks] StartStack paperless-ngx: prepared 15 env vars for compose
2026/09/24 12:15:19 manager.go:410: [INFO] [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping ensure (held by drive gate)
2026/09/24 12:15:19 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, PAPERLESS_SECRET_KEY, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, PAPERLESS_ADMIN_PASSWORD, PAPERLESS_ADMIN_USER, PAPERLESS_OCR_LANGUAGE, USERDATA_PATH, IMPORT_PATH] (15 app + 0 system)
2026/09/24 12:15:19 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/paperless-ngx)
2026/09/24 12:15:24 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true)
2026/09/24 12:15:24 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈4995MB avail≈20903MB
2026/09/24 12:15:24 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24789940 → total=25898MB avail=20903MB used=4995MB (19.3%)
2026/09/24 12:15:24 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=25.5GB avail=39.4GB (37.3%)
2026/09/24 12:15:24 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%)
2026/09/24 12:15:24 info.go:13: [DEBUG] [system] readLoadAvg: raw="4.33 3.89 3.47 5/1533 5165" → 1m=4.33 5m=3.89 15m=3.47
2026/09/24 12:15:24 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
2026/09/24 12:15:24 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 69.0°C (hwmon2)
2026/09/24 12:15:24 info.go:13: [DEBUG] [system] GetInfo done in 70ms — mem=4995MB/25898MB (19.3%), rootDisk=25.5GB/68.4GB (37.3%), load=4.33/3.89/3.47, temp=69.0°C (hwmon2), cpu=32.1%
2026/09/24 12:15:24 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:15:24 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan
2026/09/24 12:15:24 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting
2026/09/24 12:15:24 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/09/24 12:15:24 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/09/24 12:15:24 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:15:24 manager.go:851: [DEBUG] [stacks] restart-policy of down member "paperless-webserver" = "unless-stopped"
2026/09/24 12:15:24 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:24 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:24 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:24 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:24 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:24 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:24 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container)
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml
2026/09/24 12:15:24 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=true composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=false composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
2026/09/24 12:15:24 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml
2026/09/24 12:15:25 healthprobe.go:181: [DEBUG] Health probe nextcloud: API GET :80/status.php → 200 (46ms)
2026/09/24 12:15:25 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml
2026/09/24 12:15:25 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml
2026/09/24 12:15:25 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml
2026/09/24 12:15:25 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml
2026/09/24 12:15:25 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml
2026/09/24 12:15:25 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml
2026/09/24 12:15:25 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml
2026/09/24 12:15:25 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml
2026/09/24 12:15:25 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml
2026/09/24 12:15:25 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml
2026/09/24 12:15:25 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml
2026/09/24 12:15:25 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml
2026/09/24 12:15:25 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml
2026/09/24 12:15:25 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates
2026/09/24 12:15:25 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection
2026/09/24 12:15:25 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available)
2026/09/24 12:15:25 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 129ms)
2026/09/24 12:15:25 healthprobe.go:159: [WARN] Health probes: 1 ok, 1 unhealthy (of 2 probed)
2026/09/24 12:15:31 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 11.3s)
2026/09/24 12:15:31 manager.go:1212: [INFO] [stacks] Stack paperless-ngx started successfully (took 11.3s)
2026/09/24 12:15:31 installed.go:409: [DEBUG] [stacks] installed-images paperless-ngx: unchanged (3 service(s)) — app.yaml not rewritten
2026/09/24 12:15:31 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=25.5 GB avail=39.4GB (37.3%)
2026/09/24 12:15:31 backup.go:920: [INFO] [backup] Stopping privatebin for safe volume dump
2026/09/24 12:15:31 manager.go:1283: [DEBUG] [stacks] StopStack privatebin: current state=running deployed=true containers=1
2026/09/24 12:15:31 manager.go:1286: [INFO] [stacks] Stopping stack: privatebin
2026/09/24 12:15:31 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/09/24 12:15:31 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/privatebin)
2026/09/24 12:15:31 manager.go:1483: [DEBUG] Command completed: docker compose down (took 0.4s)
2026/09/24 12:15:31 manager.go:1295: [INFO] [stacks] Stack privatebin stopped successfully (took 0.4s)
2026/09/24 12:15:31 backup.go:796: [DEBUG] [backup] Dumping volume privatebin_privatebin_data for privatebin
2026/09/24 12:15:32 backup.go:821: [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 1.5 KB
2026/09/24 12:15:32 backup.go:930: [INFO] [backup] Restarting privatebin after volume dump
2026/09/24 12:15:32 manager.go:1194: [DEBUG] [stacks] StartStack privatebin: current state=stopped deployed=true
2026/09/24 12:15:32 manager.go:1197: [INFO] [stacks] Starting stack: privatebin
2026/09/24 12:15:32 manager.go:1204: [DEBUG] [stacks] StartStack privatebin: prepared 8 env vars for compose
2026/09/24 12:15:32 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/09/24 12:15:32 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/privatebin)
2026/09/24 12:15:32 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 0.4s)
2026/09/24 12:15:32 manager.go:1212: [INFO] [stacks] Stack privatebin started successfully (took 0.4s)
2026/09/24 12:15:32 installed.go:409: [DEBUG] [stacks] installed-images privatebin: unchanged (1 service(s)) — app.yaml not rewritten
2026/09/24 12:15:32 backup.go:920: [INFO] [backup] Stopping romm for safe volume dump
2026/09/24 12:15:32 manager.go:1283: [DEBUG] [stacks] StopStack romm: current state=running deployed=true containers=3
2026/09/24 12:15:32 manager.go:1286: [INFO] [stacks] Stopping stack: romm
2026/09/24 12:15:32 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, ROMM_AUTH_SECRET_KEY, SUBDOMAIN, DB_PASSWORD, USERDATA_PATH, IMPORT_PATH] (13 app + 0 system)
2026/09/24 12:15:32 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/romm)
2026/09/24 12:15:34 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, PAPERLESS_SECRET_KEY, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, PAPERLESS_ADMIN_PASSWORD, PAPERLESS_ADMIN_USER, PAPERLESS_OCR_LANGUAGE, USERDATA_PATH, IMPORT_PATH] (15 app + 0 system)
2026/09/24 12:15:34 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/paperless-ngx)
2026/09/24 12:15:34 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s)
2026/09/24 12:15:34 manager.go:1543: [INFO] [stacks] Stack paperless-ngx post-start status:
2026/09/24 12:15:34 manager.go:1546: [INFO] [stacks] paperless-postgres postgres:16-alpine running Up 14 seconds (healthy)
2026/09/24 12:15:34 manager.go:1546: [INFO] [stacks] paperless-redis redis:7-alpine running Up 14 seconds (healthy)
2026/09/24 12:15:34 manager.go:1546: [INFO] [stacks] paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 running Up 3 seconds (health: starting)
2026/09/24 12:15:34 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:15:34 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:15:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:34 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:15:34 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (6 skipped not due, 0 skipped no container)
2026/09/24 12:15:35 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/09/24 12:15:35 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/privatebin)
2026/09/24 12:15:35 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.2s)
2026/09/24 12:15:35 manager.go:1543: [INFO] [stacks] Stack privatebin post-start status:
2026/09/24 12:15:35 manager.go:1546: [INFO] [stacks] privatebin privatebin/pdo:2.0.6@sha256:4c141b2326f8b353598ce9ce7507a9cfecf2dad5c60a39fea903d430e296d8f5 running Up 3 seconds (health: starting)
2026/09/24 12:15:38 manager.go:1483: [DEBUG] Command completed: docker compose down (took 5.5s)
2026/09/24 12:15:38 manager.go:1295: [INFO] [stacks] Stack romm stopped successfully (took 5.5s)
2026/09/24 12:15:38 backup.go:796: [DEBUG] [backup] Dumping volume romm_romm_redis_data for romm
2026/09/24 12:15:38 backup.go:821: [INFO] [backup] Volume dump: romm/romm_romm_redis_data → 15.6 MB
2026/09/24 12:15:38 backup.go:796: [DEBUG] [backup] Dumping volume romm_romm_config for romm
2026/09/24 12:15:39 backup.go:821: [INFO] [backup] Volume dump: romm/romm_romm_config → 2.5 KB
2026/09/24 12:15:39 backup.go:796: [DEBUG] [backup] Dumping volume romm_romm_db_data for romm
2026/09/24 12:15:40 backup.go:821: [INFO] [backup] Volume dump: romm/romm_romm_db_data → 156.4 MB
2026/09/24 12:15:40 backup.go:930: [INFO] [backup] Restarting romm after volume dump
2026/09/24 12:15:40 manager.go:1194: [DEBUG] [stacks] StartStack romm: current state=stopped deployed=true
2026/09/24 12:15:40 manager.go:1197: [INFO] [stacks] Starting stack: romm
2026/09/24 12:15:40 manager.go:1204: [DEBUG] [stacks] StartStack romm: prepared 13 env vars for compose
2026/09/24 12:15:40 manager.go:410: [INFO] [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping ensure (held by drive gate)
2026/09/24 12:15:40 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, ROMM_AUTH_SECRET_KEY, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, USERDATA_PATH, IMPORT_PATH] (13 app + 0 system)
2026/09/24 12:15:40 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/romm)
2026/09/24 12:15:44 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:15:44 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:15:44 manager.go:851: [DEBUG] [stacks] restart-policy of down member "romm" = "unless-stopped"
2026/09/24 12:15:44 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:44 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:44 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:44 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:44 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:44 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (5 skipped not due, 0 skipped no container)
2026/09/24 12:15:44 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused
2026/09/24 12:15:44 healthprobe.go:181: [DEBUG] Health probe privatebin: HTTP GET :8080/ → 200 (5ms)
2026/09/24 12:15:44 healthprobe.go:159: [WARN] Health probes: 1 ok, 1 unhealthy (of 2 probed)
2026/09/24 12:15:51 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 11.4s)
2026/09/24 12:15:51 manager.go:1212: [INFO] [stacks] Stack romm started successfully (took 11.4s)
2026/09/24 12:15:51 installed.go:409: [DEBUG] [stacks] installed-images romm: unchanged (3 service(s)) — app.yaml not rewritten
2026/09/24 12:15:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%)
2026/09/24 12:15:51 backup.go:920: [INFO] [backup] Stopping vikunja for safe volume dump
2026/09/24 12:15:51 manager.go:1283: [DEBUG] [stacks] StopStack vikunja: current state=unhealthy deployed=true containers=1
2026/09/24 12:15:51 manager.go:1286: [INFO] [stacks] Stopping stack: vikunja
2026/09/24 12:15:51 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, VIKUNJA_SERVICE_JWTSECRET, IMPORT_PATH] (9 app + 0 system)
2026/09/24 12:15:51 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/vikunja)
2026/09/24 12:15:52 manager.go:1483: [DEBUG] Command completed: docker compose down (took 0.4s)
2026/09/24 12:15:52 manager.go:1295: [INFO] [stacks] Stack vikunja stopped successfully (took 0.4s)
2026/09/24 12:15:52 backup.go:796: [DEBUG] [backup] Dumping volume vikunja_vikunja_data for vikunja
2026/09/24 12:15:52 backup.go:821: [INFO] [backup] Volume dump: vikunja/vikunja_vikunja_data → 1.5 KB
2026/09/24 12:15:52 backup.go:796: [DEBUG] [backup] Dumping volume vikunja_vikunja_db for vikunja
2026/09/24 12:15:53 backup.go:821: [INFO] [backup] Volume dump: vikunja/vikunja_vikunja_db → 2.0 MB
2026/09/24 12:15:53 backup.go:930: [INFO] [backup] Restarting vikunja after volume dump
2026/09/24 12:15:53 manager.go:1194: [DEBUG] [stacks] StartStack vikunja: current state=stopped deployed=true
2026/09/24 12:15:53 manager.go:1197: [INFO] [stacks] Starting stack: vikunja
2026/09/24 12:15:53 manager.go:1204: [DEBUG] [stacks] StartStack vikunja: prepared 9 env vars for compose
2026/09/24 12:15:53 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, VIKUNJA_SERVICE_JWTSECRET, IMPORT_PATH] (9 app + 0 system)
2026/09/24 12:15:53 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/vikunja)
2026/09/24 12:15:54 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 0.6s)
2026/09/24 12:15:54 manager.go:1212: [INFO] [stacks] Stack vikunja started successfully (took 0.6s)
2026/09/24 12:15:54 installed.go:409: [DEBUG] [stacks] installed-images vikunja: unchanged (1 service(s)) — app.yaml not rewritten
2026/09/24 12:15:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%)
2026/09/24 12:15:54 backup.go:920: [INFO] [backup] Stopping wishlist for safe volume dump
2026/09/24 12:15:54 manager.go:1283: [DEBUG] [stacks] StopStack wishlist: current state=running deployed=true containers=1
2026/09/24 12:15:54 manager.go:1286: [INFO] [stacks] Stopping stack: wishlist
2026/09/24 12:15:54 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/09/24 12:15:54 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/wishlist)
2026/09/24 12:15:54 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, ROMM_AUTH_SECRET_KEY, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, USERDATA_PATH, IMPORT_PATH] (13 app + 0 system)
2026/09/24 12:15:54 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/romm)
2026/09/24 12:15:54 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:15:54 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/09/24 12:15:54 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/09/24 12:15:54 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:15:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 2m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:15:54 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (5 skipped not due, 0 skipped no container)
2026/09/24 12:15:54 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused
2026/09/24 12:15:54 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed)
2026/09/24 12:15:55 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.3s)
2026/09/24 12:15:55 manager.go:1543: [INFO] [stacks] Stack romm post-start status:
2026/09/24 12:15:55 manager.go:1546: [INFO] [stacks] romm rommapp/romm:5.3.1@sha256:0d66b4ea152a237c7f24b95e87459f46d3c596a7c0b35a82f26bc971487278b3 running Up 3 seconds (health: starting)
2026/09/24 12:15:55 manager.go:1546: [INFO] [stacks] romm-db mariadb:11.8@sha256:79d59758afc91b89b120b0a8904d637f5a3b3e1c4900f29b740d6d46c72fef68 running Up 14 seconds (healthy)
2026/09/24 12:15:55 manager.go:1546: [INFO] [stacks] romm-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy)
2026/09/24 12:15:55 manager.go:1483: [DEBUG] Command completed: docker compose down (took 0.9s)
2026/09/24 12:15:55 manager.go:1295: [INFO] [stacks] Stack wishlist stopped successfully (took 0.9s)
2026/09/24 12:15:55 backup.go:796: [DEBUG] [backup] Dumping volume wishlist_wishlist_data for wishlist
2026/09/24 12:15:55 manager.go:1502: [ERROR] [stacks] execCommand failed: exit status 1
2026/09/24 12:15:56 backup.go:821: [INFO] [backup] Volume dump: wishlist/wishlist_wishlist_data → 302.0 KB
2026/09/24 12:15:56 backup.go:796: [DEBUG] [backup] Dumping volume wishlist_wishlist_uploads for wishlist
2026/09/24 12:15:56 backup.go:821: [INFO] [backup] Volume dump: wishlist/wishlist_wishlist_uploads → 1.5 KB
2026/09/24 12:15:56 backup.go:930: [INFO] [backup] Restarting wishlist after volume dump
2026/09/24 12:15:56 manager.go:1194: [DEBUG] [stacks] StartStack wishlist: current state=stopped deployed=true
2026/09/24 12:15:56 manager.go:1197: [INFO] [stacks] Starting stack: wishlist
2026/09/24 12:15:56 manager.go:1204: [DEBUG] [stacks] StartStack wishlist: prepared 8 env vars for compose
2026/09/24 12:15:56 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/09/24 12:15:56 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/wishlist)
2026/09/24 12:15:57 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, VIKUNJA_SERVICE_JWTSECRET, IMPORT_PATH] (9 app + 0 system)
2026/09/24 12:15:57 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/vikunja)
2026/09/24 12:15:57 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.3s)
2026/09/24 12:15:57 manager.go:1543: [INFO] [stacks] Stack vikunja post-start status:
2026/09/24 12:15:57 manager.go:1546: [INFO] [stacks] vikunja vikunja/vikunja:2.3.0 running Up 3 seconds
2026/09/24 12:15:57 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 0.9s)
2026/09/24 12:15:57 manager.go:1212: [INFO] [stacks] Stack wishlist started successfully (took 0.9s)
2026/09/24 12:15:58 installed.go:409: [DEBUG] [stacks] installed-images wishlist: unchanged (1 service(s)) — app.yaml not rewritten
2026/09/24 12:15:58 backup.go:712: [DEBUG] [backup] chaoscrash has no named volumes — volume dump skipped
2026/09/24 12:15:58 backup.go:705: [WARN] [backup] Skipping volume dump for chaosoom — the app is HELD stopped (its restore point is preserved)
2026/09/24 12:15:58 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%)
2026/09/24 12:15:58 backup.go:920: [INFO] [backup] Stopping n8n for safe volume dump
2026/09/24 12:15:58 manager.go:1283: [DEBUG] [stacks] StopStack n8n: current state=running deployed=true containers=1
2026/09/24 12:15:58 manager.go:1286: [INFO] [stacks] Stopping stack: n8n
2026/09/24 12:15:58 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, DOMAIN, IMPORT_PATH] (9 app + 0 system)
2026/09/24 12:15:58 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/n8n)
2026/09/24 12:15:59 manager.go:1483: [DEBUG] Command completed: docker compose down (took 1.4s)
2026/09/24 12:15:59 manager.go:1295: [INFO] [stacks] Stack n8n stopped successfully (took 1.4s)
2026/09/24 12:15:59 backup.go:796: [DEBUG] [backup] Dumping volume n8n_n8n_data for n8n
2026/09/24 12:16:00 backup.go:821: [INFO] [backup] Volume dump: n8n/n8n_n8n_data → 5.4 MB
2026/09/24 12:16:00 backup.go:930: [INFO] [backup] Restarting n8n after volume dump
2026/09/24 12:16:00 manager.go:1194: [DEBUG] [stacks] StartStack n8n: current state=stopped deployed=true
2026/09/24 12:16:00 manager.go:1197: [INFO] [stacks] Starting stack: n8n
2026/09/24 12:16:00 manager.go:1204: [DEBUG] [stacks] StartStack n8n: prepared 9 env vars for compose
2026/09/24 12:16:00 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, IMPORT_PATH] (9 app + 0 system)
2026/09/24 12:16:00 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/n8n)
2026/09/24 12:16:01 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/09/24 12:16:01 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/wishlist)
2026/09/24 12:16:01 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 0.7s)
2026/09/24 12:16:01 manager.go:1212: [INFO] [stacks] Stack n8n started successfully (took 0.7s)
2026/09/24 12:16:01 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.3s)
2026/09/24 12:16:01 manager.go:1543: [INFO] [stacks] Stack wishlist post-start status:
2026/09/24 12:16:01 manager.go:1546: [INFO] [stacks] wishlist ghcr.io/cmintey/wishlist:v0.67.1@sha256:4b89bfb5b57ffb087c33978d21b09048e5c37f4506326e9ccd2705a0a1d86548 running Up 4 seconds (health: starting)
2026/09/24 12:16:01 installed.go:409: [DEBUG] [stacks] installed-images n8n: unchanged (1 service(s)) — app.yaml not rewritten
2026/09/24 12:16:01 backup.go:636: [INFO] [backup] App-data backup completed: 3 databases (1.3 MB total), 8 volume dump(s) (1m6.606s)
2026/09/24 12:16:01 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=25.4 GB avail=39.4GB (37.2%)
2026/09/24 12:16:01 recovery_unit.go:224: [INFO] [backup] Recovery unit captured for chaoscrash → /mnt/sys_drive/felhom-data/backups/primary/chaoscrash (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/09/24 12:16:04 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, IMPORT_PATH] (9 app + 0 system)
2026/09/24 12:16:04 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/n8n)
2026/09/24 12:16:04 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.2s)
2026/09/24 12:16:04 manager.go:1543: [INFO] [stacks] Stack n8n post-start status:
2026/09/24 12:16:04 manager.go:1546: [INFO] [stacks] n8n n8nio/n8n:2.31.3 running Up 4 seconds (health: starting)
2026/09/24 12:16:04 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:16:04 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:16:04 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:16:04 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:16:04 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:16:04 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:16:04 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (4 skipped not due, 0 skipped no container)
2026/09/24 12:16:14 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:16:14 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:16:14 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/24 12:16:14 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:16:14 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/24 12:16:14 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 3 targets (3 skipped not due, 0 skipped no container)
2026/09/24 12:16:14 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused
2026/09/24 12:16:14 healthprobe.go:181: [DEBUG] Health probe paperless-ngx: HTTP GET :8000/ → 302 (7ms)
2026/09/24 12:16:14 healthprobe.go:181: [DEBUG] Health probe wishlist: HTTP GET :3000/ → 307 (8ms)
2026/09/24 12:16:14 healthprobe.go:159: [WARN] Health probes: 2 ok, 1 unhealthy (of 3 probed)
2026/09/24 12:16:24 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true)
2026/09/24 12:16:24 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5260MB avail≈20638MB
2026/09/24 12:16:24 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24474944 → total=25898MB avail=20638MB used=5260MB (20.3%)
2026/09/24 12:16:24 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=25.5GB avail=39.3GB (37.3%)
2026/09/24 12:16:24 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%)
2026/09/24 12:16:24 info.go:13: [DEBUG] [system] readLoadAvg: raw="6.22 4.52 3.72 9/1506 6307" → 1m=6.22 5m=4.52 15m=3.72
2026/09/24 12:16:24 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
2026/09/24 12:16:24 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 75.0°C (hwmon2)
2026/09/24 12:16:24 info.go:13: [DEBUG] [system] GetInfo done in 60ms — mem=5260MB/25898MB (20.3%), rootDisk=25.5GB/68.4GB (37.3%), load=6.22/4.52/3.72, temp=75.0°C (hwmon2), cpu=59.7%
2026/09/24 12:16:24 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/09/24 12:16:24 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/09/24 12:16:24 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/09/24 12:16:24 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/09/24 12:16:24 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false
2026/09/24 12:16:24 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/24 12:16:24 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:16:24 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/09/24 12:16:24 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/24 12:16:24 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/24 12:16:24 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container)
2026/09/24 12:16:24 healthprobe.go:181: [DEBUG] Health probe n8n: API GET :5678/healthz → 200 (3ms)
2026/09/24 12:16:24 healthprobe.go:161: [INFO] Health probes: 1 ok (of 1 probed)
2026/09/24 12:16:25 main.go:3671: [WARN] [deadapp] chaoscrash: crash_loop — 10 in 10m0s; STOPPING it (decision 28)
2026/09/24 12:16:25 manager.go:1283: [DEBUG] [stacks] StopStack chaoscrash: current state=restarting deployed=true containers=1
2026/09/24 12:16:25 manager.go:1286: [INFO] [stacks] Stopping stack: chaoscrash
2026/09/24 12:16:25 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/09/24 12:16:25 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/chaoscrash)
2026/09/24 12:16:25 manager.go:1483: [DEBUG] Command completed: docker compose down (took 0.1s)
2026/09/24 12:16:25 manager.go:1295: [INFO] [stacks] Stack chaoscrash stopped successfully (took 0.1s)
2026/09/24 12:16:25 settings.go:1814: [WARN] [settings] restore hold SET for chaoscrash — the app stays stopped until it is cleared
2026/09/24 12:16:25 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (17837 bytes)
2026/09/24 12:16:25 settings.go:847: [INFO] [settings] Settings saved
2026/09/24 12:16:25 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (17879 bytes)
2026/09/24 12:16:25 settings.go:847: [INFO] [settings] Settings saved
2026/09/24 12:16:25 update_guard.go:776: [WARN] [backup] chaoscrash is STOPPED by the box: crash_loop (trip 1 within 24h0m0s) — Start gives it one more try (decision 28)
2026/09/24 12:16:25 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_stopped_unhealthy (severity warning) — further app_stopped_unhealthy events are logged at DEBUG only
2026/09/24 12:16:26 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaoscrash from 172.18.0.10:35772
2026/09/24 12:16:26 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaoscrash from 172.18.0.10:35772
2026/09/24 12:16:26 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaoscrash from 172.18.0.10:35772
2026/09/24 12:16:26 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaoscrash from 172.18.0.10:35772