diff --git a/STATUS.md b/STATUS.md index 46d8335e..def4dce9 100644 --- a/STATUS.md +++ b/STATUS.md @@ -24,7 +24,7 @@ - DooPlex: something sets the shared checkout to „bare" (R-935). ## What we work on -- Dated checks, run by CC. 2026-10-26: tonight's database copy reaches the restic sweep (R-232); each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes). +- Dated checks, run by CC. 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes). ## Waiting on you — decide @@ -50,13 +50,13 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14. 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12. -**Monday 2026-10-26, release list (attended).** 1. Controller v0.343.0: the restore and backup fixes (R-1251–R-1253, R-1269–R-1271), R-1250. If not: the P1 stays open. 2. Agent v0.165.0 + your vouch: BYO keeps its SSH (R-1257). 3. Hub v0.170.0: the recovery package in one step (R-1254). 4. Catalog: n8n usage data off; OpenGist 256 MB. **Day, 2026-10-23 — your do list.** 1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens. -2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-aa9466. If not: two dead entries alarm. +2. On the hub's Hosts page: Discard appliance 49 (host Tester4-aa9466 is gone). If not: a dead entry alarms. 3. Give the homelab session R-1241, R-1242, the lists at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` and `documentation/audits/dooplex-alarms-2026-10-24/README.md`, and an upload alarm on DooPlex's `eno1` (~100 Mbit/s line; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups. **The rest** - System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy. +- Publish installer 1.36.1 (tag + webpage sync): a wrong passphrase then reads „passphrase REJECTED", not a doubled code (R-1284). - Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09. - Two DooPlex leftovers: `leftovers/2026-10-14/first-post/` (keep or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). Since 2026-10-14. - Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784). diff --git a/documentation/audits/release-2026-10-26/boxes-0.343.0/demo-felhom-controller.log b/documentation/audits/release-2026-10-26/boxes-0.343.0/demo-felhom-controller.log new file mode 100644 index 00000000..1f5ce723 --- /dev/null +++ b/documentation/audits/release-2026-10-26/boxes-0.343.0/demo-felhom-controller.log @@ -0,0 +1,161 @@ +2026/10/25 17:16:37 [INFO] local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible +2026/10/25 17:16:37 [INFO] local-api: mount mp0 → /var/lib/felhom (storage=local-lvm, class=, backup=true) +2026/10/25 17:16:37 [INFO] local-api: mount mp8 → /mnt/felhom-drives (storage=/mnt/felhom-drives, class=, backup=false) +2026/10/25 17:16:37 [INFO] local-api: mount mp9 → /etc/felhom-bootstrap (storage=/var/lib/felhom-agent/guests/9201/bootstrap, class=, backup=false) +2026/10/25 17:16:37 [INFO] felhom-controller 0.343.0 starting (customer: demo-felhom, domain: demo-felhom.eu) +2026/10/25 17:16:37 [INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json +2026/10/25 17:16:37 [INFO] Encryption key loaded from /opt/docker/felhom-controller/data/encryption.key +2026/10/25 17:16:37 [INFO] [stacks] Using compose command: docker compose +2026/10/25 17:16:37 [INFO] [stacks] ScanStacks complete: 68 stacks found (5 deployed, 63 available) +2026/10/25 17:16:37 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/25 17:16:37 [INFO] [stacks] InjectMissingFields: processed 5 stacks +2026/10/25 17:16:37 [INFO] [stacks] Encryption migration: no stacks needed migration +2026/10/25 17:16:37 [INFO] [quiesce] loop started (poll 5m0s, max-quiesce 30m0s) +2026/10/25 17:16:37 [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded (state ambiguous — legacy boot behaviour retained) +2026/10/25 17:16:37 [INFO] [stacks] installed-images backfill: 0 app(s) recorded, 5 already had a record, 0 left unrecorded (could not be observed completely — unknown, which renders nothing) +2026/10/25 17:16:37 [INFO] [stacks] pin adoption: 0 pinned, 5 already pinned, 0 left unpinned (0 not completely observed, 0 running something the template no longer offers) +2026/10/25 17:16:37 [INFO] [stacks] applied-meta backfill (R-646): recorded 0 []; skipped 0 [] — not current with the catalog, their pinned version's .felhom.yml is no longer on the box +2026/10/25 17:16:37 [INFO] [sync] Starting catalog sync (repo: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git, interval: 15m0s) +2026/10/25 17:16:37 [INFO] [sync] Starting catalog sync +2026/10/25 17:16:37 [INFO] Metrics store opened at /opt/docker/felhom-controller/data/metrics.db +2026/10/25 17:16:37 [INFO] Metrics collector started (60s interval) +2026/10/25 17:16:37 [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (branch: main) +2026/10/25 17:16:37 [INFO] [tier2] start-up: no leftover second-drive copy container +2026/10/25 17:16:37 [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false) +2026/10/25 17:16:37 [INFO] [backup] app-data run record wired: /opt/docker/felhom-controller/data/appdata-run.json (an earlier cut still shown: false) +2026/10/25 17:16:37 [INFO] Notifier enabled (hub: https://hub.felhom.eu) +2026/10/25 17:16:37 [INFO] [upwatch] the host's weekly line test: 92.6 Mbit/s (measured 2026-10-24T08:02:45Z) +2026/10/25 17:16:37 [INFO] [upwatch] upload and CPU watch started (line 93 Mbit/s, qBittorrent share 50 %) +2026/10/25 17:16:37 [INFO] [selfupdate] Post-update startup: update successful (0.342.0 → 0.343.0) +2026/10/25 17:16:37 [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: status-refresh (every 10s) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: docker-socket-watch (every 15s) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: docker-socket-users (every 5m0s) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: health-probes (every 10s) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: system-health (every 5m0s) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: after-unclean-boot (every 30s) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: ring-spill (every 30s) +2026/10/25 17:16:37 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply +2026/10/25 17:16:37 [INFO] [scheduler] Daily job db-dump scheduled for 2026-10-26 02:30 CET +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: conversion-copy-release (every 1h0m0s) +2026/10/25 17:16:37 [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-10-26 03:30 CET +2026/10/25 17:16:37 [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-26 04:15 CET +2026/10/25 17:16:37 [INFO] [catch-up] controller start: no backup leg missed its last scheduled time — nothing to make up +2026/10/25 17:16:37 [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-10-26 05:10 CET +2026/10/25 17:16:37 [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-10-26 06:00 CET +2026/10/25 17:16:37 [INFO] [scheduler] Daily job offsite-proof scheduled for 2026-10-26 05:30 CET +2026/10/25 17:16:37 [INFO] [scheduler] Daily job metrics-prune scheduled for 2026-10-26 04:00 CET +2026/10/25 17:16:37 [INFO] [scheduler] Daily job fill-watch scheduled for 2026-10-26 03:30 CET +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: fill-watch-interval (every 10m0s) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: hub-report (every 15m0s) +2026/10/25 17:16:37 [INFO] Hub reporting enabled (every 15m0s to https://hub.felhom.eu) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s) +2026/10/25 17:16:37 [INFO] ========== Startup Self-Test ========== +2026/10/25 17:16:37 [INFO] Event pushed: controller_updated (info) [hu-only] — Controller frissítve: 0.342.0 → 0.343.0 +2026/10/25 17:16:37 [INFO] [PASS] Docker socket: reachable (v29.9.2) +2026/10/25 17:16:37 [INFO] [PASS] Stacks directory: /opt/docker/stacks +2026/10/25 17:16:37 [INFO] [PASS] Data directory: /opt/docker/felhom-controller/data (writable) +2026/10/25 17:16:37 [INFO] [PASS] System data path: /mnt/sys_drive +2026/10/25 17:16:37 [INFO] [PASS] Storage paths: 1 connected, 0 disconnected +2026/10/25 17:16:37 [INFO] [PASS] Git catalog: 65 app definitions found +2026/10/25 17:16:37 [INFO] [PASS] Hub connectivity: https://hub.felhom.eu reachable (HTTP 200) +2026/10/25 17:16:37 [INFO] [PASS] Metrics DB: 30.3 MB +2026/10/25 17:16:37 [INFO] ======================================== +2026/10/25 17:16:37 [INFO] Self-test complete: 8 passed, 0 warnings, 0 failed +2026/10/25 17:16:37 [INFO] [scheduler] Starting scheduler with 23 jobs +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: geo-verify (every 6h0m0s) +2026/10/25 17:16:37 [INFO] Geo-restriction support enabled (CF API token configured) +2026/10/25 17:16:37 [INFO] [report] hub wait channel active (hold ≤240s) +2026/10/25 17:16:37 [INFO] Notification preferences synced to hub: source=household status=ok email=doodoo21@freemail.hu, events=[app_stopped_unhealthy app_update_held app_update_undone backup_failed db_dump_failed disk_critical disk_warning expected_backup_missed expected_dbdump_missed health_critical node_down offbox_enlarge_blocked storage_disconnected], cooldown=6h, email_cleared=false +2026/10/25 17:16:37 [INFO] [backup] Found 0 DB dump files across drives +2026/10/25 17:16:37 [INFO] [infra] connected felhom-controller to traefik-public +2026/10/25 17:16:37 [INFO] [backup] Discovered 0 databases +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/opengist/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/prowlarr/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radarr/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/sonarr/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/qbittorrent/docker-compose.yml +2026/10/25 17:16:37 [INFO] [backup] Discovered app data: 5 apps +2026/10/25 17:16:37 [INFO] [web] sessions: restored 1 dashboard session(s) from disk (dropped 0 expired, 0 malformed) +2026/10/25 17:16:37 [INFO] [web] Auth: using password from settings.json +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s) +2026/10/25 17:16:37 [INFO] [scheduler] Registered periodic job: agent-channel-health (every 1m0s) +2026/10/25 17:16:37 [INFO] Web UI listening on :8080 +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radarr/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/sonarr/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/qbittorrent/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/10/25 17:16:37 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/opengist — 2 env vars, 0 encrypted, 0 sensitive fields +2026/10/25 17:16:37 [INFO] [stacks] SaveAppConfig: saved config for opengist +2026/10/25 17:16:37 [INFO] [stacks] opengist: install id 75d74fb92a0d5fcd recorded (an app installed before D47) +2026/10/25 17:16:37 [INFO] [backup] Recovery unit captured for opengist → /mnt/sys_drive/felhom-data/backups/primary/opengist (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/25 17:16:37 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/prowlarr — 2 env vars, 0 encrypted, 0 sensitive fields +2026/10/25 17:16:37 [INFO] [stacks] SaveAppConfig: saved config for prowlarr +2026/10/25 17:16:37 [INFO] [stacks] prowlarr: install id 4a473060b914b0e2 recorded (an app installed before D47) +2026/10/25 17:16:37 [INFO] [backup] Recovery unit captured for prowlarr → /mnt/sys_drive/felhom-data/backups/primary/prowlarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/25 17:16:37 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radarr — 3 env vars, 0 encrypted, 0 sensitive fields +2026/10/25 17:16:37 [INFO] [stacks] SaveAppConfig: saved config for radarr +2026/10/25 17:16:37 [INFO] [stacks] radarr: install id 55bfad619c700564 recorded (an app installed before D47) +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/25 17:16:37 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/10/25 17:16:37 [INFO] [backup] Recovery unit captured for radarr → /mnt/felhom-drives/hdd_1/backups/primary/radarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/25 17:16:37 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/sonarr — 3 env vars, 0 encrypted, 0 sensitive fields +2026/10/25 17:16:37 [INFO] [stacks] SaveAppConfig: saved config for sonarr +2026/10/25 17:16:37 [INFO] [stacks] sonarr: install id c0df4f095aa926d7 recorded (an app installed before D47) +2026/10/25 17:16:37 [INFO] [backup] Recovery unit captured for sonarr → /mnt/felhom-drives/hdd_1/backups/primary/sonarr (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/25 17:16:37 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/qbittorrent — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/25 17:16:37 [INFO] [stacks] SaveAppConfig: saved config for qbittorrent +2026/10/25 17:16:37 [INFO] [stacks] qbittorrent: install id f82c26e291976a7f recorded (an app installed before D47) +2026/10/25 17:16:37 [INFO] [sync] Catalog sync complete +2026/10/25 17:16:37 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +2026/10/25 17:16:37 [INFO] [infra] connected felhom-controller to felhom-filebrowser (the FileBrowser default-login check, R-1126) +2026/10/25 17:16:37 [INFO] [backup] Recovery unit captured for qbittorrent → /mnt/felhom-drives/hdd_1/backups/primary/qbittorrent (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026/10/25 17:16:37 [INFO] [backup] Backup status cache refreshed +2026/10/25 17:16:37 [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (1 storage path(s)) +2026/10/25 17:16:37 [INFO] [monitor] Health check: status=ok +2026/10/25 17:16:42 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/25 17:16:42 [INFO] [report] Building system report +2026/10/25 17:16:42 [INFO] Event pushed: controller_started (info) [hu-only] — Controller elindult (0.343.0) +2026/10/25 17:16:43 [INFO] [monitor] Health check: status=ok +2026/10/25 17:16:43 [INFO] [report] Hub report pushed successfully (5496 bytes) +2026/10/25 17:16:43 [INFO] [settings] Settings saved +2026/10/25 17:16:43 [INFO] Startup hub report sent +2026/10/25 17:16:47 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/25 17:16:47 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.343.0 (we are 0.343.0), no managed update running +2026/10/25 17:16:47 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/25 17:16:47 [INFO] Health probes: 5 ok (of 5 probed) +2026/10/25 17:16:52 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/25 17:16:52 [INFO] [bootrecon] boot window: fleet settled after 10s (3 identical samples 5s apart) — sweeping +2026/10/25 17:16:52 [INFO] [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start) +2026/10/25 17:16:57 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/25 17:17:07 [INFO] [deadapp] boot grace 1m30s: the host's last boot was clean (R-856) +2026/10/25 17:17:07 [INFO] [aftercrash] a normal (or unknown) boot — no app repair after an unclean boot (R-1007) +2026/10/25 17:17:07 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/25 17:17:08 [INFO] [selfupdate] Current version 0.343.0 is up to date +2026/10/25 17:17:17 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/25 17:17:27 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/25 17:17:37 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/25 17:17:37 [INFO] [scheduler] Running job: agent-channel-health +2026/10/25 17:17:38 [INFO] [scheduler] Job agent-channel-health completed (took 859ms) +2026/10/25 17:17:47 [INFO] [stacks] Status refresh: 8 containers across 68 stacks diff --git a/documentation/audits/release-2026-10-26/boxes-0.343.0/demo-hp-controller.log b/documentation/audits/release-2026-10-26/boxes-0.343.0/demo-hp-controller.log new file mode 100644 index 00000000..55ad995e --- /dev/null +++ b/documentation/audits/release-2026-10-26/boxes-0.343.0/demo-hp-controller.log @@ -0,0 +1,731 @@ +2026/10/25 17:16:53 main.go:3990: [INFO] local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible +2026/10/25 17:16:53 main.go:3993: [INFO] local-api: mount mp8 → /mnt/felhom-drives (storage=/mnt/felhom-drives, class=, backup=false) +2026/10/25 17:16:53 main.go:3993: [INFO] local-api: mount mp0 → /var/lib/felhom (storage=local-lvm, class=, backup=true) +2026/10/25 17:16:53 main.go:3993: [INFO] local-api: mount mp9 → /etc/felhom-bootstrap (storage=/var/lib/felhom-agent/guests/9201/bootstrap, class=, backup=false) +2026/10/25 17:16:53 main.go:355: [INFO] felhom-controller 0.343.0 starting (customer: demo-hp, domain: enkisfelhom.hu) +2026/10/25 17:16:53 settings.go:813: [INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json +2026/10/25 17:16:53 settings.go:1861: [DEBUG] [settings] AutoDiscoverStoragePaths discovered=[/mnt/felhom-drives/hdd_1] existing=1 +2026/10/25 17:16:53 main.go:381: [INFO] Encryption key loaded from /opt/docker/felhom-controller/data/encryption.key +2026/10/25 17:16:53 manager.go:396: [INFO] [stacks] Using compose command: docker compose +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=true composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=true composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=true composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "cloudflared" deployed=false composePath=/opt/docker/stacks/cloudflared/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "dawarich" deployed=false composePath=/opt/docker/stacks/dawarich/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "grimmory" deployed=false composePath=/opt/docker/stacks/grimmory/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "grocy" deployed=false composePath=/opt/docker/stacks/grocy/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=true composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "karakeep" deployed=false composePath=/opt/docker/stacks/karakeep/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=true composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "lubelogger" deployed=false composePath=/opt/docker/stacks/lubelogger/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "metube" deployed=false composePath=/opt/docker/stacks/metube/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=false composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=false composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "nzbget" deployed=false composePath=/opt/docker/stacks/nzbget/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=true composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=true composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "prowlarr" deployed=false composePath=/opt/docker/stacks/prowlarr/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "qbittorrent" deployed=false composePath=/opt/docker/stacks/qbittorrent/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "radicale" deployed=false composePath=/opt/docker/stacks/radicale/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "sabnzbd" deployed=false composePath=/opt/docker/stacks/sabnzbd/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "samba" deployed=false composePath=/opt/docker/stacks/samba/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "transmission" deployed=false composePath=/opt/docker/stacks/transmission/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=false composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/10/25 17:16:53 manager.go:676: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/10/25 17:16:53 manager.go:1960: [DEBUG] [stacks] getCatalogTemplateSlugs: found 65 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/10/25 17:16:53 manager.go:757: [DEBUG] [stacks] ScanStacks: catalog has 65 template slugs for orphan detection +2026/10/25 17:16:53 manager.go:788: [INFO] [stacks] ScanStacks complete: 69 stacks found (12 deployed, 57 available) +2026/10/25 17:16:54 manager.go:839: [INFO] [stacks] Status refresh: 23 containers across 69 stacks +2026/10/25 17:16:54 deploy.go:1389: [DEBUG] [stacks] InjectMissingFields: checking 12 stacks +2026/10/25 17:16:54 deploy.go:1411: [DEBUG] [stacks] InjectMissingFields: checking stack privatebin — 2 deploy fields, 2 existing env vars +2026/10/25 17:16:54 deploy.go:1411: [DEBUG] [stacks] InjectMissingFields: checking stack calibre-web — 5 deploy fields, 4 existing env vars +2026/10/25 17:16:54 deploy.go:1470: [WARN] [stacks] Stack calibre-web: new field ADMIN_PASSWORD (type=password) requires manual configuration +2026/10/25 17:16:54 deploy.go:1411: [DEBUG] [stacks] InjectMissingFields: checking stack home-assistant — 2 deploy fields, 2 existing env vars +2026/10/25 17:16:54 deploy.go:1411: [DEBUG] [stacks] InjectMissingFields: checking stack bentopdf — 2 deploy fields, 2 existing env vars +2026/10/25 17:16:54 deploy.go:1411: [DEBUG] [stacks] InjectMissingFields: checking stack plex — 4 deploy fields, 3 existing env vars +2026/10/25 17:16:54 deploy.go:1470: [WARN] [stacks] Stack plex: new field PLEX_CLAIM (type=text) requires manual configuration +2026/10/25 17:16:54 deploy.go:1411: [DEBUG] [stacks] InjectMissingFields: checking stack romm — 6 deploy fields, 6 existing env vars +2026/10/25 17:16:54 deploy.go:1411: [DEBUG] [stacks] InjectMissingFields: checking stack bookstack — 5 deploy fields, 4 existing env vars +2026/10/25 17:16:54 deploy.go:1470: [WARN] [stacks] Stack bookstack: new field ADMIN_PASSWORD (type=password) requires manual configuration +2026/10/25 17:16:54 deploy.go:1411: [DEBUG] [stacks] InjectMissingFields: checking stack paperless-ngx — 8 deploy fields, 8 existing env vars +2026/10/25 17:16:54 deploy.go:1411: [DEBUG] [stacks] InjectMissingFields: checking stack opengist — 2 deploy fields, 2 existing env vars +2026/10/25 17:16:54 deploy.go:1411: [DEBUG] [stacks] InjectMissingFields: checking stack docmost — 4 deploy fields, 4 existing env vars +2026/10/25 17:16:54 deploy.go:1411: [DEBUG] [stacks] InjectMissingFields: checking stack adventurelog — 4 deploy fields, 5 existing env vars +2026/10/25 17:16:54 deploy.go:1411: [DEBUG] [stacks] InjectMissingFields: checking stack kimai — 5 deploy fields, 5 existing env vars +2026/10/25 17:16:54 deploy.go:1482: [INFO] [stacks] InjectMissingFields: processed 12 stacks +2026/10/25 17:16:54 manager.go:535: [DEBUG] [stacks] MigrateEncryption: checking 12 deployed stacks for plaintext sensitive values +2026/10/25 17:16:54 manager.go:555: [DEBUG] [stacks] MigrateEncryption: checking stack "adventurelog" (2 sensitive fields) +2026/10/25 17:16:54 manager.go:555: [DEBUG] [stacks] MigrateEncryption: checking stack "kimai" (2 sensitive fields) +2026/10/25 17:16:54 manager.go:555: [DEBUG] [stacks] MigrateEncryption: checking stack "calibre-web" (2 sensitive fields) +2026/10/25 17:16:54 manager.go:555: [DEBUG] [stacks] MigrateEncryption: checking stack "romm" (3 sensitive fields) +2026/10/25 17:16:54 manager.go:555: [DEBUG] [stacks] MigrateEncryption: checking stack "bookstack" (3 sensitive fields) +2026/10/25 17:16:54 manager.go:555: [DEBUG] [stacks] MigrateEncryption: checking stack "paperless-ngx" (3 sensitive fields) +2026/10/25 17:16:54 manager.go:555: [DEBUG] [stacks] MigrateEncryption: checking stack "docmost" (2 sensitive fields) +2026/10/25 17:16:54 manager.go:579: [INFO] [stacks] Encryption migration: no stacks needed migration +2026/10/25 17:16:54 quiesce.go:192: [INFO] [quiesce] loop started (poll 5m0s, max-quiesce 30m0s) +2026/10/25 17:16:54 desiredstate.go:137: [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded (state ambiguous — legacy boot behaviour retained) +2026/10/25 17:16:54 installed.go:537: [INFO] [stacks] installed-images backfill: 0 app(s) recorded, 12 already had a record, 0 left unrecorded (could not be observed completely — unknown, which renders nothing) +2026/10/25 17:16:54 pin.go:290: [INFO] [stacks] pin adoption: 0 pinned, 12 already pinned, 0 left unpinned (0 not completely observed, 0 running something the template no longer offers) +2026/10/25 17:16:54 undo.go:676: [INFO] [stacks] applied-meta backfill (R-646): recorded 0 []; skipped 0 [] — not current with the catalog, their pinned version's .felhom.yml is no longer on the box +2026/10/25 17:16:54 sync.go:137: [INFO] [sync] Starting catalog sync (repo: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git, interval: 15m0s) +2026/10/25 17:16:54 sync.go:228: [INFO] [sync] Starting catalog sync +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] CPUCollector.Start: sampleRate=5s +2026/10/25 17:16:54 main.go:570: [INFO] Metrics store opened at /opt/docker/felhom-controller/data/metrics.db +2026/10/25 17:16:54 main.go:579: [INFO] Metrics collector started (60s interval) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/hdd_1", hasCPUCollector=true) +2026/10/25 17:16:54 sync.go:358: [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (branch: main) +2026/10/25 17:16:54 sync.go:360: [DEBUG] [sync] git fetch --depth 1 origin main in /opt/docker/felhom-controller/data/catalog-cache +2026/10/25 17:16:54 sync.go:787: [DEBUG] [sync] Running: git fetch --depth 1 origin main +2026/10/25 17:16:54 [INFO] [tier2] start-up: no leftover second-drive copy container +2026/10/25 17:16:54 restore_record_wiring.go:37: [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false) +2026/10/25 17:16:54 run_record_wiring.go:29: [INFO] [backup] app-data run record wired: /opt/docker/felhom-controller/data/appdata-run.json (an earlier cut still shown: false) +2026/10/25 17:16:54 notifier.go:81: [INFO] Notifier enabled (hub: https://hub.felhom.eu) +2026/10/25 17:16:54 service.go:79: [INFO] [upwatch] the host's weekly line test: 94.4 Mbit/s (measured 2026-10-24T08:00:33Z) +2026/10/25 17:16:54 upwatch_wiring.go:149: [INFO] [upwatch] upload and CPU watch started (line 94 Mbit/s, qBittorrent share 50 %) +2026/10/25 17:16:54 updater.go:104: [DEBUG] [selfupdate] VerifyStartup: checking update state in /opt/docker/felhom-controller/data +2026/10/25 17:16:54 updater.go:104: [DEBUG] [selfupdate] VerifyStartup: pending update found — target=0.343.0 previous=0.342.0 +2026/10/25 17:16:54 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.342.0 → 0.343.0) +2026/10/25 17:16:54 main.go:864: [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30) +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: status-refresh (every 10s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="status-refresh" interval=10s totalJobs=1 +2026/10/25 17:16:54 notifier.go:317: [DEBUG] PushEvent: type=controller_updated severity=info url=https://hub.felhom.eu/api/v1/event +2026/10/25 17:16:54 offsiteapply.go:148: [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="stack-scan" interval=2m0s totalJobs=2 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: docker-socket-watch (every 15s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="docker-socket-watch" interval=15s totalJobs=3 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: docker-socket-users (every 5m0s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="docker-socket-users" interval=5m0s totalJobs=4 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: health-probes (every 10s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="health-probes" interval=10s totalJobs=5 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: system-health (every 5m0s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="system-health" interval=5m0s totalJobs=6 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: after-unclean-boot (every 30s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="after-unclean-boot" interval=30s totalJobs=7 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="deadapp-check" interval=30s totalJobs=8 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: ring-spill (every 30s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="ring-spill" interval=30s totalJobs=9 +2026/10/25 17:16:54 scheduler.go:150: [INFO] [scheduler] Daily job db-dump scheduled for 2026-10-26 02:30 CET +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="db-dump" schedule="02:30" nextRun=2026-10-26T02:30:00+01:00 totalJobs=10 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="offsite-credential-retry" interval=5m0s totalJobs=11 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="backup-cache" interval=5m0s totalJobs=12 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: conversion-copy-release (every 1h0m0s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="conversion-copy-release" interval=1h0m0s totalJobs=13 +2026/10/25 17:16:54 scheduler.go:150: [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-10-26 03:30 CET +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="tier2-backup" schedule="03:30" nextRun=2026-10-26T03:30:00+01:00 totalJobs=14 +2026/10/25 17:16:54 scheduler.go:150: [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-26 04:15 CET +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="offbox-backup" schedule="04:15" nextRun=2026-10-26T04:15:00+01:00 totalJobs=15 +2026/10/25 17:16:54 nightchain.go:284: [INFO] [catch-up] controller start: no backup leg missed its last scheduled time — nothing to make up +2026/10/25 17:16:54 scheduler.go:150: [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-10-26 05:10 CET +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="offsite-abandon-sweep" schedule="05:10" nextRun=2026-10-26T05:10:00+01:00 totalJobs=16 +2026/10/25 17:16:54 scheduler.go:150: [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-10-26 06:00 CET +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="offsite-integrity" schedule="06:00" nextRun=2026-10-26T06:00:00+01:00 totalJobs=17 +2026/10/25 17:16:54 scheduler.go:150: [INFO] [scheduler] Daily job offsite-proof scheduled for 2026-10-26 05:30 CET +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="offsite-proof" schedule="05:30" nextRun=2026-10-26T05:30:00+01:00 totalJobs=18 +2026/10/25 17:16:54 scheduler.go:150: [INFO] [scheduler] Daily job metrics-prune scheduled for 2026-10-26 04:00 CET +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="metrics-prune" schedule="04:00" nextRun=2026-10-26T04:00:00+01:00 totalJobs=19 +2026/10/25 17:16:54 scheduler.go:150: [INFO] [scheduler] Daily job fill-watch scheduled for 2026-10-26 03:30 CET +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="fill-watch" schedule="03:30" nextRun=2026-10-26T03:30:00+01:00 totalJobs=20 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: fill-watch-interval (every 10m0s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="fill-watch-interval" interval=10m0s totalJobs=21 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: hub-report (every 15m0s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="hub-report" interval=15m0s totalJobs=22 +2026/10/25 17:16:54 main.go:1823: [INFO] Hub reporting enabled (every 15m0s to https://hub.felhom.eu) +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="selfupdate-check" interval=6h0m0s totalJobs=23 +2026/10/25 17:16:54 selftest.go:50: [INFO] ========== Startup Self-Test ========== +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈16630MB avail≈9268MB +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=10990616 → total=25898MB avail=9268MB used=16630MB (64.2%) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.7GB used=20.8GB avail=44.5GB (31.8%) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/hdd_1" bsize=4096 total=937.8GB used=225.2GB avail=665.0GB (25.3%) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.71 1.15 1.44 14/1581 105" → 1m=1.71 5m=1.15 15m=1.44 +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 58.6°C (hwmon2) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetInfo done in 116ms — mem=16630MB/25898MB (64.2%), rootDisk=20.8GB/68.7GB (31.8%), load=1.71/1.15/1.44, temp=58.6°C (hwmon2), cpu=0.0% +2026/10/25 17:16:54 notifier.go:343: [DEBUG] PushEvent: controller_updated pushed OK (HTTP 200) +2026/10/25 17:16:54 selftest.go:68: [INFO] [PASS] Docker socket: reachable (v29.9.2) +2026/10/25 17:16:54 notifier.go:355: [INFO] Event pushed: controller_updated (info) [hu-only] — Controller frissítve: 0.342.0 → 0.343.0 +2026/10/25 17:16:54 selftest.go:68: [INFO] [PASS] Stacks directory: /opt/docker/stacks +2026/10/25 17:16:54 selftest.go:68: [INFO] [PASS] Data directory: /opt/docker/felhom-controller/data (writable) +2026/10/25 17:16:54 selftest.go:68: [INFO] [PASS] System data path: /mnt/sys_drive +2026/10/25 17:16:54 selftest.go:68: [INFO] [PASS] Storage paths: 1 connected, 0 disconnected +2026/10/25 17:16:54 selftest.go:68: [INFO] [PASS] Git catalog: 65 app definitions found +2026/10/25 17:16:54 selftest.go:68: [INFO] [PASS] Hub connectivity: https://hub.felhom.eu reachable (HTTP 200) +2026/10/25 17:16:54 selftest.go:68: [INFO] [PASS] Metrics DB: 107.4 MB +2026/10/25 17:16:54 selftest.go:70: [INFO] ======================================== +2026/10/25 17:16:54 selftest.go:71: [INFO] Self-test complete: 8 passed, 0 warnings, 0 failed +2026/10/25 17:16:54 scheduler.go:224: [INFO] [scheduler] Starting scheduler with 23 jobs +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] scheduler started: periodic=15 daily=8 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: geo-verify (every 6h0m0s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="geo-verify" interval=6h0m0s totalJobs=24 +2026/10/25 17:16:54 main.go:2088: [INFO] Geo-restriction support enabled (CF API token configured) +2026/10/25 17:16:54 server.go:318: [DEBUG] [web] NewServer: initializing web server v0.343.0 +2026/10/25 17:16:54 server.go:319: [DEBUG] [web] NewServer: backup=true scheduler=true alertMgr=true notifier=true updater=true +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job offsite-abandon-sweep: next run at 2026-10-26 05:10:00 CET (waiting 10h53m6s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job db-dump: next run at 2026-10-26 02:30:00 CET (waiting 8h13m6s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job tier2-backup: next run at 2026-10-26 03:30:00 CET (waiting 9h13m6s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job offbox-backup: next run at 2026-10-26 04:15:00 CET (waiting 9h58m6s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job offsite-integrity: next run at 2026-10-26 06:00:00 CET (waiting 11h43m6s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job offsite-proof: next run at 2026-10-26 05:30:00 CET (waiting 11h13m6s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job metrics-prune: next run at 2026-10-26 04:00:00 CET (waiting 9h43m6s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] daily job fill-watch: next run at 2026-10-26 03:30:00 CET (waiting 9h13m6s) +2026/10/25 17:16:54 waiter.go:95: [INFO] [report] hub wait channel active (hold ≤240s) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/hdd_1", hasCPUCollector=true) +2026/10/25 17:16:54 backup.go:540: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [adventurelog, bentopdf, bookstack, docmost, home-assistant, kimai, opengist, privatebin] +2026/10/25 17:16:54 backup.go:540: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/hdd_1 → [calibre-web, paperless-ngx, plex, romm] +2026/10/25 17:16:54 backup.go:1246: [INFO] [backup] Found 19 DB dump files across drives +2026/10/25 17:16:54 waiter.go:132: [DEBUG] [report] wait baseline generation=1 +2026/10/25 17:16:54 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/10/25 17:16:54 server.go:399: [DEBUG] [web] loadTemplates: lang=hu loaded 99 templates, 2085 markers expanded in 63.1108ms +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈16652MB avail≈9246MB +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=10965152 → total=25898MB avail=9246MB used=16652MB (64.3%) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.7GB used=20.8GB avail=44.5GB (31.8%) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/hdd_1" bsize=4096 total=937.8GB used=225.2GB avail=665.0GB (25.3%) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.71 1.15 1.44 15/1599 194" → 1m=1.71 5m=1.15 15m=1.44 +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 59.0°C (hwmon2) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetInfo done in 90ms — mem=16652MB/25898MB (64.3%), rootDisk=20.8GB/68.7GB (31.8%), load=1.71/1.15/1.44, temp=59.0°C (hwmon2), cpu=0.0% +2026/10/25 17:16:54 healthcheck.go:128: [DEBUG] [monitor] Raw values: disk=31.8%, hdd=25.3% (configured=true), mem=64.3% (16652MB/25898MB), cpu=0.0%, temp=59.0°C (hwmon2) +2026/10/25 17:16:54 healthcheck.go:409: [DEBUG] [monitor] SSD disk: OK (32%) +2026/10/25 17:16:54 healthcheck.go:445: [DEBUG] [monitor] Memory: OK (64%) +2026/10/25 17:16:54 healthcheck.go:483: [DEBUG] [monitor] Temperature: OK (59°C) +2026/10/25 17:16:54 infra.go:378: [INFO] [infra] connected felhom-controller to traefik-public +2026/10/25 17:16:54 server.go:399: [DEBUG] [web] loadTemplates: lang=en loaded 99 templates, 2085 markers expanded in 50.250005ms +2026/10/25 17:16:54 session_store.go:113: [INFO] [web] sessions: restored 38 dashboard session(s) from disk (dropped 0 expired, 0 malformed) +2026/10/25 17:16:54 server.go:339: [INFO] [web] Auth: using password from settings.json +2026/10/25 17:16:54 sync.go:787: [DEBUG] [sync] Running: git reset --hard origin/main +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="disk-health-check" interval=1h0m0s totalJobs=25 +2026/10/25 17:16:54 scheduler.go:120: [INFO] [scheduler] Registered periodic job: agent-channel-health (every 1m0s) +2026/10/25 17:16:54 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="agent-channel-health" interval=1m0s totalJobs=26 +2026/10/25 17:16:54 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 496564edc900 felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.343.0 +ef0e57fa0965 romm romm rommapp/romm:5.3.1 +01aca2e96906 romm-redis romm redis:7-alpine +d3c8ecc410f7 romm-db romm mariadb:11.8 +0a833b20453d privatebin privatebin privatebin/pdo:2.0.6 +ffbe0c7120c6 plex plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 +a3375a5404e3 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +9619a5283d77 paperless-postgres paperless-ngx postgres:18-alpine +b4082466c0fe pape... +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.343.0, not a database) +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/10/25 17:16:54 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=d3c8ecc410f7) +2026/10/25 17:16:54 server.go:983: [DEBUG] [web] CatchAllMiddleware: controller host=felhom.enkisfelhom.hu +2026/10/25 17:16:54 main.go:2364: [INFO] Web UI listening on :8080 +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] actualbudget/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] actualbudget/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/calibre-web/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/plex/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] adventurelog/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] adventurelog/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] audiobookshelf/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] audiobookshelf/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] bentopdf/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] bentopdf/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] bookstack/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] bookstack/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] calcom/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] calcom/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] calibre-web/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] calibre-web/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 manager.go:258: [DEBUG] [integrations] ReapplyConfigForTarget: target=filebrowser integrations=0 +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] claper/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] claper/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] code-server/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] code-server/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] crafty-controller/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] crafty-controller/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] dawarich/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] dawarich/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container plex (image=plexinc/pms-docker:1.43.5.11029-6bf8af3f0, not a database) +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/10/25 17:16:54 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=9619a5283d77) +2026/10/25 17:16:54 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] docmost/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] docmost/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] emby/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] emby/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] ghost/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] ghost/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] gitea/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] gitea/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] glance/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] glance/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] gokapi/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] gokapi/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] grafana/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] grafana/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] gramps-web/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] gramps-web/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] grimmory/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] grimmory/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] grocy/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] grocy/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] home-assistant/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] home-assistant/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] homebox/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] homebox/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] homepage/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container opengist (image=ghcr.io/thomiceli/opengist:1.15, not a database) +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container home-assistant (image=ghcr.io/home-assistant/home-assistant:2026.9.4, not a database) +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost (image=docmost/docmost:0.96.0, not a database) +2026/10/25 17:16:54 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: docmost-postgres (id=dd941d6240e4) +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] homepage/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] immich/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] immich/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] jellyfin/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] jellyfin/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] karakeep/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] karakeep/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 healthcheck.go:145: [DEBUG] [monitor] Docker daemon: OK +2026/10/25 17:16:54 healthcheck.go:153: [DEBUG] [monitor] Checking 4 protected containers: [traefik cloudflared felhom-controller filebrowser] +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] kimai/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] kimai/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] komga/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] komga/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] lubelogger/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] lubelogger/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] mealie/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] mealie/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] metube/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] metube/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] n8n/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] n8n/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] navidrome/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] navidrome/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] nextcloud/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] nextcloud/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] nzbget/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] nzbget/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 dbdump.go:198: [DEBUG] DiscoverDatabases: docmost-postgres → stack=docmost, dbUser=docmost, dbName=docmost +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost-redis (image=redis:7-alpine, not a database) +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container calibre-web (image=crocodilestick/calibre-web-automated:v4.0.8, not a database) +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container bookstack (image=lscr.io/linuxserver/bookstack:26.09.1, not a database) +2026/10/25 17:16:54 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: bookstack-db (id=fbe94713ce78) +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] onlyoffice/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] onlyoffice/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] opengist/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] opengist/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] outline/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] outline/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] paperless-ngx/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] paperless-ngx/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] papra/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] papra/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] plant-it/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] plant-it/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] privatebin/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] privatebin/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] prowlarr/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] prowlarr/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] qbittorrent/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] qbittorrent/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] radarr/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] radarr/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] radicale/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] radicale/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] rallly/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] rallly/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] recipe-importer/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] recipe-importer/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] romm/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] romm/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] sabnzbd/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] sabnzbd/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] seerr/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] seerr/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] sonarr/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] sonarr/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] sparkyfitness/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] sparkyfitness/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] tandoor/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] tandoor/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] termix/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] termix/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 dbdump.go:198: [DEBUG] DiscoverDatabases: bookstack-db → stack=bookstack, dbUser=root, dbName=bookstack +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container bentopdf (image=ghcr.io/alam00000/bentopdf:v2.8.6, not a database) +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog (image=ghcr.io/seanmorley15/adventurelog-backend:v0.13.0, not a database) +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog-frontend (image=ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0, not a database) +2026/10/25 17:16:54 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: adventurelog-postgres (id=c84ab95505dc) +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] transmission/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] transmission/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] uptime-kuma/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] uptime-kuma/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] vaultwarden/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] vaultwarden/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] vikunja/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] vikunja/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] wanderer/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] wanderer/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] wger/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] wger/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] wishlist/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] wishlist/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] zipline/docker-compose.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:559: [DEBUG] [sync] zipline/.felhom.yml: hash match, skipped +2026/10/25 17:16:54 sync.go:299: [INFO] [sync] Catalog sync complete +2026/10/25 17:16:54 sync.go:142: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +2026/10/25 17:16:54 dbdump.go:198: [DEBUG] DiscoverDatabases: adventurelog-postgres → stack=adventurelog, dbUser=adventurelog, dbName=adventurelog +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database) +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container cloudflared (image=cloudflare/cloudflared:2026.9.3, not a database) +2026/10/25 17:16:54 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database) +2026/10/25 17:16:54 dbdump.go:205: [DEBUG] DiscoverDatabases: found 5 database(s), skipped 19 non-DB container(s) +2026/10/25 17:16:54 dbdump.go:208: [INFO] [backup] Discovered 5 databases +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bentopdf/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/calibre-web/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/10/25 17:16:54 handlers.go:3942: [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (1 storage path(s)) +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/home-assistant/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/kimai/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/opengist/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/plex/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/10/25 17:16:54 [INFO] [backup] Discovered app data: 12 apps +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.7 GB used=20.8 GB avail=44.5GB (31.8%) +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/adventurelog — 5 env vars, 0 encrypted, 2 sensitive fields +2026/10/25 17:16:54 setup_gate.go:511: [DEBUG] [stacks] romm: setup probe unreadable (Get "http://romm:8080/api/heartbeat": dial tcp: lookup romm on 192.168.0.250:53: no such host) — gate stays closed +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for adventurelog +2026/10/25 17:16:54 deploy.go:1682: [INFO] [stacks] adventurelog: install id 2c99f493b77aed71 recorded (an app installed before D47) +2026/10/25 17:16:54 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute +2026/10/25 17:16:54 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for adventurelog → /mnt/sys_drive/felhom-data/backups/primary/adventurelog (images=3, secrets-referenced=2, data_keys=1, portable-carried=2/2, withheld=0) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.7 GB used=20.8 GB avail=44.5GB (31.8%) +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/bentopdf — 2 env vars, 0 encrypted, 0 sensitive fields +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for bentopdf +2026/10/25 17:16:54 deploy.go:1682: [INFO] [stacks] bentopdf: install id 17f35f4c0d87a32f recorded (an app installed before D47) +2026/10/25 17:16:54 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for bentopdf → /mnt/sys_drive/felhom-data/backups/primary/bentopdf (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.7 GB used=20.8 GB avail=44.5GB (31.8%) +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/bookstack — 4 env vars, 0 encrypted, 3 sensitive fields +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for bookstack +2026/10/25 17:16:54 deploy.go:1682: [INFO] [stacks] bookstack: install id e172a289fd47c51d recorded (an app installed before D47) +2026/10/25 17:16:54 healthcheck.go:161: [DEBUG] [monitor] All protected containers running +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/hdd_1" total=937.8 GB used=225.2 GB avail=665.0GB (25.3%) +2026/10/25 17:16:54 healthcheck.go:176: [INFO] [monitor] Health check: status=ok +2026/10/25 17:16:54 healthcheck.go:180: [DEBUG] [monitor] Final status: ok (issues=0, warnings=0, info=4) +2026/10/25 17:16:54 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/hdd_1" total=937.8 GB used=225.2 GB avail=665.0GB (25.3%) +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/calibre-web — 4 env vars, 0 encrypted, 2 sensitive fields +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for calibre-web +2026/10/25 17:16:54 deploy.go:1682: [INFO] [stacks] calibre-web: install id a3ee4c39e2439d7b recorded (an app installed before D47) +2026/10/25 17:16:54 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for calibre-web → /mnt/felhom-drives/hdd_1/backups/primary/calibre-web (images=1, secrets-referenced=2, data_keys=0, portable-carried=1/1, withheld=1) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.7 GB used=20.8 GB avail=44.5GB (31.8%) +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/docmost — 4 env vars, 0 encrypted, 2 sensitive fields +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for docmost +2026/10/25 17:16:54 deploy.go:1682: [INFO] [stacks] docmost: install id 7f50f75f42a6eab1 recorded (an app installed before D47) +2026/10/25 17:16:54 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for docmost → /mnt/sys_drive/felhom-data/backups/primary/docmost (images=3, secrets-referenced=2, data_keys=0, portable-carried=2/2, withheld=0) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.7 GB used=20.8 GB avail=44.5GB (31.8%) +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/home-assistant — 2 env vars, 0 encrypted, 0 sensitive fields +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for home-assistant +2026/10/25 17:16:54 deploy.go:1682: [INFO] [stacks] home-assistant: install id 919730e70cb01fe9 recorded (an app installed before D47) +2026/10/25 17:16:54 filebrowser_gate.go:68: [INFO] [infra] connected felhom-controller to felhom-filebrowser (the FileBrowser default-login check, R-1126) +2026/10/25 17:16:54 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for home-assistant → /mnt/sys_drive/felhom-data/backups/primary/home-assistant (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.7 GB used=20.8 GB avail=44.5GB (31.8%) +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/opengist — 2 env vars, 0 encrypted, 0 sensitive fields +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for opengist +2026/10/25 17:16:54 deploy.go:1682: [INFO] [stacks] opengist: install id d956c5ae9ac7b528 recorded (an app installed before D47) +2026/10/25 17:16:54 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for opengist → /mnt/sys_drive/felhom-data/backups/primary/opengist (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/hdd_1" total=937.8 GB used=225.2 GB avail=665.0GB (25.3%) +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/paperless-ngx — 8 env vars, 0 encrypted, 3 sensitive fields +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for paperless-ngx +2026/10/25 17:16:54 deploy.go:1682: [INFO] [stacks] paperless-ngx: install id a1c1bb5a5103b37d recorded (an app installed before D47) +2026/10/25 17:16:54 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/hdd_1" total=937.8 GB used=225.2 GB avail=665.0GB (25.3%) +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/plex — 3 env vars, 0 encrypted, 0 sensitive fields +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for plex +2026/10/25 17:16:54 deploy.go:1682: [INFO] [stacks] plex: install id 8a3ddf75968d32ab recorded (an app installed before D47) +2026/10/25 17:16:54 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for plex → /mnt/felhom-drives/hdd_1/backups/primary/plex (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.7 GB used=20.8 GB avail=44.5GB (31.8%) +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/privatebin — 2 env vars, 0 encrypted, 0 sensitive fields +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for privatebin +2026/10/25 17:16:54 deploy.go:1682: [INFO] [stacks] privatebin: install id 19340734630f84cd recorded (an app installed before D47) +2026/10/25 17:16:54 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/25 17:16:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/hdd_1" total=937.8 GB used=225.2 GB avail=665.0GB (25.3%) +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/romm — 6 env vars, 0 encrypted, 3 sensitive fields +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for romm +2026/10/25 17:16:54 deploy.go:1682: [INFO] [stacks] romm: install id e4637a49405a2049 recorded (an app installed before D47) +2026/10/25 17:16:54 recovery_unit.go:297: [INFO] [backup] Recovery unit captured for romm → /mnt/felhom-drives/hdd_1/backups/primary/romm (images=3, secrets-referenced=3, data_keys=0, portable-carried=3/3, withheld=0) +2026/10/25 17:16:54 backup.go:1317: [INFO] [backup] Backup status cache refreshed +2026/10/25 17:16:55 client.go:67: [DEBUG] [agentapi] agent version seen: 0.165.0 (was "") +2026/10/25 17:16:55 client.go:1276: [DEBUG] [agentapi] GET /disks -> 200 (862ms) +2026/10/25 17:16:56 client.go:1276: [DEBUG] [agentapi] GET /disks -> 200 (840ms) +2026/10/25 17:16:59 info.go:13: [DEBUG] [system] CPUCollector: first sample — cpu=61.9% (idle=1549 total=4063) +2026/10/25 17:16:59 manager.go:839: [INFO] [stacks] Status refresh: 23 containers across 69 stacks +2026/10/25 17:16:59 builder.go:37: [INFO] [report] Building system report +2026/10/25 17:16:59 builder.go:40: [DEBUG] [report] BuildReport: starting — version=0.343.0, storagePaths=1 +2026/10/25 17:16:59 notifier.go:317: [DEBUG] PushEvent: type=controller_started severity=info url=https://hub.felhom.eu/api/v1/event +2026/10/25 17:16:59 builder.go:62: [DEBUG] [report] BuildReport: configHash=042b71a3123d... (2989 bytes) +2026/10/25 17:16:59 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/hdd_1", hasCPUCollector=true) +2026/10/25 17:16:59 notifier.go:343: [DEBUG] PushEvent: controller_started pushed OK (HTTP 200) +2026/10/25 17:16:59 notifier.go:355: [INFO] Event pushed: controller_started (info) [hu-only] — Controller elindult (0.343.0) +2026/10/25 17:16:59 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈16857MB avail≈9041MB +2026/10/25 17:16:59 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=10722164 → total=25898MB avail=9041MB used=16857MB (65.1%) +2026/10/25 17:16:59 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.7GB used=20.8GB avail=44.5GB (31.8%) +2026/10/25 17:16:59 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/hdd_1" bsize=4096 total=937.8GB used=225.2GB avail=665.0GB (25.3%) +2026/10/25 17:16:59 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.89 1.20 1.45 12/1545 459" → 1m=1.89 5m=1.20 15m=1.45 +2026/10/25 17:16:59 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/25 17:16:59 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 61.6°C (hwmon2) +2026/10/25 17:16:59 info.go:13: [DEBUG] [system] GetInfo done in 112ms — mem=16857MB/25898MB (65.1%), rootDisk=20.8GB/68.7GB (31.8%), load=1.89/1.20/1.45, temp=61.6°C (hwmon2), cpu=61.9% +2026/10/25 17:16:59 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/hdd_1" total=937.8 GB used=225.2 GB avail=665.0GB (25.3%) +2026/10/25 17:16:59 builder.go:128: [DEBUG] [report] BuildReport: system info collected — cpu=61.9%, mem=16857/25898MB, temp=61.6C +2026/10/25 17:16:59 builder.go:130: [DEBUG] [report] BuildReport: storage entries=2 +2026/10/25 17:17:00 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/hdd_1", hasCPUCollector=true) +2026/10/25 17:17:00 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈16793MB avail≈9105MB +2026/10/25 17:17:00 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=10797116 → total=25898MB avail=9105MB used=16793MB (64.8%) +2026/10/25 17:17:00 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.7GB used=20.8GB avail=44.5GB (31.8%) +2026/10/25 17:17:00 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/hdd_1" bsize=4096 total=937.8GB used=225.2GB avail=665.0GB (25.3%) +2026/10/25 17:17:00 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.89 1.20 1.45 11/1541 475" → 1m=1.89 5m=1.20 15m=1.45 +2026/10/25 17:17:00 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/25 17:17:00 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 62.5°C (hwmon2) +2026/10/25 17:17:00 info.go:13: [DEBUG] [system] GetInfo done in 61ms — mem=16793MB/25898MB (64.8%), rootDisk=20.8GB/68.7GB (31.8%), load=1.89/1.20/1.45, temp=62.5°C (hwmon2), cpu=61.9% +2026/10/25 17:17:00 healthcheck.go:128: [DEBUG] [monitor] Raw values: disk=31.8%, hdd=25.3% (configured=true), mem=64.8% (16793MB/25898MB), cpu=61.9%, temp=62.5°C (hwmon2) +2026/10/25 17:17:00 healthcheck.go:409: [DEBUG] [monitor] SSD disk: OK (32%) +2026/10/25 17:17:00 healthcheck.go:445: [DEBUG] [monitor] Memory: OK (65%) +2026/10/25 17:17:00 healthcheck.go:464: [DEBUG] [monitor] CPU: OK (62%) +2026/10/25 17:17:00 healthcheck.go:483: [DEBUG] [monitor] Temperature: OK (62°C) +2026/10/25 17:17:00 healthcheck.go:145: [DEBUG] [monitor] Docker daemon: OK +2026/10/25 17:17:00 healthcheck.go:153: [DEBUG] [monitor] Checking 4 protected containers: [traefik cloudflared felhom-controller filebrowser] +2026/10/25 17:17:00 healthcheck.go:161: [DEBUG] [monitor] All protected containers running +2026/10/25 17:17:00 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/hdd_1" total=937.8 GB used=225.2 GB avail=665.0GB (25.3%) +2026/10/25 17:17:00 healthcheck.go:176: [INFO] [monitor] Health check: status=ok +2026/10/25 17:17:00 healthcheck.go:180: [DEBUG] [monitor] Final status: ok (issues=0, warnings=0, info=5) +2026/10/25 17:17:00 logscanner.go:69: [DEBUG] [metrics] logscanner: starting log scan for 21 containers (since 15m0s) +2026/10/25 17:17:00 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned adventurelog: errors=0 warnings=0 issues=0 (took 35ms) +2026/10/25 17:17:00 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned adventurelog-frontend: errors=0 warnings=0 issues=0 (took 33ms) +2026/10/25 17:17:00 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned adventurelog-postgres: errors=0 warnings=0 issues=0 (took 31ms) +2026/10/25 17:17:00 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned bentopdf: errors=0 warnings=0 issues=0 (took 34ms) +2026/10/25 17:17:00 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned bookstack: errors=0 warnings=0 issues=0 (took 29ms) +2026/10/25 17:17:00 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned bookstack-db: errors=0 warnings=0 issues=0 (took 28ms) +2026/10/25 17:17:00 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned calibre-web: errors=0 warnings=0 issues=0 (took 31ms) +2026/10/25 17:17:00 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned docmost: errors=0 warnings=0 issues=0 (took 35ms) +2026/10/25 17:17:00 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned docmost-postgres: errors=0 warnings=0 issues=0 (took 32ms) +2026/10/25 17:17:00 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned docmost-redis: errors=0 warnings=0 issues=0 (took 41ms) +2026/10/25 17:17:01 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned home-assistant: errors=0 warnings=3 issues=1 (took 37ms) +2026/10/25 17:17:01 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned opengist: errors=0 warnings=0 issues=0 (took 31ms) +2026/10/25 17:17:01 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned paperless-webserver: errors=0 warnings=0 issues=0 (took 33ms) +2026/10/25 17:17:01 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned paperless-postgres: errors=0 warnings=0 issues=0 (took 28ms) +2026/10/25 17:17:01 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned paperless-redis: errors=0 warnings=0 issues=0 (took 37ms) +2026/10/25 17:17:01 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned plex: errors=0 warnings=0 issues=0 (took 26ms) +2026/10/25 17:17:01 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned privatebin: errors=0 warnings=0 issues=0 (took 34ms) +2026/10/25 17:17:01 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned romm: errors=0 warnings=0 issues=0 (took 35ms) +2026/10/25 17:17:01 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned romm-redis: errors=0 warnings=0 issues=0 (took 34ms) +2026/10/25 17:17:01 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned romm-db: errors=0 warnings=0 issues=0 (took 30ms) +2026/10/25 17:17:01 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned felhom-controller: errors=0 warnings=3 issues=3 (took 42ms) +2026/10/25 17:17:01 logscanner.go:69: [DEBUG] [metrics] logscanner: log scan completed: 21 containers in 694ms +2026/10/25 17:17:01 builder.go:204: [DEBUG] [report] BuildReport: complete — containers=20, health=ok, deployed=12, available=53, app_telemetry=12 +2026/10/25 17:17:01 pusher.go:106: [DEBUG] [report] Push: url=https://hub.felhom.eu/api/v1/report payload=9977 bytes +2026/10/25 17:17:01 pusher.go:142: [INFO] [report] Hub report pushed successfully (9977 bytes) +2026/10/25 17:17:01 settings.go:970: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (23056 bytes) +2026/10/25 17:17:01 settings.go:973: [INFO] [settings] Settings saved +2026/10/25 17:17:01 updater.go:104: [DEBUG] [selfupdate] SetFloor: floor "" → "0.343.0" +2026/10/25 17:17:01 updater.go:104: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.343.0 >= floor 0.343.0 — no action +2026/10/25 17:17:01 main.go:1924: [INFO] Startup hub report sent +2026/10/25 17:17:04 manager.go:839: [INFO] [stacks] Status refresh: 23 containers across 69 stacks +2026/10/25 17:17:04 offsiteapply.go:148: [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.343.0 (we are 0.343.0), no managed update running +2026/10/25 17:17:04 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/25 17:17:04 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 11 targets (0 skipped not due, 0 skipped no container) +2026/10/25 17:17:04 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/25 17:17:04 healthprobe.go:187: [DEBUG] Health probe romm: HTTP GET :8080/ → 200 (2ms) +2026/10/25 17:17:04 healthprobe.go:187: [DEBUG] Health probe calibre-web: HTTP GET :8083/ → 302 (8ms) +2026/10/25 17:17:04 healthprobe.go:187: [DEBUG] Health probe paperless-ngx: HTTP GET :8000/ → 302 (8ms) +2026/10/25 17:17:04 healthprobe.go:187: [DEBUG] Health probe plex: API GET :32400/identity → 200 (8ms) +2026/10/25 17:17:04 healthprobe.go:187: [DEBUG] Health probe opengist: API GET :6157/healthcheck → 200 (8ms) +2026/10/25 17:17:04 healthprobe.go:187: [DEBUG] Health probe adventurelog: API GET :8000/api/ → 200 (9ms) +2026/10/25 17:17:04 healthprobe.go:187: [DEBUG] Health probe privatebin: HTTP GET :8080/ → 200 (10ms) +2026/10/25 17:17:04 healthprobe.go:187: [DEBUG] Health probe docmost: HTTP GET :3000/ → 200 (10ms) +2026/10/25 17:17:04 healthprobe.go:187: [DEBUG] Health probe home-assistant: API GET :8123/api/ → 401 (10ms) +2026/10/25 17:17:04 healthprobe.go:187: [DEBUG] Health probe bentopdf: HTTP GET :8080/ → 200 (10ms) +2026/10/25 17:17:04 manager.go:839: [INFO] [stacks] Status refresh: 23 containers across 69 stacks +2026/10/25 17:17:04 healthprobe.go:187: [DEBUG] Health probe bookstack: HTTP GET :80/ → 302 (44ms) +2026/10/25 17:17:04 healthprobe.go:167: [INFO] Health probes: 11 ok (of 11 probed) +2026/10/25 17:17:09 manager.go:839: [INFO] [stacks] Status refresh: 23 containers across 69 stacks +2026/10/25 17:17:09 main.go:2844: [INFO] [bootrecon] boot window: fleet settled after 10s (3 identical samples 5s apart) — sweeping +2026/10/25 17:17:09 bootrecon.go:236: [INFO] [bootrecon] "kimai" is a boot orphan by intent but is HELD (held after a failed restore whose rollback also failed (2026-10-14T09:06:31Z) — clear the hold to start it) — NOT starting it; whatever is holding it owns its recovery +2026/10/25 17:17:09 bootrecon.go:251: [INFO] [bootrecon] Boot reconciliation: nothing to start — 1 app(s) held (absent drive / quiesce / app-data operation): [kimai] +2026/10/25 17:17:09 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/25 17:17:14 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/25 17:17:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bentopdf — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:14 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/25 17:17:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bookstack — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping home-assistant — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:14 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (11 skipped not due, 0 skipped no container) +2026/10/25 17:17:14 manager.go:839: [INFO] [stacks] Status refresh: 23 containers across 69 stacks +2026/10/25 17:17:14 setup_gate.go:516: [DEBUG] [stacks] romm: setup probe SYSTEM.SHOW_SETUP_WIZARD = "true" (done when "false") +2026/10/25 17:17:24 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/25 17:17:24 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/25 17:17:24 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/25 17:17:24 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/25 17:17:24 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/25 17:17:24 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/25 17:17:24 crashboot.go:147: [INFO] [deadapp] boot grace 1m30s: the host's last unclean boot (2026-10-23T15:32:02Z) is not the one this start followed (R-856) +2026/10/25 17:17:24 aftercrash.go:106: [INFO] [aftercrash] a normal (or unknown) boot — no app repair after an unclean boot (R-1007) +2026/10/25 17:17:24 manager.go:839: [INFO] [stacks] Status refresh: 23 containers across 69 stacks +2026/10/25 17:17:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping home-assistant — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bentopdf — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bookstack — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:24 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (11 skipped not due, 0 skipped no container) +2026/10/25 17:17:26 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous mode, url=https://gitea.dooplex.hu/v2/admin/felhom-controller/tags/list +2026/10/25 17:17:26 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous plain GET HTTP 401 +2026/10/25 17:17:26 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous token request: https://gitea.dooplex.hu/v2/token?scope=repository%3Aadmin%2Ffelhom-controller%3Apull&service=container_registry +2026/10/25 17:17:26 updater.go:104: [DEBUG] [selfupdate] queryRegistry: anonymous tags/list with Bearer HTTP 200 +2026/10/25 17:17:26 updater.go:104: [DEBUG] [selfupdate] queryRegistry: 15 tags returned: [0.331.0 0.341.0 0.341.0-rc.1 0.342.0 0.342.0-rc.1 0.342.0-rc.2 0.343.0 0.343.0-rc.1 0.343.0-rc.2 0.343.0-rc.3 0.343.0-rc.4 0.343.0-rc.5 0.343.0-rc.6 0.343.0-rc.7 latest] +2026/10/25 17:17:26 updater.go:196: [INFO] [selfupdate] Current version 0.343.0 is up to date +2026/10/25 17:17:26 updater.go:104: [DEBUG] [selfupdate] version comparison: current=0.343.0 (0.343.0), latest=0.343.0 (0.343.0), cmp=0, updateAvailable=false +2026/10/25 17:17:26 client.go:1276: [DEBUG] [agentapi] GET /disks -> 200 (796ms) +2026/10/25 17:17:27 client.go:1276: [DEBUG] [agentapi] GET /disks -> 200 (806ms) +2026/10/25 17:17:34 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/25 17:17:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping home-assistant — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bentopdf — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bookstack — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:34 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (11 skipped not due, 0 skipped no container) +2026/10/25 17:17:34 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/25 17:17:34 manager.go:839: [INFO] [stacks] Status refresh: 23 containers across 69 stacks +2026/10/25 17:17:34 setup_gate.go:516: [DEBUG] [stacks] romm: setup probe SYSTEM.SHOW_SETUP_WIZARD = "true" (done when "false") +2026/10/25 17:17:39 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/25 17:17:44 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/25 17:17:44 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/25 17:17:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping home-assistant — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bentopdf — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bookstack — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:44 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (11 skipped not due, 0 skipped no container) +2026/10/25 17:17:44 manager.go:839: [INFO] [stacks] Status refresh: 23 containers across 69 stacks +2026/10/25 17:17:54 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/hdd_1", hasCPUCollector=true) +2026/10/25 17:17:54 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈16531MB avail≈9367MB +2026/10/25 17:17:54 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=11108764 → total=25898MB avail=9367MB used=16531MB (63.8%) +2026/10/25 17:17:54 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.7GB used=20.8GB avail=44.5GB (31.8%) +2026/10/25 17:17:54 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/hdd_1" bsize=4096 total=937.8GB used=225.2GB avail=665.0GB (25.3%) +2026/10/25 17:17:54 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.24 1.12 1.41 7/1529 938" → 1m=1.24 5m=1.12 15m=1.41 +2026/10/25 17:17:54 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/25 17:17:54 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 50.0°C (hwmon0) +2026/10/25 17:17:54 info.go:13: [DEBUG] [system] GetInfo done in 72ms — mem=16531MB/25898MB (63.8%), rootDisk=20.8GB/68.7GB (31.8%), load=1.24/1.12/1.41, temp=50.0°C (hwmon0), cpu=7.6% +2026/10/25 17:17:54 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/25 17:17:54 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/25 17:17:54 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/25 17:17:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping home-assistant — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bentopdf — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bookstack — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/25 17:17:54 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (11 skipped not due, 0 skipped no container) +2026/10/25 17:17:54 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/25 17:17:54 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/25 17:17:54 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/25 17:17:54 manager.go:839: [INFO] [stacks] Status refresh: 23 containers across 69 stacks +2026/10/25 17:17:54 setup_gate.go:516: [DEBUG] [stacks] romm: setup probe SYSTEM.SHOW_SETUP_WIZARD = "true" (done when "false") +2026/10/25 17:17:54 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health +2026/10/25 17:17:54 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting +2026/10/25 17:17:55 client.go:1276: [DEBUG] [agentapi] GET /storage -> 200 (705ms) +2026/10/25 17:17:55 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 705ms) +2026/10/25 17:17:56 client.go:1276: [DEBUG] [agentapi] GET /disks -> 200 (794ms) +2026/10/25 17:17:57 client.go:1276: [DEBUG] [agentapi] GET /disks -> 200 (798ms) +2026/10/25 17:18:04 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/25 17:18:04 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/25 17:18:04 manager.go:839: [INFO] [stacks] Status refresh: 23 containers across 69 stacks +2026/10/25 17:18:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/25 17:18:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/25 17:18:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/25 17:18:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/25 17:18:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping home-assistant — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/25 17:18:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bentopdf — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/25 17:18:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/25 17:18:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/25 17:18:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bookstack — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/25 17:18:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/25 17:18:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/25 17:18:04 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (11 skipped not due, 0 skipped no container) diff --git a/documentation/audits/release-2026-10-26/boxes-0.343.0/https-only.txt b/documentation/audits/release-2026-10-26/boxes-0.343.0/https-only.txt new file mode 100644 index 00000000..8682d689 --- /dev/null +++ b/documentation/audits/release-2026-10-26/boxes-0.343.0/https-only.txt @@ -0,0 +1,33 @@ +control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -) +== tester-1 (enkicsifelhom.hu): 6 public name(s) + OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1 + OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 + OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1 + OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1 + OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1 +== demo-hp (enkisfelhom.hu): 14 public name(s) + OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1 + OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1 + OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1 + OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1 + OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1 + OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1 + OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 + OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1 + OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1 + OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1 + OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1 + OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1 + OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1 +== demo-felhom (demo-felhom.eu): 8 public name(s) + OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1 + OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1 + OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1 + OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 + OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1 + OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1 + OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1 + OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1 +== every public name redirects http -> https diff --git a/documentation/audits/release-2026-10-26/boxes-0.343.0/tester-1-controller.log b/documentation/audits/release-2026-10-26/boxes-0.343.0/tester-1-controller.log new file mode 100644 index 00000000..a5f37944 --- /dev/null +++ b/documentation/audits/release-2026-10-26/boxes-0.343.0/tester-1-controller.log @@ -0,0 +1,139 @@ +2026/10/25 17:16:53 [INFO] local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible +2026/10/25 17:16:53 [INFO] local-api: mount mp9 → /etc/felhom-bootstrap (storage=/var/lib/felhom-agent/guests/9201/bootstrap, class=, backup=false) +2026/10/25 17:16:53 [INFO] local-api: mount mp8 → /mnt/felhom-drives (storage=/mnt/felhom-drives, class=, backup=false) +2026/10/25 17:16:53 [INFO] local-api: mount mp0 → /var/lib/felhom (storage=local-lvm, class=, backup=true) +2026/10/25 17:16:53 [INFO] felhom-controller 0.343.0 starting (customer: tester-1, domain: enkicsifelhom.hu) +2026/10/25 17:16:53 [INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json +2026/10/25 17:16:53 [INFO] Encryption key loaded from /opt/docker/felhom-controller/data/encryption.key +2026/10/25 17:16:53 [INFO] [stacks] Using compose command: docker compose +2026/10/25 17:16:53 [INFO] [stacks] ScanStacks complete: 68 stacks found (3 deployed, 65 available) +2026/10/25 17:16:53 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/25 17:16:53 [INFO] [stacks] InjectMissingFields: processed 3 stacks +2026/10/25 17:16:53 [INFO] [stacks] Encryption migration: no stacks needed migration +2026/10/25 17:16:53 [INFO] [quiesce] loop started (poll 5m0s, max-quiesce 30m0s) +2026/10/25 17:16:53 [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded (state ambiguous — legacy boot behaviour retained) +2026/10/25 17:16:53 [INFO] [stacks] installed-images backfill: 0 app(s) recorded, 3 already had a record, 0 left unrecorded (could not be observed completely — unknown, which renders nothing) +2026/10/25 17:16:53 [INFO] [stacks] pin adoption: 0 pinned, 3 already pinned, 0 left unpinned (0 not completely observed, 0 running something the template no longer offers) +2026/10/25 17:16:53 [INFO] [stacks] applied-meta backfill (R-646): recorded 0 []; skipped 0 [] — not current with the catalog, their pinned version's .felhom.yml is no longer on the box +2026/10/25 17:16:53 [INFO] [sync] Starting catalog sync (repo: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git, interval: 15m0s) +2026/10/25 17:16:53 [INFO] [sync] Starting catalog sync +2026/10/25 17:16:53 [INFO] Metrics store opened at /opt/docker/felhom-controller/data/metrics.db +2026/10/25 17:16:53 [INFO] Metrics collector started (60s interval) +2026/10/25 17:16:53 [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (branch: main) +2026/10/25 17:16:53 [INFO] [tier2] start-up: no leftover second-drive copy container +2026/10/25 17:16:53 [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false) +2026/10/25 17:16:53 [INFO] [backup] app-data run record wired: /opt/docker/felhom-controller/data/appdata-run.json (an earlier cut still shown: false) +2026/10/25 17:16:53 [INFO] Notifier enabled (hub: https://hub.felhom.eu) +2026/10/25 17:16:53 [INFO] [upwatch] the host's weekly line test: 94.3 Mbit/s (measured 2026-10-24T08:00:58Z) +2026/10/25 17:16:53 [INFO] [upwatch] upload and CPU watch started (line 94 Mbit/s, qBittorrent share 50 %) +2026/10/25 17:16:53 [INFO] [selfupdate] Post-update startup: update successful (0.342.0 → 0.343.0) +2026/10/25 17:16:53 [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30) +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: status-refresh (every 10s) +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s) +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: docker-socket-watch (every 15s) +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: docker-socket-users (every 5m0s) +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: health-probes (every 10s) +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: system-health (every 5m0s) +2026/10/25 17:16:53 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: after-unclean-boot (every 30s) +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s) +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: ring-spill (every 30s) +2026/10/25 17:16:53 [INFO] [scheduler] Daily job db-dump scheduled for 2026-10-26 02:30 CET +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s) +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s) +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: conversion-copy-release (every 1h0m0s) +2026/10/25 17:16:53 [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-10-26 03:30 CET +2026/10/25 17:16:53 [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-26 04:15 CET +2026/10/25 17:16:53 [INFO] [catch-up] controller start: no backup leg missed its last scheduled time — nothing to make up +2026/10/25 17:16:53 [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-10-26 05:10 CET +2026/10/25 17:16:53 [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-10-26 06:00 CET +2026/10/25 17:16:53 [INFO] [scheduler] Daily job offsite-proof scheduled for 2026-10-26 05:30 CET +2026/10/25 17:16:53 [INFO] [scheduler] Daily job metrics-prune scheduled for 2026-10-26 04:00 CET +2026/10/25 17:16:53 [INFO] [scheduler] Daily job fill-watch scheduled for 2026-10-26 03:30 CET +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: fill-watch-interval (every 10m0s) +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: hub-report (every 15m0s) +2026/10/25 17:16:53 [INFO] Hub reporting enabled (every 15m0s to https://hub.felhom.eu) +2026/10/25 17:16:53 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s) +2026/10/25 17:16:53 [INFO] ========== Startup Self-Test ========== +2026/10/25 17:16:54 [INFO] Event pushed: controller_updated (info) [hu-only] — Controller frissítve: 0.342.0 → 0.343.0 +2026/10/25 17:16:54 [INFO] [PASS] Docker socket: reachable (v29.8.2) +2026/10/25 17:16:54 [INFO] [PASS] Stacks directory: /opt/docker/stacks +2026/10/25 17:16:54 [INFO] [PASS] Data directory: /opt/docker/felhom-controller/data (writable) +2026/10/25 17:16:54 [INFO] [PASS] System data path: /mnt/sys_drive +2026/10/25 17:16:54 [INFO] [PASS] Storage paths: 1 connected, 0 disconnected +2026/10/25 17:16:54 [INFO] [PASS] Git catalog: 65 app definitions found +2026/10/25 17:16:54 [INFO] [PASS] Hub connectivity: https://hub.felhom.eu reachable (HTTP 200) +2026/10/25 17:16:54 [INFO] [PASS] Metrics DB: 35.9 MB +2026/10/25 17:16:54 [INFO] ======================================== +2026/10/25 17:16:54 [INFO] Self-test complete: 8 passed, 0 warnings, 0 failed +2026/10/25 17:16:54 [INFO] [scheduler] Starting scheduler with 23 jobs +2026/10/25 17:16:54 [INFO] [scheduler] Registered periodic job: geo-verify (every 6h0m0s) +2026/10/25 17:16:54 [INFO] Geo-restriction support enabled (CF API token configured) +2026/10/25 17:16:54 [INFO] [report] hub wait channel active (hold ≤240s) +2026/10/25 17:16:54 [INFO] [backup] Found 2 DB dump files across drives +2026/10/25 17:16:54 [INFO] [infra] connected felhom-controller to traefik-public +2026/10/25 17:16:54 [INFO] [web] sessions: restored 12 dashboard session(s) from disk (dropped 0 expired, 0 malformed) +2026/10/25 17:16:54 [INFO] [web] Auth: using password from settings.json +2026/10/25 17:16:54 [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s) +2026/10/25 17:16:54 [INFO] [scheduler] Registered periodic job: agent-channel-health (every 1m0s) +2026/10/25 17:16:54 [INFO] Web UI listening on :8080 +2026/10/25 17:16:54 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_a not mounted — skipping userdata skeleton +2026/10/25 17:16:54 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_b not mounted — skipping userdata skeleton +2026/10/25 17:16:54 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_c not mounted — skipping userdata skeleton +2026/10/25 17:16:54 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_d not mounted — skipping userdata skeleton +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/25 17:16:54 [INFO] [backup] Discovered 2 databases +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/25 17:16:54 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/25 17:16:54 [INFO] [backup] Discovered app data: 3 apps +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/bookstack — 5 env vars, 0 encrypted, 3 sensitive fields +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for bookstack +2026/10/25 17:16:54 [INFO] [stacks] bookstack: install id 27a3384799fbbc3f recorded (an app installed before D47) +2026/10/25 17:16:54 [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/paperless-ngx — 8 env vars, 0 encrypted, 3 sensitive fields +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for paperless-ngx +2026/10/25 17:16:54 [INFO] [stacks] paperless-ngx: install id 694723ce3390099b recorded (an app installed before D47) +2026/10/25 17:16:54 [INFO] [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/10/25 17:16:54 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/privatebin — 2 env vars, 0 encrypted, 0 sensitive fields +2026/10/25 17:16:54 [INFO] [stacks] SaveAppConfig: saved config for privatebin +2026/10/25 17:16:54 [INFO] [stacks] privatebin: install id 6b00861f43793b20 recorded (an app installed before D47) +2026/10/25 17:16:54 [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/25 17:16:54 [INFO] [backup] Backup status cache refreshed +2026/10/25 17:16:54 [INFO] [sync] Catalog sync complete +2026/10/25 17:16:54 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +2026/10/25 17:16:54 [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (5 storage path(s)) +2026/10/25 17:16:54 [INFO] [monitor] Health check: status=ok +2026/10/25 17:16:54 [INFO] [infra] connected felhom-controller to felhom-filebrowser (the FileBrowser default-login check, R-1126) +2026/10/25 17:16:58 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/25 17:16:59 [INFO] [report] Building system report +2026/10/25 17:16:59 [INFO] Event pushed: controller_started (info) [hu-only] — Controller elindult (0.343.0) +2026/10/25 17:17:00 [INFO] [monitor] Health check: status=ok +2026/10/25 17:17:00 [INFO] [report] Hub report pushed successfully (5247 bytes) +2026/10/25 17:17:00 [INFO] [settings] Settings saved +2026/10/25 17:17:00 [INFO] Startup hub report sent +2026/10/25 17:17:03 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/25 17:17:03 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.343.0 (we are 0.343.0), no managed update running +2026/10/25 17:17:04 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/25 17:17:04 [INFO] Health probes: 3 ok (of 3 probed) +2026/10/25 17:17:08 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/25 17:17:08 [INFO] [bootrecon] boot window: fleet settled after 10s (3 identical samples 5s apart) — sweeping +2026/10/25 17:17:08 [INFO] [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start) +2026/10/25 17:17:14 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/25 17:17:24 [INFO] [deadapp] boot grace 1m30s: the host's last unclean boot (2026-10-24T17:44:02Z) is not the one this start followed (R-856) +2026/10/25 17:17:24 [INFO] [aftercrash] a normal (or unknown) boot — no app repair after an unclean boot (R-1007) +2026/10/25 17:17:24 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/25 17:17:25 [INFO] [selfupdate] Current version 0.343.0 is up to date +2026/10/25 17:17:34 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/25 17:17:44 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/25 17:17:54 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/25 17:17:54 [INFO] [scheduler] Running job: agent-channel-health +2026/10/25 17:17:54 [INFO] [scheduler] Job agent-channel-health completed (took 551ms) diff --git a/documentation/audits/release-2026-10-26/byo/byo-install.log b/documentation/audits/release-2026-10-26/byo/byo-install.log new file mode 100644 index 00000000..3939589c --- /dev/null +++ b/documentation/audits/release-2026-10-26/byo/byo-install.log @@ -0,0 +1,159 @@ + +[INFO] felhom-host-install v1.36.0 — mode=byo customer=Tester4 vmid=9201 + +[STEP] 1/8 pre-flight +[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve) +[INFO] node: byo-test (explicit) +[INFO] agent config: /etc/felhom-agent/agent.json +[INFO] agent: not installed yet — will be fetched + installed in step 5/8 +[INFO] local-lvm free: ~29 GiB +[WARN] local-lvm free ~29 GiB is below the recommended 120 GiB — the install continues; proceed only if you sized the grows deliberately +[INFO] free RAM: ~6285 MiB +[INFO] existing guests on this host: 0 (pct+qm) +[INFO] archive storage 'local' present +[INFO] acl storages all present or pre-positioned: local local-lvm +[INFO] dnsmasq: not present before Felhom — recorded; uninstall will remove it again if we install it +[INFO] host DNS (:53): free +[INFO] hub reachable (https://hub.felhom.eu) +[OK] customer 'Tester4' exists + passphrase valid +[INFO] golden: none local — will fetch + verify from Gitea in step 7/8 + +[STEP] BYO disclosure — installing Felhom on a host the operator does NOT own will create: + pveum: roles FelhomAgentBase/FelhomAgentGuest/FelhomAgentStore + user felhom-agent@pve + token felhom-agent@pve!agent + + scoped ACL grants (Base@/, Guest@/pool/felhom, Store@[local local-lvm]) + the 'felhom' pool + agent: service user felhom-agent + /usr/local/bin/felhom-agent (+ .bak-* backups; A/B slots via the guarded self-update) + + /etc/systemd/system/felhom-agent.service (+ felhom-agent-rollback.service + start-limit drop-in) + + /etc/sudoers.d/felhom-agent + state dir /var/lib/felhom-agent + config /etc/felhom-agent/agent.json (0600) + host: /usr/local/sbin/felhom-mkfs-guarded + /usr/local/sbin/felhom-selfupdate-guarded + + /usr/local/sbin/felhom-pbs-apply (PBS-DR apply wrapper — DR capability is baked + on every install; ACTIVATION stays a hub flag, off = zero effect on this host) + + felhom-mgmt-watchdog service+timer+script + /etc/tmpfiles.d/felhom-privsep.conf + + /usr/local/sbin/felhom-priv-apply + the guest-hook snippet under /var/lib/vz/snippets/ + (both from the agent's config bundle) + + the 'sudo' and 'age' packages if absent + install state dir /var/lib/felhom-install + wg: an OUTBOUND WireGuard tunnel to the Felhom hub (wg_tunnel.enabled=true — base + infrastructure like the cloudflared tunnel; hands-free peer registration; the + operator can disable it per-customer hub-side on request) + guest: the provisioned Felhom LXC (vmid 9201, capped 2 cores / 4096 MiB) + its volumes + + the golden vzdump imported onto storage 'local' + reuse: a golden already on 'local' is used only if its sha256 or its controller version + matches the hub's vouched golden; otherwise the vouched one is fetched (a --golden you name is refused) + update: operator-signed self-update authority: felhom-op-1 + NOT touched in byo mode: root@pam (no break-glass), host DNS (:53), OOB sshd. + +Type this host's short hostname (byo-test) to acknowledge the byo install: [OK] byo install acknowledged for host byo-test +[OK] pre-flight passed +[STEP] 2/8 Proxmox API token +[OK] token minted (secret captured, not logged) +[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm]) +[SKIP] old broad role FelhomAgent already absent +[STEP] 3/8 compute volume grows +[INFO] auto-computed from a 29 GiB thin pool (16 §7: 50/100/130 GiB) +[INFO] grows: rootfs +0G (->32G), data +26G (->50G, ONE volume) +[STEP] 4/8 host enrollment (POST /host-enroll) +[OK] host MINTED (first enroll) +[INFO] host_id: Tester4-f967d4 (api_key captured, not logged) +[SKIP] break-glass (step 4b) SKIPPED (byo: the host owner manages break-glass; root@pam untouched, nothing vaulted) +[STEP] 5/8 agent install (fetch + verify + install) +[SKIP] apt repo alignment skipped (byo — the owner manages repos) +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] manifest: agent v0.164.0 (sha 1b803d8b70f8afbe…), golden v0.331.0 +[INFO] fetching agent binary v0.164.0 from Gitea … +[OK] verified sha256 1b803d8b70f8afbe… matches the hub manifest +[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.164.0) +[INFO] installing the 'sudo' package (required for the non-root agent model) … +[OK] sudo installed (Sudo version 1.9.16p2) +[INFO] installing the 'age' package (escrow ceremony identity-wrap dependency) … +[OK] age installed (1.2.1) +[OK] created service user felhom-agent +[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify) +[OK] verified sha256 1d7b54d39d0c1aff… matches the hub manifest + os-apply: BUNDLE START agent=0.164.0 sha=1d7b54d39d0c1aff authority=installer files=32 write=26 same=2 kept=0 skipped=4 + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mkfs-guarded (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-selfupdate-guarded (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-pbs-apply (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-backup-target-apply (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-vzdump-partial-clean (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-thin-check (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-internal-place (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-crash-guard (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-priv-apply (new) + os-apply: BUNDLE WROTE /var/lib/vz/snippets/felhom-guest-hook.sh (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-shared-parent.sh (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-shared-parent.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.timer (new) + os-apply: BUNDLE WROTE /etc/felhom/crash-guard.conf (new) + os-apply: BUNDLE WROTE /etc/grub.d/01_felhom_oneshot (new) + os-apply: BUNDLE WROTE /etc/grub.d/42_felhom_oneshot (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent-rollback.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mgmt-watchdog (new) + os-apply: BUNDLE WROTE /etc/tmpfiles.d/felhom-privsep.conf (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.timer (new) + os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-agent (new) + os-apply: BUNDLE DONE agent=0.164.0 written=26 same=2 self-check=ok signers-created=False +[OK] installed every root-owned file from config bundle v0.164.0 (sha 1d7b54d39d0c1aff…; checked, previous copies kept) +[SKIP] OOB (felhom-sshd) off (byo, or appliance --no-oob) — the bundle skipped its files +[STEP] 5b/8 internal data place (Belső tárhely) +[SKIP] byo: the host owner's thin pool is not ours to divide — no internal place +[STEP] 6/8 agent config + service +[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive. +[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard. +[INFO] node=byo-test local_api=192.168.0.151:8443 tls_fp=7E:77:41:86:26:92… +[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent) +[OK] byo config asserts passed (lan_resolver.enable / oob.enabled off; deployment_mode=byo; wg_tunnel intentionally ON — base infrastructure) +[OK] agent --selftest (read-only) passed +[OK] felhom-agent service active (non-root felhom-agent reads the config OK) +[OK] wrote /etc/felhom/os-trust.json + operator-signers (felhom-op-1) — the root-owned slow-lane anchors +[STEP] 7/8 golden archive +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] fetching golden v0.331.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_10_25-17_52_14.tar.zst +[OK] verified sha256 3881de62b0986ccb… matches the hub manifest +[OK] golden imported + verified: local:backup/vzdump-lxc-9100-2026_10_25-17_52_14.tar.zst +[STEP] 8/8 provision guest 9201 +[SKIP] pool felhom already exists +=== felhom-agent 0.164.0 selftest=provision (vmid=9201 customer=Tester4 hostname=tester4) === + --- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_10_25-17_52_14.tar.zst → vmid 9201 --- +time=2026-10-25T17:53:51.729+01:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=9201 pool=felhom + [OK] front half: vmid 9201 up (boot+running) in 1m26s; MAC=BC:24:11:4B:04:4E + --- back half: mint per-guest token + populate bootstrap config mount --- +time=2026-10-25T17:53:56.751+01:00 level=INFO msg="provision: back-half complete" vmid=9201 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=192.168.0.151:8443 + [OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 9201 (host dir /var/lib/felhom-agent/guests/9201/bootstrap) + local-api endpoint 192.168.0.151:8443 · leaf fp f09a20d4c01686516db562b6acc9aa7e95581d69c2318fccf7894ed772ef4a8e · token: minted (not printed) +=== selftest=provision OK — guest 9201 provisioned + bootstrap-mounted (KEPT) === + next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller, + which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api. +[OK] provision completed +[INFO] rebooting guest 9201 so the baked controller-bootstrap unit picks up the mount +[STEP] verify +[OK] pct status: running +[OK] onboot: 1 + mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G + mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives + rootfs: local-lvm:vm-9201-disk-0,size=32G +[OK] pool: guest 9201 is a member of felhom +[OK] acl: FelhomAgentBase@/ present (user+token) +[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token) +[OK] acl: FelhomAgentStore@/storage/local present (user+token) +[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token) +[OK] authz signers: 2 (operator-signed self-update armed) +[OK] controller: Up 5 seconds (healthy) (after ~0s) +[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.342.0 +[WARN] cloudflared not visible yet +[INFO] (confirm in the hub UI that host Tester4-f967d4 reports guest 9201) + +[OK] Day-0 provision SUCCESS — vmid=9201 host_id=Tester4-f967d4 customer=Tester4 golden=local:backup/vzdump-lxc-9100-2026_10_25-17_52_14.tar.zst diff --git a/documentation/audits/release-2026-10-26/byo/byo-proof.txt b/documentation/audits/release-2026-10-26/byo/byo-proof.txt new file mode 100644 index 00000000..aa735b44 --- /dev/null +++ b/documentation/audits/release-2026-10-26/byo/byo-proof.txt @@ -0,0 +1,73 @@ +# BYO test host byo-test (VM 9371 on demo-hp), host_id Tester4-f967d4, 2026-10-25 +## planted before the install: our 0.164.0 drop-in, sha256 f0485da5… (copied from Tester 1) +felhom-agent 0.165.0 +root 644 /var/lib/felhom-install/state.json +install mode: byo +## sshd_config.d now: +total 8 +drwxr-xr-x 2 root root 4096 Oct 25 18:22 . +drwxr-xr-x 4 root root 4096 Oct 25 17:37 .. +permitrootlogin yes +passwordauthentication yes +## journal (os-apply + agent, 17:40–18:25) +Oct 25 17:52:07 byo-test felhom-os-apply[4082]: os-apply: BUNDLE START agent=0.164.0 sha=1d7b54d39d0c1aff authority=installer files=32 write=26 same=2 kept=0 skipped=4 +Oct 25 17:52:07 byo-test felhom-os-apply[4083]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mkfs-guarded (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4084]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-selfupdate-guarded (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4085]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-pbs-apply (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4086]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-backup-target-apply (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4087]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-vzdump-partial-clean (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4088]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-thin-check (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4089]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-internal-place (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4090]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-crash-guard (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4091]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-priv-apply (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4092]: os-apply: BUNDLE WROTE /var/lib/vz/snippets/felhom-guest-hook.sh (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4093]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-shared-parent.sh (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4094]: os-apply: BUNDLE WROTE /etc/systemd/system/felhom-shared-parent.service (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4095]: os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard.service (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4096]: os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.service (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4097]: os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.timer (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4098]: os-apply: BUNDLE WROTE /etc/felhom/crash-guard.conf (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4099]: os-apply: BUNDLE WROTE /etc/grub.d/01_felhom_oneshot (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4100]: os-apply: BUNDLE WROTE /etc/grub.d/42_felhom_oneshot (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4101]: os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4102]: os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent-rollback.service (new) +Oct 25 17:52:07 byo-test felhom-os-apply[4103]: os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf (new) +Oct 25 17:52:08 byo-test felhom-os-apply[4104]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mgmt-watchdog (new) +Oct 25 17:52:08 byo-test felhom-os-apply[4105]: os-apply: BUNDLE WROTE /etc/tmpfiles.d/felhom-privsep.conf (new) +Oct 25 17:52:08 byo-test felhom-os-apply[4106]: os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.service (new) +Oct 25 17:52:08 byo-test felhom-os-apply[4107]: os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.timer (new) +Oct 25 17:52:08 byo-test felhom-os-apply[4108]: os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-agent (new) +Oct 25 17:52:09 byo-test felhom-os-apply[4235]: os-apply: BUNDLE DONE agent=0.164.0 written=26 same=2 self-check=ok signers-created=False +Oct 25 17:52:11 byo-test felhom-os-apply[4417]: os-apply: REFUSED: R12 this box was installed as 'byo', not appliance — its host belongs to its owner +Oct 25 17:52:11 byo-test felhom-agent[4389]: time=2026-10-25T17:52:11.013+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REFUSED: R12 this box was installed as 'byo', not appliance — its host belongs to its owner" +Oct 25 17:52:12 byo-test felhom-agent[4389]: time=2026-10-25T17:52:12.970+01:00 level=INFO msg="capabilities self-check" ok=70 total=76 degraded=1 inactive=5 +Oct 25 18:02:41 byo-test felhom-agent[4389]: time=2026-10-25T18:02:41.293+01:00 level=INFO msg="osupdate: docker step skipped — ring 1 takes an engine set only inside a signed operator job (`11` §5.8)" run=20261025T170128Z vmid=9201 trigger=night ring=1 enabled=true +Oct 25 18:02:41 byo-test felhom-agent[4389]: time=2026-10-25T18:02:41.293+01:00 level=INFO msg="osupdate: pve step skipped — ring 1 takes a Proxmox set only inside a signed operator job (`11` §5.10)" run=20261025T170128Z vmid=9201 trigger=night ring=1 enabled=true +Oct 25 18:07:19 byo-test felhom-agent[4389]: time=2026-10-25T18:07:19.468+01:00 level=INFO msg="desired: hub reports pending signed ops (fetch/execute is slice 10B)" +Oct 25 18:07:19 byo-test felhom-agent[4389]: time=2026-10-25T18:07:19.475+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=1c622afc4aec7208 op=agent_update key_id=felhom-op-1 nonce=8abd1cd5faf79f4ce21dfc5b55d49d84 +Oct 25 18:07:20 byo-test felhom-os-apply[27473]: os-apply: AGENT-UPDATE signed by the operator: version=0.165.0 sha=e9d04a47ee85222c — handing the root copy to the A/B wrapper +Oct 25 18:07:20 byo-test felhom-agent[4389]: time=2026-10-25T18:07:20.446+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=1c622afc4aec7208 op=agent_update +Oct 25 18:07:20 byo-test felhom-agent[4389]: time=2026-10-25T18:07:20.497+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=d885aae390b02162 op=agent_config_update key_id=felhom-op-1 nonce=2cfa2580f3917ea3ed018d0c2233f81f +Oct 25 18:07:20 byo-test felhom-os-apply[27523]: os-apply: BUNDLE START agent=0.165.0 sha=8dea1f34d77ba40a authority=signed files=32 write=1 same=26 kept=1 skipped=4 +Oct 25 18:07:20 byo-test felhom-os-apply[27527]: os-apply: BUNDLE WROTE /usr/local/sbin/felhom-os-apply (replaced) +Oct 25 18:07:21 byo-test felhom-os-apply[27603]: os-apply: BUNDLE DONE agent=0.165.0 written=1 same=26 self-check=ok signers-created=False +Oct 25 18:07:21 byo-test felhom-agent[4389]: time=2026-10-25T18:07:21.547+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE START agent=0.165.0 sha=8dea1f34d77ba40a authority=signed files=32 write=1 same=26 kept=1 skipped=4" +Oct 25 18:07:21 byo-test felhom-agent[4389]: time=2026-10-25T18:07:21.547+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE WROTE /usr/local/sbin/felhom-os-apply (replaced)" +Oct 25 18:07:21 byo-test felhom-agent[4389]: time=2026-10-25T18:07:21.547+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.165.0 written=1 same=26 self-check=ok signers-created=False" +Oct 25 18:07:22 byo-test felhom-agent[4389]: time=2026-10-25T18:07:22.499+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=71 total=76 degraded=pve:store-grant:felhom-pbs +Oct 25 18:07:22 byo-test felhom-agent[4389]: time=2026-10-25T18:07:22.500+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=d885aae390b02162 op=agent_config_update +Oct 25 18:07:23 byo-test felhom-os-apply[27793]: os-apply: REFUSED: R12 this box was installed as 'byo', not appliance — its host belongs to its owner +Oct 25 18:07:23 byo-test felhom-agent[27763]: time=2026-10-25T18:07:23.572+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REFUSED: R12 this box was installed as 'byo', not appliance — its host belongs to its owner" +Oct 25 18:07:24 byo-test felhom-agent[27763]: time=2026-10-25T18:07:24.350+01:00 level=INFO msg="capabilities self-check" ok=76 total=76 degraded=0 inactive=0 +Oct 25 18:08:25 byo-test felhom-agent[27763]: time=2026-10-25T18:08:25.030+01:00 level=WARN msg="selfupdate: update committed" version=0.165.0 wrapper="" +Oct 25 18:22:34 byo-test felhom-agent[27763]: time=2026-10-25T18:22:34.999+01:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=cc915dd2020e42b0 op=agent_config_update key_id=felhom-op-1 nonce=8083346e293da88779c3c975d4e8abae +Oct 25 18:22:35 byo-test felhom-os-apply[39714]: os-apply: BUNDLE /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf: a BYO host keeps its own settings (D43) — removing the copy we wrote +Oct 25 18:22:35 byo-test felhom-os-apply[39715]: os-apply: BUNDLE START agent=0.165.0 sha=8dea1f34d77ba40a authority=signed files=32 write=0 same=26 kept=1 skipped=4 +Oct 25 18:22:35 byo-test felhom-os-apply[39716]: os-apply: BUNDLE REMOVED /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf (ours, on a BYO host — D43) +Oct 25 18:22:36 byo-test felhom-os-apply[39816]: os-apply: BUNDLE DONE agent=0.165.0 written=0 same=26 self-check=ok signers-created=False +Oct 25 18:22:36 byo-test felhom-agent[27763]: time=2026-10-25T18:22:36.194+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf: a BYO host keeps its own settings (D43) — removing the copy we wrote" +Oct 25 18:22:36 byo-test felhom-agent[27763]: time=2026-10-25T18:22:36.194+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE START agent=0.165.0 sha=8dea1f34d77ba40a authority=signed files=32 write=0 same=26 kept=1 skipped=4" +Oct 25 18:22:36 byo-test felhom-agent[27763]: time=2026-10-25T18:22:36.194+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE REMOVED /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf (ours, on a BYO host — D43)" +Oct 25 18:22:36 byo-test felhom-agent[27763]: time=2026-10-25T18:22:36.194+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.165.0 written=0 same=26 self-check=ok signers-created=False" +Oct 25 18:22:37 byo-test felhom-agent[27763]: time=2026-10-25T18:22:37.238+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=76 total=76 degraded="" +Oct 25 18:22:37 byo-test felhom-agent[27763]: time=2026-10-25T18:22:37.238+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=cc915dd2020e42b0 op=agent_config_update diff --git a/documentation/audits/release-2026-10-26/byo/install-console.txt b/documentation/audits/release-2026-10-26/byo/install-console.txt new file mode 100644 index 00000000..689452fb --- /dev/null +++ b/documentation/audits/release-2026-10-26/byo/install-console.txt @@ -0,0 +1,162 @@ + +[INFO] felhom-host-install v1.36.0 — mode=byo customer=Tester4 vmid=9201 + +[STEP] 1/8 pre-flight +[INFO] pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve) +[INFO] node: byo-test (explicit) +[INFO] agent config: /etc/felhom-agent/agent.json +[INFO] agent: not installed yet — will be fetched + installed in step 5/8 +[INFO] local-lvm free: ~29 GiB +[WARN] local-lvm free ~29 GiB is below the recommended 120 GiB — the install continues; proceed only if you sized the grows deliberately +[INFO] free RAM: ~6285 MiB +[INFO] existing guests on this host: 0 (pct+qm) +[INFO] archive storage 'local' present +[INFO] acl storages all present or pre-positioned: local local-lvm +[INFO] dnsmasq: not present before Felhom — recorded; uninstall will remove it again if we install it +[INFO] host DNS (:53): free +[INFO] hub reachable (https://hub.felhom.eu) +[OK] customer 'Tester4' exists + passphrase valid +[INFO] golden: none local — will fetch + verify from Gitea in step 7/8 +byo-test + +[STEP] BYO disclosure — installing Felhom on a host the operator does NOT own will create: + pveum: roles FelhomAgentBase/FelhomAgentGuest/FelhomAgentStore + user felhom-agent@pve + token felhom-agent@pve!agent + + scoped ACL grants (Base@/, Guest@/pool/felhom, Store@[local local-lvm]) + the 'felhom' pool + agent: service user felhom-agent + /usr/local/bin/felhom-agent (+ .bak-* backups; A/B slots via the guarded self-update) + + /etc/systemd/system/felhom-agent.service (+ felhom-agent-rollback.service + start-limit drop-in) + + /etc/sudoers.d/felhom-agent + state dir /var/lib/felhom-agent + config /etc/felhom-agent/agent.json (0600) + host: /usr/local/sbin/felhom-mkfs-guarded + /usr/local/sbin/felhom-selfupdate-guarded + + /usr/local/sbin/felhom-pbs-apply (PBS-DR apply wrapper — DR capability is baked + on every install; ACTIVATION stays a hub flag, off = zero effect on this host) + + felhom-mgmt-watchdog service+timer+script + /etc/tmpfiles.d/felhom-privsep.conf + + /usr/local/sbin/felhom-priv-apply + the guest-hook snippet under /var/lib/vz/snippets/ + (both from the agent's config bundle) + + the 'sudo' and 'age' packages if absent + install state dir /var/lib/felhom-install + wg: an OUTBOUND WireGuard tunnel to the Felhom hub (wg_tunnel.enabled=true — base + infrastructure like the cloudflared tunnel; hands-free peer registration; the + operator can disable it per-customer hub-side on request) + guest: the provisioned Felhom LXC (vmid 9201, capped 2 cores / 4096 MiB) + its volumes + + the golden vzdump imported onto storage 'local' + reuse: a golden already on 'local' is used only if its sha256 or its controller version + matches the hub's vouched golden; otherwise the vouched one is fetched (a --golden you name is refused) + update: operator-signed self-update authority: felhom-op-1 + NOT touched in byo mode: root@pam (no break-glass), host DNS (:53), OOB sshd. + +Type this host's short hostname (byo-test) to acknowledge the byo install: [OK] byo install acknowledged for host byo-test +[OK] pre-flight passed +[STEP] 2/8 Proxmox API token +[OK] token minted (secret captured, not logged) +[OK] scoped ACL applied (Base@/, Guest@/pool/felhom + /vms/990000..990009, Store@[local local-lvm]) +[SKIP] old broad role FelhomAgent already absent +[STEP] 3/8 compute volume grows +[INFO] auto-computed from a 29 GiB thin pool (16 §7: 50/100/130 GiB) +[INFO] grows: rootfs +0G (->32G), data +26G (->50G, ONE volume) +[STEP] 4/8 host enrollment (POST /host-enroll) +[OK] host MINTED (first enroll) +[INFO] host_id: Tester4-f967d4 (api_key captured, not logged) +[SKIP] break-glass (step 4b) SKIPPED (byo: the host owner manages break-glass; root@pam untouched, nothing vaulted) +[STEP] 5/8 agent install (fetch + verify + install) +[SKIP] apt repo alignment skipped (byo — the owner manages repos) +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] manifest: agent v0.164.0 (sha 1b803d8b70f8afbe…), golden v0.331.0 +[INFO] fetching agent binary v0.164.0 from Gitea … +[OK] verified sha256 1b803d8b70f8afbe… matches the hub manifest +[OK] installed /usr/local/bin/felhom-agent (felhom-agent 0.164.0) +[INFO] installing the 'sudo' package (required for the non-root agent model) … +[OK] sudo installed (Sudo version 1.9.16p2) +[INFO] installing the 'age' package (escrow ceremony identity-wrap dependency) … +[OK] age installed (1.2.1) +[OK] created service user felhom-agent +[OK] added felhom-agent to systemd-journal (unprivileged journal read for NAS verify) +[OK] verified sha256 1d7b54d39d0c1aff… matches the hub manifest + os-apply: BUNDLE START agent=0.164.0 sha=1d7b54d39d0c1aff authority=installer files=32 write=26 same=2 kept=0 skipped=4 + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mkfs-guarded (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-selfupdate-guarded (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-pbs-apply (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-backup-target-apply (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-vzdump-partial-clean (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-thin-check (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-internal-place (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-crash-guard (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-priv-apply (new) + os-apply: BUNDLE WROTE /var/lib/vz/snippets/felhom-guest-hook.sh (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-shared-parent.sh (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-shared-parent.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-crash-guard-check.timer (new) + os-apply: BUNDLE WROTE /etc/felhom/crash-guard.conf (new) + os-apply: BUNDLE WROTE /etc/grub.d/01_felhom_oneshot (new) + os-apply: BUNDLE WROTE /etc/grub.d/42_felhom_oneshot (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent-rollback.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf (new) + os-apply: BUNDLE WROTE /usr/local/sbin/felhom-mgmt-watchdog (new) + os-apply: BUNDLE WROTE /etc/tmpfiles.d/felhom-privsep.conf (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.service (new) + os-apply: BUNDLE WROTE /etc/systemd/system/felhom-mgmt-watchdog.timer (new) + os-apply: BUNDLE WROTE /etc/sudoers.d/felhom-agent (new) + os-apply: BUNDLE DONE agent=0.164.0 written=26 same=2 self-check=ok signers-created=False +[OK] installed every root-owned file from config bundle v0.164.0 (sha 1d7b54d39d0c1aff…; checked, previous copies kept) +[SKIP] OOB (felhom-sshd) off (byo, or appliance --no-oob) — the bundle skipped its files +[STEP] 5b/8 internal data place (Belső tárhely) +[SKIP] byo: the host owner's thin pool is not ours to divide — no internal place +[STEP] 6/8 agent config + service +[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive. +[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard. +[INFO] node=byo-test local_api=192.168.0.151:8443 tls_fp=7E:77:41:86:26:92… +[OK] wrote /etc/felhom-agent/agent.json (0600 felhom-agent) +[OK] byo config asserts passed (lan_resolver.enable / oob.enabled off; deployment_mode=byo; wg_tunnel intentionally ON — base infrastructure) +[OK] agent --selftest (read-only) passed +[OK] felhom-agent service active (non-root felhom-agent reads the config OK) +[OK] wrote /etc/felhom/os-trust.json + operator-signers (felhom-op-1) — the root-owned slow-lane anchors +[STEP] 7/8 golden archive +[WARN] no git credential in controller.yaml — fetching artifacts ANONYMOUSLY (they are world-readable; sha256 verification unchanged) +[INFO] fetching golden v0.331.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_10_25-17_52_14.tar.zst +[OK] verified sha256 3881de62b0986ccb… matches the hub manifest +[OK] golden imported + verified: local:backup/vzdump-lxc-9100-2026_10_25-17_52_14.tar.zst +[STEP] 8/8 provision guest 9201 +[SKIP] pool felhom already exists +=== felhom-agent 0.164.0 selftest=provision (vmid=9201 customer=Tester4 hostname=tester4) === + --- front half: bring-up (provision) local:backup/vzdump-lxc-9100-2026_10_25-17_52_14.tar.zst → vmid 9201 --- +time=2026-10-25T17:53:51.729+01:00 level=INFO msg="bring-up: pool membership re-asserted" vmid=9201 pool=felhom + [OK] front half: vmid 9201 up (boot+running) in 1m26s; MAC=BC:24:11:4B:04:4E + --- back half: mint per-guest token + populate bootstrap config mount --- +time=2026-10-25T17:53:56.751+01:00 level=INFO msg="provision: back-half complete" vmid=9201 mount=mp9 guest_path=/etc/felhom-bootstrap endpoint=192.168.0.151:8443 + [OK] back half: bootstrap mount mp9 → /etc/felhom-bootstrap on vmid 9201 (host dir /var/lib/felhom-agent/guests/9201/bootstrap) + local-api endpoint 192.168.0.151:8443 · leaf fp f09a20d4c01686516db562b6acc9aa7e95581d69c2318fccf7894ed772ef4a8e · token: minted (not printed) +=== selftest=provision OK — guest 9201 provisioned + bootstrap-mounted (KEPT) === + next: reboot the guest → the golden's baked controller-bootstrap unit deploys the controller, + which PULLS its controller.yaml from the hub (retrieval passphrase) and merges in this local_api. +[OK] provision completed +[INFO] rebooting guest 9201 so the baked controller-bootstrap unit picks up the mount +[STEP] verify +[OK] pct status: running +[OK] onboot: 1 + mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=50G + mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives + rootfs: local-lvm:vm-9201-disk-0,size=32G +[OK] pool: guest 9201 is a member of felhom +[OK] acl: FelhomAgentBase@/ present (user+token) +[OK] acl: FelhomAgentGuest@/pool/felhom present (user+token) +[OK] acl: FelhomAgentStore@/storage/local present (user+token) +[OK] acl: FelhomAgentStore@/storage/local-lvm present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990000 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990001 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990002 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990003 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990004 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990005 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990006 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990007 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990008 present (user+token) +[OK] acl: FelhomAgentGuest@/vms/990009 present (user+token) +[OK] authz signers: 2 (operator-signed self-update armed) +[OK] controller: Up 5 seconds (healthy) (after ~0s) +[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.342.0 +[WARN] cloudflared not visible yet +[INFO] (confirm in the hub UI that host Tester4-f967d4 reports guest 9201) + +[OK] Day-0 provision SUCCESS — vmid=9201 host_id=Tester4-f967d4 customer=Tester4 golden=local:backup/vzdump-lxc-9100-2026_10_25-17_52_14.tar.zst +INSTALL_RC=0 +Connection to 192.168.0.151 closed. diff --git a/documentation/audits/release-2026-10-26/homepage-gate-9202.txt b/documentation/audits/release-2026-10-26/homepage-gate-9202.txt new file mode 100644 index 00000000..04134b9c --- /dev/null +++ b/documentation/audits/release-2026-10-26/homepage-gate-9202.txt @@ -0,0 +1,10 @@ +18:05:26 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +18:06:01 [1] deployed, controller state=running, pinned={'homepage': 'ghcr.io/gethomepage/homepage:v1.13.2'} +18:06:01 family gate: homepage is family-gated — passed as the household (cookie set) +18:06:01 deployed: True, state=running +18:06:01 STRANGER (text/html): 302 location=https://felhom.enkisfelhom.hu/__family/start?rd=https%3A%2F%2Fhomepage.enkisfelhom.hu%2F +18:06:01 STRANGER (application/json): 401 location= +18:06:01 HOUSEHOLD (dashboard session -> family cookie): 200, body 48349 B, homepage marker=True +18:06:02 [X] stop -> 200 {'ok': True, 'message': 'Stack homepage stop completed'} +18:06:34 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'homepage', 'volumes_removed': ['homepage_homepage_config'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'A +18:06:42 [X] after remove: deployed=False leftovers='' diff --git a/documentation/audits/release-2026-10-26/homepage-gate-repoint-drill.txt b/documentation/audits/release-2026-10-26/homepage-gate-repoint-drill.txt new file mode 100644 index 00000000..0f6a9839 --- /dev/null +++ b/documentation/audits/release-2026-10-26/homepage-gate-repoint-drill.txt @@ -0,0 +1,10 @@ +2026/10/25 17:04:28 sync.go:535: [INFO] [sync] Updated plex/docker-compose.yml +2026/10/25 17:04:28 sync.go:535: [INFO] [sync] Updated plex/.felhom.yml +2026/10/25 17:04:28 sync.go:261: [INFO] [sync] catalog moved 227a54498f76 -> 7e3f9543ae72 (0 new, 4 stack dir(s) changed) +2026/10/25 17:04:28 sync.go:299: [INFO] [sync] Catalog sync complete +2026/10/25 17:04:28 sync.go:142: [INFO] [sync] Initial sync: Sablonok frissítve — frissítve: homepage, n8n, opengist, plex + + expect n8n/docker-compose.yml ∋ N8N_DIAGNOSTICS_ENABLED=false: 1 + expect opengist/docker-compose.yml ∋ 256M: 2 + expect homepage/.felhom.yml ∋ family_gate: true: 1 +drill main 7e3f954, box cache 7e3f954: DRILL-CONFIRMED diff --git a/documentation/audits/release-2026-10-26/homepage-gate-repoint-live.txt b/documentation/audits/release-2026-10-26/homepage-gate-repoint-live.txt new file mode 100644 index 00000000..5a9ecc43 --- /dev/null +++ b/documentation/audits/release-2026-10-26/homepage-gate-repoint-live.txt @@ -0,0 +1,6 @@ +26: repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git +-rw------- 1 root root 1994 Oct 22 14:47 /var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml.pre-release1026-drill +mirror main 7e3f9543 +cache HEAD 227a544 +https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git + diff --git a/documentation/audits/release-2026-10-26/redproof/installer-http-code-doubled.txt b/documentation/audits/release-2026-10-26/redproof/installer-http-code-doubled.txt new file mode 100644 index 00000000..33e1ecb1 --- /dev/null +++ b/documentation/audits/release-2026-10-26/redproof/installer-http-code-doubled.txt @@ -0,0 +1,5 @@ +# installer http_code: the new test against the OLD helper (felhom-host-install.sh 1.36.0), 2026-10-25 +FAIL test_http_code_prints_one_status_on_a_refusal -- http_code printed '[401401]' for a 401 (the preflight's case never matches) +PASS test_http_code_prints_the_status_on_success +41 passed, 1 failed +# live, the BYO VM's preflight with installer 1.36.0: 'unexpected hub status 404404' (wrong id) and '401401' (wrong passphrase) — byo/ in this folder diff --git a/documentation/audits/release-2026-10-26/tools/homepage_gate.py b/documentation/audits/release-2026-10-26/tools/homepage_gate.py new file mode 100644 index 00000000..25b26a70 --- /dev/null +++ b/documentation/audits/release-2026-10-26/tools/homepage_gate.py @@ -0,0 +1,22 @@ +#!/usr/bin/env python3 +"""homepage_gate.py — D46 on the drill catalog (9202): Homepage deployed with `family_gate: true` stops a stranger +(no cookie → a redirect to /__family/start) and lets the household in (dashboard session → felhom_famgate → 200). +The same household pass on the same box is the positive control. Removed through the product at the end.""" +import os, re, sys, time +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools")) +os.environ.setdefault("TARGET", "9202") +import box_walk as w + +APP = SUB = "homepage" +w.login() +try: + ok = w.deploy(APP, SUB) and w.wait_app(SUB, "/", tries=60, delay=5) + w.say(f"deployed: {ok}, state={w.stack(APP).get('state')}") + for accept in ("text/html", "application/json"): + r = w.sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", f"Accept: {accept}", "-H", f"Host: {SUB}.{w.DOMAIN}", f"{w.BASE}/"]) + st = re.match(r"HTTP/\S+ (\d+)", r.stdout or "") + w.say(f"STRANGER ({accept}): {st.group(1) if st else '?'} location={w._loc(r.stdout)[:90]}") + rc, code, body = w.app_curl(SUB, "/") + w.say(f"HOUSEHOLD (dashboard session -> family cookie): {code}, body {len(body)} B, homepage marker={'homepage' in body.lower()}") +finally: + w.remove(APP) diff --git a/documentation/audits/release-2026-10-26/tools/repoint.py b/documentation/audits/release-2026-10-26/tools/repoint.py new file mode 100644 index 00000000..f5907fd7 --- /dev/null +++ b/documentation/audits/release-2026-10-26/tools/repoint.py @@ -0,0 +1,85 @@ +#!/usr/bin/env python3 +"""repoint.py drill|live|status — (release 2026-10-26 copy: TARGET=9202, Homepage family_gate expected) point the box's controller at the drill catalog, or put the saved +controller.yaml back byte for byte. + +The drill repo on Gitea is PRIVATE (R-992) and the box holds no catalog credential, so the drill is MIRRORED onto the +box: a bare git repository in the controller's own data volume, loaded from a git bundle, and +`git.repo_url: file:///opt/docker/felhom-controller/drill-catalog.git` — the method of +`audits/burndown-3-2026-10-17/tools/repoint.py`. The first version of this file set repo_url to the Gitea URL and +believed the repo public; the box's clone then failed („could not read Username", 21:18:11) and the controller ran on +the templates it already had — the first drill pass proved nothing (README, „Drill fixes"). + +`drill` ends with a POSITIVE check, not the absence of an error: the box's catalog cache must be at the drill's main +commit, and every `EXPECT` line must be in the box's copy of that template. Prints repo_url, never a token.""" +import os, subprocess, sys, time +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools")) +os.environ.setdefault("TARGET", "9202") +import box_walk as w + +DRILL = "/mnt/5_hdd/felhom.eu/drill/app-catalog-drill" +VOL = "/var/lib/docker/volumes/felhom-controller-data/_data" # guest path of the controller's data volume +CVOL = "/opt/docker/felhom-controller" # the same, inside the controller container +SAVE = f"{VOL}/controller.yaml.pre-release1026-drill" +MIRROR_URL = f"file://{CVOL}/drill-catalog.git" +EXPECT = [("n8n/docker-compose.yml", "N8N_DIAGNOSTICS_ENABLED=false"), ("opengist/docker-compose.yml", "256M"), ("homepage/.felhom.yml", "family_gate: true")] + + +def status(): + print(w.guest(f"""grep -n 'repo_url' {VOL}/controller.yaml; ls -l {SAVE} 2>&1 | tail -1 +docker exec felhom-controller sh -c 'git -C {CVOL}/drill-catalog.git log -1 --format="mirror main %h" main 2>&1; + git -C {CVOL}/data/catalog-cache log -1 --format="cache HEAD %h" 2>&1; git -C {CVOL}/data/catalog-cache remote get-url origin'""")) + + +def mirror(): + b = f"/tmp/drill-rel1026-{os.getpid()}.bundle" + subprocess.run(["git", "-C", DRILL, "bundle", "create", b, "main"], check=True, capture_output=True) + subprocess.run(["scp", "-q", "-o", "StrictHostKeyChecking=accept-new", b, f"{w._T['ssh']}:{b}"], check=True) + subprocess.run(w.ssh_args() + [f"pct push {w.GUEST} {b} {VOL}/{os.path.basename(b)} && rm -f {b}"], check=True) + print(w.guest(f"""set -e +docker exec felhom-controller sh -c 'test -d {CVOL}/drill-catalog.git || git init -q --bare -b main {CVOL}/drill-catalog.git +git -C {CVOL}/drill-catalog.git fetch -q {CVOL}/{os.path.basename(b)} +main:main && rm -f {CVOL}/{os.path.basename(b)} +git -C {CVOL}/drill-catalog.git log -1 --format="mirror main %h" main'""")) + os.remove(b) + + +def restart_sync(): + w.guest("docker restart felhom-controller >/dev/null") + for _ in range(60): + time.sleep(3) + try: + w.login() + break + except SystemExit: + continue + w.sync_rescan() + time.sleep(5) + print(w.guest("docker logs --since 3m felhom-controller 2>&1 | grep -E '\\[sync\\]' | grep -v DEBUG | tail -6 | cut -c1-200")) + + +def drill(): + head = subprocess.run(["git", "-C", DRILL, "rev-parse", "--short=7", "main"], capture_output=True, text=True).stdout.strip() + mirror() + print(w.guest(f"""set -e +test -f {SAVE} || cp -p {VOL}/controller.yaml {SAVE} +sed -i -E 's#^(\\s+repo_url: ).*$#\\1{MIRROR_URL}#' {VOL}/controller.yaml +grep -n 'repo_url' {VOL}/controller.yaml""")) + restart_sync() + cache = w.guest(f"docker exec felhom-controller git -C {CVOL}/data/catalog-cache log -1 --format=%h").strip() + ok = cache.startswith(head) + for f, want in EXPECT: + got = w.guest(f"docker exec felhom-controller grep -c -- '{want}' {CVOL}/data/catalog-cache/templates/{f} || true").strip() + print(f" expect {f} ∋ {want}: {got}") + ok = ok and got not in ("", "0") + print(f"drill main {head}, box cache {cache}: " + ("DRILL-CONFIRMED" if ok else "DRILL-NOT-CONFIRMED")) + sys.exit(0 if ok else 1) + + +def live(): + print(w.guest(f"""set -e +cp -p {SAVE} {VOL}/controller.yaml +cmp {SAVE} {VOL}/controller.yaml && echo RESTORED-IDENTICAL; grep -n 'repo_url' {VOL}/controller.yaml""")) + restart_sync() + status() + + +{"drill": drill, "live": live, "status": status}[sys.argv[1]]() diff --git a/documentation/audits/release-2026-10-26/tools/t1probe.py b/documentation/audits/release-2026-10-26/tools/t1probe.py new file mode 100644 index 00000000..ac8c054c --- /dev/null +++ b/documentation/audits/release-2026-10-26/tools/t1probe.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +"""t1probe.py — controller 0.343.0 on Tester 1 (release brief Part 5): one local unit restore and one kept-data cycle with +a THROWAWAY Tandoor, through the endpoints the pages use. Each restore: the seeded item back, then the undo copy +released after the stable window — read from the controller's own line „ran stably for … — the restore copy is +released" (the positive observable) and the restore-undo folder gone. Removed through the product at the end. + +usage: SC= SWEEP_EV= TARGET=tester-1 python3 t1probe.py""" +import json, os, re, sys, time +TOOLS = os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../catalog-sweep-2026-10-24/tools") +sys.path.insert(0, TOOLS) +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../restore-truth-2026-10-25/tools")) +import sweep as sw # noqa: E402 +import unitprobe2 as up # noqa: E402 + +w, fxm = sw.w, sw.fxm +APP = "tandoor" + + +def wait_release(since, say, label, cap=600): + for _ in range(cap // 15): + time.sleep(15) + line = sw.g(f"docker logs --since {int(since)} felhom-controller 2>&1 | grep -E '{APP}: ran stably for .* restore copy is released' | tail -1").strip() + if line: + undo = sw.g(f"ls -d /mnt/*/felhom-data/backups/restore-undo/{APP} /mnt/felhom-drives/*/felhom-data/backups/restore-undo/{APP} 2>/dev/null").strip() + say(f" [{label}] released: {line[:200]} | undo folder now: {undo or 'gone'}") + return {"line": line, "undo_left": undo} + say(f" [{label}] NO release line within {cap}s") + return {"line": "", "undo_left": "?"} + + +def main(): + d = os.path.join(sw.EV, "_t1probe") + os.makedirs(d, exist_ok=True) + log = open(os.path.join(d, "walk.log"), "a") + + def say(*a): + line = time.strftime("%H:%M:%SZ", time.gmtime()) + " " + " ".join(str(x) for x in a) + print(line, flush=True) + log.write(sw.scrub(line) + "\n") + log.flush() + + w.say = say + fx = fxm.FIXTURES[APP] + sub = getattr(fx, "sub", APP) + rec = {"app": APP} + w.login() + up.w.SC = w.SC + t_start = int(time.time()) + try: + rec["install"] = w.deploy(APP, sub) and w.wait_app(sub, "/", tries=90, delay=5) + tok = fx.seed(w, sub, say) + rec["seed"] = bool(tok) + sw.backup_now(rec, say) + # 1. a plain local unit restore + t1 = time.time() + sw.restore_unit(APP, rec, say) + rec["unit_restore"] = rec.pop("restore") + say(" [1] unit restore: " + rec["unit_restore"].get("status", "")[:300]) + time.sleep(20) + rec["unit_restore_item"] = bool(tok) and bool(fx.verify(w, sub, tok, say)) + say(f" [1] item after the unit restore: {rec['unit_restore_item']}") + rec["unit_restore_release"] = wait_release(t1, say, "1") + # 2. the kept-data cycle: remove keeping the backups, reinstall, restore the earlier install's set-aside unit + sw.remove(APP, keep=True, say=say) + t_re = time.time() + rec["reinstall"] = w.deploy(APP, sub, dict(w.GENERATED.get(APP) or {})) + aside = "" + for _ in range(60): + time.sleep(10) + aside = sw.g(f"docker logs --since {int(t_re)} felhom-controller 2>&1 | grep -E '{APP}: the recovery unit of an earlier install' | cut -c1-300") + if aside.strip(): + break + rec["set_aside_line"] = aside.strip() + say(" [2] set aside: " + (aside.strip()[:260] or "NONE within 10 min")) + html = up.page("/backups/restore") + m = re.search(r'action="/backup/kept/restore".*?name="id" value="([0-9a-f]{16})"', html, re.S) + rec["restore_page_offers"] = bool(m) + if not m: + say(" [2] the restore page offers NO restore of the set-aside unit") + return + t2 = time.time() + head, loc = up.post_form("/backup/kept/restore", {"id": m.group(1), "confirm": "1"}) + say(f" [2] POST /backup/kept/restore -> {head} {loc[:1]}") + up.wait_restore("kept_restore", rec) + say(" [2] kept restore: " + rec["kept_restore"]["status"][:300]) + time.sleep(20) + rec["kept_restore_item"] = bool(tok) and bool(fx.verify(w, sub, tok, say)) + say(f" [2] item after the kept-data restore: {rec['kept_restore_item']}") + rec["kept_restore_release"] = wait_release(t2, say, "2") + finally: + rec["controller_lines"] = sw.scrub(sw.g( + f"docker logs --since {t_start} felhom-controller 2>&1 | grep -E '{APP}' | grep -iE 'set aside|restor|stabl|install id|D47|D48|R-1270|refused|released|KEPT' | cut -c1-300")) + sw.remove(APP, keep=False, say=say) + rec["after_remove"] = sw.leftovers(APP, sub) + rec["set_aside_after_remove"] = sw.g(f"ls -1 /mnt/*/felhom-data/backups/set-aside/{APP} 2>&1 | head -3").strip() + json.dump(rec, open(os.path.join(d, "record.json"), "w"), ensure_ascii=False, indent=1) + + +if __name__ == "__main__": + main() diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 0f071025..e7d04cd7 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -113,7 +113,7 @@ match what the reader sees is how an instrument stops being believed (R-421). It **nothing was proven on the day this line was drawn** and a stopping line that moves a status is a stopping line that lies. -## Install & onboarding — 7 rows (P3 5, P4 2) +## Install & onboarding — 8 rows (P3 6, P4 2) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -123,6 +123,7 @@ stopping line that lies. | **R-1273** | Install & onboarding | P3 | **An install that fails on the image registry's rate limit tells the household only „exit code 1”.** Catalog sweep 2026-10-24, code-server 16:01:37: `lscr.io … toomanyrequests` → the product cleaned up right (R-649) but the event and the card say „Alkalmazás telepítése nem sikerült: Code-Server — exit code 1”; nothing says the download server is busy and a later try works. The same install went through later (`audits/catalog-sweep-2026-10-24/code-server/`). | OPEN | — | Recognise `toomanyrequests` (and a pull timeout) in the deploy error and say „the app's download server is busy — try again in an hour”; a test with the stderr line | CC | | **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC | | **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator | +| **R-1284** | Install & onboarding | P3 | **The installer's preflight never words a refusal: `http_code` printed the status twice („401401"), so the 401 („passphrase REJECTED"), 404 („customer not found") and 409 (R-1167's live-box sentence) branches never matched — every refusal read „unexpected hub status 401401".** Seen live 2026-10-25 on the BYO test VM (attended release run) with installer 1.36.0: a wrong id read `404404`, a wrong passphrase `401401`. R-1167 was CLOSED 2026-10-20 on a publish whose 409 sentence could not show. With `-f`, curl prints `-w` and exits 22 on a 4xx, and the fallback curl printed the code again (line 1833 already worked around it by keeping the last 3 characters). | OPEN — **fixed on main, NOT published:** installer 1.36.1 (`http_code`: one curl without `-f`, never failing); `test_http_code_prints_one_status_on_a_refusal`, seen failing on the old helper (`audits/release-2026-10-26/redproof/installer-http-code-doubled.txt`). | the publish is a DooPlex change (tag + webpage git-sync pins + the webpage sync) — the operator's | publish installer 1.36.1 (cut `installer-v1.36.1`, bump both `--ref` in `manifests/webpage.yaml`, sync the webpage alone); then a preflight with a wrong passphrase reads „passphrase REJECTED (401)"; then close | operator | ## Apps & catalog — 17 rows (P3 7, P4 10) @@ -158,7 +159,7 @@ stopping line that lies. | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC | | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC | -## Backup & restore — 59 rows (P1 1, P2 14, P3 26, P4 18) +## Backup & restore — 60 rows (P1 1, P2 14, P3 26, P4 19) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -221,6 +222,7 @@ stopping line that lies. | **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC | | **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC | | **R-1283** | Backup & restore | P3 | **MeTube's downloads are excluded from every backup — make them `optional` (D44 #25).** Catalog `c50adfc` (R-1019) marks the download folders of qBittorrent, Transmission, NZBGet, SABnzbd and MeTube `excluded` (`app-catalog-felhom.eu/templates/metube/.felhom.yml:33`). Risk: a MeTube video may be gone from its source site, so after a drive failure it cannot be fetched again. Pick: MeTube's `media/metube` becomes `optional` — the second-drive copy carries it, the off-site copy does not; NZBGet and SABnzbd stay excluded (re-downloadable). Cost: second-drive space for MeTube's folder. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/15-media-library.md` §7. | OPEN | — | change the class in the catalog template, run the catalog gates (the download-binds gate and its decoys must accept MeTube as `optional`), check on a Tier-0 box that a second-drive copy carries the folder and the off-site run does not | CC | +| **R-1286** | Backup & restore | P4 | **demo-hp's kimai has been HELD since 2026-10-14 09:06Z („held after a failed restore whose rollback also failed") and no row names it.** Seen 2026-10-25 by the D47 upgrade count (attended release run): kimai is the one app of 12 on demo-hp without an install id, because the start-up step skips a held app; `app.yaml` says deployed, no container exists, boot reconciliation leaves it held, the night update leg has skipped it since at least 2026-10-16 (`audits/burndown-apps-2026-10-16/P1/night-chain-log.txt`). A test box; the hold is the designed fail-closed state (R-379), but 11 days with no record is how a held household app would go unseen too. | OPEN | — | read why the 2026-10-14 restore and its rollback failed (demo-hp's controller log of that day, if kept; the hold record); then clear the hold (`--clear-restore-hold kimai` + a restart) or remove the app through the product; check that a held app raises a standing operator signal, else file that | CC | ## Storage & devices — 14 rows (P3 5, P4 9) @@ -273,7 +275,7 @@ stopping line that lies. | **R-1280** | Security & access | P3 | **A signed operation's window has no maximum length — cap it at 24 h (D44 #22).** The agent (`felhom-agent/internal/authz/verifier.go`) and the root wrapper (`configs/felhom-os-apply`) check `now ∈ [issued_at, expires_at]` with a 2-minute skew on `issued_at`, but neither checks `expires_at − issued_at`; `felhom-opsign` defaults to 30 min (`cmd/felhom-opsign/main.go:57`) and the nonce store keeps a record until its expiry. Risk: a signed job made with a long window by mistake stays usable for that whole time. Pick: the agent and the root wrapper refuse a window longer than 24 h. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/04-control-plane-authorization.md` §2. | OPEN | — | one agent change in both checkers + a test that a 24 h + 1 s window is refused (red first) and a 24 h one is accepted; rides an agent release (agent releases stop for the operator's vouch) | CC | | **R-1281** | Security & access | P2 | **Every app container can reach FileBrowser directly; the only guard is the secret header name — take FileBrowser off the app networks (D44 #23).** FileBrowser trusts its proxy header (`auth.methods.proxy`); every app container can reach `filebrowser:80` without traefik; the guard is that the header NAME is a per-box secret that traefik's forwardAuth strips from client requests (`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`). Risk: an app that learns the name (a hacked app on the box) reads and writes every household file. Pick: FileBrowser joins only the network traefik uses, so no app container reaches it directly. Operator ruling D44 = all picks, 2026-10-25. Note: `architecture/13-threat-model.md` §5. | OPEN | — | a compose/network change in the controller's infra render + a test on the rendered networks; a live check on a Tier-0 box that a forged header from an app container no longer reaches FileBrowser (the 2026-10-18 measurement, now refused) while the dashboard's file manager still signs in; one controller release | CC | -## Box system & updates — 12 rows (P2 1, P3 7, P4 4) +## Box system & updates — 13 rows (P2 1, P3 7, P4 5) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -289,6 +291,7 @@ stopping line that lies. | **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC | | **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC | | **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC | +| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC | ## Monitoring & notifications — 22 rows (P2 1, P3 15, P4 6) diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index adb87295..c47ee17e 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -7,6 +7,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S ## Moved from STATUS on 2026-10-25 (attended release run) +The Monday list, run on 2026-10-25 (attended; results in `REPORT-release-2026-10-26.md`): + +**Monday 2026-10-26, release list (attended).** 1. Controller v0.343.0: the restore and backup fixes (R-1251–R-1253, R-1269–R-1271), R-1250. If not: the P1 stays open. 2. Agent v0.165.0 + your vouch: BYO keeps its SSH (R-1257). 3. Hub v0.170.0: the recovery package in one step (R-1254). 4. Catalog: n8n usage data off; OpenGist 256 MB. + Decided: **D44 = all picks**, **D45 = all picks** (operator, 2026-10-25; R-1265 and R-1266 closed, the four code picks are R-1280–R-1283). | D45 | R-1266 | Seven places where a design document says one thing and the code does another. One line each, with a pick: `documentation/audits/review-round-2-2026-10-24/D44-D45-picks.md`. Answer „all picks" or name the ones you disagree with. | **Yes — the text follows the code** (each is how the system works today; you read every changed sentence) | the code follows the text: six code changes, one of them to the box's door for you | the documents keep saying what is not true | — | diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index fe751163..bd11db83 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,3 +1,13 @@ +## 2026-10-25 — installer 1.36.1: the preflight words a refusal again (committed, NOT published) + +- **`felhom-host-install.sh` `http_code`** printed the status twice on a 4xx (`curl -f` prints `-w` and exits 22, and the + `|| curl …` fallback printed it again: „401401"), so the preflight's 401 („passphrase REJECTED"), 404 („customer not + found") and 409 (R-1167's „this customer already has a live box") branches never matched — every refusal read + „unexpected hub status 401401". Seen live on the BYO test VM (attended release run 2026-10-25). Now one curl without + `-f`, `|| true`. `test_hostinstall.py`: `test_http_code_prints_one_status_on_a_refusal` (seen failing on the old helper, + `documentation/audits/release-2026-10-26/redproof/installer-http-code-doubled.txt`). **Not published:** the public + script follows the `installer-v*` tag; a publish needs the tag and the webpage sync (felhom.eu R-1284). + ## 2026-10-24 — `ep0-iperf3/`: the tunnel-only line-speed test server on ep0 (R-1247; operator ruling 2026-10-24 07:26) - **`ep0-iperf3/felhom-iperf3.service`** (new, installed by hand on ep0 2026-10-24): Debian's `iperf3 --server` bound to diff --git a/scripts/felhom-host-install.sh b/scripts/felhom-host-install.sh index 20c424cd..9b0aef33 100644 --- a/scripts/felhom-host-install.sh +++ b/scripts/felhom-host-install.sh @@ -194,7 +194,7 @@ set -euo pipefail -SCRIPT_VERSION="1.36.0" # the SINGLE version source (F-1): -h and the run banners follow it. +SCRIPT_VERSION="1.36.1" # the SINGLE version source (F-1): -h and the run banners follow it. # The hub used to carry a copy for its Setup tab; R-94 DELETED it # (2026-08-02) because the hub cannot know which version a box runs — # the Setup command fetches this script at run time. scripts/ @@ -453,8 +453,10 @@ _state_get() { STATE_FILE="$STATE_FILE" python3 -c "import json,os,sys;d=json.load(open(os.environ['STATE_FILE']));print(d.get(sys.argv[1],''))" "$1" 2>/dev/null } -http_code() { # GET, prints status code only (read-only preflight) - curl -fsS -o /dev/null -w '%{http_code}' "$@" 2>/dev/null || curl -sS -o /dev/null -w '%{http_code}' "$@" 2>/dev/null +http_code() { # GET, prints ONE status code (read-only preflight); 000 when nothing answered + # No `-f`: with it curl prints the code AND exits 22 on a 4xx, and the old `|| curl …` fallback printed it a second + # time („401401") — the preflight's 401/404/409 branches never matched (test_http_code_prints_one_status_on_a_refusal). + curl -sS -o /dev/null -w '%{http_code}' "$@" 2>/dev/null || true } #------------------------------------------------------------------------------- diff --git a/scripts/test_hostinstall.py b/scripts/test_hostinstall.py index e9cf9ad9..f635d2bf 100644 --- a/scripts/test_hostinstall.py +++ b/scripts/test_hostinstall.py @@ -700,6 +700,32 @@ def test_internal_place_runs_after_the_bundle_and_before_provision(): assert -1 < i_bundle < i_place < i_prov, (i_bundle, i_place, i_prov) +# ── http_code prints ONE status (found 2026-10-25, attended release run) ───────────────────────────────────────────── +# With `-f`, curl prints -w's code AND exits 22 on a 4xx, so the old `curl -f … || curl …` printed the code twice +# („401401"): the preflight's 401 / 404 / 409 branches never matched and every refusal read „unexpected hub status". +def _http_code_with(status): + sb = Sandbox() + try: + # a curl stub that answers like the real one: -w prints the status; -f makes a 4xx/5xx exit 22 + body = ('f=0; for a in "$@"; do case "$a" in --*) ;; -*f*) f=1;; esac; done\n' + 'printf "%s" STATUS\n[ "$f" = 1 ] && [ STATUS -ge 400 ] && exit 22\nexit 0').replace("STATUS", status) + sb.stub("curl", body) + rc, out = sb.run(func("http_code") + 'c=$(http_code https://hub.invalid/api/v1/config/x -H "X: y"); printf "[%s]" "$c"') + return out + finally: + sb.close() + + +def test_http_code_prints_one_status_on_a_refusal(): + for status in ("401", "404", "409"): + out = _http_code_with(status) + assert out.endswith("[%s]" % status), "http_code printed %r for a %s (the preflight's case never matches)" % (out, status) + + +def test_http_code_prints_the_status_on_success(): + assert _http_code_with("200").endswith("[200]") + + def main(): tests = [(n, f) for n, f in sorted(globals().items()) if n.startswith("test_") and callable(f)] fails = 0