diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 69242973..81276469 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -846,6 +846,19 @@ under the backup manager's lock. One `[WARN]` log line and one operator-only eve restore date, size; hub `19afefab`) per delete. Controller `e3fce09`, tests `TestD62_*`. **D63** (`09` §3 decision 282): H4 and H5's failed stop ship only with a live proof, staged with a fault switch that exists only in a `-tags faultinject` test build. +**`[FACT]` 2026-10-29, Tester 1, test build `0.347.0-rc.1-d63fault` (`audits/d63-2026-10-29/`):** (H4, 08:10 UTC) a +throwaway Radicale ran the true 3.8.0 (digest `0d60d544…`) while its newest off-site snapshot held 3.8.1's data; the full +restore said „it comes back at the snapshot's version", the fault failed the definition write after the compose was +written, the log then read „the live definition is written back before the start" and only then „Starting stack"; the app +ran 3.8.0 again, no hold, and an event made before the snapshot and one made after it both read back. (H5, 08:18 UTC) a +created-only container held the data volume („Failed to remove volume … (not written into)"); the restore's own stop +passed, the put-back's stop failed — „nothing is put back; the copy stays at …; holding the app"; the app page showed the +hold sentence; on the restore page the held copy's „Törlés most" was off with „Nem törölhető: az alkalmazás le van +állítva, és ez a másolat az út vissza." (D62, 08:15 UTC) a copy kept because a second restore began inside the first one's +stable-start window was listed with „visszaállítás: 2026. 10. 29. 09:11 · 37,3 KB · automatikusan törlődik: 2026. 11. +28."; a wrong typed name → 400, the right one → 200, the copy gone, one `[WARN]` line; the event got HTTP 400 from hub +0.174.1, which does not know the type yet (hub 0.175.0 does). The release image carries no fault switch (`grep -c "FAULT +INJECTED"` = 0 on `0.347.0-rc.1`, 2 on the test image). **`[FACT]` 2026-10-29 night, scratch 9202 (`audits/night-2026-10-28/r1256-restore/h5-live/`):** vikunja's first volume held by a created-only container, the household's restore pressed through the page: „Failed to remove volume vikunja_vikunja_data before the restore … (not written into)", the restore failed, the put-back poured the live copy of diff --git a/documentation/audits/d63-2026-10-29/README.md b/documentation/audits/d63-2026-10-29/README.md new file mode 100644 index 00000000..3020cdde --- /dev/null +++ b/documentation/audits/d63-2026-10-29/README.md @@ -0,0 +1,18 @@ +# D63 + D62 live proofs — Tester 1, 2026-10-29 (attended) + +Test build `0.347.0-rc.1-d63fault` = controller `e3fce09` built with `-tags faultinject` on top of the local release +candidate `0.347.0-rc.1` (never pushed; removed from the box after). The release image carries no fault switch +(`grep -c "FAULT INJECTED"`: 0 on `0.347.0-rc.1`, 2 on the test image). Driver: `d63_live.py` (box_walk helpers). + +| File | What it shows | +|---|---| +| `run-setup.log`, `run-setup2.log` | first setup tries. A hand-edited compose at "3.8.0" kept 3.8.1's digest, and the backup refused the app (R-645: it ran a version other than its pin) — so no unit. Both reverted. These runs pushed one `backup_run_failures` event (Radicale not protected off-site) — caused by this setup. | +| `run-setup3.log` | the real setup: Radicale backed up at its catalog 3.8.1 (local unit + off-site snapshot `78002a51`). | +| `run-pull.log` | the off-site snapshot pulled to the restore scratch (the wizard's two steps). | +| `run-h4.log`, `run-h4b.log`, `run-h4c.log` | failed seeds: 401 from Radicale. Cause: the harness, not the app — the generated login lives only in the deploying process, a new process sent an empty password. NOT 3.8.0 (an early guess, withdrawn). | +| `run-h4d.log` | **H4 proven**: live 3.8.0 (true digest `0d60d544…`), snapshot 3.8.1; fault after the compose write; the live definition written back before the start; app on 3.8.0, no hold, T1 + T2 read back. | +| `run-d62.log` | first D62 try: the script typed the wrong name (a parsing bug in the script) → 400; its "copy gone" line came from a finder that could not see `/mnt/sys_drive` (wrong). | +| `run-d62b.log` | **D62 proven**: the list line with the date and „automatikusan törlődik", the warning text, wrong name 400, right name 200, the copy gone, the log line; the hub event 400 (hub 0.174.1 does not know the type). | +| `run-h5.log` | **H5's failed stop proven**: busy volume not written into; the put-back's stop failed; nothing put back; copy kept; app held; the held copy's button off. Its operator mail text is wrong for this case → R-1340. | +| `run-teardown.log`, `tester1-after.txt` | the app removed; controller back on 0.346.1; test image removed; 15 anonymous volumes (R-1267's PrivateBin class, made by this session's five backup runs) removed by name — 268 volumes, as before. The Radicale **restore hold record stays** (clearing a hold is the operator's). | +| `controller-full.log` | the controller's whole log 07:50 → teardown. | diff --git a/documentation/audits/d63-2026-10-29/controller-full.log b/documentation/audits/d63-2026-10-29/controller-full.log new file mode 100644 index 00000000..691e993d --- /dev/null +++ b/documentation/audits/d63-2026-10-29/controller-full.log @@ -0,0 +1,1310 @@ +2026/10/29 07:52:58 [INFO] local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible +2026/10/29 07:52:58 [INFO] local-api: mount mp0 → /var/lib/felhom (storage=local-lvm, class=, backup=true) +2026/10/29 07:52:58 [INFO] local-api: mount mp9 → /etc/felhom-bootstrap (storage=/var/lib/felhom-agent/guests/9201/bootstrap, class=, backup=false) +2026/10/29 07:52:58 [INFO] local-api: mount mp8 → /mnt/felhom-drives (storage=/mnt/felhom-drives, class=, backup=false) +2026/10/29 07:52:58 [INFO] felhom-controller 0.347.0-rc.1 starting (customer: tester-1, domain: enkicsifelhom.hu) +2026/10/29 07:52:58 [INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json +2026/10/29 07:52:58 [INFO] Encryption key loaded from /opt/docker/felhom-controller/data/encryption.key +2026/10/29 07:52:59 [INFO] [stacks] Using compose command: docker compose +2026/10/29 07:52:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (3 deployed, 65 available) +2026/10/29 07:52:59 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 07:52:59 [INFO] [stacks] InjectMissingFields: processed 3 stacks +2026/10/29 07:52:59 [INFO] [stacks] Encryption migration: no stacks needed migration +2026/10/29 07:52:59 [INFO] [quiesce] loop started (poll 5m0s, max-quiesce 30m0s) +2026/10/29 07:52:59 [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded (state ambiguous — legacy boot behaviour retained) +2026/10/29 07:52:59 [INFO] [stacks] installed-images backfill: 0 app(s) recorded, 3 already had a record, 0 left unrecorded (could not be observed completely — unknown, which renders nothing) +2026/10/29 07:52:59 [INFO] [stacks] pin adoption: 0 pinned, 3 already pinned, 0 left unpinned (0 not completely observed, 0 running something the template no longer offers) +2026/10/29 07:52:59 [INFO] [stacks] applied-meta backfill (R-646): recorded 0 []; skipped 0 [] — not current with the catalog, their pinned version's .felhom.yml is no longer on the box +2026/10/29 07:52:59 [INFO] [sync] Starting catalog sync (repo: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git, interval: 15m0s) +2026/10/29 07:52:59 [INFO] [sync] Starting catalog sync +2026/10/29 07:52:59 [INFO] Metrics store opened at /opt/docker/felhom-controller/data/metrics.db +2026/10/29 07:52:59 [INFO] Metrics collector started (60s interval) +2026/10/29 07:52:59 [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (branch: main) +2026/10/29 07:52:59 [INFO] [tier2] start-up: no leftover second-drive copy container +2026/10/29 07:52:59 [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false) +2026/10/29 07:52:59 [INFO] [backup] app-data run record wired: /opt/docker/felhom-controller/data/appdata-run.json (an earlier cut still shown: false) +2026/10/29 07:52:59 [INFO] Notifier enabled (hub: https://hub.felhom.eu) +2026/10/29 07:52:59 [INFO] [upwatch] the host's weekly line test: 94.0 Mbit/s (measured 2026-10-27T12:33:30Z) +2026/10/29 07:52:59 [INFO] [upwatch] upload and CPU watch started (line 94 Mbit/s, qBittorrent share 50 %) +2026/10/29 07:52:59 [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: status-refresh (every 10s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: docker-socket-watch (every 15s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: docker-socket-users (every 5m0s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: health-probes (every 10s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: system-health (every 5m0s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: after-unclean-boot (every 30s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: hold-report (every 1m0s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: ring-spill (every 30s) +2026/10/29 07:52:59 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply +2026/10/29 07:52:59 [INFO] [scheduler] Daily job db-dump scheduled for 2026-10-30 02:30 CET +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: conversion-copy-release (every 1h0m0s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: restore-copy-sweep (every 6h0m0s) +2026/10/29 07:52:59 [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-10-30 03:30 CET +2026/10/29 07:52:59 [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-30 04:15 CET +2026/10/29 07:52:59 [INFO] [catch-up] controller start: no backup leg missed its last scheduled time — nothing to make up +2026/10/29 07:52:59 [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-10-30 05:10 CET +2026/10/29 07:52:59 [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-10-30 06:00 CET +2026/10/29 07:52:59 [INFO] [scheduler] Daily job offsite-proof scheduled for 2026-10-30 05:30 CET +2026/10/29 07:52:59 [INFO] [scheduler] Daily job metrics-prune scheduled for 2026-10-30 04:00 CET +2026/10/29 07:52:59 [INFO] [scheduler] Daily job fill-watch scheduled for 2026-10-30 03:30 CET +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: fill-watch-interval (every 10m0s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: hub-report (every 15m0s) +2026/10/29 07:52:59 [INFO] Hub reporting enabled (every 15m0s to https://hub.felhom.eu) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s) +2026/10/29 07:52:59 [INFO] ========== Startup Self-Test ========== +2026/10/29 07:52:59 [INFO] [PASS] Docker socket: reachable (v29.8.2) +2026/10/29 07:52:59 [INFO] [PASS] Stacks directory: /opt/docker/stacks +2026/10/29 07:52:59 [INFO] [PASS] Data directory: /opt/docker/felhom-controller/data (writable) +2026/10/29 07:52:59 [INFO] [PASS] System data path: /mnt/sys_drive +2026/10/29 07:52:59 [INFO] [PASS] Storage paths: 1 connected, 0 disconnected +2026/10/29 07:52:59 [INFO] [PASS] Git catalog: 65 app definitions found +2026/10/29 07:52:59 [INFO] [PASS] Hub connectivity: https://hub.felhom.eu reachable (HTTP 200) +2026/10/29 07:52:59 [INFO] [PASS] Metrics DB: 41.7 MB +2026/10/29 07:52:59 [INFO] ======================================== +2026/10/29 07:52:59 [INFO] Self-test complete: 8 passed, 0 warnings, 0 failed +2026/10/29 07:52:59 [INFO] [scheduler] Starting scheduler with 25 jobs +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: geo-verify (every 6h0m0s) +2026/10/29 07:52:59 [INFO] Geo-restriction support enabled (CF API token configured) +2026/10/29 07:52:59 [INFO] [report] hub wait channel active (hold ≤240s) +2026/10/29 07:52:59 [INFO] [backup] Found 3 DB dump files across drives +2026/10/29 07:52:59 [INFO] [infra] connected felhom-controller to traefik-public +2026/10/29 07:52:59 [INFO] [web] sessions: restored 28 dashboard session(s) from disk (dropped 0 expired, 0 malformed) +2026/10/29 07:52:59 [INFO] [web] Auth: using password from settings.json +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s) +2026/10/29 07:52:59 [INFO] [scheduler] Registered periodic job: agent-channel-health (every 1m0s) +2026/10/29 07:52:59 [INFO] Web UI listening on :8080 +2026/10/29 07:52:59 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_a not mounted — skipping userdata skeleton +2026/10/29 07:52:59 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_b not mounted — skipping userdata skeleton +2026/10/29 07:52:59 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_c not mounted — skipping userdata skeleton +2026/10/29 07:52:59 [INFO] [web] FileBrowser: drive /mnt/felhom-drives/media_d not mounted — skipping userdata skeleton +2026/10/29 07:52:59 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 07:52:59 [INFO] [backup] Discovered 2 databases +2026/10/29 07:52:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 07:52:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 07:52:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 07:52:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 07:52:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 07:52:59 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 07:52:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 07:52:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 07:52:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 07:52:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 07:52:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 07:52:59 [INFO] [backup] Discovered app data: 3 apps +2026/10/29 07:52:59 [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +2026/10/29 07:52:59 [INFO] [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/10/29 07:52:59 [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/29 07:52:59 [INFO] [backup] Backup status cache refreshed +2026/10/29 07:52:59 [INFO] [sync] Catalog sync complete +2026/10/29 07:52:59 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +2026/10/29 07:52:59 [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (5 storage path(s)) +2026/10/29 07:52:59 [INFO] [infra] connected felhom-controller to felhom-filebrowser (the FileBrowser default-login check, R-1126) +2026/10/29 07:52:59 [INFO] [monitor] Health check: status=ok +2026/10/29 07:53:04 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 07:53:04 [INFO] [report] Building system report +2026/10/29 07:53:04 [INFO] Event pushed: controller_started (info) [hu-only] — Controller elindult (0.347.0-rc.1) +2026/10/29 07:53:05 [INFO] [monitor] Health check: status=ok +2026/10/29 07:53:05 [INFO] [report] Hub report pushed successfully (5275 bytes) +2026/10/29 07:53:05 [INFO] [settings] Settings saved +2026/10/29 07:53:05 [INFO] Startup hub report sent +2026/10/29 07:53:09 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.346.1 (we are 0.347.0-rc.1), no managed update running +2026/10/29 07:53:09 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 07:53:09 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 07:53:09 [INFO] Health probes: 3 ok (of 3 probed) +2026/10/29 07:53:14 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 07:53:14 [INFO] [bootrecon] boot window: fleet settled after 10s (3 identical samples 5s apart) — sweeping +2026/10/29 07:53:14 [INFO] [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start) +2026/10/29 07:53:19 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 07:53:29 [INFO] [deadapp] boot grace 1m30s: the host's last boot was clean (R-856) +2026/10/29 07:53:29 [INFO] [aftercrash] a normal (or unknown) boot — no app repair after an unclean boot (R-1007) +2026/10/29 07:53:29 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 07:53:39 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 07:53:49 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 07:53:55 [INFO] [web] Login from 192.168.0.180 +2026/10/29 07:53:58 [INFO] [api] Deploy requested for stack: radicale +2026/10/29 07:53:58 [INFO] [stacks] Memory check: total=4096MB, reserved=384MB, usable=3712MB, committed_used=840MB, new_req=40MB, remaining=2832MB +2026/10/29 07:53:58 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 1 encrypted, 1 sensitive fields +2026/10/29 07:53:58 [INFO] [stacks] Deploying stack radicale with 4 env vars: [DOMAIN, SUBDOMAIN, RADICALE_USER, RADICALE_PASSWORD] +2026/10/29 07:53:58 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 07:53:58 [INFO] Event pushed: app_deploy_started (info) [hu-only] — Alkalmazás telepítése elindult: Radicale +2026/10/29 07:53:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 07:53:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 07:53:59 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 07:53:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 07:54:00 [INFO] [scheduler] Job agent-channel-health completed (took 476ms) +2026/10/29 07:54:00 [INFO] [report] Building system report +2026/10/29 07:54:01 [INFO] [monitor] Health check: status=ok +2026/10/29 07:54:02 [INFO] [report] Hub report pushed successfully (5355 bytes) +2026/10/29 07:54:02 [INFO] [settings] Settings saved +2026/10/29 07:54:05 [INFO] [stacks] Stack radicale deployed successfully (took 7.1s) +2026/10/29 07:54:05 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 1 encrypted, 1 sensitive fields +2026/10/29 07:54:05 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 07:54:06 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/29 07:54:06 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 07:54:06 [INFO] [stacks] installed-images radicale: recorded 1 service(s) (radicale=ghcr.io/kozea/radicale:3.8.1 (sha256:54d9406cd30f…)) +2026/10/29 07:54:06 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/29 07:54:06 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 07:54:06 [INFO] [stacks] pin radicale: radicale=ghcr.io/kozea/radicale:3.8.1 +2026/10/29 07:54:06 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:54:06 [INFO] [backup] radicale: off-site copy switched ON at install (decision 50) +2026/10/29 07:54:06 [INFO] [settings] Settings saved +2026/10/29 07:54:06 [INFO] Event pushed: app_deployed (info) [hu-only] — Alkalmazás telepítve: Radicale +2026/10/29 07:54:09 [INFO] [stacks] Stack radicale post-start status: +2026/10/29 07:54:09 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting) +2026/10/29 07:54:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:54:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:54:28 [INFO] [api] restart requested for stack: radicale +2026/10/29 07:54:28 [INFO] [stacks] Restarting stack: radicale +2026/10/29 07:54:29 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/10/29 07:54:29 [INFO] [stacks] Stack radicale: started again: radicale 2026-10-29T07:54:05.7001809Z → 2026-10-29T07:54:29.086891581Z +2026/10/29 07:54:29 [INFO] [stacks] Stack radicale restarted successfully (took 0.9s) +2026/10/29 07:54:29 [WARN] Health probe radicale: API GET :5232/.web/ → Get "http://radicale:5232/.web/": dial tcp 172.18.0.7:5232: connect: connection refused +2026/10/29 07:54:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:54:29 [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/10/29 07:54:29 [INFO] [deadapp] the agent has no OOM counter route (R-528) — Docker's OOM flag only; asked again in 10m0s +2026/10/29 07:54:29 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/29 07:54:29 [INFO] [stacks] installed-images radicale: recorded 1 service(s) (radicale=ghcr.io/kozea/radicale:3.8.0 (sha256:54d9406cd30f…)) +2026/10/29 07:54:29 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 07:54:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:54:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:54:32 [INFO] [stacks] Stack radicale post-start status: +2026/10/29 07:54:32 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.0@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting) +2026/10/29 07:54:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:54:48 [INFO] [settings] Settings saved +2026/10/29 07:54:48 [INFO] [offbox] backup run started (10 app(s) toggled) +2026/10/29 07:54:48 [INFO] [settings] Settings saved +2026/10/29 07:54:48 [INFO] [backup] Starting database dump run +2026/10/29 07:54:48 [INFO] [backup] Discovered 2 databases +2026/10/29 07:54:48 [INFO] [backup] Discovered 2 database(s): paperless-postgres(postgres), bookstack-db(mariadb) +2026/10/29 07:54:48 [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (474.6 KB, 392ms, 72 tables) +2026/10/29 07:54:48 [INFO] [settings] Settings saved +2026/10/29 07:54:49 [INFO] [backup] DB dump: bookstack-db → bookstack-mariadb.sql (57.9 KB, 309ms, 41 tables) +2026/10/29 07:54:49 [INFO] [settings] Settings saved +2026/10/29 07:54:49 [INFO] [backup] Stopping bookstack for safe volume dump +2026/10/29 07:54:49 [INFO] [stacks] Stopping stack: bookstack +2026/10/29 07:54:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:54:49 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 07:54:50 [INFO] [report] Building system report +2026/10/29 07:54:51 [INFO] [monitor] Health check: status=ok +2026/10/29 07:54:51 [INFO] [report] Hub report pushed successfully (6058 bytes) +2026/10/29 07:54:51 [INFO] [settings] Settings saved +2026/10/29 07:54:53 [INFO] [stacks] Stack bookstack stopped successfully (took 4.4s) +2026/10/29 07:54:53 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/29 07:54:54 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_config → 3.4 MB +2026/10/29 07:54:54 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.4 MB +2026/10/29 07:54:54 [INFO] [backup] Restarting bookstack after volume dump +2026/10/29 07:54:54 [INFO] [stacks] Starting stack: bookstack +2026/10/29 07:54:59 [INFO] [scheduler] Running job: stack-scan +2026/10/29 07:54:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 07:54:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 07:54:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:54:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 07:54:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/29 07:54:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:54:59 [INFO] [scheduler] Job stack-scan completed (took 226ms) +2026/10/29 07:55:00 [INFO] [scheduler] Job agent-channel-health completed (took 473ms) +2026/10/29 07:55:00 [INFO] [stacks] Stack bookstack started successfully (took 6.1s) +2026/10/29 07:55:01 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:01 [INFO] [backup] Stopping paperless-ngx for safe volume dump +2026/10/29 07:55:01 [INFO] [stacks] Stopping stack: paperless-ngx +2026/10/29 07:55:04 [INFO] [stacks] Stack bookstack post-start status: +2026/10/29 07:55:04 [INFO] [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.2@sha256:2fdeeb1a2028ce7cb1274696237e32b9f849e16b6d77314d08c78356a6016fca running Up 3 seconds (health: starting) +2026/10/29 07:55:04 [INFO] [stacks] bookstack-db mariadb:12.3@sha256:d46e102fa8b9b72bfcdeb56f9dd538742092119dc8b11f31fe662605cd234cd9 running Up 9 seconds (healthy) +2026/10/29 07:55:08 [INFO] [stacks] Stack paperless-ngx stopped successfully (took 7.4s) +2026/10/29 07:55:08 [INFO] [stacks] Status refresh: 7 containers across 68 stacks +2026/10/29 07:55:09 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 11.5 MB +2026/10/29 07:55:09 [INFO] [stacks] Status refresh: 7 containers across 68 stacks +2026/10/29 07:55:10 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 69.6 MB +2026/10/29 07:55:10 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_redis_data → 7.2 MB +2026/10/29 07:55:10 [INFO] [backup] Restarting paperless-ngx after volume dump +2026/10/29 07:55:10 [INFO] [stacks] Starting stack: paperless-ngx +2026/10/29 07:55:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:21 [INFO] [stacks] Stack paperless-ngx started successfully (took 10.9s) +2026/10/29 07:55:21 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:21 [INFO] [backup] Stopping privatebin for safe volume dump +2026/10/29 07:55:21 [INFO] [stacks] Stopping stack: privatebin +2026/10/29 07:55:22 [INFO] [stacks] Stack privatebin stopped successfully (took 0.4s) +2026/10/29 07:55:22 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 07:55:22 [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 8.5 KB +2026/10/29 07:55:22 [INFO] [backup] Restarting privatebin after volume dump +2026/10/29 07:55:22 [INFO] [stacks] Starting stack: privatebin +2026/10/29 07:55:23 [INFO] [stacks] Stack privatebin started successfully (took 0.6s) +2026/10/29 07:55:23 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:23 [WARN] [backup] Skipping volume dump for radicale — it is not running its pinned version (radicale runs ghcr.io/kozea/radicale:3.8.0, pinned ghcr.io/kozea/radicale:3.8.1); the last good backup is kept (R-645) +2026/10/29 07:55:23 [INFO] [backup] App-data backup completed: 2 databases (532.5 KB total), 3 volume dump(s) (35.114s) +2026/10/29 07:55:23 [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +2026/10/29 07:55:23 [INFO] [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/10/29 07:55:23 [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/29 07:55:23 [WARN] [backup] Recovery unit NOT captured for radicale — it is not running its pinned version (radicale runs ghcr.io/kozea/radicale:3.8.0, pinned ghcr.io/kozea/radicale:3.8.1); the last good unit is kept (R-645) +2026/10/29 07:55:23 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:23 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:23 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:23 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:23 [ERROR] [backup] radicale is RUNNING but its newest local copy (/mnt/sys_drive/felhom-data/backups/primary/radicale) holds NO database dump and NO volume tar — it is not protected (Part D) +2026/10/29 07:55:23 [INFO] [offbox] pre-push dump leg completed in 35.29s — snapshot pair is coherent +2026/10/29 07:55:23 [INFO] Event pushed: backup_run_failures (error) [hu-only] — Backup holds NO DATA: radicale is running, but its newest local copy carries no database dump and no volume tar — the app is not protected on that tier. Check for a hold on a running app (R-704's shape) or a skipped dump. +2026/10/29 07:55:25 [INFO] [stacks] Stack paperless-ngx post-start status: +2026/10/29 07:55:25 [INFO] [stacks] paperless-postgres postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 running Up 13 seconds (healthy) +2026/10/29 07:55:25 [INFO] [stacks] paperless-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 13 seconds (healthy) +2026/10/29 07:55:25 [INFO] [stacks] paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15@sha256:6c86cad803970ea782683a8e80e7403444c5bf3cf70de63b4d3c8e87500db92f running Up 3 seconds (health: starting) +2026/10/29 07:55:26 [INFO] [stacks] Stack privatebin post-start status: +2026/10/29 07:55:26 [INFO] [stacks] privatebin privatebin/pdo:2.1.0@sha256:3400b0a4714d62161727a710c82a3e319cfea364ac6e591aed653f5bb6bb8055 running Up 3 seconds (health: starting) +2026/10/29 07:55:27 [WARN] [offbox] jellyfin: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 07:55:27 [WARN] [offbox] plex: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 07:55:27 [WARN] [offbox] radarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 07:55:27 [WARN] [offbox] sonarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 07:55:27 [WARN] [offbox] transmission: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 07:55:27 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 07:55:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:29 [INFO] Health probes: 2 ok (of 2 probed) +2026/10/29 07:55:30 [INFO] [settings] Settings saved +2026/10/29 07:55:30 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:30 [INFO] [offbox] backed up bookstack (/mnt/sys_drive/felhom-data/backups/primary/bookstack, 0 mandatory path(s)) +2026/10/29 07:55:30 [WARN] [offbox] jellyfin: no recovery unit found on any connected drive — skipping +2026/10/29 07:55:34 [INFO] [settings] Settings saved +2026/10/29 07:55:34 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:34 [INFO] [offbox] backed up paperless-ngx (/mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx, 1 mandatory path(s)) +2026/10/29 07:55:34 [WARN] [offbox] plex: no recovery unit found on any connected drive — skipping +2026/10/29 07:55:36 [INFO] [settings] Settings saved +2026/10/29 07:55:36 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:36 [INFO] [offbox] backed up privatebin (/mnt/sys_drive/felhom-data/backups/primary/privatebin, 0 mandatory path(s)) +2026/10/29 07:55:36 [WARN] [offbox] radarr: no recovery unit found on any connected drive — skipping +2026/10/29 07:55:36 [WARN] [offbox] sonarr: no recovery unit found on any connected drive — skipping +2026/10/29 07:55:36 [WARN] [offbox] transmission: no recovery unit found on any connected drive — skipping +2026/10/29 07:55:36 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 07:55:38 [INFO] [settings] Settings saved +2026/10/29 07:55:38 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:38 [INFO] [offbox] backed up prowlarr (/mnt/sys_drive/felhom-data/backups/primary/prowlarr, 0 mandatory path(s)) +2026/10/29 07:55:38 [WARN] [offbox] radicale: no recovery unit found on any connected drive — skipping +2026/10/29 07:55:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:40 [INFO] [offbox] retention skipped (after-run): no clean-up window now (not due (last window 2026-10-26T03:15:52Z)) — nothing deleted (decision 68) +2026/10/29 07:55:40 [INFO] [settings] Settings saved +2026/10/29 07:55:40 [INFO] [upwatch] off-site push measured: 2624728 bytes in 17 s, best 10 s 2.1 Mbit/s (counts=false); line now at least 94 Mbit/s +2026/10/29 07:55:44 [INFO] [settings] Settings saved +2026/10/29 07:55:44 [INFO] [settings] Settings saved +2026/10/29 07:55:44 [INFO] [offbox] backup OK: 4 app(s) backed up, 6 skipped (no unit), 73 snapshot(s), 52s +2026/10/29 07:55:44 [INFO] [offbox] manual run progress reporting ended after 56s +2026/10/29 07:55:44 [INFO] Event pushed: backup_run_failures (error) [hu-only] — Offsite backup INCOMPLETE: 1 app(s) had a directory they declare MANDATORY missing from the snapshot — radicale ((a teljes alkalmazás — nincs helyi mentési egysége)). The run itself succeeded and what it captured is real, but these apps are NOT fully protected off-site. Check that the declared path exists on disk and that the app's data root resolves where the backup looks (R-203). +2026/10/29 07:55:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 07:55:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 07:55:59 [INFO] [stacks] controller image retention: skipped — running version "0.347.0-rc.1" is not a release (or no repository) +2026/10/29 07:55:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:55:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 07:56:00 [INFO] [scheduler] Job agent-channel-health completed (took 478ms) +2026/10/29 07:56:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:56:09 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 07:56:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:56:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:56:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:56:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:56:59 [INFO] [scheduler] Running job: stack-scan +2026/10/29 07:56:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 07:56:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 07:56:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:56:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/29 07:56:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 07:56:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:56:59 [INFO] [scheduler] Job stack-scan completed (took 152ms) +2026/10/29 07:57:00 [INFO] [scheduler] Job agent-channel-health completed (took 491ms) +2026/10/29 07:57:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:57:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:57:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:57:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:57:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:57:59 [INFO] [scheduler] Running job: system-health +2026/10/29 07:57:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 07:57:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 07:57:59 [INFO] [scheduler] Running job: docker-socket-users +2026/10/29 07:57:59 [INFO] [scheduler] Running job: offsite-credential-retry +2026/10/29 07:57:59 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/10/29 07:57:59 [INFO] [scheduler] Running job: backup-cache +2026/10/29 07:57:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:57:59 [INFO] [backup] Found 3 DB dump files across drives +2026/10/29 07:57:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 07:57:59 [INFO] [scheduler] Job docker-socket-users completed (took 219ms) +2026/10/29 07:57:59 [INFO] [backup] Discovered 2 databases +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 07:57:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 07:57:59 [INFO] [backup] Discovered app data: 4 apps +2026/10/29 07:57:59 [INFO] [backup] Backup status cache refreshed +2026/10/29 07:57:59 [INFO] [scheduler] Job backup-cache completed (took 246ms) +2026/10/29 07:57:59 [INFO] [monitor] Health check: status=ok +2026/10/29 07:57:59 [INFO] [scheduler] Job system-health completed (took 324ms) +2026/10/29 07:58:00 [INFO] [scheduler] Job agent-channel-health completed (took 490ms) +2026/10/29 07:58:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:58:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:58:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:58:30 [INFO] [web] Login from 192.168.0.180 +2026/10/29 07:58:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:58:43 [INFO] [api] Manual app-data backup (DB dump) triggered +2026/10/29 07:58:43 [INFO] [backup] Starting database dump run +2026/10/29 07:58:43 [INFO] [backup] Discovered 2 databases +2026/10/29 07:58:43 [INFO] [backup] Discovered 2 database(s): paperless-postgres(postgres), bookstack-db(mariadb) +2026/10/29 07:58:43 [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (474.6 KB, 317ms, 72 tables) +2026/10/29 07:58:43 [INFO] [settings] Settings saved +2026/10/29 07:58:43 [INFO] [backup] DB dump: bookstack-db → bookstack-mariadb.sql (57.9 KB, 290ms, 41 tables) +2026/10/29 07:58:43 [INFO] [settings] Settings saved +2026/10/29 07:58:43 [INFO] [backup] Stopping bookstack for safe volume dump +2026/10/29 07:58:43 [INFO] [stacks] Stopping stack: bookstack +2026/10/29 07:58:48 [INFO] [stacks] Stack bookstack stopped successfully (took 4.3s) +2026/10/29 07:58:48 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/29 07:58:48 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_config → 3.1 MB +2026/10/29 07:58:49 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.4 MB +2026/10/29 07:58:49 [INFO] [backup] Restarting bookstack after volume dump +2026/10/29 07:58:49 [INFO] [stacks] Starting stack: bookstack +2026/10/29 07:58:49 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/29 07:58:55 [INFO] [stacks] Stack bookstack started successfully (took 6.2s) +2026/10/29 07:58:55 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:58:55 [INFO] [backup] Stopping paperless-ngx for safe volume dump +2026/10/29 07:58:55 [INFO] [stacks] Stopping stack: paperless-ngx +2026/10/29 07:58:58 [INFO] [stacks] Stack bookstack post-start status: +2026/10/29 07:58:58 [INFO] [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.2@sha256:2fdeeb1a2028ce7cb1274696237e32b9f849e16b6d77314d08c78356a6016fca running Up 3 seconds (health: starting) +2026/10/29 07:58:58 [INFO] [stacks] bookstack-db mariadb:12.3@sha256:d46e102fa8b9b72bfcdeb56f9dd538742092119dc8b11f31fe662605cd234cd9 running Up 9 seconds (healthy) +2026/10/29 07:58:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 07:58:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 07:58:59 [INFO] [scheduler] Running job: stack-scan +2026/10/29 07:58:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/29 07:58:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:58:59 [INFO] [scheduler] Job stack-scan completed (took 107ms) +2026/10/29 07:58:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 07:58:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:58:59 [INFO] [scheduler] Job agent-channel-health completed (took 449ms) +2026/10/29 07:59:02 [INFO] [stacks] Stack paperless-ngx stopped successfully (took 6.8s) +2026/10/29 07:59:02 [INFO] [stacks] Status refresh: 7 containers across 68 stacks +2026/10/29 07:59:03 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 11.5 MB +2026/10/29 07:59:03 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 69.6 MB +2026/10/29 07:59:04 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_redis_data → 7.2 MB +2026/10/29 07:59:04 [INFO] [backup] Restarting paperless-ngx after volume dump +2026/10/29 07:59:04 [INFO] [stacks] Starting stack: paperless-ngx +2026/10/29 07:59:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:59:15 [INFO] [stacks] Stack paperless-ngx started successfully (took 10.8s) +2026/10/29 07:59:15 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:59:15 [INFO] [backup] Stopping privatebin for safe volume dump +2026/10/29 07:59:15 [INFO] [stacks] Stopping stack: privatebin +2026/10/29 07:59:15 [INFO] [stacks] Stack privatebin stopped successfully (took 0.3s) +2026/10/29 07:59:15 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 07:59:16 [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 8.5 KB +2026/10/29 07:59:16 [INFO] [backup] Restarting privatebin after volume dump +2026/10/29 07:59:16 [INFO] [stacks] Starting stack: privatebin +2026/10/29 07:59:16 [INFO] [stacks] Stack privatebin started successfully (took 0.4s) +2026/10/29 07:59:16 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:59:16 [WARN] [backup] Skipping volume dump for radicale — it is not running its pinned version (radicale runs ghcr.io/kozea/radicale:3.8.0, pinned ghcr.io/kozea/radicale:3.8.1); the last good backup is kept (R-645) +2026/10/29 07:59:16 [INFO] [backup] App-data backup completed: 2 databases (532.5 KB total), 3 volume dump(s) (33.493s) +2026/10/29 07:59:16 [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +2026/10/29 07:59:16 [INFO] [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/10/29 07:59:16 [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/29 07:59:16 [WARN] [backup] Recovery unit NOT captured for radicale — it is not running its pinned version (radicale runs ghcr.io/kozea/radicale:3.8.0, pinned ghcr.io/kozea/radicale:3.8.1); the last good unit is kept (R-645) +2026/10/29 07:59:16 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:59:16 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:59:16 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:59:16 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:59:16 [ERROR] [backup] radicale is RUNNING but its newest local copy (/mnt/sys_drive/felhom-data/backups/primary/radicale) holds NO database dump and NO volume tar — it is not protected (Part D) +2026/10/29 07:59:18 [INFO] [stacks] Stack paperless-ngx post-start status: +2026/10/29 07:59:18 [INFO] [stacks] paperless-postgres postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 running Up 13 seconds (healthy) +2026/10/29 07:59:18 [INFO] [stacks] paperless-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 13 seconds (healthy) +2026/10/29 07:59:18 [INFO] [stacks] paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15@sha256:6c86cad803970ea782683a8e80e7403444c5bf3cf70de63b4d3c8e87500db92f running Up 3 seconds (health: starting) +2026/10/29 07:59:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:59:19 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 07:59:19 [INFO] [stacks] Stack privatebin post-start status: +2026/10/29 07:59:19 [INFO] [stacks] privatebin privatebin/pdo:2.1.0@sha256:3400b0a4714d62161727a710c82a3e319cfea364ac6e591aed653f5bb6bb8055 running Up 3 seconds (health: starting) +2026/10/29 07:59:28 [INFO] [offbox] backup run started (10 app(s) toggled) +2026/10/29 07:59:28 [INFO] [settings] Settings saved +2026/10/29 07:59:28 [INFO] [backup] Starting database dump run +2026/10/29 07:59:28 [INFO] [backup] Discovered 2 databases +2026/10/29 07:59:28 [INFO] [backup] Discovered 2 database(s): paperless-postgres(postgres), bookstack-db(mariadb) +2026/10/29 07:59:28 [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (474.6 KB, 346ms, 72 tables) +2026/10/29 07:59:28 [INFO] [settings] Settings saved +2026/10/29 07:59:29 [INFO] [backup] DB dump: bookstack-db → bookstack-mariadb.sql (57.9 KB, 380ms, 41 tables) +2026/10/29 07:59:29 [INFO] [settings] Settings saved +2026/10/29 07:59:29 [INFO] [backup] Stopping bookstack for safe volume dump +2026/10/29 07:59:29 [INFO] [stacks] Stopping stack: bookstack +2026/10/29 07:59:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:59:29 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 07:59:33 [INFO] [stacks] Stack bookstack stopped successfully (took 4.2s) +2026/10/29 07:59:33 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/29 07:59:33 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_config → 3.1 MB +2026/10/29 07:59:34 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.4 MB +2026/10/29 07:59:34 [INFO] [backup] Restarting bookstack after volume dump +2026/10/29 07:59:34 [INFO] [stacks] Starting stack: bookstack +2026/10/29 07:59:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:59:40 [INFO] [stacks] Stack bookstack started successfully (took 6.3s) +2026/10/29 07:59:41 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:59:41 [INFO] [backup] Stopping paperless-ngx for safe volume dump +2026/10/29 07:59:41 [INFO] [stacks] Stopping stack: paperless-ngx +2026/10/29 07:59:44 [INFO] [stacks] Stack bookstack post-start status: +2026/10/29 07:59:44 [INFO] [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.2@sha256:2fdeeb1a2028ce7cb1274696237e32b9f849e16b6d77314d08c78356a6016fca running Up 3 seconds (health: starting) +2026/10/29 07:59:44 [INFO] [stacks] bookstack-db mariadb:12.3@sha256:d46e102fa8b9b72bfcdeb56f9dd538742092119dc8b11f31fe662605cd234cd9 running Up 9 seconds (healthy) +2026/10/29 07:59:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:59:52 [INFO] [stacks] Stack paperless-ngx stopped successfully (took 10.9s) +2026/10/29 07:59:52 [INFO] [stacks] Status refresh: 7 containers across 68 stacks +2026/10/29 07:59:52 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 11.5 MB +2026/10/29 07:59:53 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 69.6 MB +2026/10/29 07:59:53 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_redis_data → 7.2 MB +2026/10/29 07:59:53 [INFO] [backup] Restarting paperless-ngx after volume dump +2026/10/29 07:59:53 [INFO] [stacks] Starting stack: paperless-ngx +2026/10/29 07:59:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 07:59:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 07:59:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 07:59:59 [INFO] Health probes: 2 ok (of 2 probed) +2026/10/29 07:59:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:00:00 [INFO] [scheduler] Job agent-channel-health completed (took 484ms) +2026/10/29 08:00:05 [INFO] [stacks] Stack paperless-ngx started successfully (took 11.4s) +2026/10/29 08:00:05 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:05 [INFO] [backup] Stopping privatebin for safe volume dump +2026/10/29 08:00:05 [INFO] [stacks] Stopping stack: privatebin +2026/10/29 08:00:05 [INFO] [stacks] Stack privatebin stopped successfully (took 0.4s) +2026/10/29 08:00:05 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:00:06 [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 8.5 KB +2026/10/29 08:00:06 [INFO] [backup] Restarting privatebin after volume dump +2026/10/29 08:00:06 [INFO] [stacks] Starting stack: privatebin +2026/10/29 08:00:07 [INFO] [stacks] Stack privatebin started successfully (took 0.7s) +2026/10/29 08:00:07 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:07 [WARN] [backup] Skipping volume dump for radicale — it is not running its pinned version (radicale runs ghcr.io/kozea/radicale:3.8.0, pinned ghcr.io/kozea/radicale:3.8.1); the last good backup is kept (R-645) +2026/10/29 08:00:07 [INFO] [backup] App-data backup completed: 2 databases (532.5 KB total), 3 volume dump(s) (39.211s) +2026/10/29 08:00:07 [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +2026/10/29 08:00:07 [INFO] [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/10/29 08:00:07 [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/29 08:00:07 [WARN] [backup] Recovery unit NOT captured for radicale — it is not running its pinned version (radicale runs ghcr.io/kozea/radicale:3.8.0, pinned ghcr.io/kozea/radicale:3.8.1); the last good unit is kept (R-645) +2026/10/29 08:00:07 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:07 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:07 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:07 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:07 [ERROR] [backup] radicale is RUNNING but its newest local copy (/mnt/sys_drive/felhom-data/backups/primary/radicale) holds NO database dump and NO volume tar — it is not protected (Part D) +2026/10/29 08:00:07 [INFO] [offbox] pre-push dump leg completed in 39.447s — snapshot pair is coherent +2026/10/29 08:00:08 [INFO] [stacks] Stack paperless-ngx post-start status: +2026/10/29 08:00:08 [INFO] [stacks] paperless-postgres postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 running Up 14 seconds (healthy) +2026/10/29 08:00:08 [INFO] [stacks] paperless-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy) +2026/10/29 08:00:08 [INFO] [stacks] paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15@sha256:6c86cad803970ea782683a8e80e7403444c5bf3cf70de63b4d3c8e87500db92f running Up 3 seconds (health: starting) +2026/10/29 08:00:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:10 [INFO] [stacks] Stack privatebin post-start status: +2026/10/29 08:00:10 [INFO] [stacks] privatebin privatebin/pdo:2.1.0@sha256:3400b0a4714d62161727a710c82a3e319cfea364ac6e591aed653f5bb6bb8055 running Up 3 seconds (health: starting) +2026/10/29 08:00:11 [WARN] [offbox] jellyfin: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:00:11 [WARN] [offbox] plex: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:00:11 [WARN] [offbox] radarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:00:11 [WARN] [offbox] sonarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:00:11 [WARN] [offbox] transmission: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:00:11 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:00:11 [WARN] [offbox] sonarr: no recovery unit found on any connected drive — skipping +2026/10/29 08:00:11 [WARN] [offbox] transmission: no recovery unit found on any connected drive — skipping +2026/10/29 08:00:11 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:00:13 [INFO] [settings] Settings saved +2026/10/29 08:00:13 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:13 [INFO] [offbox] backed up prowlarr (/mnt/sys_drive/felhom-data/backups/primary/prowlarr, 0 mandatory path(s)) +2026/10/29 08:00:13 [WARN] [offbox] radicale: no recovery unit found on any connected drive — skipping +2026/10/29 08:00:15 [INFO] [settings] Settings saved +2026/10/29 08:00:15 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:15 [INFO] [offbox] backed up bookstack (/mnt/sys_drive/felhom-data/backups/primary/bookstack, 0 mandatory path(s)) +2026/10/29 08:00:15 [WARN] [offbox] jellyfin: no recovery unit found on any connected drive — skipping +2026/10/29 08:00:17 [INFO] [settings] Settings saved +2026/10/29 08:00:17 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:17 [INFO] [offbox] backed up paperless-ngx (/mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx, 1 mandatory path(s)) +2026/10/29 08:00:17 [WARN] [offbox] plex: no recovery unit found on any connected drive — skipping +2026/10/29 08:00:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:19 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:00:19 [INFO] [settings] Settings saved +2026/10/29 08:00:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:19 [INFO] [offbox] backed up privatebin (/mnt/sys_drive/felhom-data/backups/primary/privatebin, 0 mandatory path(s)) +2026/10/29 08:00:19 [WARN] [offbox] radarr: no recovery unit found on any connected drive — skipping +2026/10/29 08:00:21 [INFO] [offbox] retention skipped (after-run): no clean-up window now (not due (last window 2026-10-26T03:15:52Z)) — nothing deleted (decision 68) +2026/10/29 08:00:21 [INFO] [settings] Settings saved +2026/10/29 08:00:21 [INFO] [upwatch] off-site push measured: 1429133 bytes in 14 s, best 10 s 1.1 Mbit/s (counts=false); line now at least 94 Mbit/s +2026/10/29 08:00:25 [INFO] [settings] Settings saved +2026/10/29 08:00:25 [INFO] [settings] Settings saved +2026/10/29 08:00:25 [INFO] [offbox] backup OK: 4 app(s) backed up, 6 skipped (no unit), 77 snapshot(s), 53s +2026/10/29 08:00:25 [INFO] [offbox] manual run progress reporting ended after 57s +2026/10/29 08:00:25 [INFO] Event pushed: backup_run_failures (error) [hu-only] — Offsite backup INCOMPLETE: 1 app(s) had a directory they declare MANDATORY missing from the snapshot — radicale ((a teljes alkalmazás — nincs helyi mentési egysége)). The run itself succeeded and what it captured is real, but these apps are NOT fully protected off-site. Check that the declared path exists on disk and that the app's data root resolves where the backup looks (R-203). +2026/10/29 08:00:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:44 [INFO] [api] restart requested for stack: radicale +2026/10/29 08:00:44 [INFO] [stacks] Restarting stack: radicale +2026/10/29 08:00:45 [INFO] [stacks] Stack radicale: started again: radicale 2026-10-29T07:54:29.086891581Z → 2026-10-29T08:00:45.004902445Z +2026/10/29 08:00:45 [INFO] [stacks] Stack radicale restarted successfully (took 0.8s) +2026/10/29 08:00:45 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/29 08:00:45 [INFO] [stacks] installed-images radicale: recorded 1 service(s) (radicale=ghcr.io/kozea/radicale:3.8.1 (sha256:54d9406cd30f…)) +2026/10/29 08:00:45 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 08:00:45 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:45 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:48 [INFO] [stacks] Stack radicale post-start status: +2026/10/29 08:00:48 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting) +2026/10/29 08:00:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:49 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:00:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:00:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:00:59 [INFO] [scheduler] Running job: stack-scan +2026/10/29 08:00:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/29 08:00:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:00:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:00:59 [INFO] [scheduler] Job stack-scan completed (took 168ms) +2026/10/29 08:00:59 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:01:00 [INFO] [scheduler] Job agent-channel-health completed (took 486ms) +2026/10/29 08:01:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:01:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:01:24 [INFO] [web] Login from 192.168.0.180 +2026/10/29 08:01:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:01:37 [INFO] [api] Manual app-data backup (DB dump) triggered +2026/10/29 08:01:37 [INFO] [backup] Starting database dump run +2026/10/29 08:01:37 [INFO] [backup] Discovered 2 databases +2026/10/29 08:01:37 [INFO] [backup] Discovered 2 database(s): paperless-postgres(postgres), bookstack-db(mariadb) +2026/10/29 08:01:37 [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (474.9 KB, 297ms, 72 tables) +2026/10/29 08:01:37 [INFO] [settings] Settings saved +2026/10/29 08:01:37 [INFO] [backup] DB dump: bookstack-db → bookstack-mariadb.sql (57.9 KB, 288ms, 41 tables) +2026/10/29 08:01:37 [INFO] [settings] Settings saved +2026/10/29 08:01:37 [INFO] [backup] Stopping bookstack for safe volume dump +2026/10/29 08:01:37 [INFO] [stacks] Stopping stack: bookstack +2026/10/29 08:01:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:01:41 [INFO] [stacks] Stack bookstack stopped successfully (took 3.8s) +2026/10/29 08:01:41 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/29 08:01:42 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.4 MB +2026/10/29 08:01:42 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_config → 3.1 MB +2026/10/29 08:01:42 [INFO] [backup] Restarting bookstack after volume dump +2026/10/29 08:01:42 [INFO] [stacks] Starting stack: bookstack +2026/10/29 08:01:48 [INFO] [stacks] Stack bookstack started successfully (took 6.1s) +2026/10/29 08:01:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:01:49 [INFO] [backup] Stopping paperless-ngx for safe volume dump +2026/10/29 08:01:49 [INFO] [stacks] Stopping stack: paperless-ngx +2026/10/29 08:01:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:01:52 [INFO] [stacks] Stack bookstack post-start status: +2026/10/29 08:01:52 [INFO] [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.2@sha256:2fdeeb1a2028ce7cb1274696237e32b9f849e16b6d77314d08c78356a6016fca running Up 3 seconds (health: starting) +2026/10/29 08:01:52 [INFO] [stacks] bookstack-db mariadb:12.3@sha256:d46e102fa8b9b72bfcdeb56f9dd538742092119dc8b11f31fe662605cd234cd9 running Up 9 seconds (healthy) +2026/10/29 08:01:56 [INFO] [stacks] Stack paperless-ngx stopped successfully (took 7.2s) +2026/10/29 08:01:56 [INFO] [stacks] Status refresh: 7 containers across 68 stacks +2026/10/29 08:01:56 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 11.5 MB +2026/10/29 08:01:57 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 69.6 MB +2026/10/29 08:01:57 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_redis_data → 7.2 MB +2026/10/29 08:01:57 [INFO] [backup] Restarting paperless-ngx after volume dump +2026/10/29 08:01:57 [INFO] [stacks] Starting stack: paperless-ngx +2026/10/29 08:01:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:01:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:01:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:01:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:02:00 [INFO] [scheduler] Job agent-channel-health completed (took 472ms) +2026/10/29 08:02:09 [INFO] [stacks] Stack paperless-ngx started successfully (took 11.3s) +2026/10/29 08:02:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:09 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:02:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:09 [INFO] [backup] Stopping privatebin for safe volume dump +2026/10/29 08:02:09 [INFO] [stacks] Stopping stack: privatebin +2026/10/29 08:02:09 [INFO] [stacks] Stack privatebin stopped successfully (took 0.3s) +2026/10/29 08:02:09 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:02:10 [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 8.5 KB +2026/10/29 08:02:10 [INFO] [backup] Restarting privatebin after volume dump +2026/10/29 08:02:10 [INFO] [stacks] Starting stack: privatebin +2026/10/29 08:02:10 [INFO] [stacks] Stack privatebin started successfully (took 0.5s) +2026/10/29 08:02:11 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:11 [INFO] [backup] Stopping radicale for safe volume dump +2026/10/29 08:02:11 [INFO] [stacks] Stopping stack: radicale +2026/10/29 08:02:11 [INFO] [stacks] Stack radicale stopped successfully (took 0.4s) +2026/10/29 08:02:11 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:02:11 [INFO] [backup] Volume dump: radicale/radicale_radicale_data → 12.5 KB +2026/10/29 08:02:11 [INFO] [backup] Restarting radicale after volume dump +2026/10/29 08:02:11 [INFO] [stacks] Starting stack: radicale +2026/10/29 08:02:12 [INFO] [stacks] Stack radicale started successfully (took 0.4s) +2026/10/29 08:02:12 [INFO] [stacks] Stack paperless-ngx post-start status: +2026/10/29 08:02:12 [INFO] [stacks] paperless-postgres postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 running Up 14 seconds (healthy) +2026/10/29 08:02:12 [INFO] [stacks] paperless-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy) +2026/10/29 08:02:12 [INFO] [stacks] paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15@sha256:6c86cad803970ea782683a8e80e7403444c5bf3cf70de63b4d3c8e87500db92f running Up 3 seconds (health: starting) +2026/10/29 08:02:12 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:12 [INFO] [backup] App-data backup completed: 2 databases (532.8 KB total), 4 volume dump(s) (35.916s) +2026/10/29 08:02:12 [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +2026/10/29 08:02:12 [INFO] [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/10/29 08:02:13 [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/29 08:02:13 [INFO] [backup] Recovery unit captured for radicale → /mnt/sys_drive/felhom-data/backups/primary/radicale (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026/10/29 08:02:13 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:13 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:13 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:13 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:14 [INFO] [stacks] Stack privatebin post-start status: +2026/10/29 08:02:14 [INFO] [stacks] privatebin privatebin/pdo:2.1.0@sha256:3400b0a4714d62161727a710c82a3e319cfea364ac6e591aed653f5bb6bb8055 running Up 3 seconds (health: starting) +2026/10/29 08:02:16 [INFO] [stacks] Stack radicale post-start status: +2026/10/29 08:02:16 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting) +2026/10/29 08:02:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:21 [INFO] [offbox] backup run started (10 app(s) toggled) +2026/10/29 08:02:21 [INFO] [settings] Settings saved +2026/10/29 08:02:21 [INFO] [backup] Starting database dump run +2026/10/29 08:02:21 [INFO] [backup] Discovered 2 databases +2026/10/29 08:02:21 [INFO] [backup] Discovered 2 database(s): paperless-postgres(postgres), bookstack-db(mariadb) +2026/10/29 08:02:21 [INFO] [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (474.9 KB, 343ms, 72 tables) +2026/10/29 08:02:21 [INFO] [settings] Settings saved +2026/10/29 08:02:22 [INFO] [backup] DB dump: bookstack-db → bookstack-mariadb.sql (57.9 KB, 359ms, 41 tables) +2026/10/29 08:02:22 [INFO] [settings] Settings saved +2026/10/29 08:02:22 [INFO] [backup] Stopping bookstack for safe volume dump +2026/10/29 08:02:22 [INFO] [stacks] Stopping stack: bookstack +2026/10/29 08:02:26 [INFO] [stacks] Stack bookstack stopped successfully (took 4.2s) +2026/10/29 08:02:26 [INFO] [stacks] Status refresh: 8 containers across 68 stacks +2026/10/29 08:02:26 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_config → 3.1 MB +2026/10/29 08:02:27 [INFO] [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.4 MB +2026/10/29 08:02:27 [INFO] [backup] Restarting bookstack after volume dump +2026/10/29 08:02:27 [INFO] [stacks] Starting stack: bookstack +2026/10/29 08:02:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:29 [INFO] Health probes: 2 ok (of 2 probed) +2026/10/29 08:02:33 [INFO] [stacks] Stack bookstack started successfully (took 6.3s) +2026/10/29 08:02:34 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:34 [INFO] [backup] Stopping paperless-ngx for safe volume dump +2026/10/29 08:02:34 [INFO] [stacks] Stopping stack: paperless-ngx +2026/10/29 08:02:37 [INFO] [stacks] Stack bookstack post-start status: +2026/10/29 08:02:37 [INFO] [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.2@sha256:2fdeeb1a2028ce7cb1274696237e32b9f849e16b6d77314d08c78356a6016fca running Up 3 seconds (health: starting) +2026/10/29 08:02:37 [INFO] [stacks] bookstack-db mariadb:12.3@sha256:d46e102fa8b9b72bfcdeb56f9dd538742092119dc8b11f31fe662605cd234cd9 running Up 9 seconds (healthy) +2026/10/29 08:02:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:45 [INFO] [stacks] Stack paperless-ngx stopped successfully (took 10.8s) +2026/10/29 08:02:45 [INFO] [stacks] Status refresh: 7 containers across 68 stacks +2026/10/29 08:02:45 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 11.5 MB +2026/10/29 08:02:46 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 69.6 MB +2026/10/29 08:02:46 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_redis_data → 7.2 MB +2026/10/29 08:02:46 [INFO] [backup] Restarting paperless-ngx after volume dump +2026/10/29 08:02:46 [INFO] [stacks] Starting stack: paperless-ngx +2026/10/29 08:02:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:49 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:02:57 [INFO] [stacks] Stack paperless-ngx started successfully (took 10.8s) +2026/10/29 08:02:57 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:57 [INFO] [backup] Stopping privatebin for safe volume dump +2026/10/29 08:02:58 [INFO] [stacks] Stopping stack: privatebin +2026/10/29 08:02:58 [INFO] [stacks] Stack privatebin stopped successfully (took 0.4s) +2026/10/29 08:02:58 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:02:58 [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 8.5 KB +2026/10/29 08:02:58 [INFO] [backup] Restarting privatebin after volume dump +2026/10/29 08:02:58 [INFO] [stacks] Starting stack: privatebin +2026/10/29 08:02:59 [INFO] [stacks] Stack privatebin started successfully (took 0.6s) +2026/10/29 08:02:59 [INFO] [scheduler] Running job: docker-socket-users +2026/10/29 08:02:59 [INFO] [scheduler] Running job: backup-cache +2026/10/29 08:02:59 [INFO] [scheduler] Running job: stack-scan +2026/10/29 08:02:59 [INFO] [scheduler] Running job: system-health +2026/10/29 08:02:59 [INFO] [scheduler] Running job: fill-watch-interval +2026/10/29 08:02:59 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/10/29 08:02:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:02:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:02:59 [INFO] [scheduler] Job fill-watch-interval completed (took 13ms) +2026/10/29 08:02:59 [INFO] [scheduler] Running job: offsite-credential-retry +2026/10/29 08:02:59 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/10/29 08:02:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:02:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/29 08:02:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:59 [INFO] [scheduler] Job stack-scan completed (took 345ms) +2026/10/29 08:02:59 [INFO] [scheduler] Job docker-socket-users completed (took 405ms) +2026/10/29 08:02:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:02:59 [INFO] [backup] Found 3 DB dump files across drives +2026/10/29 08:02:59 [INFO] [backup] Stopping radicale for safe volume dump +2026/10/29 08:02:59 [INFO] [stacks] Stopping stack: radicale +2026/10/29 08:02:59 [INFO] [monitor] Health check: status=ok +2026/10/29 08:02:59 [INFO] [scheduler] Job system-health completed (took 587ms) +2026/10/29 08:03:00 [INFO] [scheduler] Job agent-channel-health completed (took 553ms) +2026/10/29 08:03:00 [INFO] [backup] Discovered 2 databases +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:03:00 [INFO] [backup] Discovered app data: 4 apps +2026/10/29 08:03:00 [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +2026/10/29 08:03:00 [INFO] [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/10/29 08:03:00 [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/10/29 08:03:00 [INFO] [backup] Recovery unit captured for radicale → /mnt/sys_drive/felhom-data/backups/primary/radicale (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026/10/29 08:03:00 [INFO] [backup] Backup status cache refreshed +2026/10/29 08:03:00 [INFO] [scheduler] Job backup-cache completed (took 903ms) +2026/10/29 08:03:00 [INFO] [stacks] Stack radicale stopped successfully (took 0.8s) +2026/10/29 08:03:00 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:03:01 [INFO] [stacks] Stack paperless-ngx post-start status: +2026/10/29 08:03:01 [INFO] [stacks] paperless-postgres postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 running Up 13 seconds (healthy) +2026/10/29 08:03:01 [INFO] [stacks] paperless-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy) +2026/10/29 08:03:01 [INFO] [stacks] paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15@sha256:6c86cad803970ea782683a8e80e7403444c5bf3cf70de63b4d3c8e87500db92f running Up 3 seconds (health: starting) +2026/10/29 08:03:01 [INFO] [backup] Volume dump: radicale/radicale_radicale_data → 12.5 KB +2026/10/29 08:03:01 [INFO] [backup] Restarting radicale after volume dump +2026/10/29 08:03:01 [INFO] [stacks] Starting stack: radicale +2026/10/29 08:03:02 [INFO] [stacks] Stack radicale started successfully (took 0.9s) +2026/10/29 08:03:03 [INFO] [stacks] Stack privatebin post-start status: +2026/10/29 08:03:03 [INFO] [stacks] privatebin privatebin/pdo:2.1.0@sha256:3400b0a4714d62161727a710c82a3e319cfea364ac6e591aed653f5bb6bb8055 running Up 4 seconds (health: starting) +2026/10/29 08:03:03 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:03 [INFO] [backup] App-data backup completed: 2 databases (532.8 KB total), 4 volume dump(s) (41.921s) +2026/10/29 08:03:03 [INFO] [backup] Recovery unit captured for radicale → /mnt/sys_drive/felhom-data/backups/primary/radicale (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026/10/29 08:03:03 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:03 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:03 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:03 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:03 [INFO] [offbox] pre-push dump leg completed in 42.448s — snapshot pair is coherent +2026/10/29 08:03:06 [INFO] [stacks] Stack radicale post-start status: +2026/10/29 08:03:06 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 4 seconds (health: starting) +2026/10/29 08:03:07 [WARN] [offbox] jellyfin: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:03:07 [WARN] [offbox] plex: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:03:07 [WARN] [offbox] radarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:03:07 [WARN] [offbox] sonarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:03:07 [WARN] [offbox] transmission: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:03:07 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:03:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:09 [INFO] [settings] Settings saved +2026/10/29 08:03:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:09 [INFO] [offbox] backed up bookstack (/mnt/sys_drive/felhom-data/backups/primary/bookstack, 0 mandatory path(s)) +2026/10/29 08:03:09 [WARN] [offbox] jellyfin: no recovery unit found on any connected drive — skipping +2026/10/29 08:03:12 [INFO] [settings] Settings saved +2026/10/29 08:03:12 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:12 [INFO] [offbox] backed up paperless-ngx (/mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx, 1 mandatory path(s)) +2026/10/29 08:03:12 [WARN] [offbox] plex: no recovery unit found on any connected drive — skipping +2026/10/29 08:03:13 [INFO] [settings] Settings saved +2026/10/29 08:03:14 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:14 [INFO] [offbox] backed up privatebin (/mnt/sys_drive/felhom-data/backups/primary/privatebin, 0 mandatory path(s)) +2026/10/29 08:03:14 [WARN] [offbox] radarr: no recovery unit found on any connected drive — skipping +2026/10/29 08:03:14 [WARN] [offbox] sonarr: no recovery unit found on any connected drive — skipping +2026/10/29 08:03:14 [WARN] [offbox] transmission: no recovery unit found on any connected drive — skipping +2026/10/29 08:03:14 [WARN] [offbox] prowlarr: not deployed — offsite push is unit-only (mandatory userdata not resolvable) +2026/10/29 08:03:15 [INFO] [settings] Settings saved +2026/10/29 08:03:15 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:15 [INFO] [offbox] backed up prowlarr (/mnt/sys_drive/felhom-data/backups/primary/prowlarr, 0 mandatory path(s)) +2026/10/29 08:03:17 [INFO] [settings] Settings saved +2026/10/29 08:03:17 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:17 [INFO] [offbox] backed up radicale (/mnt/sys_drive/felhom-data/backups/primary/radicale, 0 mandatory path(s)) +2026/10/29 08:03:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:19 [INFO] Health probes: 2 ok (of 2 probed) +2026/10/29 08:03:19 [INFO] [offbox] retention skipped (after-run): no clean-up window now (not due (last window 2026-10-26T03:15:52Z)) — nothing deleted (decision 68) +2026/10/29 08:03:19 [INFO] [settings] Settings saved +2026/10/29 08:03:19 [INFO] [upwatch] off-site push measured: 2493965 bytes in 16 s, best 10 s 2.0 Mbit/s (counts=false); line now at least 94 Mbit/s +2026/10/29 08:03:23 [INFO] [settings] Settings saved +2026/10/29 08:03:23 [INFO] [settings] Settings saved +2026/10/29 08:03:23 [INFO] [offbox] backup OK: 5 app(s) backed up, 5 skipped (no unit), 82 snapshot(s), 58s +2026/10/29 08:03:23 [INFO] [offbox] manual run progress reporting ended after 1m2s +2026/10/29 08:03:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:39 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:03:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:51 [INFO] [web] Login from 192.168.0.180 +2026/10/29 08:03:57 [INFO] [web] off-box full-restore prepared for radicale (size 31.3 KB) — awaiting the customer's confirm; no restore has started +2026/10/29 08:03:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:03:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:03:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:03:59 [INFO] [deadapp] check alive: 20 scans since boot, 4 deployed app(s) evaluated, 0 currently down +2026/10/29 08:03:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:04:00 [INFO] [scheduler] Job agent-channel-health completed (took 490ms) +2026/10/29 08:04:04 [INFO] [offbox] restored radicale (78002a51, full=true) → /mnt/felhom-drives/adatlemez/backups/offsite-restore/radicale +2026/10/29 08:04:04 [INFO] [web] off-box restore radicale completed (full=true, async) +2026/10/29 08:04:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:04:12 [INFO] [web] Login from 192.168.0.180 +2026/10/29 08:04:18 [INFO] [api] restart requested for stack: radicale +2026/10/29 08:04:18 [INFO] [stacks] Restarting stack: radicale +2026/10/29 08:04:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:04:27 [INFO] [stacks] Stack radicale: started again: radicale 2026-10-29T08:03:02.167464371Z → 2026-10-29T08:04:26.837498405Z +2026/10/29 08:04:27 [INFO] [stacks] Stack radicale restarted successfully (took 8.6s) +2026/10/29 08:04:27 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/29 08:04:27 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 08:04:27 [INFO] [stacks] installed-images radicale: recorded 1 service(s) (radicale=ghcr.io/kozea/radicale:3.8.0 (sha256:0d60d544710b…)) +2026/10/29 08:04:27 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:04:27 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:04:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:04:30 [INFO] [stacks] Stack radicale post-start status: +2026/10/29 08:04:30 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468 running Up 3 seconds (health: starting) +2026/10/29 08:04:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:04:39 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:04:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:04:53 [INFO] [web] Login from 192.168.0.180 +2026/10/29 08:04:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:04:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:04:59 [INFO] [scheduler] Running job: stack-scan +2026/10/29 08:04:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:04:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/29 08:04:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:04:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:04:59 [INFO] [scheduler] Job stack-scan completed (took 158ms) +2026/10/29 08:05:00 [INFO] [scheduler] Job agent-channel-health completed (took 525ms) +2026/10/29 08:05:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:05:09 [INFO] [web] Login from 192.168.0.180 +2026/10/29 08:05:16 [INFO] [api] restart requested for stack: radicale +2026/10/29 08:05:16 [INFO] [stacks] Restarting stack: radicale +2026/10/29 08:05:16 [INFO] [stacks] Stack radicale: started again: radicale 2026-10-29T08:04:26.837498405Z → 2026-10-29T08:05:16.524386943Z +2026/10/29 08:05:16 [INFO] [stacks] Stack radicale restarted successfully (took 0.8s) +2026/10/29 08:05:16 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:05:17 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:05:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:05:20 [INFO] [stacks] Stack radicale post-start status: +2026/10/29 08:05:20 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468 running Up 3 seconds (health: starting) +2026/10/29 08:05:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:05:29 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:05:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:05:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:05:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:05:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:05:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:05:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:06:00 [INFO] [scheduler] Job agent-channel-health completed (took 478ms) +2026/10/29 08:06:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:06:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:06:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:06:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:06:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:06:59 [INFO] [scheduler] Running job: stack-scan +2026/10/29 08:06:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:06:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:06:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:06:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/29 08:06:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:06:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:06:59 [INFO] [scheduler] Job stack-scan completed (took 152ms) +2026/10/29 08:07:00 [INFO] [scheduler] Job agent-channel-health completed (took 482ms) +2026/10/29 08:07:01 [INFO] [web] Login from 192.168.0.180 +2026/10/29 08:07:07 [INFO] [api] restart requested for stack: radicale +2026/10/29 08:07:07 [INFO] [stacks] Restarting stack: radicale +2026/10/29 08:07:08 [INFO] [stacks] Stack radicale: started again: radicale 2026-10-29T08:05:16.524386943Z → 2026-10-29T08:07:08.330110078Z +2026/10/29 08:07:08 [INFO] [stacks] Stack radicale restarted successfully (took 0.8s) +2026/10/29 08:07:08 [INFO] [stacks] installed-images radicale: recorded 1 service(s) (radicale=ghcr.io/kozea/radicale:3.8.1 (sha256:54d9406cd30f…)) +2026/10/29 08:07:08 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/29 08:07:08 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 08:07:08 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:07:08 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:07:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:07:11 [INFO] [stacks] Stack radicale post-start status: +2026/10/29 08:07:11 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting) +2026/10/29 08:07:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:07:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:07:29 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:07:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:07:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:07:59 [INFO] [sync] Starting catalog sync +2026/10/29 08:07:59 [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (branch: main) +2026/10/29 08:07:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:07:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:07:59 [INFO] [scheduler] Running job: system-health +2026/10/29 08:07:59 [INFO] [scheduler] Running job: docker-socket-users +2026/10/29 08:07:59 [INFO] [scheduler] Running job: hub-report +2026/10/29 08:07:59 [INFO] [report] Building system report +2026/10/29 08:07:59 [INFO] [scheduler] Running job: offsite-credential-retry +2026/10/29 08:07:59 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/10/29 08:07:59 [INFO] [scheduler] Running job: backup-cache +2026/10/29 08:07:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:07:59 [INFO] [backup] Found 3 DB dump files across drives +2026/10/29 08:07:59 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:07:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:07:59 [INFO] [backup] Discovered 2 databases +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:07:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:07:59 [INFO] [backup] Discovered app data: 4 apps +2026/10/29 08:07:59 [INFO] [scheduler] Job docker-socket-users completed (took 301ms) +2026/10/29 08:07:59 [INFO] [sync] Catalog sync complete +2026/10/29 08:07:59 [INFO] [sync] Periodic sync: Sablonok naprakészek — nincs változás +2026/10/29 08:07:59 [INFO] [monitor] Health check: status=ok +2026/10/29 08:07:59 [INFO] [scheduler] Job system-health completed (took 391ms) +2026/10/29 08:08:00 [INFO] [backup] Backup status cache refreshed +2026/10/29 08:08:00 [INFO] [scheduler] Job backup-cache completed (took 595ms) +2026/10/29 08:08:00 [INFO] [scheduler] Job agent-channel-health completed (took 522ms) +2026/10/29 08:08:00 [INFO] [monitor] Health check: status=ok +2026/10/29 08:08:00 [INFO] [report] Hub report pushed successfully (9062 bytes) +2026/10/29 08:08:00 [INFO] [settings] Settings saved +2026/10/29 08:08:00 [INFO] [scheduler] Job hub-report completed (took 1.45s) +2026/10/29 08:08:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:08:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:08:19 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:08:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:08:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:08:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:08:49 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:08:54 [INFO] [web] Login from 192.168.0.180 +2026/10/29 08:08:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:08:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:08:59 [INFO] [scheduler] Running job: stack-scan +2026/10/29 08:08:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:08:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/29 08:08:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:08:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:08:59 [INFO] [scheduler] Job stack-scan completed (took 140ms) +2026/10/29 08:09:00 [INFO] [scheduler] Job agent-channel-health completed (took 495ms) +2026/10/29 08:09:01 [INFO] [api] restart requested for stack: radicale +2026/10/29 08:09:01 [INFO] [stacks] Restarting stack: radicale +2026/10/29 08:09:02 [INFO] [stacks] Stack radicale: started again: radicale 2026-10-29T08:07:08.330110078Z → 2026-10-29T08:09:01.852817156Z +2026/10/29 08:09:02 [INFO] [stacks] Stack radicale restarted successfully (took 1.1s) +2026/10/29 08:09:02 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:09:02 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:09:05 [INFO] [stacks] Stack radicale post-start status: +2026/10/29 08:09:05 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting) +2026/10/29 08:09:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:09:09 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:09:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:09:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:09:32 [INFO] [api] restart requested for stack: radicale +2026/10/29 08:09:32 [INFO] [stacks] Restarting stack: radicale +2026/10/29 08:09:33 [INFO] [stacks] Stack radicale: started again: radicale 2026-10-29T08:09:01.852817156Z → 2026-10-29T08:09:32.950147579Z +2026/10/29 08:09:33 [INFO] [stacks] Stack radicale restarted successfully (took 0.8s) +2026/10/29 08:09:33 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/29 08:09:33 [INFO] [stacks] installed-images radicale: recorded 1 service(s) (radicale=ghcr.io/kozea/radicale:3.8.0 (sha256:0d60d544710b…)) +2026/10/29 08:09:33 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 08:09:33 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:09:33 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:09:36 [INFO] [stacks] Stack radicale post-start status: +2026/10/29 08:09:36 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468 running Up 3 seconds (health: starting) +2026/10/29 08:09:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:09:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:09:49 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:09:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:09:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:09:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:09:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:10:00 [INFO] [scheduler] Job agent-channel-health completed (took 497ms) +2026/10/29 08:10:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:10:18 [INFO] [offbox] radicale: snapshot 78002a51 holds data of [ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c] (written 2026-10-29T08:03:01Z); the app runs [ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468] — it comes back at the snapshot's version +2026/10/29 08:10:19 [INFO] [backup] Discovered 2 databases +2026/10/29 08:10:19 [INFO] [stacks] Stopping stack: radicale +2026/10/29 08:10:19 [INFO] [stacks] Stack radicale stopped successfully (took 0.3s) +2026/10/29 08:10:19 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:10:19 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:10:19 [INFO] [offbox] radicale: the live volumes are copied before the restore (1 copied, 0 new from the snapshot) in 332ms — /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081019Z (R-893) +2026/10/29 08:10:19 [INFO] [backup] radicale: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195) +2026/10/29 08:10:19 [ERROR] [faultinject] defwrite-after-compose: FAULT INJECTED (fail left 0) +2026/10/29 08:10:19 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 1 encrypted, 1 sensitive fields +2026/10/29 08:10:19 [INFO] [stacks] Redeploying radicale from recovery unit with 4 env vars +2026/10/29 08:10:19 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 08:10:19 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/29 08:10:19 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 08:10:19 [INFO] [stacks] pin radicale: radicale=ghcr.io/kozea/radicale:3.8.0 +2026/10/29 08:10:19 [INFO] [offbox] radicale: the definition write failed (FAULT INJECTED at defwrite-after-compose (D63 test build)); the live definition is written back before the start +2026/10/29 08:10:19 [INFO] [stacks] Starting stack: radicale +2026/10/29 08:10:20 [INFO] [stacks] Stack radicale started successfully (took 0.3s) +2026/10/29 08:10:20 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/29 08:10:20 [INFO] [stacks] installed-images radicale: recorded 1 service(s) (radicale=ghcr.io/kozea/radicale:3.8.0 (sha256:0d60d544710b…)) +2026/10/29 08:10:20 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 08:10:20 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:10:20 [ERROR] [web] off-box reconstitute radicale (async): restoring radicale: writing the snapshot's definition failed: FAULT INJECTED at defwrite-after-compose (D63 test build) +2026/10/29 08:10:23 [INFO] [stacks] Stack radicale post-start status: +2026/10/29 08:10:23 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468 running Up 3 seconds (health: starting) +2026/10/29 08:10:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:10:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:10:39 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:10:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:10:56 [INFO] [web] Login from 192.168.0.180 +2026/10/29 08:10:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:10:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:10:59 [INFO] [scheduler] Running job: stack-scan +2026/10/29 08:10:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:10:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:10:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/29 08:10:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:10:59 [INFO] [scheduler] Job stack-scan completed (took 157ms) +2026/10/29 08:11:00 [INFO] [scheduler] Job agent-channel-health completed (took 484ms) +2026/10/29 08:11:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:11:11 [INFO] [offbox] radicale: snapshot 78002a51 holds data of [ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c] (written 2026-10-29T08:03:01Z); the app runs [ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468] — it comes back at the snapshot's version +2026/10/29 08:11:11 [INFO] [backup] Discovered 2 databases +2026/10/29 08:11:11 [INFO] [stacks] Stopping stack: radicale +2026/10/29 08:11:11 [INFO] [stacks] Stack radicale stopped successfully (took 0.3s) +2026/10/29 08:11:11 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:11:12 [INFO] [offbox] radicale: the live volumes are copied before the restore (1 copied, 0 new from the snapshot) in 327ms — /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081111Z (R-893) +2026/10/29 08:11:12 [INFO] [backup] radicale: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195) +2026/10/29 08:11:12 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 1 encrypted, 1 sensitive fields +2026/10/29 08:11:12 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 08:11:12 [INFO] [stacks] Redeploying radicale from recovery unit with 4 env vars +2026/10/29 08:11:12 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/29 08:11:12 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 08:11:12 [INFO] [stacks] pin radicale: radicale=ghcr.io/kozea/radicale:3.8.1 +2026/10/29 08:11:12 [INFO] [backup] Restoring Docker volume radicale_radicale_data for radicale +2026/10/29 08:11:12 [INFO] [backup] Restored 1 Docker volume(s) for radicale +2026/10/29 08:11:12 [INFO] [stacks] Starting stack: radicale +2026/10/29 08:11:13 [INFO] [stacks] Stack radicale started successfully (took 0.3s) +2026/10/29 08:11:13 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields +2026/10/29 08:11:13 [INFO] [stacks] installed-images radicale: recorded 1 service(s) (radicale=ghcr.io/kozea/radicale:3.8.1 (sha256:54d9406cd30f…)) +2026/10/29 08:11:13 [INFO] [stacks] SaveAppConfig: saved config for radicale +2026/10/29 08:11:13 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:11:16 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:11:16 [INFO] [stacks] Stack radicale post-start status: +2026/10/29 08:11:16 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting) +2026/10/29 08:11:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:11:21 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:11:21 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:11:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:11:29 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:11:31 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:11:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:11:41 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:11:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:11:51 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:11:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:11:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:11:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:11:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:12:00 [INFO] [scheduler] Job agent-channel-health completed (took 478ms) +2026/10/29 08:12:01 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:11 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:11 [INFO] [offbox] reconstituted radicale from snapshot 78002a51: 0 file(s) placed, 1 volume(s) replayed, 0 DB dump(s) replayed, safety dump=., skewed=false +2026/10/29 08:12:11 [INFO] [web] off-box reconstitute radicale completed (async): files=0 dbs=0 snapshot=78002a51 +2026/10/29 08:12:11 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:22 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:32 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:42 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:52 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:58 [WARN] [offbox] inventory: size of sonarr's newest snapshot unknown: offbox: a(z) 7223e3d2 pillanatkép mérete ismeretlen (listing it anyway) +2026/10/29 08:12:58 [WARN] [web] restore-copy delete REFUSED for /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081111Z: the typed confirmation did not match +2026/10/29 08:12:58 [WARN] [web] restore-copy delete REFUSED for /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081111Z: the typed confirmation did not match +2026/10/29 08:12:59 [INFO] [scheduler] Running job: system-health +2026/10/29 08:12:59 [INFO] [scheduler] Running job: stack-scan +2026/10/29 08:12:59 [INFO] [scheduler] Running job: docker-socket-users +2026/10/29 08:12:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:12:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:12:59 [INFO] [scheduler] Running job: offsite-credential-retry +2026/10/29 08:12:59 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/10/29 08:12:59 [INFO] [scheduler] Running job: fill-watch-interval +2026/10/29 08:12:59 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/10/29 08:12:59 [INFO] [scheduler] Job fill-watch-interval completed (took 0s) +2026/10/29 08:12:59 [INFO] [scheduler] Running job: backup-cache +2026/10/29 08:12:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/29 08:12:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:12:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:59 [INFO] [backup] Found 3 DB dump files across drives +2026/10/29 08:12:59 [INFO] [scheduler] Job stack-scan completed (took 164ms) +2026/10/29 08:12:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:12:59 [INFO] [scheduler] Job docker-socket-users completed (took 273ms) +2026/10/29 08:12:59 [INFO] [backup] Discovered 2 databases +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 08:12:59 [INFO] [monitor] Health check: status=ok +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:12:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:12:59 [INFO] [backup] Discovered app data: 4 apps +2026/10/29 08:12:59 [INFO] [scheduler] Job system-health completed (took 392ms) +2026/10/29 08:12:59 [INFO] [backup] Backup status cache refreshed +2026/10/29 08:12:59 [INFO] [scheduler] Job backup-cache completed (took 415ms) +2026/10/29 08:13:00 [INFO] [scheduler] Job agent-channel-health completed (took 473ms) +2026/10/29 08:13:02 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:09 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:13:12 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:19 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:13:22 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:32 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:34 [INFO] [web] Login from 192.168.0.180 +2026/10/29 08:13:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:42 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:48 [INFO] [backup] Discovered 2 databases +2026/10/29 08:13:48 [INFO] [stacks] Stopping stack: radicale +2026/10/29 08:13:48 [INFO] [stacks] Stack radicale stopped successfully (took 0.3s) +2026/10/29 08:13:48 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:13:49 [INFO] [offbox] radicale: the live volumes are copied before the restore (1 copied, 0 new from the snapshot) in 334ms — /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081348Z (R-893) +2026/10/29 08:13:49 [INFO] [backup] radicale: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195) +2026/10/29 08:13:49 [INFO] [backup] Restoring Docker volume radicale_radicale_data for radicale +2026/10/29 08:13:49 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:13:49 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:13:49 [INFO] [backup] Restored 1 Docker volume(s) for radicale +2026/10/29 08:13:49 [INFO] [stacks] Starting stack: radicale +2026/10/29 08:13:50 [INFO] [stacks] Stack radicale started successfully (took 0.3s) +2026/10/29 08:13:50 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:53 [INFO] [backup] radicale: a new restore began before the off-site restore's start was stable — its restore copy stays for support (/mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081111Z) +2026/10/29 08:13:53 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:53 [INFO] [stacks] Stack radicale post-start status: +2026/10/29 08:13:53 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting) +2026/10/29 08:13:58 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:58 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:13:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:13:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:13:59 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:13:59 [INFO] [deadapp] check alive: 40 scans since boot, 4 deployed app(s) evaluated, 0 currently down +2026/10/29 08:13:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:14:00 [INFO] [scheduler] Job agent-channel-health completed (took 473ms) +2026/10/29 08:14:08 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:18 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:28 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:38 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:48 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:48 [INFO] [offbox] reconstituted radicale from snapshot 78002a51: 0 file(s) placed, 1 volume(s) replayed, 0 DB dump(s) replayed, safety dump=., skewed=false +2026/10/29 08:14:48 [INFO] [web] off-box reconstitute radicale completed (async): files=0 dbs=0 snapshot=78002a51 +2026/10/29 08:14:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:59 [INFO] [scheduler] Running job: stack-scan +2026/10/29 08:14:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:14:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:14:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/29 08:14:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:14:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:14:59 [INFO] [scheduler] Job stack-scan completed (took 141ms) +2026/10/29 08:15:00 [INFO] [scheduler] Job agent-channel-health completed (took 475ms) +2026/10/29 08:15:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:15:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:15:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:15:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:15:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:15:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:15:31 [WARN] [offbox] inventory: size of bookstack's newest snapshot unknown: offbox: a(z) 9d5591f1 pillanatkép mérete ismeretlen (listing it anyway) +2026/10/29 08:15:32 [WARN] [web] restore-copy delete REFUSED for /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081111Z: the typed confirmation did not match +2026/10/29 08:15:32 [WARN] [web] restore copy DELETED by hand (D62): app=radicale restore_at=2026-10-29T08:11:11Z size=38243 path=/mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081111Z who=the household (dashboard sign-in) from 192.168.0.180 +2026/10/29 08:15:38 [WARN] Event push failed after 3 attempts (restore_copy_deleted/info): HTTP 400 +2026/10/29 08:15:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:15:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:15:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:15:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:15:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:15:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:15:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:15:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:15:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:15:59 [INFO] [scheduler] Job agent-channel-health completed (took 467ms) +2026/10/29 08:16:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:16:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:16:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:16:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:16:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:16:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:16:38 [WARN] [offbox] inventory: size of jellyfin's newest snapshot unknown: offbox: a(z) 9c51c984 pillanatkép mérete ismeretlen (listing it anyway) +2026/10/29 08:16:38 [WARN] [offbox] inventory: size of prowlarr's newest snapshot unknown: offbox: a(z) 694186ec pillanatkép mérete ismeretlen (listing it anyway) +2026/10/29 08:16:38 [WARN] [offbox] inventory: size of transmission's newest snapshot unknown: offbox: a(z) de418894 pillanatkép mérete ismeretlen (listing it anyway) +2026/10/29 08:16:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:16:40 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:16:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:16:50 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:16:59 [INFO] [scheduler] Running job: stack-scan +2026/10/29 08:16:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:16:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:16:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:16:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/29 08:16:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:16:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:16:59 [INFO] [scheduler] Job stack-scan completed (took 137ms) +2026/10/29 08:16:59 [INFO] [scheduler] Job agent-channel-health completed (took 465ms) +2026/10/29 08:17:00 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:17:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:17:10 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:17:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:17:20 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:17:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:17:30 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:17:39 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:17:40 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:17:49 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:17:50 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:17:50 [INFO] [backup] radicale: ran stably for 3m0s after its off-site restore — the restore copy is released (R-1270) +2026/10/29 08:17:59 [INFO] [scheduler] Running job: offsite-credential-retry +2026/10/29 08:17:59 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/10/29 08:17:59 [INFO] [scheduler] Running job: backup-cache +2026/10/29 08:17:59 [INFO] [scheduler] Running job: system-health +2026/10/29 08:17:59 [INFO] [scheduler] Running job: docker-socket-users +2026/10/29 08:17:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:17:59 [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 08:17:59 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:17:59 [INFO] [backup] Found 3 DB dump files across drives +2026/10/29 08:17:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:17:59 [INFO] [scheduler] Job docker-socket-users completed (took 349ms) +2026/10/29 08:17:59 [INFO] [backup] Discovered 2 databases +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:17:59 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 08:17:59 [INFO] [backup] Discovered app data: 4 apps +2026/10/29 08:17:59 [INFO] [backup] Backup status cache refreshed +2026/10/29 08:17:59 [INFO] [scheduler] Job backup-cache completed (took 395ms) +2026/10/29 08:17:59 [INFO] [monitor] Health check: status=ok +2026/10/29 08:17:59 [INFO] [scheduler] Job system-health completed (took 442ms) +2026/10/29 08:18:00 [INFO] [scheduler] Job agent-channel-health completed (took 492ms) +2026/10/29 08:18:09 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:18:10 [INFO] [web] Login from 192.168.0.180 +2026/10/29 08:18:19 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:18:19 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:18:29 [INFO] [stacks] Status refresh: 10 containers across 68 stacks +2026/10/29 08:18:29 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:18:37 [INFO] [backup] Discovered 2 databases +2026/10/29 08:18:38 [WARN] [faultinject] stop-radicale: passed (skip left 0, fail left 1) +2026/10/29 08:18:38 [INFO] [stacks] Stopping stack: radicale +2026/10/29 08:18:38 [INFO] [stacks] Stack radicale stopped successfully (took 0.3s) +2026/10/29 08:18:38 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:18:38 [INFO] [offbox] radicale: the live volumes are copied before the restore (1 copied, 0 new from the snapshot) in 345ms — /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081838Z (R-893) +2026/10/29 08:18:38 [INFO] [backup] radicale: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195) +2026/10/29 08:18:38 [INFO] [backup] Restoring Docker volume radicale_radicale_data for radicale +2026/10/29 08:18:38 [ERROR] [backup] Failed to remove volume radicale_radicale_data before the restore: Error response from daemon: remove radicale_radicale_data: volume is in use - [bcea23576117066b1fbd5e4f1c0ce85c9ce3784f26e361ea1c8454c3f0bb24b7] — exit status 1 (not written into) +2026/10/29 08:18:38 [ERROR] [offbox] radicale: stopping the app before the put-back FAILED: FAULT INJECTED at stop-radicale (D63 test build) — nothing is put back; the copy stays at /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081838Z; holding the app +2026/10/29 08:18:38 [WARN] [settings] restore hold SET for radicale — the app stays stopped until it is cleared +2026/10/29 08:18:38 [ERROR] [faultinject] stop-radicale: FAULT INJECTED (fail left 0) +2026/10/29 08:18:38 [INFO] [settings] Settings saved +2026/10/29 08:18:38 [INFO] [stacks] Stopping stack: radicale +2026/10/29 08:18:38 [INFO] [stacks] Stack radicale stopped successfully (took 0.1s) +2026/10/29 08:18:38 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:18:38 [ERROR] [offbox] radicale: the restore stopped half-way after files/volumes/version had moved — HOLDING the app stopped for support (R-893); error was: failed to restore 1 volume(s): [radicale_radicale_data] +2026/10/29 08:18:38 [WARN] [offbox] radicale: the restore's volume copy is KEPT for support at /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081838Z +2026/10/29 08:18:38 [ERROR] [web] off-box reconstitute radicale (async): a(z) radicale visszaállítása félúton megszakadt. Az alkalmazás egyes fájljai vagy adatai már a mentésből származnak, mások nem, ezért nem illenek össze. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Ehhez segítség kell: vedd fel velünk a kapcsolatot +2026/10/29 08:18:39 [INFO] Event pushed: backup_run_failures (error) [hu-only] — App "radicale" is HELD STOPPED for support: its off-site database restore failed. The database was rolled back to the pre-restore copy and the live definition written back, but its files and named volumes are the snapshot's — a mixed state (R-893). The app will not start from any path until the hold is cleared. Replay error: failed to restore 1 volume(s): [radicale_radicale_data] +2026/10/29 08:18:39 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:18:49 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:18:49 [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 08:18:59 [INFO] [scheduler] Running job: stack-scan +2026/10/29 08:18:59 [INFO] [scheduler] Running job: hold-report +2026/10/29 08:18:59 [INFO] [holdreport] radicale held (restore_mixed since 2026-10-29T08:18:38Z) — reported to the hub +2026/10/29 08:18:59 [INFO] [settings] Settings saved +2026/10/29 08:18:59 [INFO] [scheduler] Job hold-report completed (took 1ms) +2026/10/29 08:18:59 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:18:59 [INFO] Event pushed: app_held (info) [hu-only] — App radicale is held: held after an off-site restore failed its database replay (2026-10-29T08:18:38Z) — clear the hold to start it +2026/10/29 08:18:59 [INFO] [scheduler] Running job: agent-channel-health +2026/10/29 08:18:59 [INFO] [stacks] ScanStacks complete: 68 stacks found (4 deployed, 64 available) +2026/10/29 08:18:59 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:18:59 [INFO] [scheduler] Job stack-scan completed (took 336ms) +2026/10/29 08:19:00 [INFO] [scheduler] Job agent-channel-health completed (took 478ms) +2026/10/29 08:19:09 [INFO] [stacks] Status refresh: 9 containers across 68 stacks +2026/10/29 08:19:19 [INFO] [stacks] Status refresh: 9 containers across 68 stacks diff --git a/documentation/audits/d63-2026-10-29/ctl-switch.txt b/documentation/audits/d63-2026-10-29/ctl-switch.txt new file mode 100644 index 00000000..ad6b9844 --- /dev/null +++ b/documentation/audits/d63-2026-10-29/ctl-switch.txt @@ -0,0 +1 @@ +gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 50 seconds (healthy) diff --git a/documentation/audits/d63-2026-10-29/d63_live.py b/documentation/audits/d63-2026-10-29/d63_live.py new file mode 100644 index 00000000..789db299 --- /dev/null +++ b/documentation/audits/d63-2026-10-29/d63_live.py @@ -0,0 +1,260 @@ +#!/usr/bin/env python3 +"""d63_live.py — D63 live proofs on Tester 1 (2026-10-29, attended), test build with `-tags faultinject`. + +Phases: + setup deploy a throwaway Radicale, seed T1 through its own front door, run it at 3.8.0 (the stack folder's + compose edited, the household's Restart), switch it into the off-site set and press „Mentés most" for + off-site, then put the compose back at 3.8.1 and Restart — so the newest off-site snapshot holds 3.8.0's + data under 3.8.1's live definition. + pull pull the newest off-site snapshot of radicale to the restore scratch (the wizard's two steps). + h4 seed T2; arm defwrite-after-compose "0 1"; press the full restore. Want: the fault, then the live definition + written back BEFORE the start; the app runs 3.8.1; no hold; T1 and T2 read back. + d62 a full restore with no fault; the copy is listed on the restore page with its date; a wrong name is refused; + the right name deletes it; it is gone. + h5 a created-only container holds radicale's data volume (the restore cannot replace it → the put-back); + arm stop-radicale "1 1" (the restore's own stop passes, the put-back's stop fails). Want: nothing is put + back, the copy stays, the app is held with the D20 sentence; the page shows the held copy's button OFF. + teardown the blocker and the fault files go; the app is removed through the product. +State (passwords) stays in the scratchpad, never printed.""" +import json, os, re, subprocess, sys, time + +sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts") +SCR = "/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0b76c4d7-9b39-4e4e-869d-d0d2c6006331/scratchpad" +os.environ.setdefault("SC", f"{SCR}/rt-tester1") +os.environ.setdefault("TARGET", "tester-1") +import box_walk as w +import upgrade_fixtures_box as fb + +EV = f"{SCR}/d63/ev" +os.makedirs(EV, exist_ok=True) +STATE = f"{SCR}/d63/.state.json" +APP, SUB = "radicale", "radicale" +fx = fb.FIXTURES[APP] +FAULTS = "/var/lib/docker/volumes/felhom-controller-data/_data/faults" +COMPOSE = f"/opt/docker/stacks/{APP}/docker-compose.yml" +OLD, NEW = "ghcr.io/kozea/radicale:3.8.0", "ghcr.io/kozea/radicale:3.8.1" +OLD_FULL = "ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468" +NEW_FULL = "ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c" + + +def state(): + return json.load(open(STATE)) if os.path.exists(STATE) else {} + + +def save(st): + json.dump(st, open(STATE, "w")) + os.chmod(STATE, 0o600) + + +def form(path, fields, accept_json=False): + """A dashboard form post with the session and its CSRF token — what the page's button sends.""" + sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip() + csrf = open(f"{w.SC}/csrf{os.getpid()}.txt").read().strip() + args = ["curl", "-sk", "-D", "-", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}", "-H", f"X-CSRF-Token: {csrf}", "-X", "POST", + "--data-urlencode", f"_csrf={csrf}"] + if accept_json: + args += ["-H", "Accept: application/json"] + for k, v in fields.items(): + args += ["--data-urlencode", f"{k}={v}"] + bodyf = f"{SCR}/d63/.body" + r = w.sh(args + ["-o", bodyf, f"{w.BASE}{path}"], timeout=180) + out = r.stdout or "" + head = out.split("\n")[0].strip() + loc = [l.strip() for l in out.split("\n") if l.lower().startswith("location:")] + body = open(bodyf).read() if os.path.exists(bodyf) else "" + return head, loc[:1], body + + +def wait_op(label, cap=1800): + t0, last = time.time(), None + while time.time() - t0 < cap: + code, d = w.ctl("GET", "/api/backup/restore-status") + dd = d.get("data") or {} + cur = (dd.get("running"), dd.get("op"), dd.get("ok"), (dd.get("message") or "")[:300]) + if cur != last: + w.say(f" +{round(time.time()-t0):>5}s {label}: {cur}") + last = cur + if not dd.get("running") and time.time() - t0 > 5: + return dd + time.sleep(3) + return {} + + +def ctl_log(since, pat): + return w.guest(f"docker logs --since {since} felhom-controller 2>&1 | grep -E '{pat}' | tail -60") + + +def now(): + return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + + +def look(): + st = w.stack(APP) + w.say(f" app: state={st.get('state')} hold={st.get('hold_reason')!r} deployed={st.get('deployed')}") + w.say(" compose image: " + w.guest(f"grep -E '^ +image:' {COMPOSE}").strip()) + w.say(" container: " + w.guest(f"docker ps -a --filter name=^/{APP}$ --format '{{{{.Image}}}} {{{{.Status}}}}'").strip()) + w.say(" copies: " + (w.guest("find /mnt/*/felhom-data/backups/restore-undo /opt/docker/felhom-data/backups/restore-undo -mindepth 2 -maxdepth 2 -type d 2>/dev/null | head").strip() or "(none)")) + return st + + +def set_image(img, label, full=False): + if full: + w.guest(f"sed -i -E 's#^( +image: )ghcr.io/kozea/radicale:[^ ]+$#\\1{img}#' {COMPOSE}") + img = img.split("@")[0] + else: + w.guest(f"sed -i -E 's#^( +image: )ghcr.io/kozea/radicale:[0-9.]+#\\1{img}#' {COMPOSE}") + c, d = w.ctl("POST", f"/api/stacks/{APP}/restart") + w.say(f" [{label}] compose → {img}; Restart → {c} {str(d)[:120]}") + for _ in range(40): + time.sleep(3) + if img in w.guest(f"docker ps --filter name=^/{APP}$ --format '{{{{.Image}}}}'"): + break + look() + + +w.login() +phase = sys.argv[1] +st = state() +if st.get("t1"): # the deploy's generated login lives only in the deploying process; the seed reads w.GENERATED + w.GENERATED[APP] = {"RADICALE_USER": st["t1"]["user"], "RADICALE_PASSWORD": st["t1"]["pw"]} +t_start = now() +w.say(f"=== phase {phase} at {t_start}") +w.say("controller image: " + w.guest("cat /etc/felhom-controller-image; docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'").strip()) + +if phase == "setup": + assert w.deploy(APP, SUB), "deploy failed" + t1 = fx.seed(w, SUB, w.say) + assert t1, "seed T1 failed" + st["t1"] = t1 + save(st) + set_image(OLD, "setup") + w.say(f" T1 readback at 3.8.0: {fx.verify(w, SUB, t1, w.say)}") + w.say(" off-site toggle: " + str(form("/backup/offbox/toggle", {"app": APP, "enabled": "on"})[:2])) + w.say(" off-site run: " + str(form("/backup/offbox/run", {})[:2])) + t0 = time.time() + while time.time() - t0 < 3600: + time.sleep(15) + if "manual off-box backup" in ctl_log(t_start, "off-box backup|offbox.*(completed|finished|failed)") or \ + re.search(r"(?i)off-?box (run|backup) (completed|finished|done)", ctl_log(t_start, "offbox|off-box")): + break + w.say(" off-site run log: " + ctl_log(t_start, "offbox|off-box|radicale").strip()[-3000:]) + set_image(NEW, "setup") + w.say(f" T1 readback at 3.8.1: {fx.verify(w, SUB, t1, w.say)}") + +elif phase in ("setup2", "setup3"): + # The off-site run pushes recovery units; a fresh app has none until a backup run made one ("skipped (no unit)"). + # The household's „Mentés most" (whole box, the only backup press there is — R-648) makes radicale's unit at 3.8.0. + look() + c, d = w.ctl("POST", "/api/backup/run") + w.say(f" local backup press → {c} {str(d)[:200]}") + t0 = time.time() + while time.time() - t0 < 2400: + time.sleep(10) + lg = ctl_log(t_start, r"App-data backup completed|\[backup\].*(failed|FAILED)|backup run (finished|complete)") + if "App-data backup completed" in lg: + break + w.say(" local backup log: " + ctl_log(t_start, r"App-data backup completed|radicale").strip()[-2500:]) + w.say(" off-site run: " + str(form("/backup/offbox/run", {})[:2])) + t1 = now() + t0 = time.time() + while time.time() - t0 < 2400: + time.sleep(10) + if re.search(r"\[offbox\] backup (OK|failed)", ctl_log(t1, r"\[offbox\] backup")): + break + w.say(" off-site run log: " + ctl_log(t1, r"\[offbox\] backup|radicale").strip()[-2500:]) + if phase == "setup2": + set_image(NEW, "setup2") + w.say(f" T1 readback: {fx.verify(w, SUB, st['t1'], w.say)}") + w.say(" radicale's off-site snapshots: " + w.guest("ls -la /mnt/sys_drive/felhom-data/backups/primary/radicale 2>&1 | head -8")) + +elif phase == "pull": + w.say(" prepare: " + str(form("/backup/offbox/restore", {"app": APP, "mode": "full"})[:2])) + w.say(" pull: " + str(form("/backup/offbox/restore", {"app": APP, "mode": "full", "confirm": "1"})[:2])) + wait_op("pull") + +elif phase == "h4": + # the live definition moves to the TRUE 3.8.0 (its own digest) right before the restore — within one catalog-sync + # tick — so the snapshot (3.8.1) and the live definition differ and the restore writes the snapshot's definition + set_image(NEW_FULL, "h4", full=True) # T2 is seeded on 3.8.1 (09:05-09:08: the 401s were the harness's empty password in a new process, not 3.8.0) + t2 = None + for _ in range(6): + time.sleep(10) + t2 = fx.seed(w, SUB, w.say) + if t2: + break + assert t2, "seed T2 failed" + st["t2"] = t2 + save(st) + set_image(OLD_FULL, "h4", full=True) + time.sleep(15) + look() + w.guest(f"mkdir -p {FAULTS} && echo '0 1' > {FAULTS}/defwrite-after-compose && cat {FAULTS}/defwrite-after-compose") + w.say(" armed: defwrite-after-compose = 0 1") + w.say(" restore: " + str(form("/backup/offbox/reconstitute", {"app": APP, "confirm": "1"})[:2])) + rec = wait_op("h4 restore") + w.say(f" restore record: {json.dumps(rec)[:1200]}") + w.say(" controller log:\n" + ctl_log(t_start, "faultinject|radicale|R-1256|definition")) + look() + w.say(f" T1 readback: {fx.verify(w, SUB, st['t1'], w.say)}") + w.say(f" T2 readback: {fx.verify(w, SUB, t2, w.say)}") + w.say(" fault file left: " + (w.guest(f"ls {FAULTS} 2>/dev/null").strip() or "(none)")) + +elif phase == "d62": + look() + w.say(" restore (no fault): " + str(form("/backup/offbox/reconstitute", {"app": APP, "confirm": "1"})[:2])) + rec = wait_op("d62 restore") + # at once: the stable-start check releases the copy about a minute after a good restore (R-1270) + page = w.page("/backups/restore") + t_page = now() + rows = re.findall(r'
(.*?)
', page, re.S) + w.say(f" page at {t_page}: {len(rows)} deletable restore copy row(s); restore ok={(rec.get('last') or {}).get('ok')}") + m = re.search(r'data-restore-copies="true">(.*?)\s*