From 83e436f60901873795bc1267708c70ba08c7a96b Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 24 Sep 2026 14:55:06 +0200 Subject: [PATCH] night 2026-09-24: findings doc, decisions 29-30, STATUS/CONTEXT, register 335->344 (7 closed), teardown evidence, floor 0.269.1 Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- CONTEXT.md | 27 + REPORT-night-2026-09-24.md | 19 + STATUS.md | 44 +- .../architecture/09-update-architecture.md | 25 + .../audits/DRILL-night-2026-09-24.md | 150 +- .../audits/night-2026-09-24/E/arm-09.json | 1 + .../audits/night-2026-09-24/E/arm-10.json | 1 + .../audits/night-2026-09-24/E/arm-11.json | 1 + .../audits/night-2026-09-24/E/done-09.json | 1 + .../audits/night-2026-09-24/E/done-10.json | 1 + .../audits/night-2026-09-24/E/done-11.json | 1 + .../E/round-09-controller.log | 459 ++++++ .../audits/night-2026-09-24/E/round-09.json | 104 ++ .../audits/night-2026-09-24/E/round-09.log | 7 + .../audits/night-2026-09-24/E/round-09.stdout | 8 + .../E/round-09b-remove-again.txt | 7 + .../E/round-10-controller-pre.log | 591 ++++++++ .../E/round-10-controller.log | 1120 +++++++++++++++ .../audits/night-2026-09-24/E/round-10.json | 182 +++ .../audits/night-2026-09-24/E/round-10.log | 14 + .../audits/night-2026-09-24/E/round-10.stdout | 15 + .../E/round-11-controller-pre.log | 512 +++++++ .../E/round-11-controller.log | 1244 +++++++++++++++++ .../audits/night-2026-09-24/E/round-11.json | 267 ++++ .../audits/night-2026-09-24/E/round-11.log | 24 + .../audits/night-2026-09-24/E/round-11.stdout | 25 + .../E/round-12-controller.log | 431 ++++++ .../audits/night-2026-09-24/E/round-12.json | 85 ++ .../audits/night-2026-09-24/E/round-12.log | 6 + .../audits/night-2026-09-24/E/round-12.stdout | 7 + .../F/10-machine-removes.json | 16 + .../night-2026-09-24/F/10-machine-removes.log | 75 + .../F/11-drive-dirs-by-name.txt | 15 + .../F/12-9202-config-restored.txt | 14 + .../night-2026-09-24/F/13-drill-reset.txt | 2 + .../night-2026-09-24/F/20-host-after.txt | 13 + .../night-2026-09-24/F/30-floor-0.269.1.txt | 2 + .../night-2026-09-24/F/31-boxes-arriving.txt | 8 + .../audits/night-2026-09-24/tools/teardown.py | 29 + documentation/backlog/CLOSED-ITEMS.md | 7 + documentation/backlog/OPEN-ITEMS.md | 9 +- 41 files changed, 5537 insertions(+), 32 deletions(-) create mode 100644 REPORT-night-2026-09-24.md create mode 100644 documentation/audits/night-2026-09-24/E/arm-09.json create mode 100644 documentation/audits/night-2026-09-24/E/arm-10.json create mode 100644 documentation/audits/night-2026-09-24/E/arm-11.json create mode 100644 documentation/audits/night-2026-09-24/E/done-09.json create mode 100644 documentation/audits/night-2026-09-24/E/done-10.json create mode 100644 documentation/audits/night-2026-09-24/E/done-11.json create mode 100644 documentation/audits/night-2026-09-24/E/round-09-controller.log create mode 100644 documentation/audits/night-2026-09-24/E/round-09.json create mode 100644 documentation/audits/night-2026-09-24/E/round-09.log create mode 100644 documentation/audits/night-2026-09-24/E/round-09.stdout create mode 100644 documentation/audits/night-2026-09-24/E/round-09b-remove-again.txt create mode 100644 documentation/audits/night-2026-09-24/E/round-10-controller-pre.log create mode 100644 documentation/audits/night-2026-09-24/E/round-10-controller.log create mode 100644 documentation/audits/night-2026-09-24/E/round-10.json create mode 100644 documentation/audits/night-2026-09-24/E/round-10.log create mode 100644 documentation/audits/night-2026-09-24/E/round-10.stdout create mode 100644 documentation/audits/night-2026-09-24/E/round-11-controller-pre.log create mode 100644 documentation/audits/night-2026-09-24/E/round-11-controller.log create mode 100644 documentation/audits/night-2026-09-24/E/round-11.json create mode 100644 documentation/audits/night-2026-09-24/E/round-11.log create mode 100644 documentation/audits/night-2026-09-24/E/round-11.stdout create mode 100644 documentation/audits/night-2026-09-24/E/round-12-controller.log create mode 100644 documentation/audits/night-2026-09-24/E/round-12.json create mode 100644 documentation/audits/night-2026-09-24/E/round-12.log create mode 100644 documentation/audits/night-2026-09-24/E/round-12.stdout create mode 100644 documentation/audits/night-2026-09-24/F/10-machine-removes.json create mode 100644 documentation/audits/night-2026-09-24/F/10-machine-removes.log create mode 100644 documentation/audits/night-2026-09-24/F/11-drive-dirs-by-name.txt create mode 100644 documentation/audits/night-2026-09-24/F/12-9202-config-restored.txt create mode 100644 documentation/audits/night-2026-09-24/F/13-drill-reset.txt create mode 100644 documentation/audits/night-2026-09-24/F/20-host-after.txt create mode 100644 documentation/audits/night-2026-09-24/F/30-floor-0.269.1.txt create mode 100644 documentation/audits/night-2026-09-24/F/31-boxes-arriving.txt create mode 100644 documentation/audits/night-2026-09-24/tools/teardown.py diff --git a/CONTEXT.md b/CONTEXT.md index fb3bfdbc..61d43774 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -14,6 +14,33 @@ > language, one screen, no identifiers in the prose. Same subjects, different readers; merging them > would make one of the two audiences stop reading. `STATUS.md` is also a **view of `OPEN-ITEMS.md`** > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +## 2026-09-24 (night shift, run in daytime) — the second drive brings a file app back whole; the box stops a crash loop; digests on the box; the update leg spiked + +**Operator rulings recorded (`09` §3 decisions 26–28).** 26 — build one action that brings a file app back WHOLE +from the second drive. 27 — while a held app's page says support is informed, Remove offers only "keep my data". +28 — a crash loop or an OOM storm is STOPPED by the box; household + operator told; Start = one more try. + +**Decisions taken by CC unattended — operator may reverse (`09` §3 decisions 29, 30).** 29 — the crash loop is +≥ 6 restarts in 10 min from `RestartCount` (the brief's 10/10 misses a steady loop: Docker caps it at ~1/min). +30 — an INSTALLED app keeps its image digest until a guarded Update moves it (`stacks.CarryDigests`); shipped as a +SECOND controller release (v0.269.1) instead of taking v0.269.0's sync behaviour to the fleet. + +**Shipped.** Controller **v0.269.0** (`3c6b49b`): `backup.RestoreTier2Whole` (R-661), fail-closed `sameDevice` +(R-668), keep-data-only remove (R-666), `stacks.ObserveUnhealthy` + `unhealthy_stop` hold (R-667), step +`.felhom.yml` for probe/memory/applied (R-664/R-665), R-662 removed, digests rendered + badge from the tested digest +(§6.4 part 6 box half). Controller **v0.269.1** (`5b1b191`): `CarryDigests`. Hub **v0.123.0**: +`app_stopped_unhealthy` (household + operator). Catalog **`cf7cf84`**: step `.felhom.yml` files, gate rule 4b. +**Floor 0.269.1** (MinAgent 0.131.0): N100 arrived; **demo-hp 9201 did not** (R-672). + +**Spiked, not built.** `09` §6.4 part 7 is now a measured build brief (§6.4.2): chain the update leg from the +off-site job on EVERY exit path (no reliable "finished" signal exists — `offbox.go:863-897` early returns write +nothing), gate interlock beside `WindowStartFn`, per-box switch default ON; gaps R-678/R-679/R-680. + +**Found.** R-672 (P1, agent): the scheduled restore-test filled demo-hp's thin pool; 9201's disks are read-only +— **operator: stop + fsck + start 9201**. R-669 (a hold + restore leaves the failed step's probe as applied; +proven to cause a false hold), R-670, R-671, R-673–R-677, R-681 (a lost install is silent), R-682, R-683. +**Not done:** Part D (no bench — the session could not destroy one). Evidence: `audits/DRILL-night-2026-09-24.md`. + ## 2026-09-24 (morning) — the fleet on 0.267.0 then 0.268.0; the undo after a restore; option A; the ladder on the box **Operator rulings (`09` §3 decisions 24–25).** 24 — the fleet takes 0.267.0 although the chaos hour's stop rule diff --git a/REPORT-night-2026-09-24.md b/REPORT-night-2026-09-24.md new file mode 100644 index 00000000..14329ab1 --- /dev/null +++ b/REPORT-night-2026-09-24.md @@ -0,0 +1,19 @@ +# REPORT — night shift 2026-09-24 (run in daytime from 11:07 CEST) + +Full record: `documentation/audits/DRILL-night-2026-09-24.md`; step log `documentation/audits/night-2026-09-24/PROGRESS.md`. + +- **Hub v0.123.0** (`99c0709`, manifest `b33a331`, live): `app_stopped_unhealthy` allow-listed, per-app cooldowns + on both legs, app named in the mail subject, seeded once into every household (3 seeded); hu/en mail goldens; + red-proofed (`night-2026-09-24/redproofs/A3-hub.txt`). +- **Docs:** `09` §3 decisions 26–28 (operator) and 29–30 (CC unattended); §6.4 part 6 marked shipped (both + halves); part 7 rewritten as the measured build brief §6.4.2. `07` whole-copy table (the second drive is whole + for a file app since v0.269.0). `08` §6.2 the stop rule. Capability map: five rows. CONTEXT and STATUS. +- **Register:** 335 → 344 rows (679,393 → 685,662 B); opened R-668…R-683, closed R-661, R-662, R-664–R-668. +- **Floor 0.269.1** (MinAgent 0.131.0), read back; N100 arrived; demo-hp 9201 did not (R-672). +- **Interventions: 1** — an agent restart on demo-hp so its own Recover removed a leaked restore-test guest that had + filled the pool. The session's permission check refused a direct `pct destroy` and a later read of vzdump task + logs; both said so in the findings doc. +- `unproven.py --summary`: unchanged — 35 of 55 not walked. +- Teardown, three layers: machine (9202 back to live catalog + saved config, tonight's apps removed through the + product, kept drive folders removed by name, backup window 02:30), host (demo-hp `pct list` 9201 + 9202, no + bench), hub (floor; drill repo reset to live `c8025093d23c`, `has_actions: false`). diff --git a/STATUS.md b/STATUS.md index eeb5c2f6..4be97686 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,30 +1,28 @@ # STATUS — what works, what's broken, what's next -**Updated 2026-09-24 (morning session). The fleet is on controller 0.268.0. The undo works after a restore. A stopped app's page tells the truth. A box that is several updates behind now climbs one tested step per press.** +**Updated 2026-09-24 (night shift, run in daytime from 11:07). The demo boxes are on controller 0.269.1 — except demo-hp's customer box, which is damaged and needs you (first item under "What needs you").** -**Decisions I took on my own:** none. Two small changes to your wording, both below. +**Decisions I took on my own (you may reverse either).** +- **A second controller release tonight (0.269.1).** The first one (0.269.0) had a flaw I found in its own live test: when the catalog re-tested an image, the box wrote the new image into a running app's file before anyone pressed Update, so any restart would have changed the image with no backup and no undo. I fixed it and released again rather than send the flaw to the fleet. Now an installed app keeps its image until an Update moves it. +- **A crash loop is 6 restarts in 10 minutes, not the 10 you wrote.** Measured: Docker slows a steady crash loop to about one restart a minute, so 10 would never catch it. No healthy app in any test record restarted more than once while starting. -**What I did, and it worked.** -- **The demo boxes got 0.267.0**, as you said yes to. Then, after every live test passed, they got **0.268.0**. Both boxes arrived healthy about 15 seconds after each change. -- **The undo after a restore.** A restore used to rebuild an app's storage without the tag the undo looked for, so the undo copied nothing. Now the undo finds the storage by its name. A restore also puts the tag back. Tested on the scratch machine with an app restored by the OLD version, the way customer boxes are today: the failed update was undone, and data written before and after the restore came back. -- **A stopped app's page (your option A).** When no copy on the box can bring an app back with its files, the page now says so, says support is informed, and shows no restore button. You get an urgent alarm. Tested by repeating last night's exact case, in Hungarian and English. -- **Small fixes.** A stopped app no longer raises a second, false alarm. A removed app leaves no old files behind. The test bench now marks a memory test with no real load as "not proven", and it starts every run with an empty drive folder. -- **The update ladder.** One press now does one tested step, never a jump. Tested with RomM on the scratch machine: first press moved only the app, second press moved only the database engine. The data came back after each press. The page says how many steps remain. +**What I tested, and it worked.** +- **The second drive now brings an app with files back whole** (your ruling). Tested four times on the scratch machine, twice under a power cut or a nearly full disk. Every time: the account and the files came back, a file the household had edited later was kept, and nothing was deleted. +- **A stranded app can only be removed keeping its data** (your ruling). The box refuses to delete the data, in both languages. +- **The box stops a crash loop or a memory storm** (your ruling), tells the household and you, and Start gives one more try. A second stop within a day says support is informed. +- **Exact image fingerprints.** The box runs exactly the image the catalog tested, and says "update available" when a newer tested image exists for the same version name. +- **Automatic updates: measured, not built.** A test caller updated three apps (up to three steps each) and set a failing one aside in about 7 minutes. The build plan is written with the numbers. +- **A chaos hour, 12 rounds** (power cuts, killed controller, Docker restarts, a full disk, backups). Updates resumed or undid themselves correctly every time. -**Your wording, changed in two small ways.** -- English: I removed the word "please". The house rule for English copy forbids it. -- Hungarian: I kept your text exactly. It uses the formal "Ön" form; the rest of the screens use the informal form. That mismatch is already on the list. +**What broke.** +- **demo-hp's customer box (not caused by tonight's work).** This morning the host's automatic restore test copied a whole guest onto the same full disk pool. The pool filled, and the customer box's disks became read-only. I freed the pool with an agent restart (the product's own clean-up). I could not repair the box itself. +- **An install interrupted by a restart is lost without a word**, and a remove interrupted the same way is left half-done. Both recorded, not fixed. +- After a failed update is restored, the box can keep the failed version's health check, and a later undo then fails for no reason. Recorded. +- Not done: moving more apps in the catalog. It needs a throwaway test machine, and this session could not delete one afterwards. -**What I found (new, not fixed).** -- **The second drive cannot bring back an app with files, even though it holds everything.** Its restore button refuses such apps, and the file restore brings back only files, not the database. So for Nextcloud, Immich, Paperless and Calibre, only the off-site copy counts. This needs your decision. -- **A long crash loop never raises an alarm** if the app looks "up" for a moment between crashes. Seen on the scratch machine: 385 restarts, zero alarms. -- The stopped-app sentence says "do not remove the app", but the Remove button is still there. This needs your word. -- Smaller: a ladder step has no health check of its own yet; right after a restore, an update may be judged with the restored health check; one restore option ("database only") is described in the code but offered nowhere. +**Rows.** 16 opened, 7 closed. The list went from 335 to 344. -**Rows.** Seven closed, six opened. The list went from 336 to 335. - -**What needs you — two questions.** -1. **The second drive and apps with files.** (a) Build a "files plus database" restore for the second drive (my recommendation — then the second drive is a real way back, as customers would expect). (b) Rule that the second drive is files-only for these apps. If you do nothing, the page keeps naming only the off-site copy for these apps, which is true. -2. **The Remove button on a stranded app.** (a) Hide it while support is informed (my recommendation — a removal destroys what support needs). (b) Keep it and soften the sentence. If you do nothing, both stay as they are. - -**Not done.** No automatic updates yet (plan part 7). No customer box touched by hand. Nothing on Peti's machine, the off-site box or your own machine was touched, beyond the normal hub update. The scratch machine is back on the real catalog with its standing apps. +**What needs you.** +1. **Repair demo-hp's customer box:** stop it, check its two disks, start it. If you do nothing, it keeps running but cannot save anything: no backups, no logs, no updates, and it stays on the old controller. +2. **Decide how the restore test may use disk space** (for example: it must check free space first and never use the pool of the box it tests). If you do nothing, the next scheduled restore test can fill the pool again, on any box with a small disk. +3. **Optional:** the two decisions above. If you do nothing, they stay as built. diff --git a/documentation/architecture/09-update-architecture.md b/documentation/architecture/09-update-architecture.md index 02b7cc4d..b4dd4e0b 100644 --- a/documentation/architecture/09-update-architecture.md +++ b/documentation/architecture/09-update-architecture.md @@ -345,6 +345,31 @@ builds them (`audits/update-rulings-2026-09-23/`). (`upgrade-test.py`, 2026-09-23); a version move checks the memory limit (gate or checklist — §6.4); R-636's louder repeated alarm. + +### 2026-09-24 (night) — two decisions taken by CC unattended + +29. **Decision 28's crash loop is ≥ 6 restarts within 10 minutes, counted from Docker's `RestartCount`** — + *decided by CC unattended 2026-09-24 — operator may reverse.* *One sentence:* how many restarts in + what window make a crash loop the box stops? **Options:** (a) the brief's 10 in 10 min; (b) 6 in + 10 min. **Costs:** (a) misses a STEADY loop — Docker's restart back-off caps it at about one restart a + minute (gokapi measured 539 → 546 in 7 min), so a loop that has run for an hour can sit at 9–10 per 10 + min forever; (b) could stop a slow first start that restarts 6 times — none in the drill evidence + (1,831 harness samples, 40 live containers; the one ≥ 6 case, immich's first-start import, was broken, + R-676 watches it), and Start always gives one more try. **Why (b):** (a) fails the ruling's own case. + `08` §6.2; `audits/night-2026-09-24/A3/`. + +30. **An INSTALLED app keeps the image digest it runs until a guarded Update moves it; the sync never moves + it** — *decided by CC unattended 2026-09-24 — operator may reverse.* *One sentence:* when the catalog + re-tests a floating tag at a new digest, does the sync write it into an installed app's compose? + **Options:** (a) yes, as v0.269.0 did; (b) no — the sync carries the running digest over, and only + the guarded Update (backup, undo copy, health check) renders the new one; a fresh install takes the + tested digest. **Costs:** (a) the next restart (a backup's stop/start, a power cut) pulls a new image + with no backup and no undo — measured live on 9202; (b) an installed app runs the older tested image + until someone (or the automatic leg) presses Update — the badge says so. **Why (b):** decision 17 + wants the TESTED image pulled, and slice 3's ruling freezes the version until a deliberate Update; + (a) broke the second. Shipped as controller **v0.269.1** — a second release in the session, instead of + shipping (a) to the fleet with the floor. + --- ## 3b. ANSWERED 2026-09-23 — the seven questions Slices 6 and 7 needed diff --git a/documentation/audits/DRILL-night-2026-09-24.md b/documentation/audits/DRILL-night-2026-09-24.md index 6d73505d..84900842 100644 --- a/documentation/audits/DRILL-night-2026-09-24.md +++ b/documentation/audits/DRILL-night-2026-09-24.md @@ -1,7 +1,97 @@ # DRILL — night 2026-09-24 (run 2026-09-24 from 11:07 CEST): the second drive brings a file app back whole; a crash loop is stopped; exact image fingerprints; automatic updates spiked; a chaos hour -> Written in parts. The "Not done, or changed" section and the results are filled at the end; **the -> Part E schedule below was written and committed BEFORE round 1.** +Run 2026-09-24 from 11:07 CEST (daytime; the brief's clock times were applied relative to the start). +Evidence: `night-2026-09-24/` (PROGRESS.md is the step log). Controller **v0.269.0 → v0.269.1**, hub +**v0.123.0**, catalog **cf7cf84**, floor **0.269.1** (MinAgent 0.131.0). Architecture read: `09` (§3 +decisions 11–28, §6.4), `07` (the whole-copy table), `08` (the alarm ladder). + +## Not done, or changed + +- **Part D (move more apps) was NOT run.** Its bar needs a bench machine on demo-hp that is created and + destroyed tonight; this session's permission check refused host-level destroys (it refused a + `pct destroy`), so a bench could not be torn down — a half-state the rules forbid. Part D is first in + the brief's order of sacrifice. **Apps moved: 0 of 0 tried.** Part E's last step (Update on demo-hp + 9201 the standing apps Part D moved) therefore had nothing to press. +- **Two controller releases, not one** — v0.269.1 fixes a defect v0.269.0's own live proof found (the + sync moved an installed app's image digest, so a restart would have changed the image unguarded). + *Decided by CC unattended — operator may reverse.* The floor went to 0.269.1. +- **The crash-loop threshold is 6 restarts in 10 minutes, not the brief's 10** — measured: Docker's + back-off caps a steady loop at ~1 restart/min. *Decided by CC unattended — operator may reverse.* +- **Part E's "whole-box backup" accident was the box's own full backup run**, not a vzdump: demo-hp's + backup storage had ~4 GB free and its root was at 90 %. +- **Part C step 1 was measured on demo-hp 9201's real night, not on 9202**: 9202 has no off-site target, + so its off-site leg never runs. 9202 gave the db-dump leg and the window move. +- **demo-hp is damaged and needs the operator (R-672, P1):** before tonight's run the agent's scheduled + restore-test filled the host's thin pool; 9201's disks went READ-ONLY. The pool was freed (intervention + 1); 9201 still needs stop + fsck + start, and did not take the 0.269.1 floor. +- Round 2's install and round 9's remove were broken by their accidents (R-681, R-682) — recorded, not fixed. + +**Interventions: 1** (an agent restart on demo-hp so its own Recover tore down the leaked restore-test +guest). **Apps moved: 0 of 0 tried.** **The result that matters most:** a file app now comes back WHOLE +from the second drive — proven four times, twice under a power cut or a nearly full disk — and nothing the +household had was deleted or overwritten. + +## A1 — the whole restore from the second drive (decision 26, R-661) — PROVEN LIVE + +- **Spike** (`A1/00-spike*`): the Tier-2 mirror keeps a file app's drive files as plain files under + `hdd/` and `userdata/` — readable file by file. **R-668 found on the way:** the Tier-2 target + was a removed folder on the app's own disk (the same-disk check failed open) — fixed, fails closed. +- **File rules, tested** (`tier2_whole_test.go`): the whole tree fingerprinted before and after; never + delete, never overwrite a newer live file, bring back the missing, keep an older replaced file beside. + Red-proofs: `redproofs/A1-file-rules.txt`, `A1-r668.txt`. +- **Live, 9202** (`A1/21-readback.json`): restored 1, replaced 0, kept-newer 1, unchanged 137; 3/3 volumes, + 1/1 database; 38 s; the account, the deleted file (back), the edited file (the household's newer copy kept) + and an untouched file read back. **From a hold** (`A1/30-*`): the page named „második meghajtó … a + beállításokat, az adatbázist és a fájlokat" / "second drive … the settings, the database and the files"; + its restore cleared the hold in 33.8 s. **Chaos rounds 3 and 11** repeated it under a power cut and a disk + 1 GB above the floor. + +## A2 — keep-data-only Remove while support is informed (decision 27, R-666) — PROVEN LIVE + +One-drive hold (the second-drive mirror moved aside): page hu „…A Felhom ügyfélszolgálatát értesítettük — +addig ne indítsd újra, és ne töröld az adatait." / en "…Felhom support has been told — until then, do not +restart the app or delete its data."; `keep_data_only: true`; Remove with data or with backups → **409** +hu „Amíg az ügyfélszolgálat foglalkozik az alkalmazással, az adatai nem törölhetők — az alkalmazást az +adatai megtartásával eltávolíthatod." / en "While support is working on this app, its data cannot be deleted +— you can remove the app and keep its data."; the app stayed. Positive control: a hold WITH a whole copy +→ `keep_data_only` absent (`A2/10-*`). **Found (R-669, proven):** that hold was reached because the undo +judged the right old version with the FAILED step's probe, left by the previous hold + restore. + +## A3 — the box stops a crash loop / OOM storm (decision 28, R-667) — PROVEN LIVE + +gokapi (a real crash loop): stopped at „crash_loop — 6 in 10m0s", trip 1, page hu/en + Start, no +„restore needed" badge (`A3/21`); **Start** lifted it, the fresh container restarted 9 times in 32 s, stopped +again: „Az alkalmazást ismét leállítottuk … A Felhom ügyfélszolgálatát értesítettük." / "We stopped the app +again … Felhom support has been told." (`A3/30`). Detector red-proofed (`redproofs/A3-r667-detector.txt`). +Hub v0.123.0 routes `app_stopped_unhealthy` to the household and the operator (red-proofed, `A3-hub.txt`). + +## A4 — each step judged by its own file (R-664, R-665); R-662 removed — shipped, red-proofed + +`redproofs/A4-*`. Live through Part C (three apps climbed with their own step files). + +## Part B — exact image fingerprints on the box — PROVEN LIVE (on v0.269.1) + +- Compose accepts `tag@digest` for **every** template with a tested digest: 25 apps, 37 digests, 0 missing, + 0 parse differences (`B/01`). +- A floating tag (redis:7-alpine) re-tested at a new digest: badge „Frissítés elérhető" (hu/en), Update + pulled exactly that digest, badge „Naprakész", records digest-free (`B/10`, `B/21`). +- **Found and fixed:** v0.269.0's sync wrote the NEW digest into the RUNNING app's compose before Update — + a restart would have changed the image with no backup and no undo. v0.269.1 `CarryDigests` keeps an + installed app's digest; red-proofed (`redproofs/B-sync-carry.txt`); re-proven live (`B/21`: "the sync kept + the running digest: True"). + +## Part C — automatic updates, SPIKE — the build brief is `09` §6.4.2 + +Chain today: legs clock-scheduled, nothing waits. 9201's real night: off-site 04:15:05 → 04:18:33 (3 m 28 s), +gate opens 04:30 → ~11 min for an update leg (decision 20's wait is required). "Off-site finished" signal: +only the persisted `offbox.LastStatus`+`LastRun` (`offbox.go:1003-1125`), not written on five early returns +— **build: chain, do not poll.** Gate interlock: an `Options` func beside `WindowStartFn` in `quiesce.runOnce`. +Simulated night on 9202 (`C/31`): romm 3 steps 77/71/94 s; wishlist 56 s; navidrome 11/9 s; vikunja's failing +step undone in 104 s (90 s drill timeout; ≈ 6–11 min at the 5 min default) and set aside; total 7 m 18 s. +Found: stale steps-left after `done` (R-678), Update on a current app runs a full update (R-679), the box +does not remember a failed step (R-680). Default of the per-box switch: **ON** (decision 12). + +## Part D — not run (see "Not done") ## Part E — the schedule (committed before round 1) @@ -48,3 +138,59 @@ root was at 90 %, so a vzdump of 9202 could have filled the HOST (demo-hp also h **How accidents are fired:** the runner arms the round and waits; the session fires each host-level accident as its own visible command (power cut = `pct stop 9202; pct start 9202`, kill -9 of the controller PID, `systemctl restart docker` in the guest, a `fallocate` to leave 3 GB free on `/var/lib/docker`). + + +### The rounds + +| # | action × accident | app | steady | end state | data read back | events (from the box's own log) | +|---|---|---|---|---|---|---| +| 1 | oom_app × power cut | chaosoom | 185 s | stopped, `unhealthy_stop` | — | app_oom (w), app_oom_storm (e), app_stopped_unhealthy (w) | +| 2 | install × docker restart | n8n | — | **not installed, silently** (R-681); reinstalled after, 211 s | — | app_deploy_started only | +| 3 | cutoff file app × power cut at verifying | nextcloud | 204 s | the update RESUMED after boot → held (second drive) → whole restore → running | account ✓ files 2/2 ✓ | app_update_held (e) | +| 4 | failing step × controller kill | wishlist | 148 s | resumed → undone | ✓ | app_update_undone (w) | +| 5 | remove held × none | gokapi | 27 s | removed, 0 leftovers | — | app_removed (i) | +| 6 | failing step × backup run | navidrome | 110 s | undone (the backup skipped the updating app) | ✓ | app_update_undone (w) | +| 7 | crash loop × backup run | chaoscrash | 116 s | stopped, `unhealthy_stop` | — | app_stopped_unhealthy (w) | +| 8 | oom_app × disk to floor + 1 GB | chaosoomb | 102 s | stopped, `unhealthy_stop` | — | app_oom (w), app_stopped_unhealthy (w) | +| 9 | remove held × controller kill | chaoscrash | — | **half-removed**: 502 to the household, containers gone, still listed (R-682); a second Remove finished it | — | — | +| 10 | ladder step × docker restart | n8n | 155 s | resumed → done, 2.31.3 → 2.40.5, steps 2 → 1 | ✓ | — | +| 11 | cutoff file app × disk to floor + 1 GB | nextcloud | 194 s | held (fresh 14:34 second-drive copy) → whole restore → running | account ✓ files 2/2 ✓ | app_update_held (e) | +| 12 | Start after unhealthy × none | chaosoomb | 125 s | one more try → stopped again, „support is told" hu/en | — | app_oom (w), app_oom_storm (e), app_stopped_unhealthy (w) | + +Controller kills: 2 of max 3, 20 min 32 s apart. **Against `08`:** every stop sent `app_stopped_unhealthy`; +every undo `app_update_undone`, every hold `app_update_held`. **What should have fired and did not:** round 2 +(an install lost — nothing) and round 9 (a remove half-done — nothing); round 3's hold named an hour-old copy +(R-683, watch). No user file was deleted or overwritten by any restore; no data was lost. + +## Teardown — three layers + +- **Machine (9202):** tonight's 8 apps removed through the product (0 containers, 0 volumes, 0 undo copies); + the three drive folders the product kept (R-442 on a box without drive access) removed by name; backup + window back to 02:30 through the product's form; `controller.yaml` restored from `.pre-night0924` (live + catalog, no drill `update:` settings); controller stays on 0.269.1 (the floor). Standing before tonight and + kept: privatebin, paperless-ngx, filebrowser. gokapi removed in round 5 (R-644's crash loop). `F/10-12`. +- **Host (demo-hp):** `pct list` 9201 + 9202 only (as at the start, minus R-672's leaked 990000); no bench + was created; storage unchanged; the thin pool at 58.9 % after intervention 1. `F/20`. +- **Hub:** floor 0.269.1 with MinAgent 0.131.0, read back; N100 9201 arrived in ~16 s (`managed floor SERVED + … from declared`); **demo-hp 9201 did NOT arrive** (read-only disks, R-672). Drill repo reset to live + `cf7cf8456f00`, `has_actions: false`. `F/13`, `F/30-31`. + +## Claims in the brief that turned out wrong (or right), named + +1. *The Tier-2 mirror holds a file app's files in a form a restore can read file by file* — **TRUE** + (plain files under `hdd/` and `userdata/`; the merge walked 139). +2. *`RestartCount` survives the scans that reset `restarting_since`* — **TRUE per container run**; it resets + when the container is recreated (a Start after the box's stop), so the detector treats a drop as a new window. +3. *10 restarts in 10 minutes separates a crash loop from a slow start* — **WRONG**: a steady loop runs at + ~1 restart/min (gokapi 7 in 7 min) and would never reach 10; 6 chosen. No healthy first start in the + drill evidence reached 6 (immich's broken first start reached 12 — R-676). +4. *Docker Compose accepts `tag@digest` in every template's image line* — **TRUE** (25/25). What the brief + did not foresee: rendering digests in the SYNC moved a running app's image (fixed v0.269.1). +5. *The chain signal can be measured on 9202* — **WRONG**: 9202 has no off-site target. +6. *One release per repo* could not hold (see "Not done"). + +## Register + +Before **335 rows / 679,393 B**; after **344 rows / 685,662 B**. Opened R-668…R-683 (16); closed R-661, +R-662, R-664, R-665, R-666, R-667, R-668 (7, compressed into CLOSED-ITEMS). + diff --git a/documentation/audits/night-2026-09-24/E/arm-09.json b/documentation/audits/night-2026-09-24/E/arm-09.json new file mode 100644 index 00000000..8ed3d0e7 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/arm-09.json @@ -0,0 +1 @@ +{"round": 9, "kind": "controller_killed", "at": 1790252994.3224466, "why": "2 s after the remove press"} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-24/E/arm-10.json b/documentation/audits/night-2026-09-24/E/arm-10.json new file mode 100644 index 00000000..f62b2b48 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/arm-10.json @@ -0,0 +1 @@ +{"round": 10, "kind": "docker_restart", "at": 1790253189.3620627, "why": "at starting"} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-24/E/arm-11.json b/documentation/audits/night-2026-09-24/E/arm-11.json new file mode 100644 index 00000000..b2ac4ac8 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/arm-11.json @@ -0,0 +1 @@ +{"round": 11, "kind": "disk_to_floor_plus_1g", "at": 1790253306.7154226, "why": "at verifying"} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-24/E/done-09.json b/documentation/audits/night-2026-09-24/E/done-09.json new file mode 100644 index 00000000..76907d2b --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/done-09.json @@ -0,0 +1 @@ +{"kind": "controller_killed", "cmd": "kill -9 of the controller PID (2 of max 3; first 14:09:22)", "fired_at": "14:29:54", "result": "killed pid 12752 rc=0"} diff --git a/documentation/audits/night-2026-09-24/E/done-10.json b/documentation/audits/night-2026-09-24/E/done-10.json new file mode 100644 index 00000000..0f238792 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/done-10.json @@ -0,0 +1 @@ +{"kind": "docker_restart", "cmd": "pct exec 9202 -- systemctl restart docker", "fired_at": "14:33:09", "result": "rc=0"} diff --git a/documentation/audits/night-2026-09-24/E/done-11.json b/documentation/audits/night-2026-09-24/E/done-11.json new file mode 100644 index 00000000..0c061151 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/done-11.json @@ -0,0 +1 @@ +{"kind": "disk_to_floor_plus_1g", "cmd": "fallocate /var/lib/docker/chaos-fill leaving 3 GB free", "fired_at": "14:35:06", "result": "rc=0 filled 36534034432 bytes /dev/loop1 69G 62G 3.0G 96% /var/lib/docker "} diff --git a/documentation/audits/night-2026-09-24/E/round-09-controller.log b/documentation/audits/night-2026-09-24/E/round-09-controller.log new file mode 100644 index 00000000..3e39a5cb --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-09-controller.log @@ -0,0 +1,459 @@ +2026/09/24 12:29:52 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.10:41526 (X-Forwarded-For: 192.168.0.180) +2026/09/24 12:29:52 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session +2026/09/24 12:29:52 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T12:29:52Z, active_sessions=7 +2026/09/24 12:29:52 auth.go:222: [INFO] [web] Login from 172.18.0.10:41526 +2026/09/24 12:29:52 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.10:41526 +2026/09/24 12:29:52 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.10:41526 +2026/09/24 12:29:52 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaoscrash from 172.18.0.10:41526 +2026/09/24 12:29:52 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaoscrash from 172.18.0.10:41526 +2026/09/24 12:29:52 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaoscrash from 172.18.0.10:41526 +2026/09/24 12:29:52 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaoscrash from 172.18.0.10:41526 +2026/09/24 12:29:52 router.go:907: [INFO] [api] Remove requested for stack: chaoscrash +2026/09/24 12:29:52 delete.go:526: [DEBUG] [stacks] RemoveStack called: name="chaoscrash", removeHDDData=true, backupPathsToRemove=2 +2026/09/24 12:29:52 delete.go:540: [DEBUG] [stacks] RemoveStack chaoscrash: state=stopped, deployed=true, orphaned=false, deploying=false +2026/09/24 12:29:52 delete.go:597: [INFO] Removing deployed stack: chaoscrash (removeHDDData=true, hddDeclared=false, backupPaths=2) +2026/09/24 12:29:52 delete.go:609: [DEBUG] [stacks] RemoveStack chaoscrash: found 0 HDD mounts from compose file +2026/09/24 12:29:52 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system) +2026/09/24 12:29:52 manager.go:1463: [DEBUG] Running: docker compose down --volumes (in /opt/docker/stacks/chaoscrash) +2026/09/24 12:29:52 manager.go:1483: [DEBUG] Command completed: docker compose down --volumes (took 0.1s) +2026/09/24 12:29:52 delete.go:623: [DEBUG] [stacks] RemoveStack chaoscrash: compose down output: +2026/09/24 12:29:54 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:29:54 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:29:54 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:29:54 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:29:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:29:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:29:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:29:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:29:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:29:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:29:54 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:29:54 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:29:54 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:29:54 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:29:56 main.go:335: [INFO] felhom-controller 0.269.1 starting (customer: demo-hp, domain: enkisfelhom.hu) +2026/09/24 12:29:56 settings.go:709: [INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json +2026/09/24 12:29:56 settings.go:1671: [DEBUG] [settings] AutoDiscoverStoragePaths discovered=[/mnt/felhom-drives/scratch_hdd/userdata/navidrome /mnt/felhom-drives/scratch_hdd/userdata/nextcloud /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx /mnt/felhom-drives/scratch_hdd/userdata/romm] fallback="" existing=5 +2026/09/24 12:29:56 main.go:361: [INFO] Encryption key loaded from /opt/docker/felhom-controller/data/encryption.key +2026/09/24 12:29:56 manager.go:328: [INFO] [stacks] Using compose command: docker compose +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=true composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:29:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:29:56 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:29:56 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:29:56 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (11 deployed, 47 available) +2026/09/24 12:29:56 deploy.go:1104: [DEBUG] [stacks] InjectMissingFields: checking 11 stacks +2026/09/24 12:29:56 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack navidrome — 3 deploy fields, 3 existing env vars +2026/09/24 12:29:56 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack chaoscrash — 2 deploy fields, 2 existing env vars +2026/09/24 12:29:56 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack nextcloud — 7 deploy fields, 7 existing env vars +2026/09/24 12:29:56 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack paperless-ngx — 8 deploy fields, 8 existing env vars +2026/09/24 12:29:56 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack chaosoom — 2 deploy fields, 2 existing env vars +2026/09/24 12:29:56 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack chaosoomb — 2 deploy fields, 2 existing env vars +2026/09/24 12:29:56 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack vikunja — 3 deploy fields, 3 existing env vars +2026/09/24 12:29:56 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack wishlist — 2 deploy fields, 2 existing env vars +2026/09/24 12:29:56 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack privatebin — 2 deploy fields, 2 existing env vars +2026/09/24 12:29:56 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack n8n — 3 deploy fields, 3 existing env vars +2026/09/24 12:29:56 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack romm — 6 deploy fields, 6 existing env vars +2026/09/24 12:29:56 deploy.go:1189: [INFO] [stacks] InjectMissingFields: processed 11 stacks +2026/09/24 12:29:56 manager.go:467: [DEBUG] [stacks] MigrateEncryption: checking 11 deployed stacks for plaintext sensitive values +2026/09/24 12:29:56 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "n8n" (1 sensitive fields) +2026/09/24 12:29:56 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "romm" (3 sensitive fields) +2026/09/24 12:29:56 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "nextcloud" (3 sensitive fields) +2026/09/24 12:29:56 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "paperless-ngx" (3 sensitive fields) +2026/09/24 12:29:56 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "vikunja" (1 sensitive fields) +2026/09/24 12:29:56 manager.go:511: [INFO] [stacks] Encryption migration: no stacks needed migration +2026/09/24 12:29:56 desiredstate.go:137: [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded (state ambiguous — legacy boot behaviour retained) +2026/09/24 12:29:56 installed.go:537: [INFO] [stacks] installed-images backfill: 0 app(s) recorded, 11 already had a record, 0 left unrecorded (could not be observed completely — unknown, which renders nothing) +2026/09/24 12:29:56 pin.go:290: [INFO] [stacks] pin adoption: 0 pinned, 11 already pinned, 0 left unpinned (0 not completely observed, 0 running something the template no longer offers) +2026/09/24 12:29:56 undo.go:656: [INFO] [stacks] applied-meta backfill (R-646): recorded 0 []; skipped 0 [] — not current with the catalog, their pinned version's .felhom.yml is no longer on the box +2026/09/24 12:29:56 sync.go:117: [INFO] [sync] Starting catalog sync (repo: https://gitea.dooplex.hu/admin/app-catalog-drill.git, interval: 15m0s) +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] CPUCollector.Start: sampleRate=5s +2026/09/24 12:29:56 sync.go:208: [INFO] [sync] Starting catalog sync +2026/09/24 12:29:56 sync.go:296: [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-drill.git (branch: main) +2026/09/24 12:29:56 sync.go:298: [DEBUG] [sync] git fetch --depth 1 origin main in /opt/docker/felhom-controller/data/catalog-cache +2026/09/24 12:29:56 sync.go:627: [DEBUG] [sync] Running: git fetch --depth 1 origin main +2026/09/24 12:29:56 main.go:515: [INFO] Metrics store opened at /opt/docker/felhom-controller/data/metrics.db +2026/09/24 12:29:56 main.go:527: [INFO] Metrics collector started (60s interval) +2026/09/24 12:29:56 restore_record_wiring.go:35: [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false) +2026/09/24 12:29:56 notifier.go:81: [INFO] Notifier disabled (hub not configured) +2026/09/24 12:29:56 main.go:692: [INFO] [mailrelay] app-email shim unavailable (no hub configured or disabled in config) +2026/09/24 12:29:56 scheduler.go:102: [INFO] [scheduler] Registered periodic job: status-refresh (every 10s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="status-refresh" interval=10s totalJobs=1 +2026/09/24 12:29:56 scheduler.go:102: [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="stack-scan" interval=2m0s totalJobs=2 +2026/09/24 12:29:56 scheduler.go:102: [INFO] [scheduler] Registered periodic job: health-probes (every 10s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="health-probes" interval=10s totalJobs=3 +2026/09/24 12:29:56 scheduler.go:102: [INFO] [scheduler] Registered periodic job: system-health (every 5m0s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="system-health" interval=5m0s totalJobs=4 +2026/09/24 12:29:56 scheduler.go:102: [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="deadapp-check" interval=30s totalJobs=5 +2026/09/24 12:29:56 scheduler.go:102: [INFO] [scheduler] Registered periodic job: ring-spill (every 30s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="ring-spill" interval=30s totalJobs=6 +2026/09/24 12:29:56 scheduler.go:132: [INFO] [scheduler] Daily job db-dump scheduled for 2026-09-25 13:20 CEST +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="db-dump" schedule="13:20" nextRun=2026-09-25T13:20:00+02:00 totalJobs=7 +2026/09/24 12:29:56 scheduler.go:102: [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="offsite-credential-retry" interval=5m0s totalJobs=8 +2026/09/24 12:29:56 scheduler.go:102: [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="backup-cache" interval=5m0s totalJobs=9 +2026/09/24 12:29:56 scheduler.go:132: [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-09-25 14:20 CEST +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="tier2-backup" schedule="14:20" nextRun=2026-09-25T14:20:00+02:00 totalJobs=10 +2026/09/24 12:29:56 scheduler.go:132: [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-09-24 15:05 CEST +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offbox-backup" schedule="15:05" nextRun=2026-09-24T15:05:00+02:00 totalJobs=11 +2026/09/24 12:29:56 scheduler.go:132: [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-09-25 05:10 CEST +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offsite-abandon-sweep" schedule="05:10" nextRun=2026-09-25T05:10:00+02:00 totalJobs=12 +2026/09/24 12:29:56 scheduler.go:132: [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-09-25 06:00 CEST +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offsite-integrity" schedule="06:00" nextRun=2026-09-25T06:00:00+02:00 totalJobs=13 +2026/09/24 12:29:56 scheduler.go:132: [INFO] [scheduler] Daily job offsite-proof scheduled for 2026-09-25 05:30 CEST +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offsite-proof" schedule="05:30" nextRun=2026-09-25T05:30:00+02:00 totalJobs=14 +2026/09/24 12:29:56 scheduler.go:132: [INFO] [scheduler] Daily job metrics-prune scheduled for 2026-09-25 04:00 CEST +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="metrics-prune" schedule="04:00" nextRun=2026-09-25T04:00:00+02:00 totalJobs=15 +2026/09/24 12:29:56 scheduler.go:132: [INFO] [scheduler] Daily job fill-watch scheduled for 2026-09-25 03:30 CEST +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="fill-watch" schedule="03:30" nextRun=2026-09-25T03:30:00+02:00 totalJobs=16 +2026/09/24 12:29:56 selftest.go:50: [INFO] ========== Startup Self-Test ========== +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5480MB avail≈20418MB +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24215424 → total=25898MB avail=20418MB used=5480MB (21.2%) +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=25.5GB avail=39.3GB (37.3%) +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readLoadAvg: raw="2.28 2.56 3.08 8/1537 113" → 1m=2.28 5m=2.56 15m=3.08 +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 67.2°C (hwmon2) +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] GetInfo done in 81ms — mem=5480MB/25898MB (21.2%), rootDisk=25.5GB/68.4GB (37.3%), load=2.28/2.56/3.08, temp=67.2°C (hwmon2), cpu=0.0% +2026/09/24 12:29:56 selftest.go:68: [INFO] [PASS] Docker socket: reachable (v29.8.0) +2026/09/24 12:29:56 selftest.go:68: [INFO] [PASS] Stacks directory: /opt/docker/stacks +2026/09/24 12:29:56 selftest.go:68: [INFO] [PASS] Data directory: /opt/docker/felhom-controller/data (writable) +2026/09/24 12:29:56 selftest.go:68: [INFO] [PASS] System data path: /mnt/sys_drive +2026/09/24 12:29:56 selftest.go:68: [INFO] [PASS] Storage paths: 1 connected, 4 disconnected +2026/09/24 12:29:56 selftest.go:68: [INFO] [PASS] Git catalog: 56 app definitions found +2026/09/24 12:29:56 selftest.go:68: [INFO] [PASS] Hub connectivity: hub disabled, skipped +2026/09/24 12:29:56 selftest.go:68: [INFO] [PASS] Metrics DB: 8.2 MB +2026/09/24 12:29:56 selftest.go:70: [INFO] ======================================== +2026/09/24 12:29:56 selftest.go:71: [INFO] Self-test complete: 8 passed, 0 warnings, 0 failed +2026/09/24 12:29:56 scheduler.go:206: [INFO] [scheduler] Starting scheduler with 16 jobs +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] scheduler started: periodic=8 daily=8 +2026/09/24 12:29:56 server.go:250: [DEBUG] [web] NewServer: initializing web server v0.269.1 +2026/09/24 12:29:56 server.go:251: [DEBUG] [web] NewServer: backup=true scheduler=true alertMgr=true notifier=true updater=false +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job offbox-backup: next run at 2026-09-24 15:05:00 CEST (waiting 35m3s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job tier2-backup: next run at 2026-09-25 14:20:00 CEST (waiting 23h50m3s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job offsite-abandon-sweep: next run at 2026-09-25 05:10:00 CEST (waiting 14h40m3s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job offsite-proof: next run at 2026-09-25 05:30:00 CEST (waiting 15h0m3s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job offsite-integrity: next run at 2026-09-25 06:00:00 CEST (waiting 15h30m3s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job metrics-prune: next run at 2026-09-25 04:00:00 CEST (waiting 13h30m3s) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job fill-watch: next run at 2026-09-25 03:30:00 CEST (waiting 13h0m3s) +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:29:56 scheduler.go:67: [DEBUG] [scheduler] daily job db-dump: next run at 2026-09-25 13:20:00 CEST (waiting 22h50m3s) +2026/09/24 12:29:56 infra.go:67: [INFO] [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node) +2026/09/24 12:29:56 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/navidrome → [navidrome] +2026/09/24 12:29:56 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/nextcloud → [nextcloud] +2026/09/24 12:29:56 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx → [paperless-ngx] +2026/09/24 12:29:56 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [privatebin, vikunja, wishlist, chaoscrash, chaosoom, chaosoomb, n8n] +2026/09/24 12:29:56 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/romm → [romm] +2026/09/24 12:29:56 backup.go:1119: [INFO] [backup] Found 9 DB dump files across drives +2026/09/24 12:29:56 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/24 12:29:56 server.go:327: [DEBUG] [web] loadTemplates: lang=hu loaded 81 templates, 1886 markers expanded in 49.494187ms +2026/09/24 12:29:56 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 609b09c2b45b n8n n8n n8nio/n8n:2.31.3 +a0b33d0b9958 wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1 +9f47a1f5273b vikunja vikunja vikunja/vikunja:2.3.0 +1adf3ec73c9b romm romm rommapp/romm:5.3.1 +e059bbc146b0 romm-redis romm redis:7-alpine +fc5ed5428e75 romm-db romm mariadb:11.8 +361a05346765 privatebin privatebin privatebin/pdo:2.0.6 +92b06fef42c0 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +5e5c371b82fa paperless-postgres paperless-ngx postgres:16-alpine +0ef3bfdcb4... +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.31.3, not a database) +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database) +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database) +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/24 12:29:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=fc5ed5428e75) +2026/09/24 12:29:56 server.go:327: [DEBUG] [web] loadTemplates: lang=en loaded 81 templates, 1886 markers expanded in 24.412492ms +2026/09/24 12:29:56 server.go:270: [INFO] [web] Auth: using password from settings.json +2026/09/24 12:29:56 server.go:814: [DEBUG] [web] CatchAllMiddleware: controller host=felhom.enkisfelhom.hu +2026/09/24 12:29:56 main.go:1950: [INFO] Web UI listening on :8080 +2026/09/24 12:29:56 handlers.go:3492: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping userdata skeleton +2026/09/24 12:29:56 handlers.go:3492: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping userdata skeleton +2026/09/24 12:29:56 handlers.go:3492: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping userdata skeleton +2026/09/24 12:29:56 handlers.go:3492: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping userdata skeleton +2026/09/24 12:29:56 manager.go:258: [DEBUG] [integrations] ReapplyConfigForTarget: target=filebrowser integrations=0 +2026/09/24 12:29:56 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/24 12:29:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5e5c371b82fa) +2026/09/24 12:29:56 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5480MB avail≈20418MB +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24215076 → total=25898MB avail=20418MB used=5480MB (21.2%) +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=25.5GB avail=39.3GB (37.3%) +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readLoadAvg: raw="2.28 2.56 3.08 8/1543 199" → 1m=2.28 5m=2.56 15m=3.08 +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 67.4°C (hwmon2) +2026/09/24 12:29:56 info.go:13: [DEBUG] [system] GetInfo done in 125ms — mem=5480MB/25898MB (21.2%), rootDisk=25.5GB/68.4GB (37.3%), load=2.28/2.56/3.08, temp=67.4°C (hwmon2), cpu=0.0% +2026/09/24 12:29:56 healthcheck.go:86: [DEBUG] [monitor] Raw values: disk=37.3%, hdd=7.8% (configured=true), mem=21.2% (5480MB/25898MB), cpu=0.0%, temp=67.4°C (hwmon2) +2026/09/24 12:29:56 healthcheck.go:118: [DEBUG] [monitor] SSD disk: OK (37%) +2026/09/24 12:29:56 healthcheck.go:151: [DEBUG] [monitor] Memory: OK (21%) +2026/09/24 12:29:56 healthcheck.go:187: [DEBUG] [monitor] Temperature: OK (67°C) +2026/09/24 12:29:56 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database) +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/24 12:29:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=13e101052288) +2026/09/24 12:29:56 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database) +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database) +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/24 12:29:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/24 12:29:56 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 14 non-DB container(s) +2026/09/24 12:29:56 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/09/24 12:29:56 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:29:56 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:29:56 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:29:57 sync.go:627: [DEBUG] [sync] Running: git reset --hard origin/main +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:29:57 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:29:57 [INFO] [backup] Discovered app data: 11 apps +2026/09/24 12:29:57 healthcheck.go:201: [DEBUG] [monitor] Docker daemon: OK +2026/09/24 12:29:57 healthcheck.go:209: [DEBUG] [monitor] Checking 1 protected containers: [traefik] +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] actualbudget/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] actualbudget/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] adventurelog/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] adventurelog/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] audiobookshelf/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] audiobookshelf/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] bentopdf/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] bentopdf/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] bookstack/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] bookstack/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] calcom/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] calcom/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] calibre-web/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] calibre-web/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] chaoscrash/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] chaoscrash/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] chaosoom/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] chaosoom/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] chaosoomb/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%) +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] chaosoomb/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] claper/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] claper/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] code-server/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] code-server/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] crafty-controller/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] crafty-controller/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%) +2026/09/24 12:29:57 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%) +2026/09/24 12:29:57 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%) +2026/09/24 12:29:57 healthcheck.go:219: [DEBUG] [monitor] All protected containers running +2026/09/24 12:29:57 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:29:57 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:29:57 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:29:57 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:29:57 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:29:57 healthcheck.go:238: [INFO] [monitor] Health check: status=warn +2026/09/24 12:29:57 healthcheck.go:242: [DEBUG] [monitor] Final status: warn (issues=0, warnings=4, info=4) +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] docmost/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] docmost/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 backup.go:1189: [INFO] [backup] Backup status cache refreshed +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] emby/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] emby/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] ghost/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] ghost/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] gitea/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] gitea/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] glance/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] glance/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] gokapi/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] gokapi/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] grafana/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] grafana/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] gramps-web/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] gramps-web/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] home-assistant/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] home-assistant/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] homebox/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] homebox/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] homepage/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] homepage/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] immich/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] immich/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] jellyfin/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] jellyfin/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] kimai/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] kimai/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] komga/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] komga/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] mealie/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] mealie/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:533: [DEBUG] [sync] n8n: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] n8n/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] n8n/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] navidrome/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] navidrome/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] nextcloud/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] nextcloud/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] onlyoffice/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] onlyoffice/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] opengist/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] opengist/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 handlers.go:3432: [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (5 storage path(s)) +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] outline/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] outline/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] paperless-ngx/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] paperless-ngx/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] papra/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] papra/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] plant-it/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] plant-it/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] privatebin/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] privatebin/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] radarr/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] radarr/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] rallly/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] rallly/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] recipe-importer/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] recipe-importer/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] romm/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] romm/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] seerr/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] seerr/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] sonarr/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] sonarr/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] sparkyfitness/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] sparkyfitness/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] tandoor/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] tandoor/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] termix/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] termix/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] uptime-kuma/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] uptime-kuma/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] vaultwarden/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] vaultwarden/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:533: [DEBUG] [sync] vikunja: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] vikunja/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] vikunja/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] wanderer/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] wanderer/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] wger/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] wger/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] wishlist/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] wishlist/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] zipline/docker-compose.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:440: [DEBUG] [sync] zipline/.felhom.yml: hash match, skipped +2026/09/24 12:29:57 sync.go:263: [INFO] [sync] Catalog sync complete +2026/09/24 12:29:57 sync.go:122: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +2026/09/24 12:29:58 auth.go:123: [DEBUG] [web] auth: rejected GET /api/stacks/chaoscrash from 172.18.0.10:34202 (invalid/expired session) +2026/09/24 12:29:58 auth.go:126: [WARN] [api] Unauthorized request to /api/stacks/chaoscrash from 172.18.0.10:34202 +2026/09/24 12:29:58 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.10:34202 (X-Forwarded-For: 192.168.0.180) +2026/09/24 12:29:58 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session +2026/09/24 12:29:58 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T12:29:58Z, active_sessions=1 +2026/09/24 12:29:58 auth.go:222: [INFO] [web] Login from 172.18.0.10:34202 +2026/09/24 12:29:58 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.10:34202 +2026/09/24 12:29:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.10:34202 +2026/09/24 12:29:58 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.10:34202 (X-Forwarded-For: 192.168.0.180) +2026/09/24 12:29:58 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session +2026/09/24 12:29:58 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T12:29:58Z, active_sessions=2 +2026/09/24 12:29:58 auth.go:222: [INFO] [web] Login from 172.18.0.10:34202 +2026/09/24 12:29:58 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.10:34202 +2026/09/24 12:29:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.10:34202 +2026/09/24 12:29:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaoscrash from 172.18.0.10:34202 +2026/09/24 12:29:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaoscrash from 172.18.0.10:34202 +2026/09/24 12:29:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaoscrash from 172.18.0.10:34202 +2026/09/24 12:29:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaoscrash from 172.18.0.10:34202 +2026/09/24 12:30:01 info.go:13: [DEBUG] [system] CPUCollector: first sample — cpu=47.5% (idle=2069 total=3940) +2026/09/24 12:30:01 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event controller_started (severity info) — further controller_started events are logged at DEBUG only diff --git a/documentation/audits/night-2026-09-24/E/round-09.json b/documentation/audits/night-2026-09-24/E/round-09.json new file mode 100644 index 00000000..d8a20db8 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-09.json @@ -0,0 +1,104 @@ +{ + "round": 9, + "action": "remove_held", + "accident": { + "kind": "controller_killed", + "cmd": "kill -9 of the controller PID (2 of max 3; first 14:09:22)", + "fired_at": "14:29:54", + "result": "killed pid 12752 rc=0" + }, + "app": "chaoscrash", + "started": "2026-09-24T12:29:52Z", + "before": { + "state": { + "deployed": true, + "state": "stopped", + "update_phase": null, + "hold_reason": "Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.", + "hold_kind": "unhealthy_stop", + "update_error": null, + "ladder_steps_left": null + }, + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "page_hu": [ + "Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.", + "Az ügyfélszolgálat foglalkozik ezzel az alkalmazással, ezért az adatai megmaradnak: most csak magát az alkalmazást távolíthatod el." + ], + "page_en": [ + "We stopped the app because it kept crashing (or ran out of memory). Press Start to try again.", + "Support is working on this app, so its data stays: for now you can remove only the app itself." + ] + }, + "held_before": { + "deployed": true, + "hold_kind": "unhealthy_stop", + "hold_reason": "Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod." + }, + "armed_at": "14:29:54", + "armed_why": "2 s after the remove press", + "act": "remove (data + backups) -> 502 {'_raw': 'Bad Gateway'}", + "controller_back_s": 0.2, + "time_to_steady_s": 6.2, + "after": { + "state": { + "deployed": true, + "state": "stopped", + "update_phase": null, + "hold_reason": "Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.", + "hold_kind": "unhealthy_stop", + "update_error": null, + "ladder_steps_left": null + }, + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "page_hu": [ + "Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.", + "Az ügyfélszolgálat foglalkozik ezzel az alkalmazással, ezért az adatai megmaradnak: most csak magát az alkalmazást távolíthatod el." + ], + "page_en": [ + "We stopped the app because it kept crashing (or ran out of memory). Press Start to try again.", + "Support is working on this app, so its data stays: for now you can remove only the app itself." + ] + }, + "readback": null, + "events": [], + "box_log": [ + "2026-09-24T12:29:52Z INFO [api] Remove requested for stack: chaoscrash", + "2026-09-24T12:29:52Z INFO Removing deployed stack: chaoscrash (removeHDDData=true, hddDeclared=false, backupPaths=2)", + "2026-09-24T12:29:56Z INFO [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false)", + "2026-09-24T12:29:56Z INFO [scheduler] Registered periodic job: deadapp-check (every 30s)", + "2026-09-24T12:29:56Z INFO [scheduler] Registered periodic job: backup-cache (every 5m0s)", + "2026-09-24T12:29:56Z INFO [scheduler] Daily job tier2-backup scheduled for 2026-09-25 14:20 CEST", + "2026-09-24T12:29:56Z INFO [scheduler] Daily job offbox-backup scheduled for 2026-09-24 15:05 CEST", + "2026-09-24T12:29:56Z INFO [backup] Found 9 DB dump files across drives", + "2026-09-24T12:29:56Z INFO [backup] Discovered 3 databases", + "2026-09-24T12:29:57Z INFO [backup] Backup status cache refreshed", + "2026-09-24T12:29:58Z WARN [api] Unauthorized request to /api/stacks/chaoscrash from 172.18.0.10:34202" + ], + "undo_copies_left": "0" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-24/E/round-09.log b/documentation/audits/night-2026-09-24/E/round-09.log new file mode 100644 index 00000000..d87d7648 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-09.log @@ -0,0 +1,7 @@ +14:29:52 ==== ROUND 9: remove_held on chaoscrash x controller_killed (2026-09-24T12:29:52Z) +14:29:54 >>> ARMED accident controller_killed (2 s after the remove press) +14:29:56 remove (data + backups) -> 502 {'_raw': 'Bad Gateway'} +14:29:56 + 3.8s chaoscrash: deployed=None state=None updating=None phase=None hold=None +14:29:58 + 6.0s chaoscrash: deployed=True state=stopped updating=False phase=None hold=unhealthy_stop +14:29:58 >>> accident done: {'kind': 'controller_killed', 'cmd': 'kill -9 of the controller PID (2 of max 3; first 14:09:22)', 'fired_at': '14:29:54', 'result': 'killed pid 12752 rc=0'} +14:29:58 + 6.2s chaoscrash: deployed=True state=stopped updating=False phase=None hold=unhealthy_stop diff --git a/documentation/audits/night-2026-09-24/E/round-09.stdout b/documentation/audits/night-2026-09-24/E/round-09.stdout new file mode 100644 index 00000000..e79e7534 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-09.stdout @@ -0,0 +1,8 @@ +14:29:52 ==== ROUND 9: remove_held on chaoscrash x controller_killed (2026-09-24T12:29:52Z) +14:29:54 >>> ARMED accident controller_killed (2 s after the remove press) +14:29:56 remove (data + backups) -> 502 {'_raw': 'Bad Gateway'} +14:29:56 + 3.8s chaoscrash: deployed=None state=None updating=None phase=None hold=None +14:29:58 + 6.0s chaoscrash: deployed=True state=stopped updating=False phase=None hold=unhealthy_stop +14:29:58 >>> accident done: {'kind': 'controller_killed', 'cmd': 'kill -9 of the controller PID (2 of max 3; first 14:09:22)', 'fired_at': '14:29:54', 'result': 'killed pid 12752 rc=0'} +14:29:58 + 6.2s chaoscrash: deployed=True state=stopped updating=False phase=None hold=unhealthy_stop +14:30:06 ==== ROUND 9 END: steady in 6.2s; readback=None; events=[] diff --git a/documentation/audits/night-2026-09-24/E/round-09b-remove-again.txt b/documentation/audits/night-2026-09-24/E/round-09b-remove-again.txt new file mode 100644 index 00000000..b74e435b --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-09b-remove-again.txt @@ -0,0 +1,7 @@ +before: app.yaml +applied-compose.yml +applied-meta +docker-compose.yml +remove again -> 200 {'ok': True, 'data': {'removed': 'chaoscrash', 'volumes_removed': [], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott ne +after: deployed= False hold= None +leftovers: docker-compose.yml diff --git a/documentation/audits/night-2026-09-24/E/round-10-controller-pre.log b/documentation/audits/night-2026-09-24/E/round-10-controller-pre.log new file mode 100644 index 00000000..6517a573 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-10-controller-pre.log @@ -0,0 +1,591 @@ +2026/09/24 12:31:06 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:31:06 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.10:34202 (X-Forwarded-For: 192.168.0.180) +2026/09/24 12:31:06 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session +2026/09/24 12:31:06 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T12:31:06Z, active_sessions=4 +2026/09/24 12:31:06 auth.go:222: [INFO] [web] Login from 172.18.0.10:34202 +2026/09/24 12:31:06 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.10:34202 +2026/09/24 12:31:06 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.10:34202 +2026/09/24 12:31:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/n8n from 172.18.0.10:34202 +2026/09/24 12:31:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/n8n from 172.18.0.10:34202 +2026/09/24 12:31:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/n8n from 172.18.0.10:34202 +2026/09/24 12:31:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/n8n from 172.18.0.10:34202 +2026/09/24 12:31:07 router.go:1065: [INFO] [api] Manual catalog sync requested +2026/09/24 12:31:07 sync.go:208: [INFO] [sync] Starting catalog sync +2026/09/24 12:31:07 sync.go:296: [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-drill.git (branch: main) +2026/09/24 12:31:07 sync.go:298: [DEBUG] [sync] git fetch --depth 1 origin main in /opt/docker/felhom-controller/data/catalog-cache +2026/09/24 12:31:07 sync.go:627: [DEBUG] [sync] Running: git fetch --depth 1 origin main +2026/09/24 12:31:07 sync.go:627: [DEBUG] [sync] Running: git reset --hard origin/main +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] actualbudget/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] actualbudget/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] adventurelog/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] adventurelog/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] audiobookshelf/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] audiobookshelf/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] bentopdf/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] bentopdf/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] bookstack/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] bookstack/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] calcom/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] calcom/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] calibre-web/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] calibre-web/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] chaoscrash/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] chaoscrash/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] chaosoom/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] chaosoom/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] chaosoomb/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] chaosoomb/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] claper/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] claper/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] code-server/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] code-server/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] crafty-controller/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] crafty-controller/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] docmost/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] docmost/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] emby/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] emby/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] ghost/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] ghost/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] gitea/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] gitea/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] glance/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] glance/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] gokapi/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] gokapi/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] grafana/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] grafana/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] gramps-web/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] gramps-web/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] home-assistant/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] home-assistant/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] homebox/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] homebox/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] homepage/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] homepage/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] immich/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] immich/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] jellyfin/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] jellyfin/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] kimai/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] kimai/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] komga/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] komga/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] mealie/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] mealie/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:533: [DEBUG] [sync] n8n: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] n8n/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] n8n/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] navidrome/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] navidrome/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] nextcloud/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] nextcloud/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] onlyoffice/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] onlyoffice/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] opengist/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] opengist/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] outline/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] outline/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] paperless-ngx/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] paperless-ngx/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] papra/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] papra/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] plant-it/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] plant-it/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] privatebin/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] privatebin/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] radarr/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] radarr/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] rallly/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] rallly/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] recipe-importer/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] recipe-importer/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] romm/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] romm/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] seerr/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] seerr/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] sonarr/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] sonarr/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] sparkyfitness/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] sparkyfitness/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] tandoor/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] tandoor/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] termix/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] termix/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] uptime-kuma/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] uptime-kuma/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] vaultwarden/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] vaultwarden/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:533: [DEBUG] [sync] vikunja: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] vikunja/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] vikunja/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] wanderer/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] wanderer/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] wger/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] wger/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] wishlist/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] wishlist/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] zipline/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] zipline/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:263: [INFO] [sync] Catalog sync complete +2026/09/24 12:31:07 router.go:1071: [INFO] [api] Catalog sync completed: Sablonok naprakészek — nincs változás +2026/09/24 12:31:09 router.go:362: [INFO] [api] Manual stack rescan requested +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:31:09 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:31:09 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:31:09 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:31:09 router.go:369: [INFO] [api] Stack rescan completed: 58 stacks found +2026/09/24 12:31:11 router.go:591: [INFO] [api] update requested for stack: n8n +2026/09/24 12:31:11 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5582MB avail≈20316MB +2026/09/24 12:31:11 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24094292 → total=25898MB avail=20316MB used=5582MB (21.6%) +2026/09/24 12:31:11 deploy.go:1250: [INFO] [stacks] Memory check: total=25898MB, reserved=384MB, usable=25514MB, committed_used=1306MB, new_req=512MB, remaining=23696MB +2026/09/24 12:31:11 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%) +2026/09/24 12:31:11 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5532MB avail≈20366MB +2026/09/24 12:31:11 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24152196 → total=25898MB avail=20366MB used=5532MB (21.4%) +2026/09/24 12:31:12 deploy.go:1250: [INFO] [stacks] Memory check: total=25898MB, reserved=384MB, usable=25514MB, committed_used=1306MB, new_req=512MB, remaining=23696MB +2026/09/24 12:31:12 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%) +2026/09/24 12:31:12 update.go:512: [INFO] [stacks] update n8n: accepted — guarded update started +2026/09/24 12:31:12 update.go:1260: [INFO] [stacks] update n8n: phase checking +2026/09/24 12:31:12 update.go:762: [INFO] [stacks] update n8n: ladder — step 1 of 2: n8n=n8nio/n8n:2.31.3 → n8n=n8nio/n8n:2.40.5, from 9e432b2ae70c14e3.yml (probe from .felhom.yml) +2026/09/24 12:31:12 update.go:777: [INFO] [stacks] update n8n: no usable copy on any tier — younger than 1m0s and not older than this install's deploy (2026-09-24T12:00:13Z) (found: Tier 2 (second drive) at 2026-09-24T12:20:05Z (11m0s old); Tier 1 (own recovery unit) at 2026-09-24T12:16:00Z (15m0s old)) — backing up first +2026/09/24 12:31:12 update.go:1260: [INFO] [stacks] update n8n: phase backing-up +2026/09/24 12:31:12 update_guard.go:360: [INFO] [backup] update pre-backup for n8n: starting (DB dump → volume dump → unit capture → Tier 2) +2026/09/24 12:31:12 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%) +2026/09/24 12:31:12 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/24 12:31:12 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 609b09c2b45b n8n n8n n8nio/n8n:2.31.3 +a0b33d0b9958 wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1 +9f47a1f5273b vikunja vikunja vikunja/vikunja:2.3.0 +1adf3ec73c9b romm romm rommapp/romm:5.3.1 +e059bbc146b0 romm-redis romm redis:7-alpine +fc5ed5428e75 romm-db romm mariadb:11.8 +361a05346765 privatebin privatebin privatebin/pdo:2.0.6 +92b06fef42c0 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +5e5c371b82fa paperless-postgres paperless-ngx postgres:16-alpine +0ef3bfdcb4... +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.31.3, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/24 12:31:12 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=fc5ed5428e75) +2026/09/24 12:31:12 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/24 12:31:12 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5e5c371b82fa) +2026/09/24 12:31:12 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/24 12:31:12 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/24 12:31:12 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=13e101052288) +2026/09/24 12:31:12 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/24 12:31:12 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 14 non-DB container(s) +2026/09/24 12:31:12 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/09/24 12:31:12 backup.go:920: [INFO] [backup] Stopping n8n for safe volume dump +2026/09/24 12:31:12 manager.go:1283: [DEBUG] [stacks] StopStack n8n: current state=running deployed=true containers=1 +2026/09/24 12:31:12 manager.go:1286: [INFO] [stacks] Stopping stack: n8n +2026/09/24 12:31:12 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, DOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/24 12:31:12 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/n8n) +2026/09/24 12:31:12 manager.go:1483: [DEBUG] Command completed: docker compose down (took 0.7s) +2026/09/24 12:31:12 manager.go:1295: [INFO] [stacks] Stack n8n stopped successfully (took 0.7s) +2026/09/24 12:31:12 backup.go:796: [DEBUG] [backup] Dumping volume n8n_n8n_data for n8n +2026/09/24 12:31:13 backup.go:821: [INFO] [backup] Volume dump: n8n/n8n_n8n_data → 5.4 MB +2026/09/24 12:31:13 backup.go:930: [INFO] [backup] Restarting n8n after volume dump +2026/09/24 12:31:13 manager.go:1194: [DEBUG] [stacks] StartStack n8n: current state=stopped deployed=true +2026/09/24 12:31:13 manager.go:1197: [INFO] [stacks] Starting stack: n8n +2026/09/24 12:31:13 manager.go:1204: [DEBUG] [stacks] StartStack n8n: prepared 9 env vars for compose +2026/09/24 12:31:13 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, DOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/24 12:31:13 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/n8n) +2026/09/24 12:31:13 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 0.4s) +2026/09/24 12:31:13 manager.go:1212: [INFO] [stacks] Stack n8n started successfully (took 0.4s) +2026/09/24 12:31:13 installed.go:409: [DEBUG] [stacks] installed-images n8n: unchanged (1 service(s)) — app.yaml not rewritten +2026/09/24 12:31:13 update_guard.go:410: [INFO] [backup] update pre-backup for n8n: volume dump OK +2026/09/24 12:31:13 update_guard.go:416: [INFO] [backup] update pre-backup for n8n: recovery unit captured (0 database dump(s)) +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:14 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18567 bytes) +2026/09/24 12:31:14 settings.go:847: [INFO] [settings] Settings saved +2026/09/24 12:31:14 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event crossdrive_completed (severity info) — further crossdrive_completed events are logged at DEBUG only +2026/09/24 12:31:14 tier2.go:425: [INFO] [backup] Tier 2 copied n8n → /mnt/felhom-drives/scratch_hdd/backups/secondary/n8n (5.5 MB, 0 leg(s), 0s) +2026/09/24 12:31:14 update_guard.go:425: [INFO] [backup] update pre-backup for n8n: complete in 1.998s +2026/09/24 12:31:14 update.go:792: [INFO] [stacks] update n8n: precondition met after the backup — Tier 2 (second drive) copy from 2026-09-24T12:31:14Z +2026/09/24 12:31:14 update.go:1260: [INFO] [stacks] update n8n: phase safety-dump +2026/09/24 12:31:14 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/24 12:31:14 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 78107b034c81 n8n n8n n8nio/n8n:2.31.3 +a0b33d0b9958 wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1 +9f47a1f5273b vikunja vikunja vikunja/vikunja:2.3.0 +1adf3ec73c9b romm romm rommapp/romm:5.3.1 +e059bbc146b0 romm-redis romm redis:7-alpine +fc5ed5428e75 romm-db romm mariadb:11.8 +361a05346765 privatebin privatebin privatebin/pdo:2.0.6 +92b06fef42c0 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +5e5c371b82fa paperless-postgres paperless-ngx postgres:16-alpine +0ef3bfdcb4... +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.31.3, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/24 12:31:14 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=fc5ed5428e75) +2026/09/24 12:31:14 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/24 12:31:14 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5e5c371b82fa) +2026/09/24 12:31:14 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/24 12:31:14 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/24 12:31:14 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=13e101052288) +2026/09/24 12:31:14 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/24 12:31:14 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 14 non-DB container(s) +2026/09/24 12:31:14 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/09/24 12:31:14 update_guard.go:476: [INFO] [backup] update safety dump for n8n: the app has no database — nothing to copy (no-op) +2026/09/24 12:31:14 update.go:807: [INFO] [stacks] update n8n: safety dump done (0 file(s)) [] +2026/09/24 12:31:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=25.4 GB avail=39.4GB (37.2%) +2026/09/24 12:31:14 undo.go:273: [INFO] [stacks] update n8n: the undo copy will hold 1 named volume(s), 5.4 MiB +2026/09/24 12:31:14 update.go:1260: [INFO] [stacks] update n8n: phase pinning +2026/09/24 12:31:14 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/n8n — 3 env vars, 0 encrypted, 1 sensitive fields +2026/09/24 12:31:14 [INFO] [stacks] SaveAppConfig: saved config for n8n +2026/09/24 12:31:14 pin.go:373: [INFO] [stacks] update n8n: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/n8n/steps/9e432b2ae70c14e3.yml (n8n=n8nio/n8n:2.40.5) +2026/09/24 12:31:14 update.go:1260: [INFO] [stacks] update n8n: phase pulling +2026/09/24 12:31:14 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/24 12:31:14 manager.go:1463: [DEBUG] Running: docker compose pull (in /opt/docker/stacks/n8n) +2026/09/24 12:31:16 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:31:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:16 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:31:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:16 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:31:16 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:31:16 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:31:16 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, DOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/24 12:31:16 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/n8n) +2026/09/24 12:31:16 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s) +2026/09/24 12:31:16 manager.go:1543: [INFO] [stacks] Stack n8n post-start status: +2026/09/24 12:31:16 manager.go:1546: [INFO] [stacks] n8n n8nio/n8n:2.31.3 running Up 3 seconds (health: starting) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=25.9 GB avail=38.9GB (38.0%) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=25.9 GB avail=38.9GB (38.0%) +2026/09/24 12:31:26 fillwatch.go:256: [INFO] [fillwatch] checked 7 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/09/24 12:31:26 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:31:26 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:31:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:26 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:31:26 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:31:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:26 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:31:26 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:31:26 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:31:36 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:31:36 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:31:36 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:31:36 healthprobe.go:181: [DEBUG] Health probe n8n: API GET :5678/healthz → 200 (4ms) +2026/09/24 12:31:36 healthprobe.go:161: [INFO] Health probes: 1 ok (of 1 probed) +2026/09/24 12:31:46 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:31:46 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:31:46 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:31:46 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5467MB avail≈20431MB +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24230000 → total=25898MB avail=20431MB used=5467MB (21.1%) +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=26.3GB avail=38.5GB (38.5%) +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readLoadAvg: raw="3.68 2.90 3.13 9/1541 1187" → 1m=3.68 5m=2.90 15m=3.13 +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 71.9°C (hwmon2) +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] GetInfo done in 62ms — mem=5467MB/25898MB (21.1%), rootDisk=26.3GB/68.4GB (38.5%), load=3.68/2.90/3.13, temp=71.9°C (hwmon2), cpu=42.5% +2026/09/24 12:31:56 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan +2026/09/24 12:31:56 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting +2026/09/24 12:31:56 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:31:56 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:31:56 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:31:56 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:31:56 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:31:56 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:31:56 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:31:56 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 87ms) +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:31:56 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:31:56 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:32:06 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:32:16 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:32:16 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:32:16 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:32:16 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:32:26 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:32:26 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:32:26 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:32:26 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:32:26 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:32:26 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:32:36 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:32:36 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:32:46 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:32:46 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:32:46 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:32:46 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5564MB avail≈20334MB +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24115496 → total=25898MB avail=20334MB used=5564MB (21.5%) +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=27.0GB avail=37.8GB (39.5%) +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readLoadAvg: raw="4.34 3.22 3.23 5/1535 1337" → 1m=4.34 5m=3.22 15m=3.23 +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 72.0°C (hwmon2) +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] GetInfo done in 61ms — mem=5564MB/25898MB (21.5%), rootDisk=27.0GB/68.4GB (39.5%), load=4.34/3.22/3.23, temp=72.0°C (hwmon2), cpu=44.0% +2026/09/24 12:32:56 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:32:56 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:32:56 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:32:56 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:33:03 manager.go:1483: [DEBUG] Command completed: docker compose pull (took 108.8s) +2026/09/24 12:33:03 update.go:1260: [INFO] [stacks] update n8n: phase copying +2026/09/24 12:33:03 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/24 12:33:03 manager.go:1463: [DEBUG] Running: docker compose stop (in /opt/docker/stacks/n8n) +2026/09/24 12:33:04 manager.go:1483: [DEBUG] Command completed: docker compose stop (took 1.4s) +2026/09/24 12:33:04 update.go:1260: [INFO] [stacks] update n8n: phase copying +2026/09/24 12:33:05 undo.go:423: [INFO] [stacks] update n8n: copied n8n_n8n_data → n8n_n8n_data.pre-update-20260924T123304Z in 500ms +2026/09/24 12:33:05 update.go:1260: [INFO] [stacks] update n8n: phase starting +2026/09/24 12:33:05 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/24 12:33:05 manager.go:1463: [DEBUG] Running: docker compose up -d --remove-orphans (in /opt/docker/stacks/n8n) +2026/09/24 12:33:05 manager.go:1483: [DEBUG] Command completed: docker compose up -d --remove-orphans (took 0.5s) +2026/09/24 12:33:05 update.go:1260: [INFO] [stacks] update n8n: phase verifying +2026/09/24 12:33:06 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:33:06 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:33:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:06 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:33:06 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:33:06 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) diff --git a/documentation/audits/night-2026-09-24/E/round-10-controller.log b/documentation/audits/night-2026-09-24/E/round-10-controller.log new file mode 100644 index 00000000..09817b4f --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-10-controller.log @@ -0,0 +1,1120 @@ +2026/09/24 12:31:06 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:31:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:06 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:31:06 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.10:34202 (X-Forwarded-For: 192.168.0.180) +2026/09/24 12:31:06 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session +2026/09/24 12:31:06 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T12:31:06Z, active_sessions=4 +2026/09/24 12:31:06 auth.go:222: [INFO] [web] Login from 172.18.0.10:34202 +2026/09/24 12:31:06 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.10:34202 +2026/09/24 12:31:06 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.10:34202 +2026/09/24 12:31:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/n8n from 172.18.0.10:34202 +2026/09/24 12:31:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/n8n from 172.18.0.10:34202 +2026/09/24 12:31:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/n8n from 172.18.0.10:34202 +2026/09/24 12:31:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/n8n from 172.18.0.10:34202 +2026/09/24 12:31:07 router.go:1065: [INFO] [api] Manual catalog sync requested +2026/09/24 12:31:07 sync.go:208: [INFO] [sync] Starting catalog sync +2026/09/24 12:31:07 sync.go:296: [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-drill.git (branch: main) +2026/09/24 12:31:07 sync.go:298: [DEBUG] [sync] git fetch --depth 1 origin main in /opt/docker/felhom-controller/data/catalog-cache +2026/09/24 12:31:07 sync.go:627: [DEBUG] [sync] Running: git fetch --depth 1 origin main +2026/09/24 12:31:07 sync.go:627: [DEBUG] [sync] Running: git reset --hard origin/main +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] actualbudget/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] actualbudget/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] adventurelog/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] adventurelog/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] audiobookshelf/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] audiobookshelf/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] bentopdf/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] bentopdf/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] bookstack/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] bookstack/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] calcom/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] calcom/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] calibre-web/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] calibre-web/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] chaoscrash/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] chaoscrash/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] chaosoom/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] chaosoom/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] chaosoomb/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] chaosoomb/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] claper/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] claper/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] code-server/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] code-server/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] crafty-controller/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] crafty-controller/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] docmost/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] docmost/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] emby/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] emby/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] ghost/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] ghost/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] gitea/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] gitea/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] glance/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] glance/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] gokapi/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] gokapi/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] grafana/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] grafana/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] gramps-web/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] gramps-web/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] home-assistant/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] home-assistant/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] homebox/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] homebox/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] homepage/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] homepage/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] immich/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] immich/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] jellyfin/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] jellyfin/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] kimai/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] kimai/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] komga/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] komga/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] mealie/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] mealie/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:533: [DEBUG] [sync] n8n: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] n8n/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] n8n/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] navidrome/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] navidrome/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] nextcloud/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] nextcloud/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] onlyoffice/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] onlyoffice/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] opengist/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] opengist/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] outline/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] outline/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] paperless-ngx/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] paperless-ngx/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] papra/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] papra/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] plant-it/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] plant-it/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] privatebin/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] privatebin/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] radarr/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] radarr/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] rallly/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] rallly/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] recipe-importer/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] recipe-importer/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] romm/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] romm/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] seerr/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] seerr/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] sonarr/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] sonarr/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] sparkyfitness/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] sparkyfitness/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] tandoor/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] tandoor/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] termix/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] termix/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] uptime-kuma/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] uptime-kuma/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] vaultwarden/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] vaultwarden/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:533: [DEBUG] [sync] vikunja: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] vikunja/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] vikunja/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] wanderer/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] wanderer/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] wger/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] wger/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] wishlist/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] wishlist/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] zipline/docker-compose.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:440: [DEBUG] [sync] zipline/.felhom.yml: hash match, skipped +2026/09/24 12:31:07 sync.go:263: [INFO] [sync] Catalog sync complete +2026/09/24 12:31:07 router.go:1071: [INFO] [api] Catalog sync completed: Sablonok naprakészek — nincs változás +2026/09/24 12:31:09 router.go:362: [INFO] [api] Manual stack rescan requested +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:31:09 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:31:09 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:31:09 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:31:09 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:31:09 router.go:369: [INFO] [api] Stack rescan completed: 58 stacks found +2026/09/24 12:31:11 router.go:591: [INFO] [api] update requested for stack: n8n +2026/09/24 12:31:11 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5582MB avail≈20316MB +2026/09/24 12:31:11 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24094292 → total=25898MB avail=20316MB used=5582MB (21.6%) +2026/09/24 12:31:11 deploy.go:1250: [INFO] [stacks] Memory check: total=25898MB, reserved=384MB, usable=25514MB, committed_used=1306MB, new_req=512MB, remaining=23696MB +2026/09/24 12:31:11 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%) +2026/09/24 12:31:11 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5532MB avail≈20366MB +2026/09/24 12:31:11 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24152196 → total=25898MB avail=20366MB used=5532MB (21.4%) +2026/09/24 12:31:12 deploy.go:1250: [INFO] [stacks] Memory check: total=25898MB, reserved=384MB, usable=25514MB, committed_used=1306MB, new_req=512MB, remaining=23696MB +2026/09/24 12:31:12 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%) +2026/09/24 12:31:12 update.go:512: [INFO] [stacks] update n8n: accepted — guarded update started +2026/09/24 12:31:12 update.go:1260: [INFO] [stacks] update n8n: phase checking +2026/09/24 12:31:12 update.go:762: [INFO] [stacks] update n8n: ladder — step 1 of 2: n8n=n8nio/n8n:2.31.3 → n8n=n8nio/n8n:2.40.5, from 9e432b2ae70c14e3.yml (probe from .felhom.yml) +2026/09/24 12:31:12 update.go:777: [INFO] [stacks] update n8n: no usable copy on any tier — younger than 1m0s and not older than this install's deploy (2026-09-24T12:00:13Z) (found: Tier 2 (second drive) at 2026-09-24T12:20:05Z (11m0s old); Tier 1 (own recovery unit) at 2026-09-24T12:16:00Z (15m0s old)) — backing up first +2026/09/24 12:31:12 update.go:1260: [INFO] [stacks] update n8n: phase backing-up +2026/09/24 12:31:12 update_guard.go:360: [INFO] [backup] update pre-backup for n8n: starting (DB dump → volume dump → unit capture → Tier 2) +2026/09/24 12:31:12 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=25.5 GB avail=39.3GB (37.3%) +2026/09/24 12:31:12 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/24 12:31:12 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 609b09c2b45b n8n n8n n8nio/n8n:2.31.3 +a0b33d0b9958 wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1 +9f47a1f5273b vikunja vikunja vikunja/vikunja:2.3.0 +1adf3ec73c9b romm romm rommapp/romm:5.3.1 +e059bbc146b0 romm-redis romm redis:7-alpine +fc5ed5428e75 romm-db romm mariadb:11.8 +361a05346765 privatebin privatebin privatebin/pdo:2.0.6 +92b06fef42c0 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +5e5c371b82fa paperless-postgres paperless-ngx postgres:16-alpine +0ef3bfdcb4... +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.31.3, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/24 12:31:12 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=fc5ed5428e75) +2026/09/24 12:31:12 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/24 12:31:12 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5e5c371b82fa) +2026/09/24 12:31:12 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/24 12:31:12 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/24 12:31:12 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=13e101052288) +2026/09/24 12:31:12 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/24 12:31:12 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/24 12:31:12 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 14 non-DB container(s) +2026/09/24 12:31:12 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/09/24 12:31:12 backup.go:920: [INFO] [backup] Stopping n8n for safe volume dump +2026/09/24 12:31:12 manager.go:1283: [DEBUG] [stacks] StopStack n8n: current state=running deployed=true containers=1 +2026/09/24 12:31:12 manager.go:1286: [INFO] [stacks] Stopping stack: n8n +2026/09/24 12:31:12 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, DOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/24 12:31:12 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/n8n) +2026/09/24 12:31:12 manager.go:1483: [DEBUG] Command completed: docker compose down (took 0.7s) +2026/09/24 12:31:12 manager.go:1295: [INFO] [stacks] Stack n8n stopped successfully (took 0.7s) +2026/09/24 12:31:12 backup.go:796: [DEBUG] [backup] Dumping volume n8n_n8n_data for n8n +2026/09/24 12:31:13 backup.go:821: [INFO] [backup] Volume dump: n8n/n8n_n8n_data → 5.4 MB +2026/09/24 12:31:13 backup.go:930: [INFO] [backup] Restarting n8n after volume dump +2026/09/24 12:31:13 manager.go:1194: [DEBUG] [stacks] StartStack n8n: current state=stopped deployed=true +2026/09/24 12:31:13 manager.go:1197: [INFO] [stacks] Starting stack: n8n +2026/09/24 12:31:13 manager.go:1204: [DEBUG] [stacks] StartStack n8n: prepared 9 env vars for compose +2026/09/24 12:31:13 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, DOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/24 12:31:13 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/n8n) +2026/09/24 12:31:13 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 0.4s) +2026/09/24 12:31:13 manager.go:1212: [INFO] [stacks] Stack n8n started successfully (took 0.4s) +2026/09/24 12:31:13 installed.go:409: [DEBUG] [stacks] installed-images n8n: unchanged (1 service(s)) — app.yaml not rewritten +2026/09/24 12:31:13 update_guard.go:410: [INFO] [backup] update pre-backup for n8n: volume dump OK +2026/09/24 12:31:13 update_guard.go:416: [INFO] [backup] update pre-backup for n8n: recovery unit captured (0 database dump(s)) +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:13 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:14 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18567 bytes) +2026/09/24 12:31:14 settings.go:847: [INFO] [settings] Settings saved +2026/09/24 12:31:14 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event crossdrive_completed (severity info) — further crossdrive_completed events are logged at DEBUG only +2026/09/24 12:31:14 tier2.go:425: [INFO] [backup] Tier 2 copied n8n → /mnt/felhom-drives/scratch_hdd/backups/secondary/n8n (5.5 MB, 0 leg(s), 0s) +2026/09/24 12:31:14 update_guard.go:425: [INFO] [backup] update pre-backup for n8n: complete in 1.998s +2026/09/24 12:31:14 update.go:792: [INFO] [stacks] update n8n: precondition met after the backup — Tier 2 (second drive) copy from 2026-09-24T12:31:14Z +2026/09/24 12:31:14 update.go:1260: [INFO] [stacks] update n8n: phase safety-dump +2026/09/24 12:31:14 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/24 12:31:14 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 78107b034c81 n8n n8n n8nio/n8n:2.31.3 +a0b33d0b9958 wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1 +9f47a1f5273b vikunja vikunja vikunja/vikunja:2.3.0 +1adf3ec73c9b romm romm rommapp/romm:5.3.1 +e059bbc146b0 romm-redis romm redis:7-alpine +fc5ed5428e75 romm-db romm mariadb:11.8 +361a05346765 privatebin privatebin privatebin/pdo:2.0.6 +92b06fef42c0 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +5e5c371b82fa paperless-postgres paperless-ngx postgres:16-alpine +0ef3bfdcb4... +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.31.3, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/24 12:31:14 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=fc5ed5428e75) +2026/09/24 12:31:14 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/24 12:31:14 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5e5c371b82fa) +2026/09/24 12:31:14 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/24 12:31:14 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/24 12:31:14 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=13e101052288) +2026/09/24 12:31:14 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/24 12:31:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/24 12:31:14 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 14 non-DB container(s) +2026/09/24 12:31:14 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/09/24 12:31:14 update_guard.go:476: [INFO] [backup] update safety dump for n8n: the app has no database — nothing to copy (no-op) +2026/09/24 12:31:14 update.go:807: [INFO] [stacks] update n8n: safety dump done (0 file(s)) [] +2026/09/24 12:31:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=25.4 GB avail=39.4GB (37.2%) +2026/09/24 12:31:14 undo.go:273: [INFO] [stacks] update n8n: the undo copy will hold 1 named volume(s), 5.4 MiB +2026/09/24 12:31:14 update.go:1260: [INFO] [stacks] update n8n: phase pinning +2026/09/24 12:31:14 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/n8n — 3 env vars, 0 encrypted, 1 sensitive fields +2026/09/24 12:31:14 [INFO] [stacks] SaveAppConfig: saved config for n8n +2026/09/24 12:31:14 pin.go:373: [INFO] [stacks] update n8n: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/n8n/steps/9e432b2ae70c14e3.yml (n8n=n8nio/n8n:2.40.5) +2026/09/24 12:31:14 update.go:1260: [INFO] [stacks] update n8n: phase pulling +2026/09/24 12:31:14 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/24 12:31:14 manager.go:1463: [DEBUG] Running: docker compose pull (in /opt/docker/stacks/n8n) +2026/09/24 12:31:16 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:31:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:16 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:31:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:16 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:31:16 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:31:16 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:31:16 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, DOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/24 12:31:16 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/n8n) +2026/09/24 12:31:16 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s) +2026/09/24 12:31:16 manager.go:1543: [INFO] [stacks] Stack n8n post-start status: +2026/09/24 12:31:16 manager.go:1546: [INFO] [stacks] n8n n8nio/n8n:2.31.3 running Up 3 seconds (health: starting) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=25.9 GB avail=38.9GB (38.0%) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:31:26 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=25.9 GB avail=38.9GB (38.0%) +2026/09/24 12:31:26 fillwatch.go:256: [INFO] [fillwatch] checked 7 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/09/24 12:31:26 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:31:26 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:31:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:26 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:31:26 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:31:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:26 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:31:26 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:31:26 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:31:36 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:31:36 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:31:36 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:31:36 healthprobe.go:181: [DEBUG] Health probe n8n: API GET :5678/healthz → 200 (4ms) +2026/09/24 12:31:36 healthprobe.go:161: [INFO] Health probes: 1 ok (of 1 probed) +2026/09/24 12:31:46 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:31:46 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:46 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:31:46 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:31:46 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5467MB avail≈20431MB +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24230000 → total=25898MB avail=20431MB used=5467MB (21.1%) +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=26.3GB avail=38.5GB (38.5%) +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readLoadAvg: raw="3.68 2.90 3.13 9/1541 1187" → 1m=3.68 5m=2.90 15m=3.13 +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 71.9°C (hwmon2) +2026/09/24 12:31:56 info.go:13: [DEBUG] [system] GetInfo done in 62ms — mem=5467MB/25898MB (21.1%), rootDisk=26.3GB/68.4GB (38.5%), load=3.68/2.90/3.13, temp=71.9°C (hwmon2), cpu=42.5% +2026/09/24 12:31:56 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan +2026/09/24 12:31:56 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting +2026/09/24 12:31:56 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:31:56 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:31:56 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:31:56 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:31:56 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:31:56 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:31:56 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:31:56 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:31:56 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 87ms) +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:31:56 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:31:56 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:31:56 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:32:06 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:32:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:06 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:32:16 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:16 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:32:16 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:32:16 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:32:16 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:32:26 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:32:26 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:32:26 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:32:26 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:26 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:32:26 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:32:26 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:32:36 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:32:36 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:36 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:32:46 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:32:46 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:46 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:32:46 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:32:46 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5564MB avail≈20334MB +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24115496 → total=25898MB avail=20334MB used=5564MB (21.5%) +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=27.0GB avail=37.8GB (39.5%) +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readLoadAvg: raw="4.34 3.22 3.23 5/1535 1337" → 1m=4.34 5m=3.22 15m=3.23 +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 72.0°C (hwmon2) +2026/09/24 12:32:56 info.go:13: [DEBUG] [system] GetInfo done in 61ms — mem=5564MB/25898MB (21.5%), rootDisk=27.0GB/68.4GB (39.5%), load=4.34/3.22/3.23, temp=72.0°C (hwmon2), cpu=44.0% +2026/09/24 12:32:56 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:32:56 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:32:56 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:32:56 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:32:56 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:32:56 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:33:03 manager.go:1483: [DEBUG] Command completed: docker compose pull (took 108.8s) +2026/09/24 12:33:03 update.go:1260: [INFO] [stacks] update n8n: phase copying +2026/09/24 12:33:03 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/24 12:33:03 manager.go:1463: [DEBUG] Running: docker compose stop (in /opt/docker/stacks/n8n) +2026/09/24 12:33:04 manager.go:1483: [DEBUG] Command completed: docker compose stop (took 1.4s) +2026/09/24 12:33:04 update.go:1260: [INFO] [stacks] update n8n: phase copying +2026/09/24 12:33:05 undo.go:423: [INFO] [stacks] update n8n: copied n8n_n8n_data → n8n_n8n_data.pre-update-20260924T123304Z in 500ms +2026/09/24 12:33:05 update.go:1260: [INFO] [stacks] update n8n: phase starting +2026/09/24 12:33:05 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/24 12:33:05 manager.go:1463: [DEBUG] Running: docker compose up -d --remove-orphans (in /opt/docker/stacks/n8n) +2026/09/24 12:33:05 manager.go:1483: [DEBUG] Command completed: docker compose up -d --remove-orphans (took 0.5s) +2026/09/24 12:33:05 update.go:1260: [INFO] [stacks] update n8n: phase verifying +2026/09/24 12:33:06 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:33:06 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:33:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:06 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:06 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:33:06 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.8:8999: connect: connection refused +2026/09/24 12:33:06 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:33:10 main.go:1931: [INFO] Received signal terminated, shutting down... +2026/09/24 12:33:11 scheduler.go:67: [DEBUG] [scheduler] daily job tier2-backup: context cancelled, stopping +2026/09/24 12:33:11 scheduler.go:67: [DEBUG] [scheduler] daily job metrics-prune: context cancelled, stopping +2026/09/24 12:33:11 main.go:1955: [INFO] felhom-controller stopped +2026/09/24 12:33:11 scheduler.go:67: [DEBUG] [scheduler] daily job offsite-proof: context cancelled, stopping +2026/09/24 12:33:11 scheduler.go:232: [INFO] [scheduler] Stopping scheduler +2026/09/24 12:33:11 scheduler.go:67: [DEBUG] [scheduler] daily job offsite-abandon-sweep: context cancelled, stopping +2026/09/24 12:33:11 scheduler.go:67: [DEBUG] [scheduler] daily job offbox-backup: context cancelled, stopping +2026/09/24 12:33:11 scheduler.go:67: [DEBUG] [scheduler] daily job offsite-integrity: context cancelled, stopping +2026/09/24 12:33:11 scheduler.go:67: [DEBUG] [scheduler] daily job fill-watch: context cancelled, stopping +2026/09/24 12:33:11 scheduler.go:67: [DEBUG] [scheduler] daily job db-dump: context cancelled, stopping +2026/09/24 12:33:11 scheduler.go:245: [INFO] [scheduler] All jobs stopped +2026/09/24 12:33:20 main.go:335: [INFO] felhom-controller 0.269.1 starting (customer: demo-hp, domain: enkisfelhom.hu) +2026/09/24 12:33:20 settings.go:709: [INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json +2026/09/24 12:33:20 settings.go:1671: [DEBUG] [settings] AutoDiscoverStoragePaths discovered=[/mnt/felhom-drives/scratch_hdd/userdata/navidrome /mnt/felhom-drives/scratch_hdd/userdata/nextcloud /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx /mnt/felhom-drives/scratch_hdd/userdata/romm] fallback="" existing=5 +2026/09/24 12:33:20 main.go:361: [INFO] Encryption key loaded from /opt/docker/felhom-controller/data/encryption.key +2026/09/24 12:33:20 manager.go:328: [INFO] [stacks] Using compose command: docker compose +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:33:20 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:33:20 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:33:20 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:33:20 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:33:21 deploy.go:1104: [DEBUG] [stacks] InjectMissingFields: checking 10 stacks +2026/09/24 12:33:21 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack navidrome — 3 deploy fields, 3 existing env vars +2026/09/24 12:33:21 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack chaosoomb — 2 deploy fields, 2 existing env vars +2026/09/24 12:33:21 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack n8n — 3 deploy fields, 3 existing env vars +2026/09/24 12:33:21 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack paperless-ngx — 8 deploy fields, 8 existing env vars +2026/09/24 12:33:21 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack nextcloud — 7 deploy fields, 7 existing env vars +2026/09/24 12:33:21 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack romm — 6 deploy fields, 6 existing env vars +2026/09/24 12:33:21 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack wishlist — 2 deploy fields, 2 existing env vars +2026/09/24 12:33:21 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack chaosoom — 2 deploy fields, 2 existing env vars +2026/09/24 12:33:21 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack privatebin — 2 deploy fields, 2 existing env vars +2026/09/24 12:33:21 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack vikunja — 3 deploy fields, 3 existing env vars +2026/09/24 12:33:21 deploy.go:1189: [INFO] [stacks] InjectMissingFields: processed 10 stacks +2026/09/24 12:33:21 manager.go:467: [DEBUG] [stacks] MigrateEncryption: checking 10 deployed stacks for plaintext sensitive values +2026/09/24 12:33:21 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "vikunja" (1 sensitive fields) +2026/09/24 12:33:21 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "n8n" (1 sensitive fields) +2026/09/24 12:33:21 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "paperless-ngx" (3 sensitive fields) +2026/09/24 12:33:21 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "nextcloud" (3 sensitive fields) +2026/09/24 12:33:21 manager.go:487: [DEBUG] [stacks] MigrateEncryption: checking stack "romm" (3 sensitive fields) +2026/09/24 12:33:21 manager.go:511: [INFO] [stacks] Encryption migration: no stacks needed migration +2026/09/24 12:33:21 desiredstate.go:137: [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded (state ambiguous — legacy boot behaviour retained) +2026/09/24 12:33:21 installed.go:537: [INFO] [stacks] installed-images backfill: 0 app(s) recorded, 10 already had a record, 0 left unrecorded (could not be observed completely — unknown, which renders nothing) +2026/09/24 12:33:21 pin.go:290: [INFO] [stacks] pin adoption: 0 pinned, 10 already pinned, 0 left unpinned (0 not completely observed, 0 running something the template no longer offers) +2026/09/24 12:33:21 undo.go:656: [INFO] [stacks] applied-meta backfill (R-646): recorded 0 []; skipped 0 [] — not current with the catalog, their pinned version's .felhom.yml is no longer on the box +2026/09/24 12:33:21 sync.go:117: [INFO] [sync] Starting catalog sync (repo: https://gitea.dooplex.hu/admin/app-catalog-drill.git, interval: 15m0s) +2026/09/24 12:33:21 update.go:1335: [WARN] [stacks] update recovery: n8n was interrupted in verifying (started 2026-09-24T12:31:12Z) — the new version may have run; marking it Updating and RESUMING the health wait +2026/09/24 12:33:21 main.go:495: [WARN] [update] 1 interrupted update(s) will resume after the backup side is wired: [n8n] +2026/09/24 12:33:21 info.go:13: [DEBUG] [system] CPUCollector.Start: sampleRate=5s +2026/09/24 12:33:21 sync.go:208: [INFO] [sync] Starting catalog sync +2026/09/24 12:33:21 sync.go:296: [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-drill.git (branch: main) +2026/09/24 12:33:21 sync.go:298: [DEBUG] [sync] git fetch --depth 1 origin main in /opt/docker/felhom-controller/data/catalog-cache +2026/09/24 12:33:21 sync.go:627: [DEBUG] [sync] Running: git fetch --depth 1 origin main +2026/09/24 12:33:21 main.go:515: [INFO] Metrics store opened at /opt/docker/felhom-controller/data/metrics.db +2026/09/24 12:33:21 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:33:21 main.go:527: [INFO] Metrics collector started (60s interval) +2026/09/24 12:33:21 restore_record_wiring.go:35: [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false) +2026/09/24 12:33:21 main.go:576: [WARN] [update] resumed 1 interrupted update(s) +2026/09/24 12:33:21 notifier.go:81: [INFO] Notifier disabled (hub not configured) +2026/09/24 12:33:21 main.go:692: [INFO] [mailrelay] app-email shim unavailable (no hub configured or disabled in config) +2026/09/24 12:33:21 scheduler.go:102: [INFO] [scheduler] Registered periodic job: status-refresh (every 10s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="status-refresh" interval=10s totalJobs=1 +2026/09/24 12:33:21 scheduler.go:102: [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="stack-scan" interval=2m0s totalJobs=2 +2026/09/24 12:33:21 scheduler.go:102: [INFO] [scheduler] Registered periodic job: health-probes (every 10s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="health-probes" interval=10s totalJobs=3 +2026/09/24 12:33:21 scheduler.go:102: [INFO] [scheduler] Registered periodic job: system-health (every 5m0s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="system-health" interval=5m0s totalJobs=4 +2026/09/24 12:33:21 scheduler.go:102: [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="deadapp-check" interval=30s totalJobs=5 +2026/09/24 12:33:21 scheduler.go:102: [INFO] [scheduler] Registered periodic job: ring-spill (every 30s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="ring-spill" interval=30s totalJobs=6 +2026/09/24 12:33:21 scheduler.go:132: [INFO] [scheduler] Daily job db-dump scheduled for 2026-09-25 13:20 CEST +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="db-dump" schedule="13:20" nextRun=2026-09-25T13:20:00+02:00 totalJobs=7 +2026/09/24 12:33:21 scheduler.go:102: [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="offsite-credential-retry" interval=5m0s totalJobs=8 +2026/09/24 12:33:21 scheduler.go:102: [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="backup-cache" interval=5m0s totalJobs=9 +2026/09/24 12:33:21 scheduler.go:132: [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-09-25 14:20 CEST +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="tier2-backup" schedule="14:20" nextRun=2026-09-25T14:20:00+02:00 totalJobs=10 +2026/09/24 12:33:21 scheduler.go:132: [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-09-24 15:05 CEST +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offbox-backup" schedule="15:05" nextRun=2026-09-24T15:05:00+02:00 totalJobs=11 +2026/09/24 12:33:21 scheduler.go:132: [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-09-25 05:10 CEST +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offsite-abandon-sweep" schedule="05:10" nextRun=2026-09-25T05:10:00+02:00 totalJobs=12 +2026/09/24 12:33:21 scheduler.go:132: [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-09-25 06:00 CEST +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offsite-integrity" schedule="06:00" nextRun=2026-09-25T06:00:00+02:00 totalJobs=13 +2026/09/24 12:33:21 scheduler.go:132: [INFO] [scheduler] Daily job offsite-proof scheduled for 2026-09-25 05:30 CEST +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="offsite-proof" schedule="05:30" nextRun=2026-09-25T05:30:00+02:00 totalJobs=14 +2026/09/24 12:33:21 scheduler.go:132: [INFO] [scheduler] Daily job metrics-prune scheduled for 2026-09-25 04:00 CEST +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="metrics-prune" schedule="04:00" nextRun=2026-09-25T04:00:00+02:00 totalJobs=15 +2026/09/24 12:33:21 scheduler.go:132: [INFO] [scheduler] Daily job fill-watch scheduled for 2026-09-25 03:30 CEST +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job registered: name="fill-watch" schedule="03:30" nextRun=2026-09-25T03:30:00+02:00 totalJobs=16 +2026/09/24 12:33:21 selftest.go:50: [INFO] ========== Startup Self-Test ========== +2026/09/24 12:33:21 update.go:1382: [INFO] [stacks] update n8n: resuming after a controller restart — `up -d` then the health wait +2026/09/24 12:33:21 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, N8N_ENCRYPTION_KEY, SUBDOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/24 12:33:21 manager.go:1463: [DEBUG] Running: docker compose up -d --remove-orphans (in /opt/docker/stacks/n8n) +2026/09/24 12:33:21 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈4098MB avail≈21800MB +2026/09/24 12:33:21 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=25853544 → total=25898MB avail=21800MB used=4098MB (15.8%) +2026/09/24 12:33:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=26.8GB avail=38.0GB (39.2%) +2026/09/24 12:33:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:33:21 info.go:13: [DEBUG] [system] readLoadAvg: raw="4.31 3.30 3.25 31/1490 122" → 1m=4.31 5m=3.30 15m=3.25 +2026/09/24 12:33:21 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:33:21 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 80.9°C (hwmon2) +2026/09/24 12:33:21 info.go:13: [DEBUG] [system] GetInfo done in 184ms — mem=4098MB/25898MB (15.8%), rootDisk=26.8GB/68.4GB (39.2%), load=4.31/3.30/3.25, temp=80.9°C (hwmon2), cpu=0.0% +2026/09/24 12:33:21 selftest.go:68: [INFO] [PASS] Docker socket: reachable (v29.8.0) +2026/09/24 12:33:21 selftest.go:68: [INFO] [PASS] Stacks directory: /opt/docker/stacks +2026/09/24 12:33:21 selftest.go:68: [INFO] [PASS] Data directory: /opt/docker/felhom-controller/data (writable) +2026/09/24 12:33:21 selftest.go:68: [INFO] [PASS] System data path: /mnt/sys_drive +2026/09/24 12:33:21 selftest.go:68: [INFO] [PASS] Storage paths: 1 connected, 4 disconnected +2026/09/24 12:33:21 selftest.go:68: [INFO] [PASS] Git catalog: 56 app definitions found +2026/09/24 12:33:21 selftest.go:68: [INFO] [PASS] Hub connectivity: hub disabled, skipped +2026/09/24 12:33:21 selftest.go:68: [INFO] [PASS] Metrics DB: 8.2 MB +2026/09/24 12:33:21 selftest.go:70: [INFO] ======================================== +2026/09/24 12:33:21 selftest.go:71: [INFO] Self-test complete: 8 passed, 0 warnings, 0 failed +2026/09/24 12:33:21 scheduler.go:206: [INFO] [scheduler] Starting scheduler with 16 jobs +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] scheduler started: periodic=8 daily=8 +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job metrics-prune: next run at 2026-09-25 04:00:00 CEST (waiting 13h26m39s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job offsite-integrity: next run at 2026-09-25 06:00:00 CEST (waiting 15h26m39s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job offsite-proof: next run at 2026-09-25 05:30:00 CEST (waiting 14h56m39s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job db-dump: next run at 2026-09-25 13:20:00 CEST (waiting 22h46m39s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job tier2-backup: next run at 2026-09-25 14:20:00 CEST (waiting 23h46m39s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job offbox-backup: next run at 2026-09-24 15:05:00 CEST (waiting 31m39s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job offsite-abandon-sweep: next run at 2026-09-25 05:10:00 CEST (waiting 14h36m39s) +2026/09/24 12:33:21 scheduler.go:67: [DEBUG] [scheduler] daily job fill-watch: next run at 2026-09-25 03:30:00 CEST (waiting 12h56m39s) +2026/09/24 12:33:21 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:33:21 server.go:250: [DEBUG] [web] NewServer: initializing web server v0.269.1 +2026/09/24 12:33:21 server.go:251: [DEBUG] [web] NewServer: backup=true scheduler=true alertMgr=true notifier=true updater=false +2026/09/24 12:33:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/navidrome → [navidrome] +2026/09/24 12:33:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/nextcloud → [nextcloud] +2026/09/24 12:33:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx → [paperless-ngx] +2026/09/24 12:33:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [privatebin, vikunja, wishlist, chaosoom, chaosoomb, n8n] +2026/09/24 12:33:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/romm → [romm] +2026/09/24 12:33:21 backup.go:1119: [INFO] [backup] Found 9 DB dump files across drives +2026/09/24 12:33:21 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/24 12:33:21 manager.go:1483: [DEBUG] Command completed: docker compose up -d --remove-orphans (took 0.5s) +2026/09/24 12:33:21 update.go:1260: [INFO] [stacks] update n8n: phase verifying +2026/09/24 12:33:21 server.go:327: [DEBUG] [web] loadTemplates: lang=hu loaded 81 templates, 1886 markers expanded in 114.432128ms +2026/09/24 12:33:21 server.go:327: [DEBUG] [web] loadTemplates: lang=en loaded 81 templates, 1886 markers expanded in 53.897017ms +2026/09/24 12:33:21 server.go:270: [INFO] [web] Auth: using password from settings.json +2026/09/24 12:33:21 server.go:814: [DEBUG] [web] CatchAllMiddleware: controller host=felhom.enkisfelhom.hu +2026/09/24 12:33:21 main.go:1950: [INFO] Web UI listening on :8080 +2026/09/24 12:33:21 handlers.go:3492: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping userdata skeleton +2026/09/24 12:33:21 handlers.go:3492: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping userdata skeleton +2026/09/24 12:33:21 handlers.go:3492: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping userdata skeleton +2026/09/24 12:33:21 handlers.go:3492: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping userdata skeleton +2026/09/24 12:33:21 manager.go:258: [DEBUG] [integrations] ReapplyConfigForTarget: target=filebrowser integrations=0 +2026/09/24 12:33:21 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 517b7ba21f80 n8n n8n n8nio/n8n:2.40.5 +a0b33d0b9958 wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1 +9f47a1f5273b vikunja vikunja vikunja/vikunja:2.3.0 +1adf3ec73c9b romm romm rommapp/romm:5.3.1 +e059bbc146b0 romm-redis romm redis:7-alpine +fc5ed5428e75 romm-db romm mariadb:11.8 +361a05346765 privatebin privatebin privatebin/pdo:2.0.6 +92b06fef42c0 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +5e5c371b82fa paperless-postgres paperless-ngx postgres:16-alpine +0ef3bfdcb4... +2026/09/24 12:33:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.40.5, not a database) +2026/09/24 12:33:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database) +2026/09/24 12:33:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database) +2026/09/24 12:33:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/24 12:33:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/24 12:33:21 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=fc5ed5428e75) +2026/09/24 12:33:21 infra.go:67: [INFO] [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node) +2026/09/24 12:33:22 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/24 12:33:22 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/24 12:33:22 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/24 12:33:22 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5e5c371b82fa) +2026/09/24 12:33:22 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/24 12:33:22 sync.go:627: [DEBUG] [sync] Running: git reset --hard origin/main +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] actualbudget/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] actualbudget/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] adventurelog/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] adventurelog/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] audiobookshelf/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] audiobookshelf/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] bentopdf/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] bentopdf/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] bookstack/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] bookstack/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] calcom/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] calcom/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] calibre-web/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] calibre-web/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] chaoscrash/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] chaoscrash/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈4198MB avail≈21700MB +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=25734588 → total=25898MB avail=21700MB used=4198MB (16.2%) +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=26.8GB avail=38.0GB (39.2%) +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] chaosoom/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="4.31 3.30 3.25 16/1506 263" → 1m=4.31 5m=3.30 15m=3.25 +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] chaosoom/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] chaosoomb/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] chaosoomb/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 80.2°C (hwmon2) +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] GetInfo done in 806ms — mem=4198MB/25898MB (16.2%), rootDisk=26.8GB/68.4GB (39.2%), load=4.31/3.30/3.25, temp=80.2°C (hwmon2), cpu=0.0% +2026/09/24 12:33:22 healthcheck.go:86: [DEBUG] [monitor] Raw values: disk=39.2%, hdd=7.8% (configured=true), mem=16.2% (4198MB/25898MB), cpu=0.0%, temp=80.2°C (hwmon2) +2026/09/24 12:33:22 healthcheck.go:118: [DEBUG] [monitor] SSD disk: OK (39%) +2026/09/24 12:33:22 healthcheck.go:151: [DEBUG] [monitor] Memory: OK (16%) +2026/09/24 12:33:22 healthcheck.go:179: [WARN] [monitor] Temperature threshold breached: 80°C (limit: 75°C) +2026/09/24 12:33:22 healthcheck.go:182: [DEBUG] [monitor] Temperature: WARN (80°C >= 75°C) +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] claper/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] claper/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] code-server/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] code-server/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] crafty-controller/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] crafty-controller/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] docmost/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] docmost/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] emby/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] emby/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] ghost/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] ghost/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] gitea/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] gitea/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] glance/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] glance/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] gokapi/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] gokapi/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] grafana/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] grafana/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] gramps-web/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] gramps-web/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] home-assistant/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] home-assistant/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] homebox/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] homebox/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] homepage/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] homepage/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] immich/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] immich/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] jellyfin/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] jellyfin/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 handlers.go:3432: [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (5 storage path(s)) +2026/09/24 12:33:22 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/24 12:33:22 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/24 12:33:22 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database) +2026/09/24 12:33:22 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/24 12:33:22 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=13e101052288) +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] kimai/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] kimai/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] komga/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] komga/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] mealie/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] mealie/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:533: [DEBUG] [sync] n8n: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] n8n/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] n8n/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] navidrome/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] navidrome/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] nextcloud/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] nextcloud/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] onlyoffice/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] onlyoffice/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] opengist/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] opengist/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] outline/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] outline/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] paperless-ngx/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] paperless-ngx/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] papra/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] papra/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] plant-it/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] plant-it/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] privatebin/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] privatebin/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] radarr/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] radarr/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] rallly/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] rallly/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] recipe-importer/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] recipe-importer/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] romm/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] romm/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] seerr/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] seerr/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] sonarr/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] sonarr/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] sparkyfitness/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] sparkyfitness/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] tandoor/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] tandoor/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] termix/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] termix/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] uptime-kuma/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] uptime-kuma/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] vaultwarden/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] vaultwarden/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:533: [DEBUG] [sync] vikunja: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] vikunja/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] vikunja/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] wanderer/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] wanderer/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] wger/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] wger/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] wishlist/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] wishlist/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] zipline/docker-compose.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:440: [DEBUG] [sync] zipline/.felhom.yml: hash match, skipped +2026/09/24 12:33:22 sync.go:263: [INFO] [sync] Catalog sync complete +2026/09/24 12:33:22 sync.go:122: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +2026/09/24 12:33:22 healthcheck.go:201: [DEBUG] [monitor] Docker daemon: OK +2026/09/24 12:33:22 healthcheck.go:209: [DEBUG] [monitor] Checking 1 protected containers: [traefik] +2026/09/24 12:33:22 auth.go:123: [DEBUG] [web] auth: rejected GET /api/stacks/n8n from 172.18.0.2:51556 (invalid/expired session) +2026/09/24 12:33:22 auth.go:126: [WARN] [api] Unauthorized request to /api/stacks/n8n from 172.18.0.2:51556 +2026/09/24 12:33:22 healthcheck.go:219: [DEBUG] [monitor] All protected containers running +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:33:22 healthcheck.go:238: [INFO] [monitor] Health check: status=warn +2026/09/24 12:33:22 healthcheck.go:242: [DEBUG] [monitor] Final status: warn (issues=0, warnings=5, info=3) +2026/09/24 12:33:22 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/24 12:33:22 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database) +2026/09/24 12:33:22 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database) +2026/09/24 12:33:22 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/24 12:33:22 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/24 12:33:22 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 14 non-DB container(s) +2026/09/24 12:33:22 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/09/24 12:33:22 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.2:51556 (X-Forwarded-For: 192.168.0.180) +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:33:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:33:22 [INFO] [backup] Discovered app data: 10 apps +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=26.8 GB avail=38.0GB (39.2%) +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=26.8 GB avail=38.0GB (39.2%) +2026/09/24 12:33:22 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=26.8 GB avail=38.0GB (39.2%) +2026/09/24 12:33:22 backup.go:1189: [INFO] [backup] Backup status cache refreshed +2026/09/24 12:33:22 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session +2026/09/24 12:33:22 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T12:33:22Z, active_sessions=1 +2026/09/24 12:33:22 auth.go:222: [INFO] [web] Login from 172.18.0.2:51556 +2026/09/24 12:33:22 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.2:51556 +2026/09/24 12:33:22 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.2:51556 +2026/09/24 12:33:26 info.go:13: [DEBUG] [system] CPUCollector: first sample — cpu=96.3% (idle=146 total=3979) +2026/09/24 12:33:26 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event controller_started (severity info) — further controller_started events are logged at DEBUG only +2026/09/24 12:33:31 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:33:31 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 3 targets (0 skipped not due, 0 skipped no container) +2026/09/24 12:33:31 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:33:31 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:33:31 healthprobe.go:181: [DEBUG] Health probe navidrome: API GET :4533/ping → 200 (3ms) +2026/09/24 12:33:31 healthprobe.go:181: [DEBUG] Health probe privatebin: HTTP GET :8080/ → 200 (14ms) +2026/09/24 12:33:31 healthprobe.go:159: [WARN] Health probes: 2 ok, 1 unhealthy (of 3 probed) +2026/09/24 12:33:36 main.go:2356: [DEBUG] [bootrecon] boot window: fleet still changing (10 app(s)) — resampling +2026/09/24 12:33:37 healthprobe.go:181: [DEBUG] Health probe n8n: API GET :5678/healthz → 200 (157ms) +2026/09/24 12:33:37 update.go:928: [INFO] [stacks] update n8n: healthy after 16s (the app's health check passed) +2026/09/24 12:33:37 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/n8n — 3 env vars, 0 encrypted, 1 sensitive fields +2026/09/24 12:33:37 [INFO] [stacks] SaveAppConfig: saved config for n8n +2026/09/24 12:33:37 installed.go:420: [INFO] [stacks] installed-images n8n: recorded 1 service(s) (n8n=n8nio/n8n:2.40.5 (sha256:9f693fd55655…)) +2026/09/24 12:33:37 update.go:936: [INFO] [stacks] update n8n: DONE in 2m25s +2026/09/24 12:33:37 router.go:1065: [INFO] [api] Manual catalog sync requested +2026/09/24 12:33:39 router.go:362: [INFO] [api] Manual stack rescan requested +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:33:39 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:33:39 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:33:39 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:33:39 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:33:39 router.go:369: [INFO] [api] Stack rescan completed: 58 stacks found +2026/09/24 12:33:41 main.go:2356: [DEBUG] [bootrecon] boot window: fleet still changing (10 app(s)) — resampling +2026/09/24 12:33:41 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:33:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 4s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:33:41 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 3 targets (3 skipped not due, 0 skipped no container) +2026/09/24 12:33:41 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:33:41 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:33:41 healthprobe.go:181: [DEBUG] Health probe wishlist: HTTP GET :3000/ → 307 (8ms) +2026/09/24 12:33:41 healthprobe.go:181: [DEBUG] Health probe nextcloud: API GET :80/status.php → 200 (42ms) +2026/09/24 12:33:41 healthprobe.go:159: [WARN] Health probes: 2 ok, 1 unhealthy (of 3 probed) +2026/09/24 12:33:41 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.2:51556 (X-Forwarded-For: 192.168.0.180) +2026/09/24 12:33:42 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session +2026/09/24 12:33:42 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T12:33:42Z, active_sessions=2 +2026/09/24 12:33:42 auth.go:222: [INFO] [web] Login from 172.18.0.2:51556 +2026/09/24 12:33:42 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.2:51556 +2026/09/24 12:33:42 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.2:51556 +2026/09/24 12:33:42 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/n8n from 172.18.0.2:51556 +2026/09/24 12:33:42 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/n8n from 172.18.0.2:51556 +2026/09/24 12:33:42 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/n8n from 172.18.0.2:51556 +2026/09/24 12:33:42 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/n8n from 172.18.0.2:51556 diff --git a/documentation/audits/night-2026-09-24/E/round-10.json b/documentation/audits/night-2026-09-24/E/round-10.json new file mode 100644 index 00000000..fee45165 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-10.json @@ -0,0 +1,182 @@ +{ + "round": 10, + "action": "ladder_step", + "accident": { + "kind": "docker_restart", + "cmd": "pct exec 9202 -- systemctl restart docker", + "fired_at": "14:33:09", + "result": "rc=0" + }, + "app": "n8n", + "started": "2026-09-24T12:31:06Z", + "before": { + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "hold_kind": null, + "update_error": null, + "ladder_steps_left": 2 + }, + "badges": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — 1 napja" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — 1 day ago" + } + ] + }, + "page_hu": [ + "Az ügyfélszolgálat foglalkozik ezzel az alkalmazással, ezért az adatai megmaradnak: most csak magát az alkalmazást távolíthatod el." + ], + "page_en": [ + "Support is working on this app, so its data stays: for now you can remove only the app itself." + ] + }, + "steps_left_before": 2, + "pinned_before": { + "n8n": "n8nio/n8n:2.31.3" + }, + "act": "Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}", + "armed_at": "14:33:05", + "armed_why": "at starting", + "phases": [ + [ + 4.9, + "backing-up" + ], + [ + 7.0, + "safety-dump" + ], + [ + 7.6, + "pulling" + ], + [ + 116.4, + "copying" + ], + [ + 118.5, + "starting" + ], + [ + 122.8, + "verifying" + ], + [ + 150.5, + "done" + ] + ], + "steps_left_after": 1, + "pinned_after": { + "n8n": "n8nio/n8n:2.40.5" + }, + "controller_back_s": 0.2, + "time_to_steady_s": 155.0, + "after": { + "state": { + "deployed": true, + "state": "running", + "update_phase": "done", + "hold_reason": null, + "hold_kind": null, + "update_error": null, + "ladder_steps_left": 1 + }, + "badges": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — 1 napja" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — 1 day ago" + } + ] + }, + "page_hu": [ + "Temperature high: 80°C (hwmon2) Rendszermonitor →", + "+ 1 további figyelmeztetés →", + "Az ügyfélszolgálat foglalkozik ezzel az alkalmazással, ezért az adatai megmaradnak: most csak magát az alkalmazást távolíthatod el." + ], + "page_en": [ + "Temperature high: 80°C (hwmon2) System monitor →", + "+ 1 more warning →", + "Support is working on this app, so its data stays: for now you can remove only the app itself." + ] + }, + "readback": { + "A": true + }, + "events": [ + "2026-09-24T12:33:26Z controller_started (info)" + ], + "box_log": [ + "2026-09-24T12:31:11Z INFO [api] update requested for stack: n8n", + "2026-09-24T12:31:12Z INFO [stacks] update n8n: accepted — guarded update started", + "2026-09-24T12:31:12Z INFO [stacks] update n8n: phase checking", + "2026-09-24T12:31:12Z INFO [stacks] update n8n: ladder — step 1 of 2: n8n=n8nio/n8n:2.31.3 → n8n=n8nio/n8n:2.40.5, from 9e432b2ae70c14e3.yml (probe from .felhom.yml)", + "2026-09-24T12:31:12Z INFO [stacks] update n8n: no usable copy on any tier — younger than 1m0s and not older than this install's deploy (2026-09-24T12:00:13Z) (found: Tier 2 (second drive) at 2026-09-24T12:20:05Z (11m0s old); Tier 1 (own recovery unit) at 2026-09-24T12:16:00Z (15m0s old)) — backing up first", + "2026-09-24T12:31:12Z INFO [stacks] update n8n: phase backing-up", + "2026-09-24T12:31:12Z INFO [backup] update pre-backup for n8n: starting (DB dump → volume dump → unit capture → Tier 2)", + "2026-09-24T12:31:12Z INFO [backup] Discovered 3 databases", + "2026-09-24T12:31:12Z INFO [backup] Stopping n8n for safe volume dump", + "2026-09-24T12:31:12Z INFO [stacks] Stopping stack: n8n", + "2026-09-24T12:31:12Z INFO [stacks] Stack n8n stopped successfully (took 0.7s)", + "2026-09-24T12:31:13Z INFO [backup] Volume dump: n8n/n8n_n8n_data → 5.4 MB", + "2026-09-24T12:31:13Z INFO [backup] Restarting n8n after volume dump", + "2026-09-24T12:31:13Z INFO [stacks] Starting stack: n8n", + "2026-09-24T12:31:13Z INFO [stacks] Stack n8n started successfully (took 0.4s)", + "2026-09-24T12:31:13Z INFO [backup] update pre-backup for n8n: volume dump OK", + "2026-09-24T12:31:13Z INFO [backup] update pre-backup for n8n: recovery unit captured (0 database dump(s))", + "2026-09-24T12:31:14Z INFO [backup] Tier 2 copied n8n → /mnt/felhom-drives/scratch_hdd/backups/secondary/n8n (5.5 MB, 0 leg(s), 0s)", + "2026-09-24T12:31:14Z INFO [backup] update pre-backup for n8n: complete in 1.998s", + "2026-09-24T12:31:14Z INFO [stacks] update n8n: precondition met after the backup — Tier 2 (second drive) copy from 2026-09-24T12:31:14Z", + "2026-09-24T12:31:14Z INFO [stacks] update n8n: phase safety-dump", + "2026-09-24T12:31:14Z INFO [backup] Discovered 3 databases", + "2026-09-24T12:31:14Z INFO [backup] update safety dump for n8n: the app has no database — nothing to copy (no-op)", + "2026-09-24T12:31:14Z INFO [stacks] update n8n: safety dump done (0 file(s)) []", + "2026-09-24T12:31:14Z INFO [stacks] update n8n: the undo copy will hold 1 named volume(s), 5.4 MiB", + "2026-09-24T12:31:14Z INFO [stacks] update n8n: phase pinning", + "2026-09-24T12:31:14Z INFO [stacks] update n8n: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/n8n/steps/9e432b2ae70c14e3.yml (n8n=n8nio/n8n:2.40.5)", + "2026-09-24T12:31:14Z INFO [stacks] update n8n: phase pulling", + "2026-09-24T12:31:16Z INFO [stacks] Stack n8n post-start status:", + "2026-09-24T12:31:16Z INFO [stacks] n8n n8nio/n8n:2.31.3 running Up 3 seconds (health: starting)", + "2026-09-24T12:33:03Z INFO [stacks] update n8n: phase copying", + "2026-09-24T12:33:04Z INFO [stacks] update n8n: phase copying", + "2026-09-24T12:33:05Z INFO [stacks] update n8n: copied n8n_n8n_data → n8n_n8n_data.pre-update-20260924T123304Z in 500ms", + "2026-09-24T12:33:05Z INFO [stacks] update n8n: phase starting", + "2026-09-24T12:33:05Z INFO [stacks] update n8n: phase verifying", + "2026-09-24T12:33:21Z WARN [stacks] update recovery: n8n was interrupted in verifying (started 2026-09-24T12:31:12Z) — the new version may have run; marking it Updating and RESUMING the health wait", + "2026-09-24T12:33:21Z WARN [update] 1 interrupted update(s) will resume after the backup side is wired: [n8n]", + "2026-09-24T12:33:21Z INFO [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false)", + "2026-09-24T12:33:21Z INFO [scheduler] Registered periodic job: deadapp-check (every 30s)", + "2026-09-24T12:33:21Z INFO [scheduler] Registered periodic job: backup-cache (every 5m0s)", + "2026-09-24T12:33:21Z INFO [scheduler] Daily job tier2-backup scheduled for 2026-09-25 14:20 CEST", + "2026-09-24T12:33:21Z INFO [scheduler] Daily job offbox-backup scheduled for 2026-09-24 15:05 CEST", + "2026-09-24T12:33:21Z INFO [stacks] update n8n: resuming after a controller restart — `up -d` then the health wait", + "2026-09-24T12:33:21Z INFO [backup] Found 9 DB dump files across drives", + "2026-09-24T12:33:21Z INFO [stacks] update n8n: phase verifying", + "2026-09-24T12:33:22Z WARN [monitor] Temperature threshold breached: 80°C (limit: 75°C)", + "2026-09-24T12:33:22Z WARN [api] Unauthorized request to /api/stacks/n8n from 172.18.0.2:51556", + "2026-09-24T12:33:22Z INFO [backup] Discovered 3 databases", + "2026-09-24T12:33:22Z INFO [backup] Backup status cache refreshed", + "2026-09-24T12:33:37Z INFO [stacks] update n8n: healthy after 16s (the app's health check passed)", + "2026-09-24T12:33:37Z INFO [stacks] installed-images n8n: recorded 1 service(s) (n8n=n8nio/n8n:2.40.5 (sha256:9f693fd55655…))", + "2026-09-24T12:33:37Z INFO [stacks] update n8n: DONE in 2m25s" + ], + "undo_copies_left": "0" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-24/E/round-10.log b/documentation/audits/night-2026-09-24/E/round-10.log new file mode 100644 index 00000000..d093f747 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-10.log @@ -0,0 +1,14 @@ +14:31:06 ==== ROUND 10: ladder_step on n8n x docker_restart (2026-09-24T12:31:06Z) +14:31:12 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +14:31:12 + 4.9s phase=backing-up +14:31:14 + 7.0s phase=safety-dump +14:31:14 + 7.6s phase=pulling +14:33:03 + 116.4s phase=copying +14:33:05 + 118.5s phase=starting +14:33:05 >>> ARMED accident docker_restart (at starting) +14:33:09 + 122.8s phase=verifying +14:33:37 + 150.5s phase=done +14:33:37 + 150.5s n8n: deployed=True state=running updating=False phase=done hold=None +14:33:41 >>> accident done: {'kind': 'docker_restart', 'cmd': 'pct exec 9202 -- systemctl restart docker', 'fired_at': '14:33:09', 'result': 'rc=0'} +14:33:42 + 155.0s n8n: deployed=True state=running updating=False phase=done hold=None +14:33:42 n8n: login as the seeded owner http=200 ok=True diff --git a/documentation/audits/night-2026-09-24/E/round-10.stdout b/documentation/audits/night-2026-09-24/E/round-10.stdout new file mode 100644 index 00000000..580070ce --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-10.stdout @@ -0,0 +1,15 @@ +14:31:06 ==== ROUND 10: ladder_step on n8n x docker_restart (2026-09-24T12:31:06Z) +14:31:12 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +14:31:12 + 4.9s phase=backing-up +14:31:14 + 7.0s phase=safety-dump +14:31:14 + 7.6s phase=pulling +14:33:03 + 116.4s phase=copying +14:33:05 + 118.5s phase=starting +14:33:05 >>> ARMED accident docker_restart (at starting) +14:33:09 + 122.8s phase=verifying +14:33:37 + 150.5s phase=done +14:33:37 + 150.5s n8n: deployed=True state=running updating=False phase=done hold=None +14:33:41 >>> accident done: {'kind': 'docker_restart', 'cmd': 'pct exec 9202 -- systemctl restart docker', 'fired_at': '14:33:09', 'result': 'rc=0'} +14:33:42 + 155.0s n8n: deployed=True state=running updating=False phase=done hold=None +14:33:42 n8n: login as the seeded owner http=200 ok=True +14:33:51 ==== ROUND 10 END: steady in 155.0s; readback={'A': True}; events=['2026-09-24T12:33:26Z controller_started (info)'] diff --git a/documentation/audits/night-2026-09-24/E/round-11-controller-pre.log b/documentation/audits/night-2026-09-24/E/round-11-controller-pre.log new file mode 100644 index 00000000..e5bb638d --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-11-controller-pre.log @@ -0,0 +1,512 @@ +2026/09/24 12:34:07 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.2:51556 (X-Forwarded-For: 192.168.0.180) +2026/09/24 12:34:07 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session +2026/09/24 12:34:07 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T12:34:07Z, active_sessions=3 +2026/09/24 12:34:07 auth.go:222: [INFO] [web] Login from 172.18.0.2:51556 +2026/09/24 12:34:07 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.2:51556 +2026/09/24 12:34:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.2:51556 +2026/09/24 12:34:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:34:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:34:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:34:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:34:08 router.go:1065: [INFO] [api] Manual catalog sync requested +2026/09/24 12:34:08 sync.go:208: [INFO] [sync] Starting catalog sync +2026/09/24 12:34:08 sync.go:296: [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-drill.git (branch: main) +2026/09/24 12:34:08 sync.go:298: [DEBUG] [sync] git fetch --depth 1 origin main in /opt/docker/felhom-controller/data/catalog-cache +2026/09/24 12:34:08 sync.go:627: [DEBUG] [sync] Running: git fetch --depth 1 origin main +2026/09/24 12:34:08 sync.go:627: [DEBUG] [sync] Running: git reset --hard origin/main +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] actualbudget/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] actualbudget/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] adventurelog/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] adventurelog/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] audiobookshelf/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] audiobookshelf/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] bentopdf/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] bentopdf/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] bookstack/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] bookstack/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] calcom/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] calcom/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] calibre-web/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] calibre-web/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] chaoscrash/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] chaoscrash/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] chaosoom/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] chaosoom/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] chaosoomb/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] chaosoomb/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] claper/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] claper/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] code-server/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] code-server/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] crafty-controller/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] crafty-controller/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] docmost/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] docmost/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] emby/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] emby/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] ghost/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] ghost/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] gitea/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] gitea/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] glance/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] glance/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] gokapi/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] gokapi/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] grafana/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] grafana/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] gramps-web/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] gramps-web/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] home-assistant/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] home-assistant/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] homebox/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] homebox/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] homepage/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] homepage/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] immich/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] immich/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] jellyfin/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] jellyfin/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] kimai/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] kimai/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] komga/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] komga/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] mealie/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] mealie/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:533: [DEBUG] [sync] n8n: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] n8n/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] n8n/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] navidrome/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] navidrome/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:533: [DEBUG] [sync] nextcloud: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] nextcloud/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:416: [INFO] [sync] Updated nextcloud/.felhom.yml +2026/09/24 12:34:08 sync.go:565: [DEBUG] [sync] nextcloud/.felhom.yml: src=75a58fb7c5e1639e, dst=75a58fb7c5e1639e (changed) +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] onlyoffice/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] onlyoffice/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] opengist/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] opengist/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] outline/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] outline/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] paperless-ngx/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] paperless-ngx/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] papra/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] papra/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] plant-it/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] plant-it/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] privatebin/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] privatebin/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] radarr/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] radarr/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] rallly/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] rallly/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] recipe-importer/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] recipe-importer/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] romm/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] romm/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] seerr/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] seerr/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] sonarr/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] sonarr/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] sparkyfitness/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] sparkyfitness/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] tandoor/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] tandoor/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] termix/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] termix/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] uptime-kuma/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] uptime-kuma/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] vaultwarden/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] vaultwarden/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:533: [DEBUG] [sync] vikunja: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] vikunja/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] vikunja/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] wanderer/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] wanderer/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] wger/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] wger/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] wishlist/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] wishlist/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] zipline/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] zipline/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:34:08 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:34:08 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:34:08 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:34:08 sync.go:244: [DEBUG] [sync] Post-sync hook: triggering missing field injection for 1 stack(s): [nextcloud] +2026/09/24 12:34:08 deploy.go:1104: [DEBUG] [stacks] InjectMissingFields: checking 1 stacks +2026/09/24 12:34:08 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack nextcloud — 7 deploy fields, 7 existing env vars +2026/09/24 12:34:08 deploy.go:1189: [INFO] [stacks] InjectMissingFields: processed 1 stacks +2026/09/24 12:34:08 sync.go:263: [INFO] [sync] Catalog sync complete +2026/09/24 12:34:08 router.go:1071: [INFO] [api] Catalog sync completed: Sablonok frissítve — frissítve: nextcloud +2026/09/24 12:34:10 router.go:362: [INFO] [api] Manual stack rescan requested +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:34:10 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:34:10 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:34:10 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:34:11 router.go:369: [INFO] [api] Stack rescan completed: 58 stacks found +2026/09/24 12:34:11 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:34:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:34:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 34s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:11 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:34:11 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:34:11 healthprobe.go:181: [DEBUG] Health probe romm: HTTP GET :8080/ → 200 (3ms) +2026/09/24 12:34:11 healthprobe.go:181: [DEBUG] Health probe paperless-ngx: HTTP GET :8000/ → 302 (11ms) +2026/09/24 12:34:11 healthprobe.go:161: [INFO] Health probes: 2 ok (of 2 probed) +2026/09/24 12:34:13 router.go:591: [INFO] [api] update requested for stack: nextcloud +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5305MB avail≈20593MB +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24422308 → total=25898MB avail=20593MB used=5305MB (20.5%) +2026/09/24 12:34:13 deploy.go:1250: [INFO] [stacks] Memory check: total=25898MB, reserved=384MB, usable=25514MB, committed_used=1562MB, new_req=256MB, remaining=23696MB +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=26.8 GB avail=38.0GB (39.3%) +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5300MB avail≈20598MB +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24428424 → total=25898MB avail=20598MB used=5300MB (20.5%) +2026/09/24 12:34:13 deploy.go:1250: [INFO] [stacks] Memory check: total=25898MB, reserved=384MB, usable=25514MB, committed_used=1562MB, new_req=256MB, remaining=23696MB +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=26.8 GB avail=38.0GB (39.3%) +2026/09/24 12:34:13 update.go:512: [INFO] [stacks] update nextcloud: accepted — guarded update started +2026/09/24 12:34:13 update.go:1260: [INFO] [stacks] update nextcloud: phase checking +2026/09/24 12:34:13 update.go:762: [INFO] [stacks] update nextcloud: ladder — the installed version nextcloud=nextcloud:34.0.4-apache, nextcloud-db=mariadb:12.3, nextcloud-redis=redis:7-alpine matches no update_ladder entry (2 entries) — an app older than the ladder has no record to climb; the catalog's current definition +2026/09/24 12:34:13 update.go:777: [INFO] [stacks] update nextcloud: no usable copy on any tier — younger than 1m0s and not older than this install's deploy (2026-09-24T12:07:07Z) (found: Tier 2 (second drive) at 2026-09-24T12:20:03Z (14m0s old); Tier 1 (own recovery unit) at 2026-09-24T12:15:03Z (19m0s old)) — backing up first +2026/09/24 12:34:13 update.go:1260: [INFO] [stacks] update nextcloud: phase backing-up +2026/09/24 12:34:13 update_guard.go:360: [INFO] [backup] update pre-backup for nextcloud: starting (DB dump → volume dump → unit capture → Tier 2) +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:34:13 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/24 12:34:13 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 517b7ba21f80 n8n n8n n8nio/n8n:2.40.5 +a0b33d0b9958 wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1 +9f47a1f5273b vikunja vikunja vikunja/vikunja:2.3.0 +1adf3ec73c9b romm romm rommapp/romm:5.3.1 +e059bbc146b0 romm-redis romm redis:7-alpine +fc5ed5428e75 romm-db romm mariadb:11.8 +361a05346765 privatebin privatebin privatebin/pdo:2.0.6 +92b06fef42c0 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +5e5c371b82fa paperless-postgres paperless-ngx postgres:16-alpine +0ef3bfdcb4... +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.40.5, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/24 12:34:13 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=fc5ed5428e75) +2026/09/24 12:34:13 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/24 12:34:13 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5e5c371b82fa) +2026/09/24 12:34:13 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/24 12:34:13 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/24 12:34:13 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=13e101052288) +2026/09/24 12:34:13 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/24 12:34:13 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 14 non-DB container(s) +2026/09/24 12:34:13 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/09/24 12:34:13 dbdump.go:266: [DEBUG] DumpOne: starting dump for container=nextcloud-db, stack=nextcloud, dbType=mariadb, dest=/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/nextcloud-mariadb.sql +2026/09/24 12:34:13 dbdump.go:321: [DEBUG] DumpOne: mariadb-dump command: docker exec 13e101052288 mariadb-dump -u root -p*** --single-transaction --routines --triggers nextcloud +2026/09/24 12:34:14 dbdump.go:405: [DEBUG] DumpOne: completed nextcloud-db → nextcloud-mariadb.sql (size=855.4 KB, valid=true, tables=131, duration=686ms) +2026/09/24 12:34:14 dbdump.go:410: [INFO] [backup] DB dump: nextcloud-db → nextcloud-mariadb.sql (855.4 KB, 686ms, 131 tables) +2026/09/24 12:34:14 update_guard.go:399: [INFO] [backup] update pre-backup for nextcloud: database dump OK (nextcloud-db, 855.4 KB) +2026/09/24 12:34:14 backup.go:920: [INFO] [backup] Stopping nextcloud for safe volume dump +2026/09/24 12:34:14 manager.go:1283: [DEBUG] [stacks] StopStack nextcloud: current state=running deployed=true containers=3 +2026/09/24 12:34:14 manager.go:1286: [INFO] [stacks] Stopping stack: nextcloud +2026/09/24 12:34:14 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:34:14 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/nextcloud) +2026/09/24 12:34:16 manager.go:1483: [DEBUG] Command completed: docker compose down (took 1.8s) +2026/09/24 12:34:16 manager.go:1295: [INFO] [stacks] Stack nextcloud stopped successfully (took 1.8s) +2026/09/24 12:34:16 backup.go:796: [DEBUG] [backup] Dumping volume nextcloud_nextcloud_html for nextcloud +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5217MB avail≈20681MB +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24525888 → total=25898MB avail=20681MB used=5217MB (20.1%) +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=26.8GB avail=38.0GB (39.3%) +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=74.1GB avail=816.0GB (7.9%) +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readLoadAvg: raw="5.95 4.00 3.50 12/1454 768" → 1m=5.95 5m=4.00 15m=3.50 +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 78.4°C (hwmon2) +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] GetInfo done in 92ms — mem=5217MB/25898MB (20.1%), rootDisk=26.8GB/68.4GB (39.3%), load=5.95/4.00/3.50, temp=78.4°C (hwmon2), cpu=58.9% +2026/09/24 12:34:21 backup.go:821: [INFO] [backup] Volume dump: nextcloud/nextcloud_nextcloud_html → 761.6 MB +2026/09/24 12:34:21 backup.go:796: [DEBUG] [backup] Dumping volume nextcloud_nextcloud_redis_data for nextcloud +2026/09/24 12:34:21 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:34:21 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:34:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 44s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:21 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:34:21 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:34:21 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:34:21 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:34:21 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:34:22 backup.go:821: [INFO] [backup] Volume dump: nextcloud/nextcloud_nextcloud_redis_data → 277.0 KB +2026/09/24 12:34:22 backup.go:796: [DEBUG] [backup] Dumping volume nextcloud_nextcloud_db_data for nextcloud +2026/09/24 12:34:24 backup.go:821: [INFO] [backup] Volume dump: nextcloud/nextcloud_nextcloud_db_data → 164.9 MB +2026/09/24 12:34:24 backup.go:930: [INFO] [backup] Restarting nextcloud after volume dump +2026/09/24 12:34:24 manager.go:1194: [DEBUG] [stacks] StartStack nextcloud: current state=stopped deployed=true +2026/09/24 12:34:24 manager.go:1197: [INFO] [stacks] Starting stack: nextcloud +2026/09/24 12:34:24 manager.go:1204: [DEBUG] [stacks] StartStack nextcloud: prepared 14 env vars for compose +2026/09/24 12:34:24 manager.go:410: [INFO] [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping ensure (held by drive gate) +2026/09/24 12:34:24 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:34:24 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/nextcloud) +2026/09/24 12:34:30 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 6.5s) +2026/09/24 12:34:30 manager.go:1212: [INFO] [stacks] Stack nextcloud started successfully (took 6.5s) +2026/09/24 12:34:30 installed.go:409: [DEBUG] [stacks] installed-images nextcloud: unchanged (3 service(s)) — app.yaml not rewritten +2026/09/24 12:34:30 update_guard.go:410: [INFO] [backup] update pre-backup for nextcloud: volume dump OK +2026/09/24 12:34:30 recovery_unit.go:224: [INFO] [backup] Recovery unit captured for nextcloud → /mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/09/24 12:34:30 update_guard.go:416: [INFO] [backup] update pre-backup for nextcloud: recovery unit captured (1 database dump(s)) +2026/09/24 12:34:30 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=26.8 GB avail=38.0GB (39.3%) +2026/09/24 12:34:31 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 54s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:31 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:31 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:34:33 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:34:33 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/nextcloud) +2026/09/24 12:34:33 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s) +2026/09/24 12:34:33 manager.go:1543: [INFO] [stacks] Stack nextcloud post-start status: +2026/09/24 12:34:33 manager.go:1546: [INFO] [stacks] nextcloud nextcloud:34.0.4-apache@sha256:a5ace30c695afe48c2c406e940ee7886a81e13fa382e57cd68b2416d1a66914c running Up 3 seconds (health: starting) +2026/09/24 12:34:33 manager.go:1546: [INFO] [stacks] nextcloud-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy) +2026/09/24 12:34:33 manager.go:1546: [INFO] [stacks] nextcloud-redis redis:7-alpine@sha256:c35af3bbcef51a62c8bae5a9a563c6f1b60d7ebaea4cb5a3ccbcc157580ae098 running Up 9 seconds (healthy) +2026/09/24 12:34:34 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18567 bytes) +2026/09/24 12:34:34 settings.go:847: [INFO] [settings] Settings saved +2026/09/24 12:34:34 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event crossdrive_completed (severity info) — further crossdrive_completed events are logged at DEBUG only +2026/09/24 12:34:34 tier2.go:425: [INFO] [backup] Tier 2 copied nextcloud → /mnt/sys_drive/felhom-data/backups/secondary/nextcloud (1.0 GB, 1 leg(s), 3s) [SSD: state-only] +2026/09/24 12:34:34 update_guard.go:425: [INFO] [backup] update pre-backup for nextcloud: complete in 20.653s +2026/09/24 12:34:34 update.go:792: [INFO] [stacks] update nextcloud: precondition met after the backup — Tier 2 (second drive) copy from 2026-09-24T12:34:34Z +2026/09/24 12:34:36 update.go:1260: [INFO] [stacks] update nextcloud: phase safety-dump +2026/09/24 12:34:36 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/24 12:34:36 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 101ac4b42fdc nextcloud nextcloud nextcloud:34.0.4-apache +061bb4d3beb2 nextcloud-db nextcloud mariadb:12.3 +08859b9002cf nextcloud-redis nextcloud redis:7-alpine +517b7ba21f80 n8n n8n n8nio/n8n:2.40.5 +a0b33d0b9958 wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1 +9f47a1f5273b vikunja vikunja vikunja/vikunja:2.3.0 +1adf3ec73c9b romm romm rommapp/romm:5.3.1 +e059bbc146b0 romm-redis romm redis:7-alpine +fc5ed5428e75 romm-db romm mariadb:11.8 +361a05346765 privatebin privatebin privatebin/pdo:2.0.6 +92b06f... +2026/09/24 12:34:36 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database) +2026/09/24 12:34:36 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=061bb4d3beb2) +2026/09/24 12:34:37 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.40.5, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/24 12:34:37 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=fc5ed5428e75) +2026/09/24 12:34:37 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/24 12:34:37 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5e5c371b82fa) +2026/09/24 12:34:37 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/24 12:34:37 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/24 12:34:37 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 14 non-DB container(s) +2026/09/24 12:34:37 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/09/24 12:34:37 dbdump.go:266: [DEBUG] DumpOne: starting dump for container=nextcloud-db, stack=nextcloud, dbType=mariadb, dest=/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/pre-restore-20260924T123437Z-nextcloud-mariadb.sql +2026/09/24 12:34:37 dbdump.go:321: [DEBUG] DumpOne: mariadb-dump command: docker exec 061bb4d3beb2 mariadb-dump -u root -p*** --single-transaction --routines --triggers nextcloud +2026/09/24 12:34:37 dbdump.go:405: [DEBUG] DumpOne: completed nextcloud-db → pre-restore-20260924T123437Z-nextcloud-mariadb.sql (size=855.4 KB, valid=true, tables=131, duration=632ms) +2026/09/24 12:34:37 dbdump.go:410: [INFO] [backup] DB dump: nextcloud-db → pre-restore-20260924T123437Z-nextcloud-mariadb.sql (855.4 KB, 632ms, 131 tables) +2026/09/24 12:34:37 offbox_reconstitute.go:255: [INFO] [offbox] nextcloud: pre-restore safety dump written → pre-restore-20260924T123437Z-nextcloud-mariadb.sql (855.4 KB) +2026/09/24 12:34:37 update_guard.go:478: [INFO] [backup] update safety dump for nextcloud: 1 file(s) [/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/pre-restore-20260924T123437Z-nextcloud-mariadb.sql] +2026/09/24 12:34:37 update.go:807: [INFO] [stacks] update nextcloud: safety dump done (1 file(s)) [/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/pre-restore-20260924T123437Z-nextcloud-mariadb.sql] +2026/09/24 12:34:39 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=26.8 GB avail=38.0GB (39.3%) +2026/09/24 12:34:39 undo.go:273: [INFO] [stacks] update nextcloud: the undo copy will hold 3 named volume(s), 928.8 MiB +2026/09/24 12:34:39 update.go:1260: [INFO] [stacks] update nextcloud: phase pinning +2026/09/24 12:34:39 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/nextcloud — 7 env vars, 0 encrypted, 3 sensitive fields +2026/09/24 12:34:39 [INFO] [stacks] SaveAppConfig: saved config for nextcloud +2026/09/24 12:34:39 pin.go:373: [INFO] [stacks] update nextcloud: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/nextcloud/docker-compose.yml (nextcloud=nextcloud:34.0.4-apache, nextcloud-db=mariadb:12.3, nextcloud-redis=redis:7.4.0-alpine) +2026/09/24 12:34:39 update.go:1260: [INFO] [stacks] update nextcloud: phase pulling +2026/09/24 12:34:39 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:34:39 manager.go:1463: [DEBUG] Running: docker compose pull (in /opt/docker/stacks/nextcloud) +2026/09/24 12:34:40 manager.go:1483: [DEBUG] Command completed: docker compose pull (took 1.3s) +2026/09/24 12:34:40 update.go:1260: [INFO] [stacks] update nextcloud: phase copying +2026/09/24 12:34:40 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:34:40 manager.go:1463: [DEBUG] Running: docker compose stop (in /opt/docker/stacks/nextcloud) +2026/09/24 12:34:41 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:34:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m4s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:41 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:34:41 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:34:41 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:34:41 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:34:42 manager.go:1483: [DEBUG] Command completed: docker compose stop (took 1.6s) +2026/09/24 12:34:42 update.go:1260: [INFO] [stacks] update nextcloud: phase copying +2026/09/24 12:34:43 undo.go:423: [INFO] [stacks] update nextcloud: copied nextcloud_nextcloud_db_data → nextcloud_nextcloud_db_data.pre-update-20260924T123442Z in 1.346s +2026/09/24 12:34:43 update.go:1260: [INFO] [stacks] update nextcloud: phase copying +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%) +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%) +2026/09/24 12:34:51 fillwatch.go:256: [INFO] [fillwatch] checked 7 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/09/24 12:34:51 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:34:51 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:34:51 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:34:51 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:34:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m14s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:51 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:34:51 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:34:51 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:34:52 undo.go:423: [INFO] [stacks] update nextcloud: copied nextcloud_nextcloud_html → nextcloud_nextcloud_html.pre-update-20260924T123442Z in 8.62s +2026/09/24 12:34:52 update.go:1260: [INFO] [stacks] update nextcloud: phase copying +2026/09/24 12:34:52 undo.go:423: [INFO] [stacks] update nextcloud: copied nextcloud_nextcloud_redis_data → nextcloud_nextcloud_redis_data.pre-update-20260924T123442Z in 455ms +2026/09/24 12:34:52 update.go:1260: [INFO] [stacks] update nextcloud: phase starting +2026/09/24 12:34:52 manager.go:410: [INFO] [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping ensure (held by drive gate) +2026/09/24 12:34:52 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:34:52 manager.go:1463: [DEBUG] Running: docker compose up -d --remove-orphans (in /opt/docker/stacks/nextcloud) +2026/09/24 12:34:58 manager.go:1483: [DEBUG] Command completed: docker compose up -d --remove-orphans (took 6.3s) +2026/09/24 12:34:58 update.go:1260: [INFO] [stacks] update nextcloud: phase verifying +2026/09/24 12:35:01 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:35:01 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m24s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:01 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container) diff --git a/documentation/audits/night-2026-09-24/E/round-11-controller.log b/documentation/audits/night-2026-09-24/E/round-11-controller.log new file mode 100644 index 00000000..73985a09 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-11-controller.log @@ -0,0 +1,1244 @@ +2026/09/24 12:34:07 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.2:51556 (X-Forwarded-For: 192.168.0.180) +2026/09/24 12:34:07 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session +2026/09/24 12:34:07 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T12:34:07Z, active_sessions=3 +2026/09/24 12:34:07 auth.go:222: [INFO] [web] Login from 172.18.0.2:51556 +2026/09/24 12:34:07 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.2:51556 +2026/09/24 12:34:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.2:51556 +2026/09/24 12:34:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:34:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:34:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:34:07 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:34:08 router.go:1065: [INFO] [api] Manual catalog sync requested +2026/09/24 12:34:08 sync.go:208: [INFO] [sync] Starting catalog sync +2026/09/24 12:34:08 sync.go:296: [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-drill.git (branch: main) +2026/09/24 12:34:08 sync.go:298: [DEBUG] [sync] git fetch --depth 1 origin main in /opt/docker/felhom-controller/data/catalog-cache +2026/09/24 12:34:08 sync.go:627: [DEBUG] [sync] Running: git fetch --depth 1 origin main +2026/09/24 12:34:08 sync.go:627: [DEBUG] [sync] Running: git reset --hard origin/main +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] actualbudget/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] actualbudget/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] adventurelog/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] adventurelog/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] audiobookshelf/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] audiobookshelf/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] bentopdf/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] bentopdf/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] bookstack/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] bookstack/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] calcom/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] calcom/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] calibre-web/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] calibre-web/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] chaoscrash/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] chaoscrash/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] chaosoom/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] chaosoom/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] chaosoomb/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] chaosoomb/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] claper/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] claper/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] code-server/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] code-server/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] crafty-controller/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] crafty-controller/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] docmost/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] docmost/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] emby/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] emby/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] ghost/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] ghost/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] gitea/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] gitea/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] glance/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] glance/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] gokapi/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] gokapi/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] grafana/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] grafana/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] gramps-web/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] gramps-web/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] home-assistant/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] home-assistant/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] homebox/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] homebox/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] homepage/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] homepage/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] immich/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] immich/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] jellyfin/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] jellyfin/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] kimai/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] kimai/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] komga/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] komga/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] mealie/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] mealie/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:533: [DEBUG] [sync] n8n: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] n8n/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] n8n/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] navidrome/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] navidrome/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:533: [DEBUG] [sync] nextcloud: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] nextcloud/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:416: [INFO] [sync] Updated nextcloud/.felhom.yml +2026/09/24 12:34:08 sync.go:565: [DEBUG] [sync] nextcloud/.felhom.yml: src=75a58fb7c5e1639e, dst=75a58fb7c5e1639e (changed) +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] onlyoffice/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] onlyoffice/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] opengist/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] opengist/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] outline/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] outline/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] paperless-ngx/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] paperless-ngx/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] papra/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] papra/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] plant-it/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] plant-it/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] privatebin/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] privatebin/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] radarr/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] radarr/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] rallly/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] rallly/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] recipe-importer/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] recipe-importer/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] romm/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] romm/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] seerr/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] seerr/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] sonarr/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] sonarr/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] sparkyfitness/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] sparkyfitness/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] tandoor/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] tandoor/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] termix/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] termix/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] uptime-kuma/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] uptime-kuma/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] vaultwarden/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] vaultwarden/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:533: [DEBUG] [sync] vikunja: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] vikunja/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] vikunja/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] wanderer/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] wanderer/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] wger/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] wger/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] wishlist/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] wishlist/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] zipline/docker-compose.yml: hash match, skipped +2026/09/24 12:34:08 sync.go:440: [DEBUG] [sync] zipline/.felhom.yml: hash match, skipped +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:34:08 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:34:08 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:34:08 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:34:08 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:34:08 sync.go:244: [DEBUG] [sync] Post-sync hook: triggering missing field injection for 1 stack(s): [nextcloud] +2026/09/24 12:34:08 deploy.go:1104: [DEBUG] [stacks] InjectMissingFields: checking 1 stacks +2026/09/24 12:34:08 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack nextcloud — 7 deploy fields, 7 existing env vars +2026/09/24 12:34:08 deploy.go:1189: [INFO] [stacks] InjectMissingFields: processed 1 stacks +2026/09/24 12:34:08 sync.go:263: [INFO] [sync] Catalog sync complete +2026/09/24 12:34:08 router.go:1071: [INFO] [api] Catalog sync completed: Sablonok frissítve — frissítve: nextcloud +2026/09/24 12:34:10 router.go:362: [INFO] [api] Manual stack rescan requested +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:34:10 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:34:10 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:34:10 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:34:10 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:34:11 router.go:369: [INFO] [api] Stack rescan completed: 58 stacks found +2026/09/24 12:34:11 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:34:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:34:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 34s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:11 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:34:11 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:34:11 healthprobe.go:181: [DEBUG] Health probe romm: HTTP GET :8080/ → 200 (3ms) +2026/09/24 12:34:11 healthprobe.go:181: [DEBUG] Health probe paperless-ngx: HTTP GET :8000/ → 302 (11ms) +2026/09/24 12:34:11 healthprobe.go:161: [INFO] Health probes: 2 ok (of 2 probed) +2026/09/24 12:34:13 router.go:591: [INFO] [api] update requested for stack: nextcloud +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5305MB avail≈20593MB +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24422308 → total=25898MB avail=20593MB used=5305MB (20.5%) +2026/09/24 12:34:13 deploy.go:1250: [INFO] [stacks] Memory check: total=25898MB, reserved=384MB, usable=25514MB, committed_used=1562MB, new_req=256MB, remaining=23696MB +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=26.8 GB avail=38.0GB (39.3%) +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5300MB avail≈20598MB +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24428424 → total=25898MB avail=20598MB used=5300MB (20.5%) +2026/09/24 12:34:13 deploy.go:1250: [INFO] [stacks] Memory check: total=25898MB, reserved=384MB, usable=25514MB, committed_used=1562MB, new_req=256MB, remaining=23696MB +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=26.8 GB avail=38.0GB (39.3%) +2026/09/24 12:34:13 update.go:512: [INFO] [stacks] update nextcloud: accepted — guarded update started +2026/09/24 12:34:13 update.go:1260: [INFO] [stacks] update nextcloud: phase checking +2026/09/24 12:34:13 update.go:762: [INFO] [stacks] update nextcloud: ladder — the installed version nextcloud=nextcloud:34.0.4-apache, nextcloud-db=mariadb:12.3, nextcloud-redis=redis:7-alpine matches no update_ladder entry (2 entries) — an app older than the ladder has no record to climb; the catalog's current definition +2026/09/24 12:34:13 update.go:777: [INFO] [stacks] update nextcloud: no usable copy on any tier — younger than 1m0s and not older than this install's deploy (2026-09-24T12:07:07Z) (found: Tier 2 (second drive) at 2026-09-24T12:20:03Z (14m0s old); Tier 1 (own recovery unit) at 2026-09-24T12:15:03Z (19m0s old)) — backing up first +2026/09/24 12:34:13 update.go:1260: [INFO] [stacks] update nextcloud: phase backing-up +2026/09/24 12:34:13 update_guard.go:360: [INFO] [backup] update pre-backup for nextcloud: starting (DB dump → volume dump → unit capture → Tier 2) +2026/09/24 12:34:13 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:34:13 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/24 12:34:13 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 517b7ba21f80 n8n n8n n8nio/n8n:2.40.5 +a0b33d0b9958 wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1 +9f47a1f5273b vikunja vikunja vikunja/vikunja:2.3.0 +1adf3ec73c9b romm romm rommapp/romm:5.3.1 +e059bbc146b0 romm-redis romm redis:7-alpine +fc5ed5428e75 romm-db romm mariadb:11.8 +361a05346765 privatebin privatebin privatebin/pdo:2.0.6 +92b06fef42c0 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +5e5c371b82fa paperless-postgres paperless-ngx postgres:16-alpine +0ef3bfdcb4... +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.40.5, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/24 12:34:13 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=fc5ed5428e75) +2026/09/24 12:34:13 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/24 12:34:13 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5e5c371b82fa) +2026/09/24 12:34:13 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/24 12:34:13 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/24 12:34:13 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=13e101052288) +2026/09/24 12:34:13 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/24 12:34:13 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/24 12:34:13 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 14 non-DB container(s) +2026/09/24 12:34:13 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/09/24 12:34:13 dbdump.go:266: [DEBUG] DumpOne: starting dump for container=nextcloud-db, stack=nextcloud, dbType=mariadb, dest=/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/nextcloud-mariadb.sql +2026/09/24 12:34:13 dbdump.go:321: [DEBUG] DumpOne: mariadb-dump command: docker exec 13e101052288 mariadb-dump -u root -p*** --single-transaction --routines --triggers nextcloud +2026/09/24 12:34:14 dbdump.go:405: [DEBUG] DumpOne: completed nextcloud-db → nextcloud-mariadb.sql (size=855.4 KB, valid=true, tables=131, duration=686ms) +2026/09/24 12:34:14 dbdump.go:410: [INFO] [backup] DB dump: nextcloud-db → nextcloud-mariadb.sql (855.4 KB, 686ms, 131 tables) +2026/09/24 12:34:14 update_guard.go:399: [INFO] [backup] update pre-backup for nextcloud: database dump OK (nextcloud-db, 855.4 KB) +2026/09/24 12:34:14 backup.go:920: [INFO] [backup] Stopping nextcloud for safe volume dump +2026/09/24 12:34:14 manager.go:1283: [DEBUG] [stacks] StopStack nextcloud: current state=running deployed=true containers=3 +2026/09/24 12:34:14 manager.go:1286: [INFO] [stacks] Stopping stack: nextcloud +2026/09/24 12:34:14 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:34:14 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/nextcloud) +2026/09/24 12:34:16 manager.go:1483: [DEBUG] Command completed: docker compose down (took 1.8s) +2026/09/24 12:34:16 manager.go:1295: [INFO] [stacks] Stack nextcloud stopped successfully (took 1.8s) +2026/09/24 12:34:16 backup.go:796: [DEBUG] [backup] Dumping volume nextcloud_nextcloud_html for nextcloud +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5217MB avail≈20681MB +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24525888 → total=25898MB avail=20681MB used=5217MB (20.1%) +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=26.8GB avail=38.0GB (39.3%) +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=74.1GB avail=816.0GB (7.9%) +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readLoadAvg: raw="5.95 4.00 3.50 12/1454 768" → 1m=5.95 5m=4.00 15m=3.50 +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 78.4°C (hwmon2) +2026/09/24 12:34:21 info.go:13: [DEBUG] [system] GetInfo done in 92ms — mem=5217MB/25898MB (20.1%), rootDisk=26.8GB/68.4GB (39.3%), load=5.95/4.00/3.50, temp=78.4°C (hwmon2), cpu=58.9% +2026/09/24 12:34:21 backup.go:821: [INFO] [backup] Volume dump: nextcloud/nextcloud_nextcloud_html → 761.6 MB +2026/09/24 12:34:21 backup.go:796: [DEBUG] [backup] Dumping volume nextcloud_nextcloud_redis_data for nextcloud +2026/09/24 12:34:21 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:34:21 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:34:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 44s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:21 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:34:21 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:34:21 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:34:21 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:34:21 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:34:22 backup.go:821: [INFO] [backup] Volume dump: nextcloud/nextcloud_nextcloud_redis_data → 277.0 KB +2026/09/24 12:34:22 backup.go:796: [DEBUG] [backup] Dumping volume nextcloud_nextcloud_db_data for nextcloud +2026/09/24 12:34:24 backup.go:821: [INFO] [backup] Volume dump: nextcloud/nextcloud_nextcloud_db_data → 164.9 MB +2026/09/24 12:34:24 backup.go:930: [INFO] [backup] Restarting nextcloud after volume dump +2026/09/24 12:34:24 manager.go:1194: [DEBUG] [stacks] StartStack nextcloud: current state=stopped deployed=true +2026/09/24 12:34:24 manager.go:1197: [INFO] [stacks] Starting stack: nextcloud +2026/09/24 12:34:24 manager.go:1204: [DEBUG] [stacks] StartStack nextcloud: prepared 14 env vars for compose +2026/09/24 12:34:24 manager.go:410: [INFO] [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping ensure (held by drive gate) +2026/09/24 12:34:24 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:34:24 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/nextcloud) +2026/09/24 12:34:30 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 6.5s) +2026/09/24 12:34:30 manager.go:1212: [INFO] [stacks] Stack nextcloud started successfully (took 6.5s) +2026/09/24 12:34:30 installed.go:409: [DEBUG] [stacks] installed-images nextcloud: unchanged (3 service(s)) — app.yaml not rewritten +2026/09/24 12:34:30 update_guard.go:410: [INFO] [backup] update pre-backup for nextcloud: volume dump OK +2026/09/24 12:34:30 recovery_unit.go:224: [INFO] [backup] Recovery unit captured for nextcloud → /mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/09/24 12:34:30 update_guard.go:416: [INFO] [backup] update pre-backup for nextcloud: recovery unit captured (1 database dump(s)) +2026/09/24 12:34:30 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=26.8 GB avail=38.0GB (39.3%) +2026/09/24 12:34:31 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 54s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:31 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:34:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:31 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:34:33 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:34:33 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/nextcloud) +2026/09/24 12:34:33 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s) +2026/09/24 12:34:33 manager.go:1543: [INFO] [stacks] Stack nextcloud post-start status: +2026/09/24 12:34:33 manager.go:1546: [INFO] [stacks] nextcloud nextcloud:34.0.4-apache@sha256:a5ace30c695afe48c2c406e940ee7886a81e13fa382e57cd68b2416d1a66914c running Up 3 seconds (health: starting) +2026/09/24 12:34:33 manager.go:1546: [INFO] [stacks] nextcloud-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy) +2026/09/24 12:34:33 manager.go:1546: [INFO] [stacks] nextcloud-redis redis:7-alpine@sha256:c35af3bbcef51a62c8bae5a9a563c6f1b60d7ebaea4cb5a3ccbcc157580ae098 running Up 9 seconds (healthy) +2026/09/24 12:34:34 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18567 bytes) +2026/09/24 12:34:34 settings.go:847: [INFO] [settings] Settings saved +2026/09/24 12:34:34 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event crossdrive_completed (severity info) — further crossdrive_completed events are logged at DEBUG only +2026/09/24 12:34:34 tier2.go:425: [INFO] [backup] Tier 2 copied nextcloud → /mnt/sys_drive/felhom-data/backups/secondary/nextcloud (1.0 GB, 1 leg(s), 3s) [SSD: state-only] +2026/09/24 12:34:34 update_guard.go:425: [INFO] [backup] update pre-backup for nextcloud: complete in 20.653s +2026/09/24 12:34:34 update.go:792: [INFO] [stacks] update nextcloud: precondition met after the backup — Tier 2 (second drive) copy from 2026-09-24T12:34:34Z +2026/09/24 12:34:36 update.go:1260: [INFO] [stacks] update nextcloud: phase safety-dump +2026/09/24 12:34:36 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/24 12:34:36 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 101ac4b42fdc nextcloud nextcloud nextcloud:34.0.4-apache +061bb4d3beb2 nextcloud-db nextcloud mariadb:12.3 +08859b9002cf nextcloud-redis nextcloud redis:7-alpine +517b7ba21f80 n8n n8n n8nio/n8n:2.40.5 +a0b33d0b9958 wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1 +9f47a1f5273b vikunja vikunja vikunja/vikunja:2.3.0 +1adf3ec73c9b romm romm rommapp/romm:5.3.1 +e059bbc146b0 romm-redis romm redis:7-alpine +fc5ed5428e75 romm-db romm mariadb:11.8 +361a05346765 privatebin privatebin privatebin/pdo:2.0.6 +92b06f... +2026/09/24 12:34:36 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database) +2026/09/24 12:34:36 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=061bb4d3beb2) +2026/09/24 12:34:37 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.40.5, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/24 12:34:37 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=fc5ed5428e75) +2026/09/24 12:34:37 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/24 12:34:37 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5e5c371b82fa) +2026/09/24 12:34:37 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/24 12:34:37 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/24 12:34:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/24 12:34:37 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 14 non-DB container(s) +2026/09/24 12:34:37 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/09/24 12:34:37 dbdump.go:266: [DEBUG] DumpOne: starting dump for container=nextcloud-db, stack=nextcloud, dbType=mariadb, dest=/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/pre-restore-20260924T123437Z-nextcloud-mariadb.sql +2026/09/24 12:34:37 dbdump.go:321: [DEBUG] DumpOne: mariadb-dump command: docker exec 061bb4d3beb2 mariadb-dump -u root -p*** --single-transaction --routines --triggers nextcloud +2026/09/24 12:34:37 dbdump.go:405: [DEBUG] DumpOne: completed nextcloud-db → pre-restore-20260924T123437Z-nextcloud-mariadb.sql (size=855.4 KB, valid=true, tables=131, duration=632ms) +2026/09/24 12:34:37 dbdump.go:410: [INFO] [backup] DB dump: nextcloud-db → pre-restore-20260924T123437Z-nextcloud-mariadb.sql (855.4 KB, 632ms, 131 tables) +2026/09/24 12:34:37 offbox_reconstitute.go:255: [INFO] [offbox] nextcloud: pre-restore safety dump written → pre-restore-20260924T123437Z-nextcloud-mariadb.sql (855.4 KB) +2026/09/24 12:34:37 update_guard.go:478: [INFO] [backup] update safety dump for nextcloud: 1 file(s) [/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/pre-restore-20260924T123437Z-nextcloud-mariadb.sql] +2026/09/24 12:34:37 update.go:807: [INFO] [stacks] update nextcloud: safety dump done (1 file(s)) [/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/pre-restore-20260924T123437Z-nextcloud-mariadb.sql] +2026/09/24 12:34:39 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=26.8 GB avail=38.0GB (39.3%) +2026/09/24 12:34:39 undo.go:273: [INFO] [stacks] update nextcloud: the undo copy will hold 3 named volume(s), 928.8 MiB +2026/09/24 12:34:39 update.go:1260: [INFO] [stacks] update nextcloud: phase pinning +2026/09/24 12:34:39 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/nextcloud — 7 env vars, 0 encrypted, 3 sensitive fields +2026/09/24 12:34:39 [INFO] [stacks] SaveAppConfig: saved config for nextcloud +2026/09/24 12:34:39 pin.go:373: [INFO] [stacks] update nextcloud: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/nextcloud/docker-compose.yml (nextcloud=nextcloud:34.0.4-apache, nextcloud-db=mariadb:12.3, nextcloud-redis=redis:7.4.0-alpine) +2026/09/24 12:34:39 update.go:1260: [INFO] [stacks] update nextcloud: phase pulling +2026/09/24 12:34:39 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:34:39 manager.go:1463: [DEBUG] Running: docker compose pull (in /opt/docker/stacks/nextcloud) +2026/09/24 12:34:40 manager.go:1483: [DEBUG] Command completed: docker compose pull (took 1.3s) +2026/09/24 12:34:40 update.go:1260: [INFO] [stacks] update nextcloud: phase copying +2026/09/24 12:34:40 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:34:40 manager.go:1463: [DEBUG] Running: docker compose stop (in /opt/docker/stacks/nextcloud) +2026/09/24 12:34:41 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:34:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m4s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:41 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:34:41 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:34:41 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:34:41 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:34:42 manager.go:1483: [DEBUG] Command completed: docker compose stop (took 1.6s) +2026/09/24 12:34:42 update.go:1260: [INFO] [stacks] update nextcloud: phase copying +2026/09/24 12:34:43 undo.go:423: [INFO] [stacks] update nextcloud: copied nextcloud_nextcloud_db_data → nextcloud_nextcloud_db_data.pre-update-20260924T123442Z in 1.346s +2026/09/24 12:34:43 update.go:1260: [INFO] [stacks] update nextcloud: phase copying +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%) +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:34:51 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%) +2026/09/24 12:34:51 fillwatch.go:256: [INFO] [fillwatch] checked 7 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026/09/24 12:34:51 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:34:51 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:34:51 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:34:51 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:34:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m14s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:34:51 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:34:51 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:34:51 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:34:52 undo.go:423: [INFO] [stacks] update nextcloud: copied nextcloud_nextcloud_html → nextcloud_nextcloud_html.pre-update-20260924T123442Z in 8.62s +2026/09/24 12:34:52 update.go:1260: [INFO] [stacks] update nextcloud: phase copying +2026/09/24 12:34:52 undo.go:423: [INFO] [stacks] update nextcloud: copied nextcloud_nextcloud_redis_data → nextcloud_nextcloud_redis_data.pre-update-20260924T123442Z in 455ms +2026/09/24 12:34:52 update.go:1260: [INFO] [stacks] update nextcloud: phase starting +2026/09/24 12:34:52 manager.go:410: [INFO] [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping ensure (held by drive gate) +2026/09/24 12:34:52 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:34:52 manager.go:1463: [DEBUG] Running: docker compose up -d --remove-orphans (in /opt/docker/stacks/nextcloud) +2026/09/24 12:34:58 manager.go:1483: [DEBUG] Command completed: docker compose up -d --remove-orphans (took 6.3s) +2026/09/24 12:34:58 update.go:1260: [INFO] [stacks] update nextcloud: phase verifying +2026/09/24 12:35:01 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:35:01 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m24s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:35:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:01 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:35:08 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:35:11 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:35:11 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:35:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m34s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 3s ago, effective interval 10s, healthy=false +2026/09/24 12:35:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:11 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:35:11 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:35:11 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:35:13 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:35:19 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:35:21 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:35:21 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5466MB avail≈20432MB +2026/09/24 12:35:21 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24231272 → total=25898MB avail=20432MB used=5466MB (21.1%) +2026/09/24 12:35:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=61.8GB avail=3.0GB (90.5%) +2026/09/24 12:35:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:35:21 info.go:13: [DEBUG] [system] readLoadAvg: raw="5.79 4.31 3.64 8/1517 1396" → 1m=5.79 5m=4.31 15m=3.64 +2026/09/24 12:35:21 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:35:21 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 69.9°C (hwmon2) +2026/09/24 12:35:21 info.go:13: [DEBUG] [system] GetInfo done in 78ms — mem=5466MB/25898MB (21.1%), rootDisk=61.8GB/68.4GB (90.5%), load=5.79/4.31/3.64, temp=69.9°C (hwmon2), cpu=24.7% +2026/09/24 12:35:21 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:35:21 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan +2026/09/24 12:35:21 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting +2026/09/24 12:35:21 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:35:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m44s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2s ago, effective interval 10s, healthy=false +2026/09/24 12:35:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:35:21 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:35:21 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:35:21 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:35:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:35:21 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:35:21 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:35:21 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:35:21 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 87ms) +2026/09/24 12:35:24 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:35:29 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:35:31 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:35:31 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:35:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m54s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2s ago, effective interval 10s, healthy=false +2026/09/24 12:35:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:31 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:35:31 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:35:31 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:35:34 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:35:39 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:35:41 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:35:41 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:35:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 2m4s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2s ago, effective interval 10s, healthy=false +2026/09/24 12:35:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:35:41 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:35:44 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:35:49 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:35:51 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:35:51 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:35:51 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:35:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 2m14s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2s ago, effective interval 10s, healthy=false +2026/09/24 12:35:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:35:51 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:35:51 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:35:51 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:35:51 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:35:54 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:35:59 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:36:01 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:36:01 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:36:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:36:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 2m24s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2s ago, effective interval 10s, healthy=false +2026/09/24 12:36:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:01 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:36:04 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:36:09 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:36:11 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:36:11 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:36:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 2m34s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2s ago, effective interval 10s, healthy=false +2026/09/24 12:36:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:11 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:36:11 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:36:11 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:36:14 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:36:19 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:36:21 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:36:21 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5418MB avail≈20480MB +2026/09/24 12:36:21 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24287572 → total=25898MB avail=20480MB used=5418MB (20.9%) +2026/09/24 12:36:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=61.8GB avail=3.0GB (90.5%) +2026/09/24 12:36:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:36:21 info.go:13: [DEBUG] [system] readLoadAvg: raw="3.07 3.80 3.51 4/1518 1678" → 1m=3.07 5m=3.80 15m=3.51 +2026/09/24 12:36:21 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:36:21 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 65.1°C (hwmon2) +2026/09/24 12:36:21 info.go:13: [DEBUG] [system] GetInfo done in 70ms — mem=5418MB/25898MB (20.9%), rootDisk=61.8GB/68.4GB (90.5%), load=3.07/3.80/3.51, temp=65.1°C (hwmon2), cpu=23.8% +2026/09/24 12:36:21 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:36:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:36:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 2m44s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2s ago, effective interval 10s, healthy=false +2026/09/24 12:36:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:21 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:36:21 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:36:21 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:36:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:21 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:36:24 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:36:29 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:36:29 update.go:969: [ERROR] [stacks] update nextcloud FAILED after the new version was started: not healthy: not healthy within 1m30s (last: health check failing) +2026/09/24 12:36:29 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:36:29 manager.go:1463: [DEBUG] Running: docker compose logs --no-color --tail 400 (in /opt/docker/stacks/nextcloud) +2026/09/24 12:36:29 manager.go:1483: [DEBUG] Command completed: docker compose logs --no-color --tail 400 (took 0.1s) +2026/09/24 12:36:29 update.go:961: [INFO] [stacks] update nextcloud: kept 6809 bytes of the app's own log at /opt/docker/stacks/nextcloud/hold-logs/20260924T123629Z/compose-logs.txt before stopping it (R-621) +2026/09/24 12:36:29 update.go:1260: [INFO] [stacks] update nextcloud: phase undoing +2026/09/24 12:36:29 undo.go:501: [WARN] [stacks] update nextcloud: UNDO — putting back the previous version and its 3 volume copy(ies) (reason: not healthy: not healthy within 1m30s (last: health check failing)) +2026/09/24 12:36:29 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:36:29 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/nextcloud) +2026/09/24 12:36:31 manager.go:1483: [DEBUG] Command completed: docker compose down (took 1.8s) +2026/09/24 12:36:31 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:36:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 2m54s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 2s ago, effective interval 10s, healthy=false +2026/09/24 12:36:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:31 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:36:31 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:36:31 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:36:31 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:36:32 manager.go:1502: [ERROR] [stacks] execCommand failed: exit status 1 +2026/09/24 12:36:32 undo.go:507: [ERROR] [stacks] update nextcloud: the undo copy nextcloud_nextcloud_redis_data.pre-update-20260924T123442Z has no finished-marker — it is cut off or missing; NOTHING is put back +2026/09/24 12:36:32 update.go:981: [ERROR] [stacks] update nextcloud: the UNDO failed too (untouched) — HOLDING the app; the undo copies are kept: [{nextcloud_nextcloud_db_data nextcloud_nextcloud_db_data.pre-update-20260924T123442Z} {nextcloud_nextcloud_html nextcloud_nextcloud_html.pre-update-20260924T123442Z} {nextcloud_nextcloud_redis_data nextcloud_nextcloud_redis_data.pre-update-20260924T123442Z}] +2026/09/24 12:36:32 settings.go:1814: [WARN] [settings] restore hold SET for nextcloud — the app stays stopped until it is cleared +2026/09/24 12:36:32 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18873 bytes) +2026/09/24 12:36:32 settings.go:847: [INFO] [settings] Settings saved +2026/09/24 12:36:32 update_guard.go:560: [WARN] [backup] nextcloud is HELD STOPPED after a failed update (restore point: tier 2 "második meghajtó", 2026-09-24T12:34:34Z; holds: "a beállításokat, az adatbázist és a fájlokat tartalmazza"; undo: "untouched") +2026/09/24 12:36:32 settings.go:1814: [WARN] [settings] restore hold SET for nextcloud — the app stays stopped until it is cleared +2026/09/24 12:36:32 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18988 bytes) +2026/09/24 12:36:32 settings.go:847: [INFO] [settings] Settings saved +2026/09/24 12:36:32 undo.go:619: [INFO] [stacks] update nextcloud: event app_update_held (hold recorded: true) +2026/09/24 12:36:32 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_update_held (severity error) — further app_update_held events are logged at DEBUG only +2026/09/24 12:36:34 router.go:1065: [INFO] [api] Manual catalog sync requested +2026/09/24 12:36:34 sync.go:208: [INFO] [sync] Starting catalog sync +2026/09/24 12:36:34 sync.go:296: [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-drill.git (branch: main) +2026/09/24 12:36:34 sync.go:298: [DEBUG] [sync] git fetch --depth 1 origin main in /opt/docker/felhom-controller/data/catalog-cache +2026/09/24 12:36:34 sync.go:627: [DEBUG] [sync] Running: git fetch --depth 1 origin main +2026/09/24 12:36:34 sync.go:627: [DEBUG] [sync] Running: git reset --hard origin/main +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] actualbudget/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] actualbudget/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] adventurelog/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] adventurelog/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] audiobookshelf/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] audiobookshelf/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] bentopdf/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] bentopdf/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] bookstack/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] bookstack/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] calcom/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] calcom/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] calibre-web/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] calibre-web/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] chaoscrash/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] chaoscrash/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] chaosoom/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] chaosoom/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] chaosoomb/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] chaosoomb/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] claper/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] claper/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] code-server/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] code-server/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] crafty-controller/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] crafty-controller/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] docmost/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] docmost/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] emby/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] emby/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] ghost/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] ghost/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] gitea/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] gitea/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] glance/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] glance/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] gokapi/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] gokapi/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] grafana/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] grafana/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] gramps-web/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] gramps-web/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] home-assistant/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] home-assistant/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] homebox/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] homebox/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] homepage/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] homepage/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] immich/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] immich/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] jellyfin/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] jellyfin/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] kimai/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] kimai/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] komga/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] komga/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] mealie/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] mealie/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:533: [DEBUG] [sync] n8n: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] n8n/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] n8n/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] navidrome/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] navidrome/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:533: [DEBUG] [sync] nextcloud: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] nextcloud/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:416: [INFO] [sync] Updated nextcloud/.felhom.yml +2026/09/24 12:36:34 sync.go:565: [DEBUG] [sync] nextcloud/.felhom.yml: src=3009031bf29d605a, dst=3009031bf29d605a (changed) +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] onlyoffice/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] onlyoffice/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] opengist/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] opengist/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] outline/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] outline/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] paperless-ngx/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] paperless-ngx/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] papra/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] papra/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] plant-it/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] plant-it/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] privatebin/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] privatebin/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] radarr/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] radarr/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] rallly/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] rallly/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] recipe-importer/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] recipe-importer/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] romm/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] romm/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] seerr/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] seerr/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] sonarr/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] sonarr/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] sparkyfitness/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] sparkyfitness/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] tandoor/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] tandoor/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] termix/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] termix/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] uptime-kuma/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] uptime-kuma/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] vaultwarden/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] vaultwarden/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:533: [DEBUG] [sync] vikunja: catalog has moved past the pin — rendering the stored applied definition +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] vikunja/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] vikunja/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] wanderer/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] wanderer/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] wger/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] wger/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] wishlist/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] wishlist/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] zipline/docker-compose.yml: hash match, skipped +2026/09/24 12:36:34 sync.go:440: [DEBUG] [sync] zipline/.felhom.yml: hash match, skipped +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:36:34 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:36:34 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:36:34 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:36:34 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:36:34 sync.go:244: [DEBUG] [sync] Post-sync hook: triggering missing field injection for 1 stack(s): [nextcloud] +2026/09/24 12:36:34 deploy.go:1104: [DEBUG] [stacks] InjectMissingFields: checking 1 stacks +2026/09/24 12:36:34 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack nextcloud — 7 deploy fields, 7 existing env vars +2026/09/24 12:36:34 deploy.go:1189: [INFO] [stacks] InjectMissingFields: processed 1 stacks +2026/09/24 12:36:34 sync.go:263: [INFO] [sync] Catalog sync complete +2026/09/24 12:36:34 router.go:1071: [INFO] [api] Catalog sync completed: Sablonok frissítve — frissítve: nextcloud +2026/09/24 12:36:36 router.go:362: [INFO] [api] Manual stack rescan requested +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:36:36 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:36:36 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:36:36 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:36:36 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:36:36 router.go:369: [INFO] [api] Stack rescan completed: 58 stacks found +2026/09/24 12:36:38 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:36:38 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:36:38 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:36:38 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:36:38 server.go:545: [DEBUG] [web] ServeHTTP: POST /backup/tier2/unit-restore from 172.18.0.2:51556 +2026/09/24 12:36:38 handlers.go:2063: [WARN] [web] Tier-2 WHOLE restore requested (async, files + database): stack=nextcloud from 172.18.0.2:51556 +2026/09/24 12:36:38 tier2_whole.go:304: [WARN] [backup] WHOLE restore for nextcloud from the second drive /mnt/sys_drive/felhom-data/backups/secondary/nextcloud: files by the four rules (never delete, never overwrite newer), then the unit +2026/09/24 12:36:38 manager.go:1283: [DEBUG] [stacks] StopStack nextcloud: current state=stopped deployed=true containers=0 +2026/09/24 12:36:38 manager.go:1286: [INFO] [stacks] Stopping stack: nextcloud +2026/09/24 12:36:38 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:36:38 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/nextcloud) +2026/09/24 12:36:38 manager.go:1483: [DEBUG] Command completed: docker compose down (took 0.1s) +2026/09/24 12:36:38 manager.go:1295: [INFO] [stacks] Stack nextcloud stopped successfully (took 0.1s) +2026/09/24 12:36:39 tier2_whole.go:326: [INFO] [backup] whole restore nextcloud: files restored=0 replaced=0 (live kept beside as *.felhom-20260924T123638Z) kept-newer=0 unchanged=142 +2026/09/24 12:36:39 restore_unit.go:379: [INFO] [backup] Restoring nextcloud from recovery unit /mnt/sys_drive/felhom-data/backups/secondary/nextcloud/recovery-unit: images=3, secrets recovered=3/3, data_keys=0 +2026/09/24 12:36:39 manager.go:1283: [DEBUG] [stacks] StopStack nextcloud: current state=stopped deployed=true containers=0 +2026/09/24 12:36:39 manager.go:1286: [INFO] [stacks] Stopping stack: nextcloud +2026/09/24 12:36:39 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, DB_PASSWORD, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:36:39 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/nextcloud) +2026/09/24 12:36:39 manager.go:1483: [DEBUG] Command completed: docker compose down (took 0.1s) +2026/09/24 12:36:39 manager.go:1295: [INFO] [stacks] Stack nextcloud stopped successfully (took 0.1s) +2026/09/24 12:36:39 restore.go:152: [INFO] [backup] Restoring Docker volume nextcloud_nextcloud_db_data for nextcloud +2026/09/24 12:36:41 restore.go:190: [DEBUG] [backup] Volume nextcloud_nextcloud_db_data restored successfully +2026/09/24 12:36:41 restore.go:152: [INFO] [backup] Restoring Docker volume nextcloud_nextcloud_html for nextcloud +2026/09/24 12:36:41 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:36:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:41 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:36:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:36:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 3m4s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:41 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:36:51 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:36:51 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:36:51 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:36:51 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:36:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 3m14s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:36:51 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:36:51 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:36:51 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:36:52 restore.go:190: [DEBUG] [backup] Volume nextcloud_nextcloud_html restored successfully +2026/09/24 12:36:52 restore.go:152: [INFO] [backup] Restoring Docker volume nextcloud_nextcloud_redis_data for nextcloud +2026/09/24 12:36:52 restore.go:190: [DEBUG] [backup] Volume nextcloud_nextcloud_redis_data restored successfully +2026/09/24 12:36:52 restore.go:195: [INFO] [backup] Restored 3 Docker volume(s) for nextcloud +2026/09/24 12:36:52 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/nextcloud — 7 env vars, 3 encrypted, 3 sensitive fields +2026/09/24 12:36:52 [INFO] [stacks] SaveAppConfig: saved config for nextcloud +2026/09/24 12:36:52 deploy.go:645: [INFO] [stacks] Redeploying nextcloud from recovery unit with 7 env vars +2026/09/24 12:36:52 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/nextcloud — 7 env vars, 0 encrypted, 3 sensitive fields +2026/09/24 12:36:52 pin.go:93: [INFO] [stacks] pin nextcloud: nextcloud=nextcloud:34.0.4-apache, nextcloud-db=mariadb:12.3, nextcloud-redis=redis:7-alpine +2026/09/24 12:36:52 [INFO] [stacks] SaveAppConfig: saved config for nextcloud +2026/09/24 12:36:52 manager.go:1250: [INFO] [stacks] Starting stack nextcloud services only: [nextcloud-db] +2026/09/24 12:36:52 manager.go:410: [INFO] [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping ensure (held by drive gate) +2026/09/24 12:36:52 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:36:52 manager.go:1463: [DEBUG] Running: docker compose up -d nextcloud-db (in /opt/docker/stacks/nextcloud) +2026/09/24 12:36:53 manager.go:1483: [DEBUG] Command completed: docker compose up -d nextcloud-db (took 0.5s) +2026/09/24 12:36:53 manager.go:1261: [INFO] [stacks] Stack nextcloud services [nextcloud-db] started (took 0.5s) +2026/09/24 12:36:53 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/24 12:36:53 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: f990f2f4a0e6 nextcloud-db nextcloud mariadb:12.3 +517b7ba21f80 n8n n8n n8nio/n8n:2.40.5 +a0b33d0b9958 wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1 +9f47a1f5273b vikunja vikunja vikunja/vikunja:2.3.0 +1adf3ec73c9b romm romm rommapp/romm:5.3.1 +e059bbc146b0 romm-redis romm redis:7-alpine +fc5ed5428e75 romm-db romm mariadb:11.8 +361a05346765 privatebin privatebin privatebin/pdo:2.0.6 +92b06fef42c0 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +5e5c371b82fa paperless-pos... +2026/09/24 12:36:53 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=f990f2f4a0e6) +2026/09/24 12:36:53 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/24 12:36:53 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.40.5, not a database) +2026/09/24 12:36:53 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database) +2026/09/24 12:36:53 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database) +2026/09/24 12:36:53 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/24 12:36:53 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/24 12:36:53 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=fc5ed5428e75) +2026/09/24 12:36:53 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/24 12:36:53 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/24 12:36:53 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/24 12:36:53 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5e5c371b82fa) +2026/09/24 12:36:53 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/24 12:36:53 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/24 12:36:53 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/24 12:36:53 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database) +2026/09/24 12:36:53 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database) +2026/09/24 12:36:53 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/24 12:36:53 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/24 12:36:53 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 12 non-DB container(s) +2026/09/24 12:36:53 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/09/24 12:36:53 restore_db.go:87: [INFO] [backup] Restore nextcloud: replaying DB dump into nextcloud-db (mariadb) +2026/09/24 12:37:00 dbdump.go:781: [DEBUG] [backup] ImportDump: importing /mnt/sys_drive/felhom-data/backups/secondary/nextcloud/recovery-unit/db-dumps/nextcloud-mariadb.sql into nextcloud-db (mariadb) +2026/09/24 12:37:01 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:37:01 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:37:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 3m24s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:01 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:37:01 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:37:01 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:37:05 dbdump.go:801: [INFO] [backup] Imported DB dump nextcloud-mariadb.sql into nextcloud-db (mariadb) +2026/09/24 12:37:05 restore_db.go:97: [INFO] [backup] Restore nextcloud: replayed 1 DB dump(s) +2026/09/24 12:37:05 manager.go:1194: [DEBUG] [stacks] StartStack nextcloud: current state=starting deployed=true +2026/09/24 12:37:05 manager.go:1197: [INFO] [stacks] Starting stack: nextcloud +2026/09/24 12:37:05 manager.go:1204: [DEBUG] [stacks] StartStack nextcloud: prepared 14 env vars for compose +2026/09/24 12:37:05 manager.go:410: [INFO] [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping ensure (held by drive gate) +2026/09/24 12:37:05 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:37:05 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/nextcloud) +2026/09/24 12:37:11 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 6.3s) +2026/09/24 12:37:11 manager.go:1212: [INFO] [stacks] Stack nextcloud started successfully (took 6.3s) +2026/09/24 12:37:11 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:37:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:37:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 3m34s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:11 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:37:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:11 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:37:11 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/nextcloud — 7 env vars, 0 encrypted, 3 sensitive fields +2026/09/24 12:37:11 [INFO] [stacks] SaveAppConfig: saved config for nextcloud +2026/09/24 12:37:11 installed.go:420: [INFO] [stacks] installed-images nextcloud: recorded 3 service(s) (nextcloud=nextcloud:34.0.4-apache (sha256:a5ace30c695a…), nextcloud-db=mariadb:12.3 (sha256:805c8e104bd5…), nextcloud-redis=redis:7-alpine (sha256:c35af3bbcef5…)) +2026/09/24 12:37:14 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/24 12:37:14 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/nextcloud) +2026/09/24 12:37:14 restore.go:225: [DEBUG] [backup] Post-restore health check: nextcloud not yet running, waiting... +2026/09/24 12:37:14 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s) +2026/09/24 12:37:14 manager.go:1543: [INFO] [stacks] Stack nextcloud post-start status: +2026/09/24 12:37:14 manager.go:1546: [INFO] [stacks] nextcloud nextcloud:34.0.4-apache@sha256:a5ace30c695afe48c2c406e940ee7886a81e13fa382e57cd68b2416d1a66914c running Up 3 seconds (health: starting) +2026/09/24 12:37:14 manager.go:1546: [INFO] [stacks] nextcloud-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 21 seconds (healthy) +2026/09/24 12:37:14 manager.go:1546: [INFO] [stacks] nextcloud-redis redis:7-alpine@sha256:c35af3bbcef51a62c8bae5a9a563c6f1b60d7ebaea4cb5a3ccbcc157580ae098 running Up 9 seconds (healthy) +2026/09/24 12:37:19 restore.go:220: [DEBUG] [backup] Post-restore health check: nextcloud is running +2026/09/24 12:37:19 restore_unit.go:464: [INFO] [backup] Restore-from-unit completed: nextcloud — 3 volume(s) of 3 listed, 1 database(s) of 1 listed +2026/09/24 12:37:19 settings.go:1848: [INFO] [settings] restore hold CLEARED for nextcloud +2026/09/24 12:37:19 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18567 bytes) +2026/09/24 12:37:19 settings.go:847: [INFO] [settings] Settings saved +2026/09/24 12:37:19 update_guard.go:609: [INFO] [backup] nextcloud: restore completed — the update hold (set 2026-09-24T12:36:32Z) is CLEARED +2026/09/24 12:37:19 tier2_restore.go:251: [INFO] [backup] nextcloud: primary unit already carries data — left untouched (R-403 never overwrites a richer package with a poorer one) +2026/09/24 12:37:19 handlers.go:2074: [INFO] [web] Tier-2 whole restore completed (async): stack=nextcloud in 40.750999537s (files restored 0, replaced 0, kept-newer 0, unchanged 142; volumes 3/3, dbs 1/1) +2026/09/24 12:37:21 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:37:21 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5477MB avail≈20421MB +2026/09/24 12:37:21 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24217712 → total=25898MB avail=20421MB used=5477MB (21.1%) +2026/09/24 12:37:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=61.8GB avail=3.0GB (90.5%) +2026/09/24 12:37:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:37:21 info.go:13: [DEBUG] [system] readLoadAvg: raw="3.30 3.73 3.50 3/1518 2510" → 1m=3.30 5m=3.73 15m=3.50 +2026/09/24 12:37:21 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:37:21 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 66.2°C (hwmon2) +2026/09/24 12:37:21 info.go:13: [DEBUG] [system] GetInfo done in 72ms — mem=5477MB/25898MB (21.1%), rootDisk=61.8GB/68.4GB (90.5%), load=3.30/3.73/3.50, temp=66.2°C (hwmon2), cpu=26.0% +2026/09/24 12:37:21 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:37:21 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan +2026/09/24 12:37:21 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting +2026/09/24 12:37:21 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:37:21 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:37:21 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:37:21 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:37:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:37:21 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:37:21 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:37:21 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:37:21 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 84ms) +2026/09/24 12:37:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 3m44s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:21 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:37:21 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:37:21 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:37:21 healthprobe.go:159: [WARN] Health probes: 0 ok, 2 unhealthy (of 2 probed) +2026/09/24 12:37:21 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:37:21 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:37:21 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.2:51556 +2026/09/24 12:37:31 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:37:31 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:37:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:37:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 3m54s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:37:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:37:31 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) diff --git a/documentation/audits/night-2026-09-24/E/round-11.json b/documentation/audits/night-2026-09-24/E/round-11.json new file mode 100644 index 00000000..31f999d4 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-11.json @@ -0,0 +1,267 @@ +{ + "round": 11, + "action": "cutoff_undo_file_app", + "accident": { + "kind": "disk_to_floor_plus_1g", + "cmd": "fallocate /var/lib/docker/chaos-fill leaving 3 GB free", + "fired_at": "14:35:06", + "result": "rc=0 filled 36534034432 bytes /dev/loop1 69G 62G 3.0G 96% /var/lib/docker " + }, + "app": "nextcloud", + "started": "2026-09-24T12:34:07Z", + "before": { + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "hold_kind": null, + "update_error": null, + "ladder_steps_left": null + }, + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "page_hu": [ + "Temperature high: 80°C (hwmon2) Rendszermonitor →", + "+ 1 további figyelmeztetés →", + "Az ügyfélszolgálat foglalkozik ezzel az alkalmazással, ezért az adatai megmaradnak: most csak magát az alkalmazást távolíthatod el." + ], + "page_en": [ + "Temperature high: 80°C (hwmon2) System monitor →", + "+ 1 more warning →", + "Support is working on this app, so its data stays: for now you can remove only the app itself." + ] + }, + "badge_catchup_s": 4.7, + "act": "Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}", + "cutoff": "copy: nextcloud_nextcloud_redis_data.pre-update-20260924T123442Z\ndata", + "armed_at": "14:35:03", + "armed_why": "at verifying", + "phases": [ + [ + 6.2, + "backing-up" + ], + [ + 29.8, + "safety-dump" + ], + [ + 32.0, + "pulling" + ], + [ + 33.5, + "copying" + ], + [ + 45.6, + "starting" + ], + [ + 51.9, + "verifying" + ], + [ + 142.6, + "undoing" + ], + [ + 145.8, + "failed" + ] + ], + "held": { + "update_phase": "failed", + "hold_kind": "update_failed", + "hold_no_whole_copy": null, + "hold_reason": "A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-24 14:36-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: második meghajtó, 2026-09-24 14:34 — ez a másolat a beállításokat, az adatbázist és a fájlokat tartalmazza." + }, + "held_page": { + "state": { + "deployed": true, + "state": "stopped", + "update_phase": "failed", + "hold_reason": "A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-24 14:36-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: második meghajtó, 2026-09-24 14:34 — ez a másolat a beállításokat, az adatbázist és a fájlokat tartalmazza.", + "hold_kind": "update_failed", + "update_error": "A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-24 14:36-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: második meghajtó, 2026-09-24 14:34 — ez a másolat a beállításokat, az adatbázist és a fájlokat tartalmazza.", + "ladder_steps_left": null + }, + "badges": { + "hu": [ + { + "title": "A frissítés nem sikerült, és az automatikus visszaállítás sem.", + "text": "Megállítva — visszaállítás szükséges" + } + ], + "en": [ + { + "title": "The update did not succeed, and the automatic undo did not either.", + "text": "Stopped — restore needed" + } + ] + }, + "page_hu": [ + "Temperature high: 80°C (hwmon2) Rendszermonitor →", + "+ 1 további figyelmeztetés →", + "A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-24 14:36-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: második meghajtó, 2026-09-2", + "Az ügyfélszolgálat foglalkozik ezzel az alkalmazással, ezért az adatai megmaradnak: most csak magát az alkalmazást távolíthatod el." + ], + "page_en": [ + "Temperature high: 80°C (hwmon2) System monitor →", + "+ 1 more warning →", + "The update did not succeed, and the automatic undo did not either. The data is as the new version left it. The update of nextcloud at 2026-09-24 14:36 did not succeed, and the app did not start on the new version. The app stays stopped for safety, so that its data is not damaged. It can be restored on the Backups page from this backup: second drive, 2026-09-24 14:34 — this copy holds the settings,", + "Support is working on this app, so its data stays: for now you can remove only the app itself." + ] + }, + "whole_restore_press": "302", + "whole_restore": { + "op": "tier2-whole-restore", + "stack": "nextcloud", + "ok": true, + "message": "A(z) nextcloud: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult. A meghajtó fájljai: 0 visszahozva, 0 kicserélve (a régebbi élő példány megmaradt mellette, .felhom-… végződéssel), 0 újabb példány érintetlenül hagyva, 142 változatlan. Fájl nem lett törölve.", + "finished_at": "2026-09-24T12:37:19.572384468Z" + }, + "time_to_steady_s": 194.1, + "after": { + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "hold_kind": null, + "update_error": null, + "ladder_steps_left": null + }, + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "page_hu": [ + "Temperature high: 80°C (hwmon2) Rendszermonitor →", + "+ 1 további figyelmeztetés →", + "Az ügyfélszolgálat foglalkozik ezzel az alkalmazással, ezért az adatai megmaradnak: most csak magát az alkalmazást távolíthatod el." + ], + "page_en": [ + "Temperature high: 80°C (hwmon2) System monitor →", + "+ 1 more warning →", + "Support is working on this app, so its data stays: for now you can remove only the app itself." + ] + }, + "readback": { + "A": true, + "files": [ + { + "felhom-seed-0-e19728.txt": true, + "felhom-seed-2-8620d3.txt": true + }, + "404" + ] + }, + "events": [], + "box_log": [ + "2026-09-24T12:34:37Z INFO [backup] update safety dump for nextcloud: 1 file(s) [/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/pre-restore-20260924T123437Z-nextcloud-mariadb.sql]", + "2026-09-24T12:34:37Z INFO [stacks] update nextcloud: safety dump done (1 file(s)) [/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/pre-restore-20260924T123437Z-nextcloud-mariadb.sql]", + "2026-09-24T12:34:39Z INFO [stacks] update nextcloud: the undo copy will hold 3 named volume(s), 928.8 MiB", + "2026-09-24T12:34:39Z INFO [stacks] update nextcloud: phase pinning", + "2026-09-24T12:34:39Z INFO [stacks] update nextcloud: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/nextcloud/docker-compose.yml (nextcloud=nextcloud:34.0.4-apache, nextcloud-db=mariadb:12.3, nextcloud-redis=redis:7.4.0-alpine)", + "2026-09-24T12:34:39Z INFO [stacks] update nextcloud: phase pulling", + "2026-09-24T12:34:40Z INFO [stacks] update nextcloud: phase copying", + "2026-09-24T12:34:42Z INFO [stacks] update nextcloud: phase copying", + "2026-09-24T12:34:43Z INFO [stacks] update nextcloud: copied nextcloud_nextcloud_db_data → nextcloud_nextcloud_db_data.pre-update-20260924T123442Z in 1.346s", + "2026-09-24T12:34:43Z INFO [stacks] update nextcloud: phase copying", + "2026-09-24T12:34:52Z INFO [stacks] update nextcloud: copied nextcloud_nextcloud_html → nextcloud_nextcloud_html.pre-update-20260924T123442Z in 8.62s", + "2026-09-24T12:34:52Z INFO [stacks] update nextcloud: phase copying", + "2026-09-24T12:34:52Z INFO [stacks] update nextcloud: copied nextcloud_nextcloud_redis_data → nextcloud_nextcloud_redis_data.pre-update-20260924T123442Z in 455ms", + "2026-09-24T12:34:52Z INFO [stacks] update nextcloud: phase starting", + "2026-09-24T12:34:52Z INFO [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping ensure (held by drive gate)", + "2026-09-24T12:34:58Z INFO [stacks] update nextcloud: phase verifying", + "2026-09-24T12:35:08Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:35:13Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:35:19Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:35:24Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:35:29Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:35:34Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:35:39Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:35:44Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:35:49Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:35:54Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:35:59Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:36:04Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:36:09Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:36:14Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:36:19Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:36:24Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:36:29Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused", + "2026-09-24T12:36:29Z ERROR [stacks] update nextcloud FAILED after the new version was started: not healthy: not healthy within 1m30s (last: health check failing)", + "2026-09-24T12:36:29Z INFO [stacks] update nextcloud: kept 6809 bytes of the app's own log at /opt/docker/stacks/nextcloud/hold-logs/20260924T123629Z/compose-logs.txt before stopping it (R-621)", + "2026-09-24T12:36:29Z INFO [stacks] update nextcloud: phase undoing", + "2026-09-24T12:36:29Z WARN [stacks] update nextcloud: UNDO — putting back the previous version and its 3 volume copy(ies) (reason: not healthy: not healthy within 1m30s (last: health check failing))", + "2026-09-24T12:36:32Z ERROR [stacks] update nextcloud: the undo copy nextcloud_nextcloud_redis_data.pre-update-20260924T123442Z has no finished-marker — it is cut off or missing; NOTHING is put back", + "2026-09-24T12:36:32Z ERROR [stacks] update nextcloud: the UNDO failed too (untouched) — HOLDING the app; the undo copies are kept: [{nextcloud_nextcloud_db_data nextcloud_nextcloud_db_data.pre-update-20260924T123442Z} {nextcloud_nextcloud_html nextcloud_nextcloud_html.pre-update-20260924T123442Z} {nextcloud_nextcloud_redis_da", + "2026-09-24T12:36:32Z WARN [settings] restore hold SET for nextcloud — the app stays stopped until it is cleared", + "2026-09-24T12:36:32Z WARN [backup] nextcloud is HELD STOPPED after a failed update (restore point: tier 2 \"második meghajtó\", 2026-09-24T12:34:34Z; holds: \"a beállításokat, az adatbázist és a fájlokat tartalmazza\"; undo: \"untouched\")", + "2026-09-24T12:36:32Z WARN [settings] restore hold SET for nextcloud — the app stays stopped until it is cleared", + "2026-09-24T12:36:32Z INFO [stacks] update nextcloud: event app_update_held (hold recorded: true)", + "2026-09-24T12:36:34Z INFO [sync] Updated nextcloud/.felhom.yml", + "2026-09-24T12:36:34Z INFO [api] Catalog sync completed: Sablonok frissítve — frissítve: nextcloud", + "2026-09-24T12:36:38Z WARN [web] Tier-2 WHOLE restore requested (async, files + database): stack=nextcloud from 172.18.0.2:51556", + "2026-09-24T12:36:38Z WARN [backup] WHOLE restore for nextcloud from the second drive /mnt/sys_drive/felhom-data/backups/secondary/nextcloud: files by the four rules (never delete, never overwrite newer), then the unit", + "2026-09-24T12:36:38Z INFO [stacks] Stopping stack: nextcloud", + "2026-09-24T12:36:38Z INFO [stacks] Stack nextcloud stopped successfully (took 0.1s)", + "2026-09-24T12:36:39Z INFO [backup] whole restore nextcloud: files restored=0 replaced=0 (live kept beside as *.felhom-20260924T123638Z) kept-newer=0 unchanged=142", + "2026-09-24T12:36:39Z INFO [backup] Restoring nextcloud from recovery unit /mnt/sys_drive/felhom-data/backups/secondary/nextcloud/recovery-unit: images=3, secrets recovered=3/3, data_keys=0", + "2026-09-24T12:36:39Z INFO [stacks] Stopping stack: nextcloud", + "2026-09-24T12:36:39Z INFO [stacks] Stack nextcloud stopped successfully (took 0.1s)", + "2026-09-24T12:36:39Z INFO [backup] Restoring Docker volume nextcloud_nextcloud_db_data for nextcloud", + "2026-09-24T12:36:41Z INFO [backup] Restoring Docker volume nextcloud_nextcloud_html for nextcloud", + "2026-09-24T12:36:52Z INFO [backup] Restoring Docker volume nextcloud_nextcloud_redis_data for nextcloud", + "2026-09-24T12:36:52Z INFO [backup] Restored 3 Docker volume(s) for nextcloud", + "2026-09-24T12:36:52Z INFO [stacks] Redeploying nextcloud from recovery unit with 7 env vars", + "2026-09-24T12:36:52Z INFO [stacks] pin nextcloud: nextcloud=nextcloud:34.0.4-apache, nextcloud-db=mariadb:12.3, nextcloud-redis=redis:7-alpine", + "2026-09-24T12:36:52Z INFO [stacks] Starting stack nextcloud services only: [nextcloud-db]", + "2026-09-24T12:36:52Z INFO [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping ensure (held by drive gate)", + "2026-09-24T12:36:53Z INFO [stacks] Stack nextcloud services [nextcloud-db] started (took 0.5s)", + "2026-09-24T12:36:53Z INFO [backup] Discovered 3 databases", + "2026-09-24T12:36:53Z INFO [backup] Restore nextcloud: replaying DB dump into nextcloud-db (mariadb)", + "2026-09-24T12:37:05Z INFO [backup] Imported DB dump nextcloud-mariadb.sql into nextcloud-db (mariadb)", + "2026-09-24T12:37:05Z INFO [backup] Restore nextcloud: replayed 1 DB dump(s)", + "2026-09-24T12:37:05Z INFO [stacks] Starting stack: nextcloud", + "2026-09-24T12:37:05Z INFO [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping ensure (held by drive gate)", + "2026-09-24T12:37:11Z INFO [stacks] Stack nextcloud started successfully (took 6.3s)", + "2026-09-24T12:37:11Z INFO [stacks] installed-images nextcloud: recorded 3 service(s) (nextcloud=nextcloud:34.0.4-apache (sha256:a5ace30c695a…), nextcloud-db=mariadb:12.3 (sha256:805c8e104bd5…), nextcloud-redis=redis:7-alpine (sha256:c35af3bbcef5…))", + "2026-09-24T12:37:14Z INFO [stacks] Stack nextcloud post-start status:", + "2026-09-24T12:37:14Z INFO [stacks] nextcloud nextcloud:34.0.4-apache@sha256:a5ace30c695afe48c2c406e940ee7886a81e13fa382e57cd68b2416d1a66914c running Up 3 seconds (health: starting)", + "2026-09-24T12:37:14Z INFO [stacks] nextcloud-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 21 seconds (healthy)", + "2026-09-24T12:37:14Z INFO [stacks] nextcloud-redis redis:7-alpine@sha256:c35af3bbcef51a62c8bae5a9a563c6f1b60d7ebaea4cb5a3ccbcc157580ae098 running Up 9 seconds (healthy)", + "2026-09-24T12:37:19Z INFO [backup] Restore-from-unit completed: nextcloud — 3 volume(s) of 3 listed, 1 database(s) of 1 listed", + "2026-09-24T12:37:19Z INFO [settings] restore hold CLEARED for nextcloud", + "2026-09-24T12:37:19Z INFO [backup] nextcloud: restore completed — the update hold (set 2026-09-24T12:36:32Z) is CLEARED", + "2026-09-24T12:37:19Z INFO [backup] nextcloud: primary unit already carries data — left untouched (R-403 never overwrites a richer package with a poorer one)", + "2026-09-24T12:37:19Z INFO [web] Tier-2 whole restore completed (async): stack=nextcloud in 40.750999537s (files restored 0, replaced 0, kept-newer 0, unchanged 142; volumes 3/3, dbs 1/1)", + "2026-09-24T12:37:21Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.11:8999: connect: connection refused" + ], + "undo_copies_left": "12" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-24/E/round-11.log b/documentation/audits/night-2026-09-24/E/round-11.log new file mode 100644 index 00000000..7930a28e --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-11.log @@ -0,0 +1,24 @@ +14:34:07 ==== ROUND 11: cutoff_undo_file_app on nextcloud x disk_to_floor_plus_1g (2026-09-24T12:34:07Z) +14:34:08 drill: nextcloud: failing step redis:7-alpine -> redis:7.4.0-alpine + probe 8999 +14:34:13 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +14:34:13 + 6.2s phase=backing-up +14:34:37 + 29.8s phase=safety-dump +14:34:39 + 32.0s phase=pulling +14:34:40 + 33.5s phase=copying +14:34:52 + 45.6s phase=starting +14:34:59 + 51.9s phase=verifying +14:35:03 >>> finished-marker removed from the NEWEST undo copy: +copy: nextcloud_nextcloud_redis_data.pre-update-20260924T123442Z +data + +14:35:03 >>> ARMED accident disk_to_floor_plus_1g (at verifying) +14:36:29 + 142.6s phase=undoing +14:36:33 + 145.8s phase=failed +14:36:33 + 145.8s nextcloud: deployed=True state=stopped updating=False phase=failed hold=update_failed +14:36:34 drill: nextcloud: back to redis:7-alpine and its real probe +14:36:38 whole restore (the second drive) -> 302 +14:37:21 whole restore: {'op': 'tier2-whole-restore', 'stack': 'nextcloud', 'ok': True, 'message': 'A(z) nextcloud: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult. A meghajtó fájljai: 0 visszahozva, 0 kicserélve (a régebbi élő példány megmaradt mellette, .felhom-… végződéssel), 0 újabb példány érintet +14:37:21 + 194.1s nextcloud: deployed=True state=running updating=False phase=None hold=None +14:37:21 >>> accident done: {'kind': 'disk_to_floor_plus_1g', 'cmd': 'fallocate /var/lib/docker/chaos-fill leaving 3 GB free', 'fired_at': '14:35:06', 'result': 'rc=0 filled 36534034432 bytes /dev/loop1 69G 62G 3.0G 96% /var/lib/docker '} +14:37:29 nextcloud: readback of the seeded user found=True +14:37:31 nextcloud files read back: 2/2 (negative control http=404) diff --git a/documentation/audits/night-2026-09-24/E/round-11.stdout b/documentation/audits/night-2026-09-24/E/round-11.stdout new file mode 100644 index 00000000..1da2d704 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-11.stdout @@ -0,0 +1,25 @@ +14:34:07 ==== ROUND 11: cutoff_undo_file_app on nextcloud x disk_to_floor_plus_1g (2026-09-24T12:34:07Z) +14:34:08 drill: nextcloud: failing step redis:7-alpine -> redis:7.4.0-alpine + probe 8999 +14:34:13 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +14:34:13 + 6.2s phase=backing-up +14:34:37 + 29.8s phase=safety-dump +14:34:39 + 32.0s phase=pulling +14:34:40 + 33.5s phase=copying +14:34:52 + 45.6s phase=starting +14:34:59 + 51.9s phase=verifying +14:35:03 >>> finished-marker removed from the NEWEST undo copy: +copy: nextcloud_nextcloud_redis_data.pre-update-20260924T123442Z +data + +14:35:03 >>> ARMED accident disk_to_floor_plus_1g (at verifying) +14:36:29 + 142.6s phase=undoing +14:36:33 + 145.8s phase=failed +14:36:33 + 145.8s nextcloud: deployed=True state=stopped updating=False phase=failed hold=update_failed +14:36:34 drill: nextcloud: back to redis:7-alpine and its real probe +14:36:38 whole restore (the second drive) -> 302 +14:37:21 whole restore: {'op': 'tier2-whole-restore', 'stack': 'nextcloud', 'ok': True, 'message': 'A(z) nextcloud: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult. A meghajtó fájljai: 0 visszahozva, 0 kicserélve (a régebbi élő példány megmaradt mellette, .felhom-… végződéssel), 0 újabb példány érintet +14:37:21 + 194.1s nextcloud: deployed=True state=running updating=False phase=None hold=None +14:37:21 >>> accident done: {'kind': 'disk_to_floor_plus_1g', 'cmd': 'fallocate /var/lib/docker/chaos-fill leaving 3 GB free', 'fired_at': '14:35:06', 'result': 'rc=0 filled 36534034432 bytes /dev/loop1 69G 62G 3.0G 96% /var/lib/docker '} +14:37:29 nextcloud: readback of the seeded user found=True +14:37:31 nextcloud files read back: 2/2 (negative control http=404) +14:37:37 ==== ROUND 11 END: steady in 194.1s; readback={'A': True, 'files': ({'felhom-seed-0-e19728.txt': True, 'felhom-seed-2-8620d3.txt': True}, '404')}; events=[] diff --git a/documentation/audits/night-2026-09-24/E/round-12-controller.log b/documentation/audits/night-2026-09-24/E/round-12-controller.log new file mode 100644 index 00000000..62cef77a --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-12-controller.log @@ -0,0 +1,431 @@ +2026/09/24 12:37:58 auth.go:154: [DEBUG] [web] login attempt from 172.18.0.2:51556 (X-Forwarded-For: 192.168.0.180) +2026/09/24 12:37:58 auth.go:202: [DEBUG] [web] login successful from 192.168.0.180, creating session +2026/09/24 12:37:58 auth.go:268: [DEBUG] [web] session created, expires=2026-10-01T12:37:58Z, active_sessions=4 +2026/09/24 12:37:58 auth.go:222: [INFO] [web] Login from 172.18.0.2:51556 +2026/09/24 12:37:58 server.go:545: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.2:51556 +2026/09/24 12:37:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.2:51556 +2026/09/24 12:37:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556 +2026/09/24 12:37:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556 +2026/09/24 12:37:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556 +2026/09/24 12:37:58 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556 +2026/09/24 12:37:58 router.go:591: [INFO] [api] start requested for stack: chaosoomb +2026/09/24 12:37:58 settings.go:2677: [INFO] [settings] unhealthy-stop hold LIFTED for chaosoomb (Start) +2026/09/24 12:37:58 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18389 bytes) +2026/09/24 12:37:58 settings.go:847: [INFO] [settings] Settings saved +2026/09/24 12:37:58 router.go:615: [INFO] [api] chaosoomb: the unhealthy stop is LIFTED by Start — one more try (decision 28) +2026/09/24 12:37:58 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5507MB avail≈20391MB +2026/09/24 12:37:58 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24182664 → total=25898MB avail=20391MB used=5507MB (21.3%) +2026/09/24 12:37:58 manager.go:1194: [DEBUG] [stacks] StartStack chaosoomb: current state=stopped deployed=true +2026/09/24 12:37:58 manager.go:1197: [INFO] [stacks] Starting stack: chaosoomb +2026/09/24 12:37:58 manager.go:1204: [DEBUG] [stacks] StartStack chaosoomb: prepared 8 env vars for compose +2026/09/24 12:37:58 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system) +2026/09/24 12:37:58 manager.go:1463: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/chaosoomb) +2026/09/24 12:37:58 manager.go:1483: [DEBUG] Command completed: docker compose up -d (took 0.4s) +2026/09/24 12:37:58 manager.go:1212: [INFO] [stacks] Stack chaosoomb started successfully (took 0.4s) +2026/09/24 12:37:59 installed.go:409: [DEBUG] [stacks] installed-images chaosoomb: unchanged (1 service(s)) — app.yaml not rewritten +2026/09/24 12:37:59 lifecycle.go:70: [DEBUG] [integrations] OnStackStart: stack=chaosoomb, waiting 5s for state refresh +2026/09/24 12:38:01 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 4m24s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:01 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:38:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:01 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/24 12:38:01 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:38:01 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/24 12:38:02 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system) +2026/09/24 12:38:02 manager.go:1463: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/chaosoomb) +2026/09/24 12:38:02 manager.go:1483: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s) +2026/09/24 12:38:02 manager.go:1543: [INFO] [stacks] Stack chaosoomb post-start status: +2026/09/24 12:38:02 manager.go:1546: [INFO] [stacks] chaosoomb alpine:3.20 running Up 3 seconds +2026/09/24 12:38:04 lifecycle.go:86: [DEBUG] [integrations] OnStackStart: stack=chaosoomb integrationsFound=0 +2026/09/24 12:38:11 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:38:11 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:38:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 4m34s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:11 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:38:11 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:38:11 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:38:11 healthprobe.go:159: [WARN] Health probes: 0 ok, 2 unhealthy (of 2 probed) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5431MB avail≈20467MB +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24273396 → total=25898MB avail=20467MB used=5431MB (21.0%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=27.8GB avail=37.0GB (40.7%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readLoadAvg: raw="2.69 3.49 3.43 3/1532 2820" → 1m=2.69 5m=3.49 15m=3.43 +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 68.5°C (hwmon2) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetInfo done in 67ms — mem=5431MB/25898MB (21.0%), rootDisk=27.8GB/68.4GB (40.7%), load=2.69/3.49/3.43, temp=68.5°C (hwmon2), cpu=30.3% +2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:38:21 scheduler.go:346: [INFO] [scheduler] Running job: offsite-credential-retry +2026/09/24 12:38:21 scheduler.go:346: [INFO] [scheduler] Running job: backup-cache +2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job offsite-credential-retry: execution starting +2026/09/24 12:38:21 scheduler.go:363: [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job backup-cache: execution starting +2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:38:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 4m44s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:21 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:38:21 scheduler.go:346: [INFO] [scheduler] Running job: system-health +2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job system-health: execution starting +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:38:21 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:38:21 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:38:21 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:38:21 healthprobe.go:159: [WARN] Health probes: 0 ok, 2 unhealthy (of 2 probed) +2026/09/24 12:38:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/nextcloud → [nextcloud] +2026/09/24 12:38:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx → [paperless-ngx] +2026/09/24 12:38:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [privatebin, vikunja, wishlist, chaosoom, chaosoomb, n8n] +2026/09/24 12:38:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/romm → [romm] +2026/09/24 12:38:21 backup.go:467: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/navidrome → [navidrome] +2026/09/24 12:38:21 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18389 bytes) +2026/09/24 12:38:21 settings.go:847: [INFO] [settings] Settings saved +2026/09/24 12:38:21 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18617 bytes) +2026/09/24 12:38:21 settings.go:847: [INFO] [settings] Settings saved +2026/09/24 12:38:21 backup.go:1119: [INFO] [backup] Found 10 DB dump files across drives +2026/09/24 12:38:21 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5439MB avail≈20459MB +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24263608 → total=25898MB avail=20459MB used=5439MB (21.0%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=27.8GB avail=37.0GB (40.7%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readLoadAvg: raw="2.69 3.49 3.43 5/1557 2877" → 1m=2.69 5m=3.49 15m=3.43 +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 68.0°C (hwmon2) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetInfo done in 78ms — mem=5439MB/25898MB (21.0%), rootDisk=27.8GB/68.4GB (40.7%), load=2.69/3.49/3.43, temp=68.0°C (hwmon2), cpu=31.6% +2026/09/24 12:38:21 healthcheck.go:86: [DEBUG] [monitor] Raw values: disk=40.7%, hdd=7.8% (configured=true), mem=21.0% (5439MB/25898MB), cpu=31.6%, temp=68.0°C (hwmon2) +2026/09/24 12:38:21 healthcheck.go:118: [DEBUG] [monitor] SSD disk: OK (41%) +2026/09/24 12:38:21 healthcheck.go:151: [DEBUG] [monitor] Memory: OK (21%) +2026/09/24 12:38:21 healthcheck.go:169: [DEBUG] [monitor] CPU: OK (32%) +2026/09/24 12:38:21 healthcheck.go:187: [DEBUG] [monitor] Temperature: OK (68°C) +2026/09/24 12:38:21 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 231d20f4db9e chaosoomb chaosoomb alpine:3.20 +ec2856735161 nextcloud nextcloud nextcloud:34.0.4-apache +560874757b4e nextcloud-redis nextcloud redis:7-alpine +f990f2f4a0e6 nextcloud-db nextcloud mariadb:12.3 +517b7ba21f80 n8n n8n n8nio/n8n:2.40.5 +a0b33d0b9958 wishlist wishlist ghcr.io/cmintey/wishlist:v0.67.1 +9f47a1f5273b vikunja vikunja vikunja/vikunja:2.3.0 +1adf3ec73c9b romm romm rommapp/romm:5.3.1 +e059bbc146b0 romm-redis romm redis:7-alpine +fc5ed5428e75 romm-db romm mariadb:11.8 +361a05346765 priv... +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container chaosoomb (image=alpine:3.20, not a database) +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database) +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/24 12:38:21 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=f990f2f4a0e6) +2026/09/24 12:38:21 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container n8n (image=n8nio/n8n:2.40.5, not a database) +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container wishlist (image=ghcr.io/cmintey/wishlist:v0.67.1, not a database) +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.3.0, not a database) +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/24 12:38:21 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=fc5ed5428e75) +2026/09/24 12:38:21 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/24 12:38:21 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5e5c371b82fa) +2026/09/24 12:38:21 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/24 12:38:21 healthcheck.go:201: [DEBUG] [monitor] Docker daemon: OK +2026/09/24 12:38:21 healthcheck.go:209: [DEBUG] [monitor] Checking 1 protected containers: [traefik] +2026/09/24 12:38:21 main.go:872: [WARN] [deadapp] chaosoomb: container chaosoomb was OOM-killed (started 2026-09-24T12:37:58.784734494Z; kills 18, limit 32M, peak 32M) +2026/09/24 12:38:21 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_oom (severity warning) — further app_oom events are logged at DEBUG only +2026/09/24 12:38:21 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.1, not a database) +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.269.1, not a database) +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/24 12:38:21 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/24 12:38:21 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 15 non-DB container(s) +2026/09/24 12:38:21 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:38:21 healthcheck.go:219: [DEBUG] [monitor] All protected containers running +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:38:21 healthcheck.go:238: [INFO] [monitor] Health check: status=warn +2026/09/24 12:38:21 healthcheck.go:242: [DEBUG] [monitor] Final status: warn (issues=0, warnings=4, info=5) +2026/09/24 12:38:21 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event health_change (severity warning) — further health_change events are logged at DEBUG only +2026/09/24 12:38:21 scheduler.go:363: [INFO] [scheduler] Job system-health completed (took 192ms) +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:38:21 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:38:21 [INFO] [backup] Discovered app data: 10 apps +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:38:21 offbox_reconstitute.go:319: [INFO] [backup] nextcloud: pruned old undo copy pre-restore-20260924T105053Z-nextcloud-mariadb.sql (keeping the newest 3) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=73.4 GB avail=816.7GB (7.8%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%) +2026/09/24 12:38:21 recovery_unit.go:224: [INFO] [backup] Recovery unit captured for chaosoomb → /mnt/sys_drive/felhom-data/backups/primary/chaosoomb (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026/09/24 12:38:21 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=27.8 GB avail=37.0GB (40.7%) +2026/09/24 12:38:21 recovery_unit.go:224: [INFO] [backup] Recovery unit captured for n8n → /mnt/sys_drive/felhom-data/backups/primary/n8n (images=1, secrets-referenced=1, data_keys=0, portable-carried=1/1, withheld=0) +2026/09/24 12:38:21 backup.go:1189: [INFO] [backup] Backup status cache refreshed +2026/09/24 12:38:21 scheduler.go:363: [INFO] [scheduler] Job backup-cache completed (took 222ms) +2026/09/24 12:38:21 infra.go:67: [INFO] [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node) +2026/09/24 12:38:31 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:38:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 4m54s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:38:31 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:38:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:38:31 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:38:31 healthprobe.go:181: [DEBUG] Health probe navidrome: API GET :4533/ping → 200 (2ms) +2026/09/24 12:38:31 healthprobe.go:181: [DEBUG] Health probe privatebin: HTTP GET :8080/ → 200 (5ms) +2026/09/24 12:38:31 healthprobe.go:161: [INFO] Health probes: 2 ok (of 2 probed) +2026/09/24 12:38:41 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:38:41 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:38:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:41 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 4 targets (4 skipped not due, 0 skipped no container) +2026/09/24 12:38:41 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:38:41 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:38:41 healthprobe.go:181: [DEBUG] Health probe n8n: API GET :5678/healthz → 200 (1ms) +2026/09/24 12:38:41 healthprobe.go:181: [DEBUG] Health probe wishlist: HTTP GET :3000/ → 307 (4ms) +2026/09/24 12:38:41 healthprobe.go:159: [WARN] Health probes: 2 ok, 2 unhealthy (of 4 probed) +2026/09/24 12:38:51 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:38:51 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:38:51 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:38:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:38:51 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:38:51 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:38:51 main.go:872: [WARN] [deadapp] chaosoomb: container chaosoomb was OOM-killed (started 2026-09-24T12:38:41.277810467Z; kills 8, limit 32M, peak 32M) +2026/09/24 12:38:51 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_oom (severity warning) +2026/09/24 12:39:01 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:39:01 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:39:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:01 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:39:01 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:39:01 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:39:01 healthprobe.go:159: [WARN] Health probes: 0 ok, 2 unhealthy (of 2 probed) +2026/09/24 12:39:11 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:11 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:39:11 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:11 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:39:21 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5509MB avail≈20389MB +2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=24179764 → total=25898MB avail=20389MB used=5509MB (21.3%) +2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=27.8GB avail=37.0GB (40.7%) +2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=73.4GB avail=816.7GB (7.8%) +2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readLoadAvg: raw="3.02 3.42 3.41 7/1539 3107" → 1m=3.02 5m=3.42 15m=3.41 +2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/24 12:39:21 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 69.4°C (hwmon2) +2026/09/24 12:39:21 info.go:13: [DEBUG] [system] GetInfo done in 80ms — mem=5509MB/25898MB (21.3%), rootDisk=27.8GB/68.4GB (40.7%), load=3.02/3.42/3.41, temp=69.4°C (hwmon2), cpu=34.7% +2026/09/24 12:39:21 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:39:21 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:39:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:21 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:21 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:39:21 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:39:21 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan +2026/09/24 12:39:21 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting +2026/09/24 12:39:21 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 4 targets (4 skipped not due, 0 skipped no container) +2026/09/24 12:39:21 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:39:21 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:39:21 healthprobe.go:181: [DEBUG] Health probe romm: HTTP GET :8080/ → 200 (2ms) +2026/09/24 12:39:21 healthprobe.go:181: [DEBUG] Health probe paperless-ngx: HTTP GET :8000/ → 302 (6ms) +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=true composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=true composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=true composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=true composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/24 12:39:21 manager.go:608: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/24 12:39:21 manager.go:1690: [DEBUG] [stacks] getCatalogTemplateSlugs: found 56 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/24 12:39:21 manager.go:687: [DEBUG] [stacks] ScanStacks: catalog has 56 template slugs for orphan detection +2026/09/24 12:39:21 manager.go:718: [INFO] [stacks] ScanStacks complete: 58 stacks found (10 deployed, 48 available) +2026/09/24 12:39:21 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 108ms) +2026/09/24 12:39:21 healthprobe.go:159: [WARN] Health probes: 2 ok, 2 unhealthy (of 4 probed) +2026/09/24 12:39:21 main.go:872: [WARN] [deadapp] chaosoomb: container chaosoomb was OOM-killed (started 2026-09-24T12:39:20.327438775Z; kills 1, limit 32M, peak 32M) +2026/09/24 12:39:21 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_oom (severity warning) +2026/09/24 12:39:31 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:39:31 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:39:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:31 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:31 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:39:31 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:39:31 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:39:31 healthprobe.go:159: [WARN] Health probes: 0 ok, 2 unhealthy (of 2 probed) +2026/09/24 12:39:41 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:39:41 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:41 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 10s, healthy=false +2026/09/24 12:39:41 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/24 12:39:51 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:39:51 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/24 12:39:51 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/24 12:39:51 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:39:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:51 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/24 12:39:51 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:39:51 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:39:51 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:39:51 healthprobe.go:159: [WARN] Health probes: 0 ok, 2 unhealthy (of 2 probed) +2026/09/24 12:39:51 main.go:872: [WARN] [deadapp] chaosoomb: container chaosoomb was OOM-killed (started 2026-09-24T12:39:20.327438775Z; kills 22, limit 32M, peak 32M) +2026/09/24 12:39:51 notifier.go:738: [ERROR] [notify] chaosoomb: container chaosoomb OOM STORM — 22 kills in 30 min (limit 32M, peak 32M) +2026/09/24 12:39:51 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_oom_storm (severity error) — further app_oom_storm events are logged at DEBUG only +2026/09/24 12:39:51 main.go:3671: [WARN] [deadapp] chaosoomb: oom_storm — 21 in 30m0s; STOPPING it (decision 28) +2026/09/24 12:39:51 manager.go:1283: [DEBUG] [stacks] StopStack chaosoomb: current state=running deployed=true containers=1 +2026/09/24 12:39:51 manager.go:1286: [INFO] [stacks] Stopping stack: chaosoomb +2026/09/24 12:39:51 manager.go:1454: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system) +2026/09/24 12:39:51 manager.go:1463: [DEBUG] Running: docker compose down (in /opt/docker/stacks/chaosoomb) +2026/09/24 12:40:01 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/24 12:40:01 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/24 12:40:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping n8n — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:40:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:40:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/24 12:40:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping wishlist — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/24 12:40:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:40:01 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/24 12:40:01 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container) +2026/09/24 12:40:01 healthprobe.go:190: [WARN] Health probe vikunja: API GET :8999/api/v1/info → Get "http://vikunja:8999/api/v1/info": dial tcp 172.18.0.5:8999: connect: connection refused +2026/09/24 12:40:01 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.11:8999: connect: connection refused +2026/09/24 12:40:01 healthprobe.go:159: [WARN] Health probes: 0 ok, 2 unhealthy (of 2 probed) +2026/09/24 12:40:01 manager.go:1483: [DEBUG] Command completed: docker compose down (took 10.3s) +2026/09/24 12:40:01 manager.go:1295: [INFO] [stacks] Stack chaosoomb stopped successfully (took 10.3s) +2026/09/24 12:40:01 settings.go:1814: [WARN] [settings] restore hold SET for chaosoomb — the app stays stopped until it is cleared +2026/09/24 12:40:01 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18794 bytes) +2026/09/24 12:40:01 settings.go:847: [INFO] [settings] Settings saved +2026/09/24 12:40:01 settings.go:844: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (18794 bytes) +2026/09/24 12:40:01 settings.go:847: [INFO] [settings] Settings saved +2026/09/24 12:40:01 update_guard.go:776: [WARN] [backup] chaosoomb is STOPPED by the box: oom_storm (trip 2 within 24h0m0s) — Start gives it one more try (decision 28) +2026/09/24 12:40:01 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_stopped_unhealthy (severity warning) — further app_stopped_unhealthy events are logged at DEBUG only +2026/09/24 12:40:03 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556 +2026/09/24 12:40:03 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556 +2026/09/24 12:40:03 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556 +2026/09/24 12:40:03 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/chaosoomb from 172.18.0.2:51556 diff --git a/documentation/audits/night-2026-09-24/E/round-12.json b/documentation/audits/night-2026-09-24/E/round-12.json new file mode 100644 index 00000000..cb018764 --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-12.json @@ -0,0 +1,85 @@ +{ + "round": 12, + "action": "start_after_unhealthy", + "accident": "none", + "app": "chaosoomb", + "started": "2026-09-24T12:37:58Z", + "before": { + "state": "stopped", + "hold_kind": "unhealthy_stop", + "hold_reason": "Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod." + }, + "act": "Start -> 200 {'ok': True, 'data': {'state': 'running'}, 'message': 'Stack chaosoomb start requested — state now: running'}", + "after_start": { + "state": "running", + "hold_kind": null + }, + "time_to_steady_s": 125.3, + "after": { + "state": { + "deployed": true, + "state": "stopped", + "update_phase": null, + "hold_reason": "Az alkalmazást ismét leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). A Felhom ügyfélszolgálatát értesítettük. Az Indítás gombbal újra megpróbálhatod.", + "hold_kind": "unhealthy_stop", + "update_error": null, + "ladder_steps_left": null + }, + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "page_hu": [ + "Az alkalmazást ismét leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). A Felhom ügyfélszolgálatát értesítettük. Az Indítás gombbal újra megpróbálhatod.", + "Az ügyfélszolgálat foglalkozik ezzel az alkalmazással, ezért az adatai megmaradnak: most csak magát az alkalmazást távolíthatod el." + ], + "page_en": [ + "We stopped the app again because it kept crashing (or ran out of memory). Felhom support has been told. Press Start to try again.", + "Support is working on this app, so its data stays: for now you can remove only the app itself." + ] + }, + "readback": null, + "events": [ + "2026-09-24T12:39:21Z app_oom (warning)", + "2026-09-24T12:39:51Z app_oom_storm (error)", + "2026-09-24T12:40:01Z app_stopped_unhealthy (warning)" + ], + "box_log": [ + "2026-09-24T12:37:58Z INFO [api] start requested for stack: chaosoomb", + "2026-09-24T12:37:58Z INFO [settings] unhealthy-stop hold LIFTED for chaosoomb (Start)", + "2026-09-24T12:37:58Z INFO [api] chaosoomb: the unhealthy stop is LIFTED by Start — one more try (decision 28)", + "2026-09-24T12:37:58Z INFO [stacks] Starting stack: chaosoomb", + "2026-09-24T12:37:58Z INFO [stacks] Stack chaosoomb started successfully (took 0.4s)", + "2026-09-24T12:38:02Z INFO [stacks] Stack chaosoomb post-start status:", + "2026-09-24T12:38:02Z INFO [stacks] chaosoomb alpine:3.20 running Up 3 seconds", + "2026-09-24T12:38:21Z INFO [scheduler] Running job: backup-cache", + "2026-09-24T12:38:21Z INFO [backup] Found 10 DB dump files across drives", + "2026-09-24T12:38:21Z WARN [deadapp] chaosoomb: container chaosoomb was OOM-killed (started 2026-09-24T12:37:58.784734494Z; kills 18, limit 32M, peak 32M)", + "2026-09-24T12:38:21Z INFO [backup] Discovered 3 databases", + "2026-09-24T12:38:21Z INFO [backup] nextcloud: pruned old undo copy pre-restore-20260924T105053Z-nextcloud-mariadb.sql (keeping the newest 3)", + "2026-09-24T12:38:21Z INFO [backup] Recovery unit captured for chaosoomb → /mnt/sys_drive/felhom-data/backups/primary/chaosoomb (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)", + "2026-09-24T12:38:21Z INFO [backup] Recovery unit captured for n8n → /mnt/sys_drive/felhom-data/backups/primary/n8n (images=1, secrets-referenced=1, data_keys=0, portable-carried=1/1, withheld=0)", + "2026-09-24T12:38:21Z INFO [backup] Backup status cache refreshed", + "2026-09-24T12:38:21Z INFO [scheduler] Job backup-cache completed (took 222ms)", + "2026-09-24T12:38:51Z WARN [deadapp] chaosoomb: container chaosoomb was OOM-killed (started 2026-09-24T12:38:41.277810467Z; kills 8, limit 32M, peak 32M)", + "2026-09-24T12:39:21Z WARN [deadapp] chaosoomb: container chaosoomb was OOM-killed (started 2026-09-24T12:39:20.327438775Z; kills 1, limit 32M, peak 32M)", + "2026-09-24T12:39:51Z WARN [deadapp] chaosoomb: container chaosoomb was OOM-killed (started 2026-09-24T12:39:20.327438775Z; kills 22, limit 32M, peak 32M)", + "2026-09-24T12:39:51Z ERROR [notify] chaosoomb: container chaosoomb OOM STORM — 22 kills in 30 min (limit 32M, peak 32M)", + "2026-09-24T12:39:51Z WARN [deadapp] chaosoomb: oom_storm — 21 in 30m0s; STOPPING it (decision 28)", + "2026-09-24T12:39:51Z INFO [stacks] Stopping stack: chaosoomb", + "2026-09-24T12:40:01Z INFO [stacks] Stack chaosoomb stopped successfully (took 10.3s)", + "2026-09-24T12:40:01Z WARN [settings] restore hold SET for chaosoomb — the app stays stopped until it is cleared", + "2026-09-24T12:40:01Z WARN [backup] chaosoomb is STOPPED by the box: oom_storm (trip 2 within 24h0m0s) — Start gives it one more try (decision 28)" + ], + "undo_copies_left": "0" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-24/E/round-12.log b/documentation/audits/night-2026-09-24/E/round-12.log new file mode 100644 index 00000000..faacaf1b --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-12.log @@ -0,0 +1,6 @@ +14:37:58 ==== ROUND 12: start_after_unhealthy on chaosoomb x none (2026-09-24T12:37:58Z) +14:37:59 Start -> 200 {'ok': True, 'data': {'state': 'running'}, 'message': 'Stack chaosoomb start requested — state now: running'} +14:38:02 + 3.8s chaosoomb: deployed=True state=running updating=False phase=None hold=None +14:38:42 + 44.3s chaosoomb: deployed=True state=restarting updating=False phase=None hold=None +14:38:52 + 54.5s chaosoomb: deployed=True state=running updating=False phase=None hold=None +14:40:03 + 125.3s chaosoomb: deployed=True state=stopped updating=False phase=None hold=unhealthy_stop diff --git a/documentation/audits/night-2026-09-24/E/round-12.stdout b/documentation/audits/night-2026-09-24/E/round-12.stdout new file mode 100644 index 00000000..6d3ce61f --- /dev/null +++ b/documentation/audits/night-2026-09-24/E/round-12.stdout @@ -0,0 +1,7 @@ +14:37:58 ==== ROUND 12: start_after_unhealthy on chaosoomb x none (2026-09-24T12:37:58Z) +14:37:59 Start -> 200 {'ok': True, 'data': {'state': 'running'}, 'message': 'Stack chaosoomb start requested — state now: running'} +14:38:02 + 3.8s chaosoomb: deployed=True state=running updating=False phase=None hold=None +14:38:42 + 44.3s chaosoomb: deployed=True state=restarting updating=False phase=None hold=None +14:38:52 + 54.5s chaosoomb: deployed=True state=running updating=False phase=None hold=None +14:40:03 + 125.3s chaosoomb: deployed=True state=stopped updating=False phase=None hold=unhealthy_stop +14:40:11 ==== ROUND 12 END: steady in 125.3s; readback=None; events=['2026-09-24T12:39:21Z app_oom (warning)', '2026-09-24T12:39:51Z app_oom_storm (error)', '2026-09-24T12:40:01Z app_stopped_unhealthy (warning)'] diff --git a/documentation/audits/night-2026-09-24/F/10-machine-removes.json b/documentation/audits/night-2026-09-24/F/10-machine-removes.json new file mode 100644 index 00000000..71743a73 --- /dev/null +++ b/documentation/audits/night-2026-09-24/F/10-machine-removes.json @@ -0,0 +1,16 @@ +{ + "before": "nextcloud containers=3 volumes=15 undo_copies=12\nwishlist containers=1 volumes=2 undo_copies=0\nnavidrome containers=1 volumes=1 undo_copies=0\nromm containers=3 volumes=3 undo_copies=0\nvikunja containers=1 volumes=2 undo_copies=0\nn8n containers=1 volumes=1 undo_copies=0\nchaosoom containers=0 volumes=0 undo_copies=0\nchaosoomb containers=0 volumes=0 undo_copies=0\nfelhom-controller filebrowser n8n navidrome nextcloud nextcloud-db nextcloud-redis paperless-postgres paperless-redis paperless-webserver privatebin romm romm-db romm-redis traefik vikunja wishlist ", + "remove": { + "nextcloud": "200", + "wishlist": "200", + "navidrome": "200", + "romm": "200", + "vikunja": "200", + "n8n": "200", + "chaosoom": "200", + "chaosoomb": "200" + }, + "after": "nextcloud containers=0 volumes=0 undo_copies=0\nwishlist containers=0 volumes=0 undo_copies=0\nnavidrome containers=0 volumes=0 undo_copies=0\nromm containers=0 volumes=0 undo_copies=0\nvikunja containers=0 volumes=0 undo_copies=0\nn8n containers=0 volumes=0 undo_copies=0\nchaosoom containers=0 volumes=0 undo_copies=0\nchaosoomb containers=0 volumes=0 undo_copies=0\nfelhom-controller filebrowser paperless-postgres paperless-redis paperless-webserver privatebin traefik ", + "dirs": "/mnt/felhom-drives/scratch_hdd/userdata/calibre-web/\n/mnt/felhom-drives/scratch_hdd/userdata/documents/\n/mnt/felhom-drives/scratch_hdd/userdata/downloads/\n/mnt/felhom-drives/scratch_hdd/userdata/emby/\n/mnt/felhom-drives/scratch_hdd/userdata/immich/\n/mnt/felhom-drives/scratch_hdd/userdata/jellyfin/\n/mnt/felhom-drives/scratch_hdd/userdata/komga/\n/mnt/felhom-drives/scratch_hdd/userdata/media/\n/mnt/felhom-drives/scratch_hdd/userdata/navidrome/\n/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/\n/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx/\n/mnt/felhom-drives/scratch_hdd/userdata/plex/\n/mnt/felhom-drives/scratch_hdd/userdata/radarr/\n/mnt/felhom-drives/scratch_hdd/userdata/romm/\n/mnt/felhom-drives/scratch_hdd/userdata/roms/\n/mnt/felhom-drives/scratch_hdd/userdata/sonarr/\npaperless-ngx\n", + "window": "2026/09/24 12:46:45 scheduler.go:67: [DEBUG] [scheduler] daily job tier2-backup: rescheduled \u2014 recomputing next run\n2026/09/24 12:46:45 scheduler.go:67: [DEBUG] [scheduler] daily job db-dump: rescheduled \u2014 recomputing next run\n2026/09/24 12:46:45 scheduler.go:67: [DEBUG] [scheduler] daily job offbox-backup: rescheduled \u2014 recomputing next run\n2026/09/24 12:46:45 backup_handlers.go:64: [INFO] [web] backup window set to 02:30 (legs 02:30/03:30/04:15)\n" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-24/F/10-machine-removes.log b/documentation/audits/night-2026-09-24/F/10-machine-removes.log new file mode 100644 index 00000000..c04a1db7 --- /dev/null +++ b/documentation/audits/night-2026-09-24/F/10-machine-removes.log @@ -0,0 +1,75 @@ +14:40:59 before: +nextcloud containers=3 volumes=15 undo_copies=12 +wishlist containers=1 volumes=2 undo_copies=0 +navidrome containers=1 volumes=1 undo_copies=0 +romm containers=3 volumes=3 undo_copies=0 +vikunja containers=1 volumes=2 undo_copies=0 +n8n containers=1 volumes=1 undo_copies=0 +chaosoom containers=0 volumes=0 undo_copies=0 +chaosoomb containers=0 volumes=0 undo_copies=0 +felhom-controller filebrowser n8n navidrome nextcloud nextcloud-db nextcloud-redis paperless-postgres paperless-redis paperless-webserver privatebin romm romm-db romm-redis traefik vikunja wishlist +14:41:01 [X] stop -> 200 {'ok': True, 'message': 'Stack nextcloud stop completed'} +14:41:06 [X] remove (with drive data) -> 409 {'ok': False, 'error': 'A(z) /mnt/felhom-drives/scratch_hdd/userdata/nextcloud tárhely jelenleg nem elérhető — az alkalmazás nem távolítható el, amíg a meghajtó +14:41:06 [X] refused because the drive path cannot be resolved (R-442, fail-closed and right) — removing the app and KEEPING the drive data instead +14:41:39 [X] remove (keeping drive data) -> 200 {'ok': True, 'data': {'removed': 'nextcloud', 'volumes_removed': ['nextcloud_nextcloud_db_data', 'nextcloud_nextcloud_html', 'nextcloud_nextcloud_redis_data'], +14:41:47 [X] after remove: deployed=False leftovers='/opt/docker/stacks/nextcloud' +14:41:48 [X] stop -> 200 {'ok': True, 'message': 'Stack wishlist stop completed'} +14:42:20 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'wishlist', 'volumes_removed': ['wishlist_wishlist_data', 'wishlist_wishlist_uploads'], 'hdd_paths_removed': [], 'hdd_paths_pre +14:42:28 [X] after remove: deployed=False leftovers='/opt/docker/stacks/wishlist' +14:42:29 [X] stop -> 200 {'ok': True, 'message': 'Stack navidrome stop completed'} +14:42:34 [X] remove (with drive data) -> 409 {'ok': False, 'error': 'A(z) /mnt/felhom-drives/scratch_hdd/userdata/navidrome tárhely jelenleg nem elérhető — az alkalmazás nem távolítható el, amíg a meghajtó +14:42:34 [X] refused because the drive path cannot be resolved (R-442, fail-closed and right) — removing the app and KEEPING the drive data instead +14:43:01 [X] remove (keeping drive data) -> 200 {'ok': True, 'data': {'removed': 'navidrome', 'volumes_removed': ['navidrome_navidrome_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'backup_paths +14:43:09 [X] after remove: deployed=False leftovers='/opt/docker/stacks/navidrome' +14:43:15 [X] stop -> 200 {'ok': True, 'message': 'Stack romm stop completed'} +14:43:20 [X] remove (with drive data) -> 409 {'ok': False, 'error': 'A(z) /mnt/felhom-drives/scratch_hdd/userdata/romm tárhely jelenleg nem elérhető — az alkalmazás nem távolítható el, amíg a meghajtó viss +14:43:20 [X] refused because the drive path cannot be resolved (R-442, fail-closed and right) — removing the app and KEEPING the drive data instead +14:43:46 [X] remove (keeping drive data) -> 200 {'ok': True, 'data': {'removed': 'romm', 'volumes_removed': ['romm_romm_config', 'romm_romm_db_data', 'romm_romm_redis_data'], 'hdd_paths_removed': [], 'hdd_pat +14:43:55 [X] after remove: deployed=False leftovers='/opt/docker/stacks/romm' +14:43:55 [X] stop -> 200 {'ok': True, 'message': 'Stack vikunja stop completed'} +14:44:27 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'vikunja', 'volumes_removed': ['vikunja_vikunja_data', 'vikunja_vikunja_db'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [ +14:44:36 [X] after remove: deployed=False leftovers='/opt/docker/stacks/vikunja' +14:44:37 [X] stop -> 200 {'ok': True, 'message': 'Stack n8n stop completed'} +14:45:08 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'n8n', 'volumes_removed': ['n8n_n8n_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem +14:45:17 [X] after remove: deployed=False leftovers='/opt/docker/stacks/n8n' +14:45:17 [X] stop -> 200 {'ok': True, 'message': 'Stack chaosoom stop completed'} +14:45:49 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'chaosoom', 'volumes_removed': [], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem tárolt sa +14:45:58 [X] after remove: deployed=False leftovers='/opt/docker/stacks/chaosoom' +14:45:58 [X] stop -> 200 {'ok': True, 'message': 'Stack chaosoomb stop completed'} +14:46:29 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'chaosoomb', 'volumes_removed': [], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem tárolt s +14:46:38 [X] after remove: deployed=False leftovers='/opt/docker/stacks/chaosoomb' +14:46:42 after the product's removes: +nextcloud containers=0 volumes=0 undo_copies=0 +wishlist containers=0 volumes=0 undo_copies=0 +navidrome containers=0 volumes=0 undo_copies=0 +romm containers=0 volumes=0 undo_copies=0 +vikunja containers=0 volumes=0 undo_copies=0 +n8n containers=0 volumes=0 undo_copies=0 +chaosoom containers=0 volumes=0 undo_copies=0 +chaosoomb containers=0 volumes=0 undo_copies=0 +felhom-controller filebrowser paperless-postgres paperless-redis paperless-webserver privatebin traefik +14:46:45 drive dirs left (read by name): +/mnt/felhom-drives/scratch_hdd/userdata/calibre-web/ +/mnt/felhom-drives/scratch_hdd/userdata/documents/ +/mnt/felhom-drives/scratch_hdd/userdata/downloads/ +/mnt/felhom-drives/scratch_hdd/userdata/emby/ +/mnt/felhom-drives/scratch_hdd/userdata/immich/ +/mnt/felhom-drives/scratch_hdd/userdata/jellyfin/ +/mnt/felhom-drives/scratch_hdd/userdata/komga/ +/mnt/felhom-drives/scratch_hdd/userdata/media/ +/mnt/felhom-drives/scratch_hdd/userdata/navidrome/ +/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/ +/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx/ +/mnt/felhom-drives/scratch_hdd/userdata/plex/ +/mnt/felhom-drives/scratch_hdd/userdata/radarr/ +/mnt/felhom-drives/scratch_hdd/userdata/romm/ +/mnt/felhom-drives/scratch_hdd/userdata/roms/ +/mnt/felhom-drives/scratch_hdd/userdata/sonarr/ +paperless-ngx + +14:46:50 backup window -> 02:30: 303 +2026/09/24 12:46:45 scheduler.go:67: [DEBUG] [scheduler] daily job tier2-backup: rescheduled — recomputing next run +2026/09/24 12:46:45 scheduler.go:67: [DEBUG] [scheduler] daily job db-dump: rescheduled — recomputing next run +2026/09/24 12:46:45 scheduler.go:67: [DEBUG] [scheduler] daily job offbox-backup: rescheduled — recomputing next run +2026/09/24 12:46:45 backup_handlers.go:64: [INFO] [web] backup window set to 02:30 (legs 02:30/03:30/04:15) + diff --git a/documentation/audits/night-2026-09-24/F/11-drive-dirs-by-name.txt b/documentation/audits/night-2026-09-24/F/11-drive-dirs-by-name.txt new file mode 100644 index 00000000..c6645bc8 --- /dev/null +++ b/documentation/audits/night-2026-09-24/F/11-drive-dirs-by-name.txt @@ -0,0 +1,15 @@ +removed nextcloud +removed navidrome +removed romm +calibre-web +documents +downloads +emby +immich +jellyfin +komga +media +plex +radarr +roms +sonarr diff --git a/documentation/audits/night-2026-09-24/F/12-9202-config-restored.txt b/documentation/audits/night-2026-09-24/F/12-9202-config-restored.txt new file mode 100644 index 00000000..f71cb611 --- /dev/null +++ b/documentation/audits/night-2026-09-24/F/12-9202-config-restored.txt @@ -0,0 +1,14 @@ +git: + branch: main + repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git + sync_interval: 15m + token: + username: "" +hub: +0 + +gitea.dooplex.hu/admin/felhom-controller:0.269.1 +Up 29 seconds (healthy) +68: health_timeout_seconds: 60 + +sync 429 {'ok': False, 'error': 'Túl gyakori szinkronizálás — várj 30 másodpercet'} diff --git a/documentation/audits/night-2026-09-24/F/13-drill-reset.txt b/documentation/audits/night-2026-09-24/F/13-drill-reset.txt new file mode 100644 index 00000000..56e9a9d9 --- /dev/null +++ b/documentation/audits/night-2026-09-24/F/13-drill-reset.txt @@ -0,0 +1,2 @@ +drill main = cf7cf8456f00 | live main = cf7cf8456f00 | has_actions: False private: True +drill=c8025093d23c live=c8025093d23c diff --git a/documentation/audits/night-2026-09-24/F/20-host-after.txt b/documentation/audits/night-2026-09-24/F/20-host-after.txt new file mode 100644 index 00000000..7abd3774 --- /dev/null +++ b/documentation/audits/night-2026-09-24/F/20-host-after.txt @@ -0,0 +1,13 @@ +Thu Sep 24 14:48:50 CEST 2026 + LANG = "en_US.UTF-8" +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch + LANG = "en_US.UTF-8" +Name Type Status Total (KiB) Used (KiB) Available (KiB) % +felhom-pbs pbs active 0 0 0 0.00% +local dir active 40453376 34129460 4236800 84.37% +local-lvm lvmthin active 56487936 33277043 23210892 58.91% +nvme-scratch dir active 983379700 74629268 858723820 7.59% + LV Data% + data 58.91 diff --git a/documentation/audits/night-2026-09-24/F/30-floor-0.269.1.txt b/documentation/audits/night-2026-09-24/F/30-floor-0.269.1.txt new file mode 100644 index 00000000..889eefab --- /dev/null +++ b/documentation/audits/night-2026-09-24/F/30-floor-0.269.1.txt @@ -0,0 +1,2 @@ +POST /configuration/global-floor min_controller_version=0.269.1 min_agent=0.131.0 -> 303 location: /configuration?flash=floor_set +read back on /configuration: ['0.269.1'] diff --git a/documentation/audits/night-2026-09-24/F/31-boxes-arriving.txt b/documentation/audits/night-2026-09-24/F/31-boxes-arriving.txt new file mode 100644 index 00000000..6c948792 --- /dev/null +++ b/documentation/audits/night-2026-09-24/F/31-boxes-arriving.txt @@ -0,0 +1,8 @@ +Thu Sep 24 14:49:43 CEST 2026 +== hub +2026/09/24 14:49:25 [INFO] Global controller-version floor set to "0.269.1" (declared MinAgent "0.131.0") +2026/09/24 14:49:27 [INFO] managed floor SERVED for demo-felhom: floor 0.269.1, agent requirement "0.131.0" from declared (golden 0.258.0) +== N100 9201 +gitea.dooplex.hu/admin/felhom-controller:0.269.1 Up 10 seconds (healthy) +== demo-hp 9201 +gitea.dooplex.hu/admin/felhom-controller:0.268.0 Up 6 hours (healthy) diff --git a/documentation/audits/night-2026-09-24/tools/teardown.py b/documentation/audits/night-2026-09-24/tools/teardown.py new file mode 100644 index 00000000..a52b43dc --- /dev/null +++ b/documentation/audits/night-2026-09-24/tools/teardown.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python3 +"""Part F, the MACHINE layer on 9202: tonight's apps removed THROUGH THE PRODUCT, leftovers read by name, +the backup window put back to 02:30 through the product's own form. Apps present before tonight stay: +privatebin, paperless-ngx, filebrowser (gokapi was removed in chaos round 5 — R-644's disposition).""" +import json, sys, time +sys.path.insert(0, ".") +import walk as w +w.login() +apps = ["nextcloud", "wishlist", "navidrome", "romm", "vikunja", "n8n", "chaosoom", "chaosoomb"] +q = lambda: w.guest("for p in " + " ".join(apps) + "; do echo \"$p containers=$(docker ps -aq --filter label=com.docker.compose.project=$p | wc -l) volumes=$(docker volume ls -q | grep -c ^${p}_) undo_copies=$(docker volume ls -q --filter label=felhom.undo-copy-of=$p | wc -l)\"; done; docker ps --format '{{.Names}}' | sort | tr '\\n' ' '") +before = q(); w.say("before:\n" + before) +res = {} +for a in apps: + st = w.stack(a) + if not st.get("deployed"): + res[a] = "not deployed"; continue + res[a] = w.remove(a) +after = q(); w.say("after the product's removes:\n" + after) +dirs = w.guest("ls -d /mnt/felhom-drives/scratch_hdd/userdata/*/ 2>/dev/null; ls /mnt/sys_drive/felhom-data/backups/secondary/ 2>/dev/null; ls -d /mnt/felhom-drives/scratch_hdd/userdata/*/backups/secondary/* 2>/dev/null") +w.say("drive dirs left (read by name):\n" + dirs) +sess = open(f"{w.SC}/sess{__import__('os').getpid()}.txt").read().strip() +csrf = open(f"{w.SC}/csrf{__import__('os').getpid()}.txt").read().strip() +r = w.sh(["curl", "-sk", "-o", "/dev/null", "-w", "%{http_code}", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}", "-X", "POST", + "--data-urlencode", f"_csrf={csrf}", "--data-urlencode", "window_start=02:30", f"{w.BASE}/backups/window"]) +time.sleep(2) +win = w.guest("docker logs --since 1m felhom-controller 2>&1 | grep -E 'rescheduled|window set' | tail -4") +w.say(f"backup window -> 02:30: {r.stdout}\n{win}") +json.dump({"before": before, "remove": res, "after": after, "dirs": dirs, "window": win}, open("../F/10-machine-removes.json", "w"), indent=2) +open("../F/10-machine-removes.log", "w").write("\n".join(w.LOG) + "\n") diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 3359feca..54eaccc6 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -375,4 +375,11 @@ Compressed here to title, shipping version, evidence, and the sentences that sta | **R-656** | **The bench re-used an app's scratch drive folder (P3).** Catalog `5ed599c`: `clear_scratch_folders` removes the app's own `${HDD_PATH}`/`${USERDATA_PATH}`/`${IMPORT_PATH}` bind folders before FROM and says so; never a bare root, never outside. Unit-tested and red-proofed; no bench run this session. | catalog, 2026-09-24 | `git show 500488cad673:documentation/backlog/OPEN-ITEMS.md` | | **R-40** | **The update path could not express a multi-hop upgrade (P2).** Superseded by `09` §3 decisions 13–14 and shipped as §6.4 part 5 (v0.268.0): the catalog records every tested step with its own definition, and one press climbs one. **Rule kept:** a hop exists for a box only when the catalog holds it as a TESTED step — a >1-major catalog move without its intermediate steps is still a jump for a box below it. | v0.268.0, 2026-09-24 | `git show 500488cad673:documentation/backlog/OPEN-ITEMS.md` | | **R-663** | **Two catalog test suites had been red since the night of 2026-09-23 (P3).** Filed and fixed the same session: `test_gate_decoys.py` (kimai-db 11.6 → 11.8) and `test_ladder_writer.py` (navidrome 0.64.0 → 0.64.1) typed the live pins as literals; they now READ them. 84 decoy cases OK. **Rule:** a fixture that names a live pin reads it; the catalog moves under it every night. | catalog `5ed599c`, 2026-09-24 | this entry | +| **R-661** | **A file app could not come back WHOLE from the second drive (decision 26).** v0.269.0 `backup.RestoreTier2Whole`: the mirror's files by four rules (never delete; never write an older file over a newer one; bring back every missing one; an older, different live file is replaced and KEPT beside as `.felhom-`), then the unit from the mirror; refuses before anything moves without a proven, openable mirror with file legs or 2 GB free. Live on 9202: restored 1, kept-newer 1, unchanged 137, 3/3 volumes + 1/1 db in 38 s; again from a hold naming the second drive; again after a power cut mid-update (chaos round 3) and with the disk 1 GB above the floor (round 11). **Rule:** a restore never deletes a user file and never writes an older file over a newer one; `rsyncMirror` (`--delete`) is never reused for a restore. | v0.269.0, 2026-09-24 | `git show 4502af6bb109:documentation/backlog/OPEN-ITEMS.md`; `audits/night-2026-09-24/A1/`, `E/round-03.json`, `E/round-11.json` | +| **R-662** | **A dead „database and settings only" second step lingered in the second drive's restore (P3).** Removed in v0.269.0 (decision 25 ruled that restore out); `accept_missing_files` is read by no handler now — only `RestoreTier2Whole` passes it internally. | v0.269.0, 2026-09-24 | `git show 4502af6bb109:documentation/backlog/OPEN-ITEMS.md` | +| **R-664** | **A ladder step had no `.felhom.yml` of its own (P3).** Catalog `cf7cf84`: `steps/.felhom.yml` beside each step's compose, written by `--write-ladder`, refused absent by `check-test-record.py` rule 4b, 8 backfilled; controller v0.269.0 reads the step's own file for the probe, the memory request and the applied record. | v0.269.0 + catalog cf7cf84, 2026-09-24 | `git show 4502af6bb109:documentation/backlog/OPEN-ITEMS.md`; `audits/night-2026-09-24/redproofs/A4-*` | +| **R-665** | **The update judged the new version with the stack dir's `.felhom.yml`, which a restore rewrites (P3).** v0.269.0 journals the new version's own file (`entry.NewMeta`) and `verifyAndConclude` loads exactly it. Live (Part C): wishlist, navidrome and romm steps judged by their own files. | v0.269.0, 2026-09-24 | `git show 4502af6bb109:documentation/backlog/OPEN-ITEMS.md`; `audits/night-2026-09-24/redproofs/A4-r665-r664.txt` | +| **R-666** | **While support is informed, Remove offered to delete the data (decision 27).** v0.269.0: the dialog reads `keep_data_only` and offers only „remove the app, keep my data"; the API refuses data or backup deletion with 409 (hu + en); the no-whole-copy sentence is informal. Live on 9202 (a one-drive hold). | v0.269.0, 2026-09-24 | `git show 4502af6bb109:documentation/backlog/OPEN-ITEMS.md`; `audits/night-2026-09-24/A2/10-held-no-copy-keep-data.*` | +| **R-667** | **A crash loop never reached the alarm, and nothing stopped it (decision 28).** v0.269.0 + hub v0.123.0: ≥ 6 restarts in 10 min (`RestartCount`, not the resettable `restarting_since`) or an OOM storm → the box stops the app, holds it (`unhealthy_stop`), tells household + operator (`app_stopped_unhealthy`); Start = one more try; a repeat in 24 h says support is informed. Live: gokapi trip 1 and 2; chaos rounds 1, 7, 8, 12. **Rule:** Docker's back-off caps a steady loop at ~1 restart/min, so a threshold must be below 10 per 10 min. | v0.269.0 / hub v0.123.0, 2026-09-24 | `git show 4502af6bb109:documentation/backlog/OPEN-ITEMS.md`; `audits/night-2026-09-24/A3/`, `08` §6.2 | +| **R-668** | **The Tier-2 copy chose a registered path that no longer existed, on the app's own disk (P2).** v0.269.0: the same-disk check fails CLOSED. Live: the next copy went to the SSD. Residual: an older same-disk record counts until the next Tier-2 run replaces it. | v0.269.0, 2026-09-24 | `git show 4502af6bb109:documentation/backlog/OPEN-ITEMS.md`; `audits/night-2026-09-24/A1/01-find-mirror.txt` | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 014de1d7..1babdcc8 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -806,13 +806,6 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-654** | **[P3-LOW] opengist 1.15 moved every page under `/-/` — a household's `/login` bookmark answers 404 after the update.** MEASURED 2026-09-23 night: 1.13 serves `/login`, `/register`, `/all`; 1.15.2 answers **404** on all three and serves `/-/login`, `/-/register`, `/-/all`; `/` redirects to `/-/all`. The app, its data and its probe (`/healthcheck`) are fine, and a household arriving at the root lands correctly — only a deep link breaks. 1.15 also marks its session cookie `Secure`. **Needs:** a line in opengist's `app_info` if the operator wants households told; nothing in the product. Evidence: `apps/opengist-oldfixture/`, `apps/opengist/`. | **READY — P3; owner: operator (copy decision) / CC (writes it)** | | **R-655** | **[P2-MEDIUM] adventurelog v0.13.0 cannot become healthy in the catalog's template — its update is undone on every box.** MEASURED 2026-09-23 night, both venues. v0.13.0's FRONTEND image adds its own `HEALTHCHECK` (`node -e fetch('http://127.0.0.1:3000/health')`), and `/health` answers **503 `{"ok":false,"backend":"unreachable"}`** unless the backend's `/health/` answers OK to the frontend's own request (read from the image's `django-proxy` chunk: `fetch(${getServerEndpoint()}/health/)`). On the bench the backend served `/api/` 200, the seed read back, the migration ran (6 lines) — and the frontend stayed `unhealthy` for 420 s; on 9202 the guarded Update went `verifying` → **`undoing` → `undone`** (under the drill's 90 s timeout). **MEASURED LATER THE SAME NIGHT — two causes, and the first hypothesis was wrong.** (1) The backend's `/health/` answers `200 {"ok": true}` to the frontend (read on a fresh v0.13.0 install); the unhealthy frontend is **our template's own healthcheck override**, `["CMD", "/nodejs/bin/node", …]` — v0.12.1's distroless image keeps node there, v0.13.0 moved it to `/usr/bin/node`, and Docker's health log reads `exec: "/nodejs/bin/node": no such file or directory` 19 times in a row. (2) With the override removed, a second bench run hit a harder wall: v0.13.0's backend runs `download-countries` at EVERY start, fetching world data from the internet, and a cut-short download (`ijson.common.IncompleteJSONError: Incomplete JSON content`) crash-loops the entrypoint — the backend never became healthy in 15 min. The first bench run's download had succeeded. **So v0.13.0's boot depends on an outside download, and whether an update succeeds depends on it too.** **The catalog did NOT move adventurelog.** **Needs:** the override dropped (or pointed at `/usr/bin/node`) in the SAME commit as the image move; and a measured answer on whether `download-countries` can be skipped or pre-seeded (an env switch, or the data in the volume) before the edge is re-proven on both venues. Evidence: `audits/night-2026-09-23/apps/adventurelog/`. | **READY — P2; owner: CC (catalog)** | | **R-657** | **[P2-MEDIUM] "Remove the app, keep my data", then install it again: nextcloud never installs, and the box only says „unhealthy".** MEASURED 2026-09-23 night on 9202 (v0.267.0): nextcloud was removed through the product keeping its drive folder (the remove with data was refused — R-442's fail-closed guard, as on every drill on 9202 — and the product's keep-data remove taken). An hour later a fresh install of nextcloud on the same box: the template binds `${HDD_PATH}/appdata/nextcloud` to `/var/www/html/data`, the kept folder still holds `admin/`, `appdata_*`, `.ncdata` and a 145 MB `nextcloud.log`, and the image's installer loops **„Login is invalid because files already exist for this user — Retrying install..."**; `occ status` reads `installed: false`. The controller records the deploy as done and the app as `unhealthy`; nothing tells the household that their kept files are what blocks the new install, or what to do. **Why it matters:** keep-data is the choice the product OFFERS a household at remove time — and for nextcloud the kept data makes the app uninstallable. **Needs:** decide the product's promise for a reinstall over kept data, per app class (adopt the data? refuse with a sentence? offer to move it aside?); at minimum a deploy-time refusal or warning when the app's drive folder is not empty. Evidence: `audits/night-2026-09-23/chaos/00-nextcloud-reinstall-over-kept-data.txt`. | **READY — P2; owner: operator (the promise) / CC (the build)** | -| **R-661** | **[P2-MEDIUM] The second drive holds a file app WHOLE, and no single action brings the app back from it.** MEASURED FROM SOURCE 2026-09-24 (v0.267.0/v0.268.0): the Tier-2 mirror carries the unit AND the drive files, but „Teljes visszaállítás" there is `RestoreTier2Unit` → `RestoreFromRecoveryUnitAt`, whose R-538 guard refuses any app with declared drive files; the file restore (`RestoreTier2Files`) only ADDS missing files and replays no database. So for nextcloud/immich/paperless-ngx/calibre-web the second drive is not a way back after a failed update — which is why R-659's hold (v0.268.0) names only the off-site copy for those apps (`backup.WholeOnTier`). **Needs:** a decision on a Tier-2 "files + database" restore (the off-site full restore's shape), or a ruling that the second drive is files-only for these apps. Evidence: `audits/ladder-2026-09-24/README.md` (the truth table); `backup/update_guard.go` R-659 block. | **WAITING-ON-OPERATOR — P2; owner: operator (the promise) / CC (the build)** | -| **R-662** | **[P3-LOW] The unit restore's documented second step („csak az adatbázist és a beállításokat", `accept_missing_files=1`) is dead.** MEASURED FROM SOURCE 2026-09-24: no template sends the field (`grep accept_missing_files internal/web/templates` → 0), and when a request does, the web handler skips its own refusal but calls `RestoreFromRecoveryUnit`, which passes `UnitRestoreOptions{}` — so the backup layer refuses anyway. Nothing ever calls `AcceptMissingFiles: true` outside a test. **Needs:** either wire it end to end (with its own confirm, R-48) or remove the flag and the comment that promises it. Evidence: `web/handlers.go` backupRestoreHandler; `backup/restore_unit.go` L226. | **READY — P3; owner: CC (controller)** | -| **R-664** | **[P3-LOW] A ladder step has no `.felhom.yml` of its own.** v0.268.0 pins a step's own COMPOSE file (`steps/.yml`); its health probe, memory check and the recorded `applied-meta/` still come from the catalog's CURRENT `.felhom.yml`. Harmless while a step's probe and limits equal the head's (true for all 8 step files today); wrong the day a step's probe differs. Also: the log line for a step pin says „pin advanced to the catalog's current definition" (it names the step file one line earlier). **Needs:** `steps/.felhom.yml` (or the probe in the ladder entry) and the log line naming the source. Evidence: `audits/ladder-2026-09-24/partD/10-romm-two-steps.log`. | **READY — P3; owner: CC (controller + catalog)** | -| **R-665** | **[P3-LOW] An update pressed soon after a restore was judged with the RESTORED `.felhom.yml`.** OBSERVED 2026-09-24 on 9202 (v0.268.0), not diagnosed: vikunja restored from its unit at 06:19; the drill catalog's `.felhom.yml` named probe port 8999 (a deliberately failing edge); the periodic probe used 8999 until 06:20:41, and the update pressed at 06:21:07 probed 3456 (the unit's own file) and ended `done`. The restore apparently writes the unit's `.felhom.yml` into the stack dir, and the next catalog sync has not yet put the catalog's back. Consequence bounded (the OLD probe, ≤ 15 min), but an update's verdict should not depend on how long ago a restore ran. **Needs:** a measurement of what the restore writes and when the sync overwrites it. Evidence: `audits/ladder-2026-09-24/partA/03-why-g-was-done.txt`. | **READY — P3; owner: CC (controller)** | -| **R-666** | **[P3-LOW] A held app whose box has no whole copy is told „ne törölje az alkalmazást" — and its card still offers Eltávolítás.** v0.268.0 (R-659) hides the Mentések button beside the no-whole-copy sentence; the Remove button on the stacks card stays (the household may remove its own app — a promise, not CC's to take away). Measured live on 9202: the page and the sentence disagree. **Needs:** an operator word — hide Remove while support is informed, or soften the sentence. Evidence: `audits/ladder-2026-09-24/partB/01-round11-reproduced.json`. | **WAITING-ON-OPERATOR — P3; owner: operator** | -| **R-667** | **[P2-MEDIUM] A crash-looping app whose container reads `running` for a moment between restarts never reaches the crash-loop alarm.** MEASURED 2026-09-24 on 9202 (v0.268.0): `gokapi` (R-644) at **385 restarts**, `restarting_since` = 06:46:30Z at 06:47 — the clock resets whenever a scan catches the container up — while the dead-app heartbeat said *„4 deployed app(s) evaluated, 0 currently down"*. `Stack.CrashLooping` needs `crashLoopAfter` (5 min) of CONTINUOUS `restarting`. Likely also the unattributed second `app_start_failed` at 06:31:41Z (a scan that caught gokapi `exited`): the dropped-event line names no app, so this is inference. **Needs:** crash-loop judged on the container's RestartCount growth over a window, not on an uninterrupted state; a test with a flapping fixture. Evidence: `audits/ladder-2026-09-24/partB/04-states-and-banner.txt`, `02-r660-positive-observables.txt`. | **READY — P2; owner: CC (controller)** | -| **R-668** | **[P2-MEDIUM] The Tier-2 copy chose a registered storage path that no longer existed — on the SAME disk as the app.** MEASURED 2026-09-24 on 9202 (v0.268.0): `sameDevice` failed OPEN when it could not `stat` a path, so a removed folder read as "another disk" and a file app's "second drive" copy landed beside its own data. **FIXED v0.269.0:** the check fails CLOSED (an unreadable path is the same disk) — `r668_missing_path_test.go`, red-proofed; LIVE on 9202 the next copy went to the SSD (dev 1793 vs 66304). Residual, recorded not fixed: a Tier-2 record written on the same disk BEFORE the fix still counts as a copy until the next Tier-2 run replaces it. `audits/night-2026-09-24/A1/01-find-mirror.txt`, `redproofs/A1-r668.txt` | **CLOSED 2026-09-24 — v0.269.0, proven live** | | **R-669** | **[P2-MEDIUM] After a failed update ends in a restore, the box keeps the FAILED step's health check as the pinned version's — and the NEXT failed update's undo judges the correct old version with it and holds the app.** PROVEN LIVE 2026-09-24 on 9202 (v0.269.0): a failing step (probe port 8999) → hold → the second drive's whole restore cleared the hold, but `applied-meta/.felhom.yml` still carried port 8999 (written by `advancePinTo` at 10:39:21; no restore path rewrites it), and the stack's `.felhom.yml` came back from the unit, which the sync had already filled with the failing step's file. The next failed update's undo put the right version and the right data back and then called it unhealthy on port 8999 → **a false hold** (`not_started`). Data safe; the app stopped for nothing. **Fix direction:** a restore that recreates the definition also sets the applied record to the `.felhom.yml` of the RESTORED pin (the ladder step's `steps/.felhom.yml` for those refs, or the catalog's when the catalog head equals the pin), never the unit's copy. Pre-existing since v0.263.2; not a v0.269.0 regression. `audits/night-2026-09-24/A2/11-applied-meta-wrong-probe.txt` | **READY — P2; owner: CC (controller)** | | **R-670** | **[P3-LOW] Every undo (and every step-file verify) logs `[ERROR] .felhom.yml backup block rejected … docker-compose.yml unreadable`.** `LoadMetadata` validates the backup block against a compose file that the pre-update-meta directory (undo.go:528, since v0.263.0) and the scratch dir of `loadMetadataFile` (v0.269.0) never hold. The health check it feeds is unaffected; an operator reading ERROR lines after an undo is misled. Seen 10:24:41Z and 10:46:24Z on 9202. **Fix:** a probe-only loader that skips the backup-block validation, or copy the compose beside it. `audits/night-2026-09-24/A2/11-applied-meta-wrong-probe.txt` | **READY — P3; owner: CC (controller)** | | **R-671** | **[P3-LOW] The undo copies kept by a hold survive the hold's clearing by a restore.** MEASURED 2026-09-24 on 9202: three `nextcloud_*.pre-update-20260924T103924Z` volumes (~0.9 GiB) were still present after the whole restore cleared that hold at 10:41:52Z, and a second set joined them 6 minutes later. Nothing names them on a page; on a small disk they are the difference between the next update's copy fitting or not. **Fix direction:** the restore that clears an update hold removes that hold's undo copies (they describe the state the restore just replaced), logged. `audits/night-2026-09-24/A2/11-applied-meta-wrong-probe.txt` (volume list) | **READY — P3; owner: CC (controller)** | @@ -826,6 +819,8 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-679** | **[P2-MEDIUM] An Update pressed on an app that is already current runs the whole guarded update — backup, pull, restart — and reports `done`.** MEASURED 2026-09-24 on 9202: navidrome at the head was pressed four times by the stale-read caller (R-678); each press made a safety dump and restarted the app (8.5 s of downtime each) and changed nothing. There is no `current` refusal (`UpdatePreflight`, `update.go:366`; `UpdateOrderCurrent` falls through). Harmless by hand, costly for the automatic leg (`09` §6.4.2). **Fix:** the preflight refuses with reason `current` when the pin equals the catalog head and no newer tested digest exists. `audits/night-2026-09-24/C/30-night-run1.json` | **READY — P2; owner: CC (controller)** | | **R-680** | **[P2-MEDIUM] The box does not remember a failed update step — after an undo it offers the same step again.** MEASURED 2026-09-24 on 9202: vikunja's failing step was undone (104 s) and its badge went straight back to „Frissítés elérhető"; only the test caller's own memory stopped a re-press. Decision 15 („a failed step is never pressed again") therefore holds for a person only by their judgement and not at all for the automatic leg. **Fix (part 7 (b), `09` §6.4.2):** record the failed `to` per app; the leg skips it until the catalog's ladder for that app changes; the page says the step was tried and put back. `audits/night-2026-09-24/C/31-night.json` | **READY — P2; owner: CC (controller, with part 7)** | | **R-681** | **[P2-MEDIUM] An install interrupted by a controller restart is lost SILENTLY — no event, no page sentence, and its half-written files stay.** MEASURED 2026-09-24 on 9202 (Part E round 2, v0.269.1): n8n's install pressed at 11:48:42Z ("Deploying stack n8n … checking 1 images"); `systemctl restart docker` 20 s later took the controller down with it. After the restart the box logged NOTHING about n8n, the app read `not_deployed`, and the household's page showed it as never installed — while `app.yaml` (with its generated `N8N_ENCRYPTION_KEY`), `applied-compose.yml` and `applied-meta/` stayed in the stack dir. A fresh install through the product afterwards worked (211 s) and was not blocked by the leftovers. Compare the update path, which journals and RESUMES after the same accident (round 3/4). **Fix direction:** journal the install like the update; on boot either resume it or say on the page (and in an event) that it was interrupted, and clean the half-written files. `audits/night-2026-09-24/E/round-02*` | **READY — P2; owner: CC (controller)** | +| **R-682** | **[P3-LOW] A Remove interrupted by a controller kill leaves the app half-removed: containers gone, the app still listed as installed (and held).** MEASURED 2026-09-24 on 9202 (chaos round 9): the kill 2 s after the Remove press answered the household `502 Bad Gateway`; after the restart `chaoscrash` read deployed, stopped, `unhealthy_stop`, with NO container left. Pressing Remove again completed it cleanly (200, only the catalog template left). Recoverable by the household's own second press; nothing tells them to press it. **Fix direction:** the remove journals its intent and finishes (or says it was interrupted) at boot, as the update does. `audits/night-2026-09-24/E/round-09*.json`, `E/round-09b-remove-again.txt` | **READY — P3; owner: CC (controller)** | +| **R-683** | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** |