diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 3a95e0c2..0ec38f97 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -821,6 +821,32 @@ household's newest data („gen2") and its download on the drive were there, and nothing. Run 1 of the same proof found the journal left as `undo.json.tmp` (the rename lived only in the page cache) — the app was held for support, never started on the cut data; the folder sync above fixed it, run 2 passed. +**`[DESIGN]` 2026-10-28 night — four restore edges (R-1256, review round 2 HUNT-1 H4/H5/H6/H8; controller `342a394`, +unreleased — the morning list).** (H4) An off-site restore whose definition write fails after the compose was written +writes the LIVE definition back before it starts the app; a write-back that fails too HOLDS the app (`restore_mixed`) — +never a start on the snapshot's definition over the live data. (H5) A put-back whose stop fails pours nothing, keeps the +copy and holds the app (off-site, local unit, and a restore cut by a power loss); a failed definition write-back keeps the +copy (D20); a `docker volume rm -f` that fails (the volume is in use) fails that volume — the `volume create` after it was +a no-op and the tar was unpacked ON TOP of the old content. (H6) A restore op begins in ONE step with the check for every +other operation (a restore, the backup run, a second-drive copy — under the lock the mirror asks under); every restore +button and the kept loads begin that way, and the install-time „use my kept data" asks about the second-drive copy too. +(H8) An unused restore copy goes after 14 days (every 6 h): only when the app has no restore hold of any kind, the +app-stop marker is readable and does not name the app, and no restore runs. **Decided by CC unattended 2026-10-28 — +operator may reverse:** *How long does an unused restore copy stay?* (a) 14 days; (b) for ever (as before); (c) 3 days. +(a) holds up to two weeks of disk per restore and gives support two weeks; (b) grows without end; (c) is shorter than a +support reply. A held or waiting copy is never touched under any option. Tests `TestR1256_*`, each red first. +**`[FACT]` 2026-10-29 night, scratch 9202 (`audits/night-2026-10-28/r1256-restore/h5-live/`):** vikunja's first volume +held by a created-only container, the household's restore pressed through the page: „Failed to remove volume +vikunja_vikunja_data before the restore … (not written into)", the restore failed, the put-back poured the live copy of +both volumes, the app ran, no hold, and a project made before the restore point AND one made after it both read back. H4 +(off-site only, and a fault there races the write-back) and H5's failed stop are proven by tests only; their live proof +is owed to an attended session. + +**`[FACT]` 2026-10-29 03:29 UTC, scratch 9202 (`audits/night-2026-10-28/r1256-restore/h8-live/`):** three restore +copies planted by hand (dated 2026-10-01, 2026-10-05 and 2026-10-28); the first six-hour sweep after the controller start +removed the two older than 14 days, one of them under the standing app paperless, and kept the young one; the job took +8 ms. The young copy was then removed by hand. + **[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.** Recorded here rather than only in a closed register row, because a decision that survives only inside a closed work item is a decision nobody will find. diff --git a/documentation/architecture/where-felhom-stands.html b/documentation/architecture/where-felhom-stands.html index 9cd02b13..117a335c 100644 --- a/documentation/architecture/where-felhom-stands.html +++ b/documentation/architecture/where-felhom-stands.html @@ -7,7 +7,7 @@

Where Felhom stands

What we built, what happens when things go wrong, and what is still missing. Generated from where-felhom-stands.yaml, which cites the capability map, register row or evidence document behind every claim — and which is checked by scripts/check_stands.py.
Verified 2026-08-22 against felhom-agent 40d857b527, felhom-controller 2da259af38, hub 877fcd2a38. 17 claim(s) carry a recorded status move (7 re-checked in this pass) — each is marked on the page.
-
Walked (23)done end to end on real hardware, evidence on file
Built (13)shipped and tested, the real path never walked
Partial (15)some walked, some not — the note says which
Missing (4)does not exist
needs-hardwareonly a running box could settle it
+
Walked (24)done end to end on real hardware, evidence on file
Built (12)shipped and tested, the real path never walked
Partial (15)some walked, some not — the note says which
Missing (4)does not exist
needs-hardwareonly a running box could settle it
The journey
Seven stages, left to right, as they happen to a person.
@@ -16,7 +16,7 @@
3  Using it
Apps installed from a catalog of about 52, with a memory guard and health-aware progress53 templates listed on the rebuilt box 2026-08-09; deploy driven live.capability-map Deploy an app from the catalog (env config, memory guard, h… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
Start, stop, restart, update, logs, remove — and the parts that must not be stopped cannot bechanged 2026-08-09, was builtWalked 2026-10-28 night on scratch 9202: install, stop, start, restart, logs; traefik and cloudflared refuse a stop (403); remove with data leaves no volume and no container. The update half: the same day's guarded updates on 9202 (bench + box + undo walks).why it moved: a walk document is now cited (2026-10-28 night)capability-map App lifecycle: start/stop/restart/update/logs/remove/redeploy · register R-108 · evidence audits/night-2026-10-28/part5-claims/lifecycle-walk.log · evidence audits/dockerhub-bigsteps-2026-10-28
Reachable from anywhere through a tunnel, per-app addressesVerified tonight: the rebuilt box answered on its public URL from outside.capability-map Remote access via Cloudflare Tunnel + Traefik (per-app subd… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
Reachable on the home network when the internet is downneeds-hardwareOnly an observation on a running box with WAN pulled could settle it. Boxes are off.capability-map LAN access when internet is down (lan_resolver)
Phone photos, documents with text recognition, files from Windows Explorer or a Maccapability-map Files from Windows Explorer / Mac Finder (SMB server) · evidence audits/SPIKE-lan-discovery-2026-07-18.md
A one-tap launcher, and a read-only guest link for visitorschanged 2026-08-09, was builtWalked 2026-10-28 night on scratch 9202 through the pages the browser receives: the launcher lists each openable app; the guest link opens with no login and no admin link, a wrong token is 404, rotate and switch-off kill the old link.why it moved: walked live 2026-10-28 night (map E8, E9 → PROVEN-LIVE)capability-map Indítópult (app launcher) — one-tap grid of the household's… · evidence audits/night-2026-10-28/part5-claims/launcher-walk.log
Media to a TVMap: MISSING.capability-map Media to TV via DLNA
Separate accounts per household memberMap: MISSING.capability-map Multiple household users / per-person accounts
4  Drives
A new drive is found, offered, formatted, mounted and enrolled — including on awkward older boot layoutsmoved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walkedApplies to a NEW drive. Re-attaching an existing one after a reinstall is R-280 and fails.why it moved: receipt found 2026-08-10: a live throwaway drive on felhom-pve walked scan -> format -> mount -> PVE dir storage -> one-click Regisztralas, with the resulting storage.cfg entry and systemd mount unit recorded. The map already read PROVEN-LIVE; the dataset was behind it. Scope unchanged: this is a NEW drive. Re-attaching an existing one after a reinstall is R-280 and still fails.capability-map Drive wizard: scan/format/mount/enroll, incl. legacy-boot L… · register R-220 · evidence audits/SPIKE-raw-drive-enroll-2026-06-15.md
Moving data between drives, crash-safe; removing a drive safely; unplug detectedchanged 2026-08-09, was walkedwhy it moved: no walk document citedcapability-map Data migration between drives (all / per-app), crash-safe
A network drive can be browsed and hold bulk media, but may not hold an app's data — enforcedRefuseAsAppNamespace is the fail-closed predicate.capability-map Network storage (NAS) is browse + bulk-media only · register R-108 · evidence audits/R108-network-app-namespace-2026-07-30.md
After a reinstall the data drive cannot be re-attached through any dashboard routeWarning card. /api/disks/candidates returns empty; the restore page promises two clicks.register R-280 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
5  Backing up
App data on the machine, nightly database dumps, a copy on a second drivemoved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walkedTrue for every app but one, and that one was found on 2026-08-21: paperless-ngx's 72-table PostgreSQL was dumped nightly and written into a folder named after an app that does not exist, so it never entered the recovery unit, the second-drive copy or the off-site copy. Fixed and proven live 2026-08-22 (R-355) — the dump is now in the app's own unit and in the off-site snapshot for the first time. A catalogue-wide sweep, itself proven able to convict a planted case, says 1 of 53 was affected.why it moved: receipt found 2026-08-10: an adversarial, destructive, unattended overnight campaign across both boxes and ep0, with A2 (one quiesce, two tiers) proven end to end. Map already read PROVEN-LIVE.capability-map Tier-2 secondary-drive copy: class-driven legs · register R-355 · evidence audits/CAMPAIGN-8-backup-restore-2026-07-27.md · evidence audits/DRILL-backup-truth-2026-08-21/evidence/phase4-paperl…
A whole-machine archive that lands off the guest's own disk — a single-drive machine is recorded as degraded rather than pretendingchanged 2026-08-09, was walkedwhy it moved: no walk document citedcapability-map Whole-guest backup lands OFF the guest's own physical device · register R-165
An encrypted off-site copy, sealed with a key the operator cannot readchanged 2026-08-09, was walkedThe COPY is real and the key is still unreadable to us — what failed is DAILY and UNBROKEN. The 2026-08-09 note said '18 snapshots, daily, unbroken'; the next snapshot after 2026-08-09 08:30 was 2026-08-21 22:17, put there by hand during the drill. Twelve days, no alarm. Two causes in series: the 2026-08-21 rebuild lost the off-box target (R-193's shape), and after the self-heal restored it EVERY per-app switch was still off, so the first run logged 'backup OK: 0 app(s) backed up, 14s'.why it moved: the claim is about a CONTINUING daily copy; a 12-day silent gap was found on 2026-08-21 and nothing reported itcapability-map Offsite (restic → Hetzner Storage Box) · register R-199 · register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
The backups prove themselves: a restore is actually performed, unattended, on every tier, on both machineschanged 2026-08-09, was walkedSTILL GREY, and for a sharper reason than in August. The scheduler runs and the check works — it failed again on 2026-08-21, unprompted, and named the cause exactly: after the reinstall the box presents a different PBS key than its own older archives were sealed with, so those archives cannot be opened at all (R-366). A tier whose archives are orphaned is a different alarm from a tier whose test failed, and only the second is being said.proof decay: PROOF-DECAY RULE FIRED (first time it has). A receipt EXISTS - architecture/_recovery-inventory-2026-07-28.md carries live journal lines for scheduled restore-tests on both boxes and both tiers - but it is superseded by later observation: demo-hp logged restore_test_failed on 2026-08-05, and the box has since been wiped and reinstalled (2026-08-09). The claim is about a CONTINUING scheduled behaviour, so a 2026-07-28 observation cannot carry it. Stays grey until a scheduled restore-test is seen passing on the rebuilt box. THE CAPABILITY MAP STILL READS PROVEN-LIVE (2026-08-03) AND IS NOW THE THING OUT OF STEP.why it moved: no walk document cited, and the last recorded restore-test on demo-hp FAILED (2026-08-05)capability-map Restore-proof is UNATTENDED — the scheduler covers EVERY tier · register R-86 · register R-366 · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
The customer is warned before a drive fills, per drive, in their own languagechanged 2026-08-09, was walkedThe warning is real and was SEEN firing on 2026-08-21 with the right Hungarian copy, naming the drive and the free space. What 'BEFORE' cannot survive is the cadence: the watcher runs once a day at 03:30 plus once at startup (R-363), so a filesystem that fills at 03:31 goes unannounced for ~24 h. Watched live: the 69 GB volume carrying all 40-class app data was filled to 99% and the watcher said nothing, while the backup reserve was already refusing an app per run and telling the hub about it.why it moved: a daily check cannot carry the word BEFORE; observed silent for the whole window a filesystem sat at 99%capability-map The customer is warned BEFORE a filesystem fills · register R-167 · register R-363 · evidence audits/SPIKE-r165-mp1-merge-2026-08-02.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
A backup that covered nothing still calls itself successfulWarning card — and the drill found two more of it, both live. (1) An off-site run with no app selected logs 'backup OK: 0 app(s) backed up'; the card does say 'nincs kijelölt alkalmazás' beside the green tick, so this one is honest if you read past the tick. (2) A restore that placed nothing reported success: '0 fájl visszaállítva', ok=true. The second is the one that matters and it is R-353, still open. The volume half of it is fixed (R-354): the message now names what came back.register R-240 · register R-353 · register R-354 · evidence audits/DRILL-backup-truth-2026-08-21/evidence/phase3-experi…
-
6  When something goes wrong
The machine watches itself and repairs some faults without telling anyone it had tochanged 2026-08-09, was walkedR-264 records that the self-heal counters reach the hub and are decoded nowhere.why it moved: no walk document citedcapability-map Box survives an unattended app or guest-network failure · register R-264
Failures reach the operator by email, one mail per run, every failing app namedchanged 2026-08-09, was walkedCONFIRMED against source: backup_run_failures digest is allowlisted (api/handler.go:1837), operator-only (dispatcher.go:423), templated (templates.go:48); recovery_unit_capture_failed is record-only (dispatcher.go:376). NOTE: R-182's register row still describes the PRE-FIX behaviour — see R-289.why it moved: the digest is wired and source-verified, but no run of it has been observed deliveringcapability-map A failed per-app Tier-1 backup reaches the OPERATOR — EVERY… · register R-182
Failures reach the customerThe customer leg is built; today's log shows customer-channel rows skipped as operator_only.capability-map App crashes → customer notified (one event per transition, …
An already-paired box no longer tells anyone to pair it — its console says there is nothing to dochanged 2026-08-09, was partialWas a warning card (the console kept showing the pairing code). Fixed by ISO 1.28.0 (R-535; R-214, R-235 closed). Walked 2026-10-29 00:04 on Tester 1 (paired, claimed): the console reads, in Hungarian and English, that there is nothing to do on this machine and no need to log in — no pairing code.why it moved: the defect is closed and the console was read live on a paired box (2026-10-29)register R-214 · register R-235 · evidence audits/night-2026-10-28/part5-claims/tester1-console-2026-1…
+
6  When something goes wrong
The machine watches itself and repairs some faults without telling anyone it had tochanged 2026-08-09, was builtWalked 2026-10-29 night on scratch 9202: a crashed app part (kill -9 of paperless-redis) ran again in 1 s and the health probe read the app healthy; a guest reboot brought every app back in under 2 minutes, the boot sweep found nothing to start; no event, no banner, no mail. Docker's restart policy did the repair in both legs; the controller's own boot repair had nothing to do (its last proof: 6 of 6 hard resets, 2026-08-02). R-264 records that the self-heal counters reach the hub and are decoded nowhere.why it moved: walked 2026-10-29 on scratch 9202 (a walk document is now cited)capability-map Box survives an unattended app or guest-network failure · register R-264 · evidence audits/night-2026-10-28/part5-claims/selfheal/WALK.md · evidence audits/DIAG-bootrecon-drive-absent-2026-08-02.md
Failures reach the operator by email, one mail per run, every failing app namedchanged 2026-08-09, was walkedCONFIRMED against source: backup_run_failures digest is allowlisted (api/handler.go:1837), operator-only (dispatcher.go:423), templated (templates.go:48); recovery_unit_capture_failed is record-only (dispatcher.go:376). NOTE: R-182's register row still describes the PRE-FIX behaviour — see R-289.why it moved: the digest is wired and source-verified, but no run of it has been observed deliveringcapability-map A failed per-app Tier-1 backup reaches the OPERATOR — EVERY… · register R-182
Failures reach the customerThe customer leg is built; today's log shows customer-channel rows skipped as operator_only.capability-map App crashes → customer notified (one event per transition, …
An already-paired box no longer tells anyone to pair it — its console says there is nothing to dochanged 2026-08-09, was partialWas a warning card (the console kept showing the pairing code). Fixed by ISO 1.28.0 (R-535; R-214, R-235 closed). Walked 2026-10-29 00:04 on Tester 1 (paired, claimed): the console reads, in Hungarian and English, that there is nothing to do on this machine and no need to log in — no pairing code.why it moved: the defect is closed and the console was read live on a paired box (2026-10-29)register R-214 · register R-235 · evidence audits/night-2026-10-28/part5-claims/tester1-console-2026-1…
7  Getting everything back
A rebuilt machine shows a full-page recovery screen without anyone looking for it, and says plainly that nobody can replace a lost recovery codeSeen unsought on the rebuilt demo-hp 2026-08-09, seal date matching host_escrow.created_at.register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
The customer's code opens the sealed package and the data returns byte for byte — including accented Hungarian filenames, verified as raw bytesStill true, and re-proven 2026-08-22 (5/5 byte-identical, both accented names as raw bytes) — but the SCOPE is narrower than the sentence sounds and was silently narrower still until v0.218.0. The drill found the off-site restore had NO named-volume leg at all: the archive sat in the unit, the snapshot and the checking folder and was never replayed, under a success message (R-354, fixed and proven 2026-08-22). AND 40 of the 53 catalogue apps STILL cannot run this route at all — it refuses first, saying a running app is not installed (R-356, open). So: proven for an app that declares a data drive; unproven and currently unreachable for the class whose entire dataset is a named volume.register R-201 · register R-354 · register R-356 · evidence tests/walk5-r201-2026-08-07/journal.md · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
Walked end to end with no command line inside the machine (2026-08-07)contestedCONTESTED-RESOLVED: not a contradiction. The walk proves the ROUTE needs no guest shell; the map's MISSING row is about a NON-OPERATOR doing it, which has still never happened. Two questions, one word 'customer'.register R-201 · evidence tests/walk5-r201-2026-08-07/journal.md · capability-map A customer (not the operator) performs a restore via UI alone
Putting restored files back where they belong is still manualWarning card. Confirmed 2026-08-09: the restore lands in a verification folder and says so.register R-213
The tripwire that says someone is opening this customer's backups does fireCONFIRMED tonight from the hub store: escrow_blob_served 2026-08-09 10:19:41Z = 12:19 CEST in the operator's mailbox. R-281's original claim of silence is WITHDRAWN.register R-281 · register R-285 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
When things go wrong
diff --git a/documentation/architecture/where-felhom-stands.yaml b/documentation/architecture/where-felhom-stands.yaml index fb1d07f6..e315035a 100644 --- a/documentation/architecture/where-felhom-stands.yaml +++ b/documentation/architecture/where-felhom-stands.yaml @@ -418,17 +418,19 @@ claims: band: journey stage: 6 title: "The machine watches itself and repairs some faults without telling anyone it had to" - status: built - note: "R-264 records that the self-heal counters reach the hub and are decoded nowhere." + status: walked + note: "Walked 2026-10-29 night on scratch 9202: a crashed app part (kill -9 of paperless-redis) ran again in 1 s and the health probe read the app healthy; a guest reboot brought every app back in under 2 minutes, the boot sweep found nothing to start; no event, no banner, no mail. Docker's restart policy did the repair in both legs; the controller's own boot repair had nothing to do (its last proof: 6 of 6 hard resets, 2026-08-02). R-264 records that the self-heal counters reach the hub and are decoded nowhere." sources: - capability-map: "Box survives an unattended app or guest-network failure" - register: "R-264" + - evidence: "audits/night-2026-10-28/part5-claims/selfheal/WALK.md" + - evidence: "audits/DIAG-bootrecon-drive-absent-2026-08-02.md" changed: - from: walked - reason: "no walk document cited" + from: built + reason: "walked 2026-10-29 on scratch 9202 (a walk document is now cited)" verified: - date: 2026-08-09 - verdict: downgraded + date: 2026-10-29 + verdict: upgraded depth: source-read - id: fault.operator-email band: journey diff --git a/documentation/audits/night-2026-10-28/part5-claims/selfheal/WALK.md b/documentation/audits/night-2026-10-28/part5-claims/selfheal/WALK.md new file mode 100644 index 00000000..073aba2d --- /dev/null +++ b/documentation/audits/night-2026-10-28/part5-claims/selfheal/WALK.md @@ -0,0 +1,23 @@ +# Walk — fault.selfheal, scratch 9202, 2026-10-29 03:31–03:37 UTC + +Box: scratch guest 9202 on demo-hp, controller test build 0.347.0-rc.n1028, no hub (a scratch box). +Method: two real faults, read back through docker and the controller log. Nobody pressed anything. + +1. **A dead app part.** 03:31:55 `kill -9` of paperless-redis's main process (a crash, not `docker stop`). + 03:31:56 the container ran again (restart count 0 → 1), healthy within 20 s. The controller's + health probe read paperless at 03:33:44: `HTTP GET :8000/ → 302`, `1 ok (of 1 probed)`. No event, no + banner, no mail (`before.txt`, `after-kill.txt`, `controller-after-kill.log`). +2. **A guest reboot.** 03:34:11 `pct reboot 9202`. At 03:35:56 every container was up and healthy (paperless + is drive-backed: media and export on `scratch_hdd`, mp8). 03:35:39 `[bootrecon] boot window: fleet settled + after 10s … sweeping` → `no boot-orphaned apps (nothing to start)`. The only event was + `controller_started` (info), dropped because 9202 has no hub (`ps-after-reboot.txt`, + `controller-after-reboot.log`). + +Verdict: the box repaired both faults and told nobody. Docker's restart policy did the repair in both legs; +the controller watched (probe, boot sweep) and had nothing to do. The controller's own boot repair (an app +whose containers the reboot left stopped) did not have to act here; its last proof is 6 of 6 hard resets, +`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`. R-264 (the self-heal counters are decoded nowhere) stays. + +Side observation: at boot the controller logs `FileBrowser: drive …/userdata/ not mounted — skipping +userdata skeleton` for nine per-app folders on a drive that IS mounted. The file browser still shows them +through the drive's own mount (`/srv/scratch_hdd`). Filed as a register row (wrong words in a log line). diff --git a/documentation/audits/night-2026-10-28/part5-claims/selfheal/after-kill.txt b/documentation/audits/night-2026-10-28/part5-claims/selfheal/after-kill.txt new file mode 100644 index 00000000..c005f4a9 --- /dev/null +++ b/documentation/audits/night-2026-10-28/part5-claims/selfheal/after-kill.txt @@ -0,0 +1,18 @@ +/paperless-redis running health=starting started=2026-10-29T03:31:56.419425354Z rc=1 +/paperless-webserver running health=healthy started=2026-10-29T01:30:29.001125536Z rc=0 +--- +/paperless-redis running health=healthy started=2026-10-29T03:31:56.419425354Z rc=1 +/paperless-webserver running health=healthy started=2026-10-29T01:30:29.001125536Z rc=0 +--- +/paperless-redis running health=healthy started=2026-10-29T03:31:56.419425354Z rc=1 +/paperless-webserver running health=healthy started=2026-10-29T01:30:29.001125536Z rc=0 +--- +/paperless-redis running health=healthy started=2026-10-29T03:31:56.419425354Z rc=1 +/paperless-webserver running health=healthy started=2026-10-29T01:30:29.001125536Z rc=0 +--- +/paperless-redis running health=healthy started=2026-10-29T03:31:56.419425354Z rc=1 +/paperless-webserver running health=healthy started=2026-10-29T01:30:29.001125536Z rc=0 +--- +/paperless-redis running health=healthy started=2026-10-29T03:31:56.419425354Z rc=1 +/paperless-webserver running health=healthy started=2026-10-29T01:30:29.001125536Z rc=0 +--- diff --git a/documentation/audits/night-2026-10-28/part5-claims/selfheal/before.txt b/documentation/audits/night-2026-10-28/part5-claims/selfheal/before.txt new file mode 100644 index 00000000..34866ac3 --- /dev/null +++ b/documentation/audits/night-2026-10-28/part5-claims/selfheal/before.txt @@ -0,0 +1,3 @@ +/paperless-redis pid=2767106 restart=unless-stopped started=2026-10-29T01:30:18.591810514Z rc=0 +/paperless-webserver pid=2767370 restart=unless-stopped started=2026-10-29T01:30:29.001125536Z rc=0 +/paperless-postgres pid=2767174 restart=unless-stopped started=2026-10-29T01:30:18.791461778Z rc=0 diff --git a/documentation/audits/night-2026-10-28/part5-claims/selfheal/controller-after-kill.log b/documentation/audits/night-2026-10-28/part5-claims/selfheal/controller-after-kill.log new file mode 100644 index 00000000..4ba68e49 --- /dev/null +++ b/documentation/audits/night-2026-10-28/part5-claims/selfheal/controller-after-kill.log @@ -0,0 +1,310 @@ +2026/10/29 03:31:04 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:31:04 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:31:04 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:31:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/10/29 03:31:04 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:31:14 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/10/29 03:31:14 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈12538MB avail≈13360MB +2026/10/29 03:31:14 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=15844784 → total=25898MB avail=13360MB used=12538MB (48.4%) +2026/10/29 03:31:14 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=9.8GB avail=55.0GB (15.2%) +2026/10/29 03:31:14 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=235.7GB avail=654.4GB (26.5%) +2026/10/29 03:31:14 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.66 1.26 1.61 5/1617 86572" → 1m=0.66 5m=1.26 15m=1.61 +2026/10/29 03:31:14 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/29 03:31:14 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 50.0°C (hwmon0) +2026/10/29 03:31:14 info.go:13: [DEBUG] [system] GetInfo done in 61ms — mem=12538MB/25898MB (48.4%), rootDisk=9.8GB/68.4GB (15.2%), load=0.66/1.26/1.61, temp=50.0°C (hwmon0), cpu=6.7% +2026/10/29 03:31:14 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/29 03:31:14 scheduler.go:450: [INFO] [scheduler] Running job: stack-scan +2026/10/29 03:31:14 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:31:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/10/29 03:31:14 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:31:14 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:31:14 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:31:14 scheduler.go:85: [DEBUG] [scheduler] job stack-scan: execution starting +2026/10/29 03:31:14 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/29 03:31:14 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/29 03:31:14 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 03:31:14 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/29 03:31:14 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/29 03:31:14 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=false composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=false composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "dawarich" deployed=false composePath=/opt/docker/stacks/dawarich/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grimmory" deployed=false composePath=/opt/docker/stacks/grimmory/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grocy" deployed=false composePath=/opt/docker/stacks/grocy/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "karakeep" deployed=false composePath=/opt/docker/stacks/karakeep/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "lubelogger" deployed=false composePath=/opt/docker/stacks/lubelogger/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "metube" deployed=false composePath=/opt/docker/stacks/metube/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=false composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=false composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "nzbget" deployed=false composePath=/opt/docker/stacks/nzbget/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=false composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "prowlarr" deployed=false composePath=/opt/docker/stacks/prowlarr/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "qbittorrent" deployed=false composePath=/opt/docker/stacks/qbittorrent/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "radicale" deployed=false composePath=/opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=false composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sabnzbd" deployed=false composePath=/opt/docker/stacks/sabnzbd/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "transmission" deployed=false composePath=/opt/docker/stacks/transmission/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=false composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/10/29 03:31:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/10/29 03:31:14 manager.go:1964: [DEBUG] [stacks] getCatalogTemplateSlugs: found 65 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/10/29 03:31:14 manager.go:761: [DEBUG] [stacks] ScanStacks: catalog has 65 template slugs for orphan detection +2026/10/29 03:31:14 manager.go:792: [INFO] [stacks] ScanStacks complete: 70 stacks found (1 deployed, 69 available) +2026/10/29 03:31:14 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:31:14 scheduler.go:467: [INFO] [scheduler] Job stack-scan completed (took 140ms) +2026/10/29 03:31:24 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:31:24 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:31:24 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:31:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/10/29 03:31:24 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:31:29 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:31:34 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:31:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/10/29 03:31:34 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:31:34 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:31:34 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:31:44 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/29 03:31:44 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:31:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/10/29 03:31:44 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:31:44 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/29 03:31:44 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:31:44 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/29 03:31:44 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:31:44 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:31:54 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:31:54 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:31:54 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:31:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/10/29 03:31:54 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:31:59 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:32:04 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:32:04 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:32:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/10/29 03:32:04 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:32:04 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:32:14 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/10/29 03:32:14 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈12572MB avail≈13326MB +2026/10/29 03:32:14 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=15803912 → total=25898MB avail=13326MB used=12572MB (48.5%) +2026/10/29 03:32:14 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=9.8GB avail=55.0GB (15.2%) +2026/10/29 03:32:14 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=235.7GB avail=654.4GB (26.5%) +2026/10/29 03:32:14 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.69 1.15 1.55 11/1601 86735" → 1m=0.69 5m=1.15 15m=1.55 +2026/10/29 03:32:14 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/29 03:32:14 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 50.0°C (hwmon0) +2026/10/29 03:32:14 info.go:13: [DEBUG] [system] GetInfo done in 69ms — mem=12572MB/25898MB (48.5%), rootDisk=9.8GB/68.4GB (15.2%), load=0.69/1.15/1.55, temp=50.0°C (hwmon0), cpu=6.9% +2026/10/29 03:32:14 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/29 03:32:14 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:32:14 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/29 03:32:14 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/29 03:32:14 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 03:32:14 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:32:14 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/29 03:32:14 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:32:14 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/29 03:32:14 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:32:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/10/29 03:32:14 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:32:24 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:32:24 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:32:24 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:32:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/10/29 03:32:24 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:32:29 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:32:34 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:32:34 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:32:34 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:32:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/10/29 03:32:34 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:32:44 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:32:44 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/29 03:32:44 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/29 03:32:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/10/29 03:32:44 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:32:44 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:32:44 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:32:44 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/29 03:32:44 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:32:54 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:32:54 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:32:54 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:32:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/10/29 03:32:54 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:32:59 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:33:04 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:33:04 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:33:04 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:33:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/10/29 03:33:04 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:33:14 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/10/29 03:33:14 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈12588MB avail≈13310MB +2026/10/29 03:33:14 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=15785712 → total=25898MB avail=13310MB used=12588MB (48.6%) +2026/10/29 03:33:14 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=9.8GB avail=55.0GB (15.2%) +2026/10/29 03:33:14 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=235.7GB avail=654.4GB (26.5%) +2026/10/29 03:33:14 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.71 1.08 1.50 6/1605 86895" → 1m=0.71 5m=1.08 15m=1.50 +2026/10/29 03:33:14 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/29 03:33:14 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 50.0°C (hwmon0) +2026/10/29 03:33:14 info.go:13: [DEBUG] [system] GetInfo done in 62ms — mem=12588MB/25898MB (48.6%), rootDisk=9.8GB/68.4GB (15.2%), load=0.71/1.08/1.50, temp=50.0°C (hwmon0), cpu=7.4% +2026/10/29 03:33:14 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:33:14 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/29 03:33:14 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/29 03:33:14 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:33:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/10/29 03:33:14 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:33:14 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/29 03:33:14 scheduler.go:450: [INFO] [scheduler] Running job: stack-scan +2026/10/29 03:33:14 scheduler.go:85: [DEBUG] [scheduler] job stack-scan: execution starting +2026/10/29 03:33:14 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:33:14 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/29 03:33:14 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/29 03:33:14 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=false composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=false composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "dawarich" deployed=false composePath=/opt/docker/stacks/dawarich/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grimmory" deployed=false composePath=/opt/docker/stacks/grimmory/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grocy" deployed=false composePath=/opt/docker/stacks/grocy/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "karakeep" deployed=false composePath=/opt/docker/stacks/karakeep/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "lubelogger" deployed=false composePath=/opt/docker/stacks/lubelogger/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "metube" deployed=false composePath=/opt/docker/stacks/metube/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=false composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=false composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "nzbget" deployed=false composePath=/opt/docker/stacks/nzbget/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=false composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "prowlarr" deployed=false composePath=/opt/docker/stacks/prowlarr/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "qbittorrent" deployed=false composePath=/opt/docker/stacks/qbittorrent/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "radicale" deployed=false composePath=/opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=false composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sabnzbd" deployed=false composePath=/opt/docker/stacks/sabnzbd/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "transmission" deployed=false composePath=/opt/docker/stacks/transmission/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=false composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/10/29 03:33:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/10/29 03:33:14 manager.go:1964: [DEBUG] [stacks] getCatalogTemplateSlugs: found 65 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/10/29 03:33:14 manager.go:761: [DEBUG] [stacks] ScanStacks: catalog has 65 template slugs for orphan detection +2026/10/29 03:33:14 manager.go:792: [INFO] [stacks] ScanStacks complete: 70 stacks found (1 deployed, 69 available) +2026/10/29 03:33:14 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:33:14 scheduler.go:467: [INFO] [scheduler] Job stack-scan completed (took 113ms) +2026/10/29 03:33:14 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:33:24 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:33:24 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:33:24 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:33:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/10/29 03:33:24 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:33:29 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:33:34 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:33:34 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:33:34 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:33:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/10/29 03:33:34 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:33:44 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:33:44 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/29 03:33:44 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:33:44 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:33:44 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/29 03:33:44 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/29 03:33:44 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:33:44 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (0 skipped not due, 0 skipped no container) +2026/10/29 03:33:44 healthprobe.go:187: [DEBUG] Health probe paperless-ngx: HTTP GET :8000/ → 302 (5ms) +2026/10/29 03:33:44 healthprobe.go:167: [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 03:33:54 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:33:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/29 03:33:54 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:33:54 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:33:54 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks diff --git a/documentation/audits/night-2026-10-28/part5-claims/selfheal/controller-after-reboot.log b/documentation/audits/night-2026-10-28/part5-claims/selfheal/controller-after-reboot.log new file mode 100644 index 00000000..96fb8c2e --- /dev/null +++ b/documentation/audits/night-2026-10-28/part5-claims/selfheal/controller-after-reboot.log @@ -0,0 +1,611 @@ +2026/10/29 03:35:24 main.go:356: [INFO] felhom-controller 0.347.0-rc.n1028 starting (customer: demo-hp, domain: enkisfelhom.hu) +2026/10/29 03:35:24 settings.go:821: [INFO] [settings] Loaded settings from /opt/docker/felhom-controller/data/settings.json +2026/10/29 03:35:24 settings.go:1869: [DEBUG] [settings] AutoDiscoverStoragePaths discovered=[/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx] existing=11 +2026/10/29 03:35:24 main.go:382: [INFO] Encryption key loaded from /opt/docker/felhom-controller/data/encryption.key +2026/10/29 03:35:24 manager.go:400: [INFO] [stacks] Using compose command: docker compose +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=false composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=false composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "dawarich" deployed=false composePath=/opt/docker/stacks/dawarich/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grimmory" deployed=false composePath=/opt/docker/stacks/grimmory/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grocy" deployed=false composePath=/opt/docker/stacks/grocy/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "karakeep" deployed=false composePath=/opt/docker/stacks/karakeep/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "lubelogger" deployed=false composePath=/opt/docker/stacks/lubelogger/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "metube" deployed=false composePath=/opt/docker/stacks/metube/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=false composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=false composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "nzbget" deployed=false composePath=/opt/docker/stacks/nzbget/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=false composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "prowlarr" deployed=false composePath=/opt/docker/stacks/prowlarr/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "qbittorrent" deployed=false composePath=/opt/docker/stacks/qbittorrent/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "radicale" deployed=false composePath=/opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=false composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sabnzbd" deployed=false composePath=/opt/docker/stacks/sabnzbd/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "transmission" deployed=false composePath=/opt/docker/stacks/transmission/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=false composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/10/29 03:35:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/10/29 03:35:24 manager.go:1964: [DEBUG] [stacks] getCatalogTemplateSlugs: found 65 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/10/29 03:35:24 manager.go:761: [DEBUG] [stacks] ScanStacks: catalog has 65 template slugs for orphan detection +2026/10/29 03:35:24 manager.go:792: [INFO] [stacks] ScanStacks complete: 70 stacks found (1 deployed, 69 available) +2026/10/29 03:35:24 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:35:24 deploy.go:1389: [DEBUG] [stacks] InjectMissingFields: checking 1 stacks +2026/10/29 03:35:24 deploy.go:1411: [DEBUG] [stacks] InjectMissingFields: checking stack paperless-ngx — 8 deploy fields, 8 existing env vars +2026/10/29 03:35:24 deploy.go:1482: [INFO] [stacks] InjectMissingFields: processed 1 stacks +2026/10/29 03:35:24 manager.go:539: [DEBUG] [stacks] MigrateEncryption: checking 1 deployed stacks for plaintext sensitive values +2026/10/29 03:35:24 manager.go:559: [DEBUG] [stacks] MigrateEncryption: checking stack "paperless-ngx" (3 sensitive fields) +2026/10/29 03:35:24 manager.go:583: [INFO] [stacks] Encryption migration: no stacks needed migration +2026/10/29 03:35:24 desiredstate.go:137: [INFO] [stacks] desired-state backfill: 0 app(s) recorded as running, 0 left unrecorded (state ambiguous — legacy boot behaviour retained) +2026/10/29 03:35:24 installed.go:537: [INFO] [stacks] installed-images backfill: 0 app(s) recorded, 1 already had a record, 0 left unrecorded (could not be observed completely — unknown, which renders nothing) +2026/10/29 03:35:24 pin.go:290: [INFO] [stacks] pin adoption: 0 pinned, 1 already pinned, 0 left unpinned (0 not completely observed, 0 running something the template no longer offers) +2026/10/29 03:35:24 undo.go:676: [INFO] [stacks] applied-meta backfill (R-646): recorded 0 []; skipped 0 [] — not current with the catalog, their pinned version's .felhom.yml is no longer on the box +2026/10/29 03:35:24 sync.go:137: [INFO] [sync] Starting catalog sync (repo: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git, interval: 15m0s) +2026/10/29 03:35:24 sync.go:228: [INFO] [sync] Starting catalog sync +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] CPUCollector.Start: sampleRate=5s +2026/10/29 03:35:24 main.go:571: [INFO] Metrics store opened at /opt/docker/felhom-controller/data/metrics.db +2026/10/29 03:35:24 main.go:580: [INFO] Metrics collector started (60s interval) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/10/29 03:35:24 sync.go:358: [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (branch: main) +2026/10/29 03:35:24 sync.go:360: [DEBUG] [sync] git fetch --depth 1 origin main in /opt/docker/felhom-controller/data/catalog-cache +2026/10/29 03:35:24 sync.go:787: [DEBUG] [sync] Running: git fetch --depth 1 origin main +2026/10/29 03:35:24 [INFO] [tier2] start-up: no leftover second-drive copy container +2026/10/29 03:35:24 restore_record_wiring.go:37: [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false) +2026/10/29 03:35:24 run_record_wiring.go:29: [INFO] [backup] app-data run record wired: /opt/docker/felhom-controller/data/appdata-run.json (an earlier cut still shown: false) +2026/10/29 03:35:24 notifier.go:83: [INFO] Notifier disabled (hub not configured) +2026/10/29 03:35:24 upwatch_wiring.go:154: [INFO] [upwatch] upload and CPU watch started (line unknown — no weekly line test yet and no counting off-site push, qBittorrent share 50 %) +2026/10/29 03:35:24 main.go:821: [INFO] [mailrelay] app-email shim unavailable (no hub configured or disabled in config) +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: status-refresh (every 10s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="status-refresh" interval=10s totalJobs=1 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: stack-scan (every 2m0s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="stack-scan" interval=2m0s totalJobs=2 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: docker-socket-watch (every 15s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="docker-socket-watch" interval=15s totalJobs=3 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: docker-socket-users (every 5m0s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="docker-socket-users" interval=5m0s totalJobs=4 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: health-probes (every 10s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="health-probes" interval=10s totalJobs=5 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: system-health (every 5m0s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="system-health" interval=5m0s totalJobs=6 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: after-unclean-boot (every 30s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="after-unclean-boot" interval=30s totalJobs=7 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: hold-report (every 1m0s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="hold-report" interval=1m0s totalJobs=8 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: deadapp-check (every 30s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="deadapp-check" interval=30s totalJobs=9 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: ring-spill (every 30s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="ring-spill" interval=30s totalJobs=10 +2026/10/29 03:35:24 scheduler.go:150: [INFO] [scheduler] Daily job db-dump scheduled for 2026-10-30 02:30 CET +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="db-dump" schedule="02:30" nextRun=2026-10-30T02:30:00+01:00 totalJobs=11 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="offsite-credential-retry" interval=5m0s totalJobs=12 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="backup-cache" interval=5m0s totalJobs=13 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: conversion-copy-release (every 1h0m0s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="conversion-copy-release" interval=1h0m0s totalJobs=14 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: restore-copy-sweep (every 6h0m0s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="restore-copy-sweep" interval=6h0m0s totalJobs=15 +2026/10/29 03:35:24 scheduler.go:150: [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-10-30 03:30 CET +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="tier2-backup" schedule="03:30" nextRun=2026-10-30T03:30:00+01:00 totalJobs=16 +2026/10/29 03:35:24 scheduler.go:150: [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-10-30 04:15 CET +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="offbox-backup" schedule="04:15" nextRun=2026-10-30T04:15:00+01:00 totalJobs=17 +2026/10/29 03:35:24 nightchain.go:284: [INFO] [catch-up] controller start: no backup leg missed its last scheduled time — nothing to make up +2026/10/29 03:35:24 scheduler.go:150: [INFO] [scheduler] Daily job offsite-abandon-sweep scheduled for 2026-10-29 05:10 CET +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="offsite-abandon-sweep" schedule="05:10" nextRun=2026-10-29T05:10:00+01:00 totalJobs=18 +2026/10/29 03:35:24 scheduler.go:150: [INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-10-29 06:00 CET +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="offsite-integrity" schedule="06:00" nextRun=2026-10-29T06:00:00+01:00 totalJobs=19 +2026/10/29 03:35:24 scheduler.go:150: [INFO] [scheduler] Daily job offsite-proof scheduled for 2026-10-29 05:30 CET +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="offsite-proof" schedule="05:30" nextRun=2026-10-29T05:30:00+01:00 totalJobs=20 +2026/10/29 03:35:24 scheduler.go:150: [INFO] [scheduler] Daily job metrics-prune scheduled for 2026-10-30 04:00 CET +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="metrics-prune" schedule="04:00" nextRun=2026-10-30T04:00:00+01:00 totalJobs=21 +2026/10/29 03:35:24 scheduler.go:150: [INFO] [scheduler] Daily job fill-watch scheduled for 2026-10-30 03:30 CET +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job registered: name="fill-watch" schedule="03:30" nextRun=2026-10-30T03:30:00+01:00 totalJobs=22 +2026/10/29 03:35:24 scheduler.go:120: [INFO] [scheduler] Registered periodic job: fill-watch-interval (every 10m0s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="fill-watch-interval" interval=10m0s totalJobs=23 +2026/10/29 03:35:24 selftest.go:50: [INFO] ========== Startup Self-Test ========== +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈12071MB avail≈13827MB +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=16398084 → total=25898MB avail=13827MB used=12071MB (46.6%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=9.8GB avail=55.0GB (15.1%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=235.7GB avail=654.4GB (26.5%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readLoadAvg: raw="2.20 1.39 1.55 9/1608 127" → 1m=2.20 5m=1.39 15m=1.55 +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 66.6°C (hwmon2) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetInfo done in 100ms — mem=12071MB/25898MB (46.6%), rootDisk=9.8GB/68.4GB (15.1%), load=2.20/1.39/1.55, temp=66.6°C (hwmon2), cpu=0.0% +2026/10/29 03:35:24 selftest.go:68: [INFO] [PASS] Docker socket: reachable (v29.8.2) +2026/10/29 03:35:24 selftest.go:68: [INFO] [PASS] Stacks directory: /opt/docker/stacks +2026/10/29 03:35:24 selftest.go:68: [INFO] [PASS] Data directory: /opt/docker/felhom-controller/data (writable) +2026/10/29 03:35:24 selftest.go:68: [INFO] [PASS] System data path: /mnt/sys_drive +2026/10/29 03:35:24 selftest.go:68: [INFO] [PASS] Storage paths: 1 connected, 10 disconnected +2026/10/29 03:35:24 selftest.go:68: [INFO] [PASS] Git catalog: 68 app definitions found +2026/10/29 03:35:24 selftest.go:68: [INFO] [PASS] Hub connectivity: hub disabled, skipped +2026/10/29 03:35:24 selftest.go:68: [INFO] [PASS] Metrics DB: 43.3 MB +2026/10/29 03:35:24 selftest.go:70: [INFO] ======================================== +2026/10/29 03:35:24 selftest.go:71: [INFO] Self-test complete: 8 passed, 0 warnings, 0 failed +2026/10/29 03:35:24 scheduler.go:224: [INFO] [scheduler] Starting scheduler with 23 jobs +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] scheduler started: periodic=15 daily=8 +2026/10/29 03:35:24 server.go:318: [DEBUG] [web] NewServer: initializing web server v0.347.0-rc.n1028 +2026/10/29 03:35:24 server.go:319: [DEBUG] [web] NewServer: backup=true scheduler=true alertMgr=true notifier=true updater=false +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job db-dump: next run at 2026-10-30 02:30:00 CET (waiting 21h54m35s) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job offbox-backup: next run at 2026-10-30 04:15:00 CET (waiting 23h39m35s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job offsite-proof: next run at 2026-10-29 05:30:00 CET (waiting 54m35s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job offsite-abandon-sweep: next run at 2026-10-29 05:10:00 CET (waiting 34m35s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job offsite-integrity: next run at 2026-10-29 06:00:00 CET (waiting 1h24m35s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job metrics-prune: next run at 2026-10-30 04:00:00 CET (waiting 23h24m35s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job tier2-backup: next run at 2026-10-30 03:30:00 CET (waiting 22h54m35s) +2026/10/29 03:35:24 scheduler.go:85: [DEBUG] [scheduler] daily job fill-watch: next run at 2026-10-30 03:30:00 CET (waiting 22h54m35s) +2026/10/29 03:35:24 backup.go:540: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx → [paperless-ngx] +2026/10/29 03:35:24 backup.go:1246: [INFO] [backup] Found 2 DB dump files across drives +2026/10/29 03:35:24 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/10/29 03:35:24 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: a60825d83a11 felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.n1028 +817bf55354d0 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +95bdf0a25695 paperless-redis paperless-ngx redis:7-alpine +b85f77872d9e paperless-postgres paperless-ngx postgres:18-alpine +ee3380605608 filebrowser filebrowser gtstef/filebrowser:1.5.8-stable +e27bdddefe5e traefik traefik traefik:v3.7.16 +2026/10/29 03:35:24 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.n1028, not a database) +2026/10/29 03:35:24 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/10/29 03:35:24 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/10/29 03:35:24 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=b85f77872d9e) +2026/10/29 03:35:24 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/10/29 03:35:24 server.go:399: [DEBUG] [web] loadTemplates: lang=hu loaded 99 templates, 2087 markers expanded in 46.379875ms +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈12095MB avail≈13803MB +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=16369048 → total=25898MB avail=13803MB used=12095MB (46.7%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=9.8GB avail=55.0GB (15.1%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=235.7GB avail=654.4GB (26.5%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readLoadAvg: raw="2.20 1.39 1.55 18/1617 204" → 1m=2.20 5m=1.39 15m=1.55 +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 66.9°C (hwmon2) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetInfo done in 108ms — mem=12095MB/25898MB (46.7%), rootDisk=9.8GB/68.4GB (15.1%), load=2.20/1.39/1.55, temp=66.9°C (hwmon2), cpu=0.0% +2026/10/29 03:35:24 healthcheck.go:128: [DEBUG] [monitor] Raw values: disk=15.1%, hdd=26.5% (configured=true), mem=46.7% (12095MB/25898MB), cpu=0.0%, temp=66.9°C (hwmon2) +2026/10/29 03:35:24 healthcheck.go:409: [DEBUG] [monitor] SSD disk: OK (15%) +2026/10/29 03:35:24 healthcheck.go:445: [DEBUG] [monitor] Memory: OK (47%) +2026/10/29 03:35:24 healthcheck.go:483: [DEBUG] [monitor] Temperature: OK (67°C) +2026/10/29 03:35:24 server.go:399: [DEBUG] [web] loadTemplates: lang=en loaded 99 templates, 2087 markers expanded in 52.993162ms +2026/10/29 03:35:24 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/10/29 03:35:24 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.8-stable, not a database) +2026/10/29 03:35:24 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.16, not a database) +2026/10/29 03:35:24 dbdump.go:205: [DEBUG] DiscoverDatabases: found 1 database(s), skipped 5 non-DB container(s) +2026/10/29 03:35:24 dbdump.go:208: [INFO] [backup] Discovered 1 databases +2026/10/29 03:35:24 session_store.go:113: [INFO] [web] sessions: restored 218 dashboard session(s) from disk (dropped 0 expired, 0 malformed) +2026/10/29 03:35:24 server.go:339: [INFO] [web] Auth: using password from settings.json +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 03:35:24 server.go:983: [DEBUG] [web] CatchAllMiddleware: controller host=felhom.enkisfelhom.hu +2026/10/29 03:35:24 main.go:2387: [INFO] Web UI listening on :8080 +2026/10/29 03:35:24 [INFO] [backup] Discovered app data: 1 apps +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:35:24 handlers.go:4017: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping userdata skeleton +2026/10/29 03:35:24 handlers.go:4017: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping userdata skeleton +2026/10/29 03:35:24 handlers.go:4017: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping userdata skeleton +2026/10/29 03:35:24 handlers.go:4017: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping userdata skeleton +2026/10/29 03:35:24 handlers.go:4017: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf not mounted — skipping userdata skeleton +2026/10/29 03:35:24 handlers.go:4017: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/immich not mounted — skipping userdata skeleton +2026/10/29 03:35:24 handlers.go:4017: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/calibre-web not mounted — skipping userdata skeleton +2026/10/29 03:35:24 handlers.go:4017: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/grimmory not mounted — skipping userdata skeleton +2026/10/29 03:35:24 handlers.go:4017: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/metube not mounted — skipping userdata skeleton +2026/10/29 03:35:24 backup.go:1317: [INFO] [backup] Backup status cache refreshed +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 03:35:24 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/10/29 03:35:24 healthcheck.go:145: [DEBUG] [monitor] Docker daemon: OK +2026/10/29 03:35:24 healthcheck.go:153: [DEBUG] [monitor] Checking 1 protected containers: [traefik] +2026/10/29 03:35:24 infra.go:378: [INFO] [infra] connected felhom-controller to traefik-public +2026/10/29 03:35:24 infra.go:97: [INFO] [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node) +2026/10/29 03:35:24 healthcheck.go:161: [DEBUG] [monitor] All protected containers running +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/immich" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/calibre-web" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:35:24 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sizing-bench" total=31.2 GB used=1.8 GB avail=27.7GB (6.2%) +2026/10/29 03:35:24 healthcheck.go:176: [INFO] [monitor] Health check: status=warn +2026/10/29 03:35:24 healthcheck.go:180: [DEBUG] [monitor] Final status: warn (issues=0, warnings=10, info=4) +2026/10/29 03:35:24 filebrowser_gate.go:70: [INFO] [infra] connected felhom-controller to felhom-filebrowser (the FileBrowser default-login check, R-1126) +2026/10/29 03:35:24 sync.go:787: [DEBUG] [sync] Running: git reset --hard origin/main +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] actualbudget/docker-compose.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] actualbudget/.felhom.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] adventurelog/docker-compose.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] adventurelog/.felhom.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] audiobookshelf/docker-compose.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] audiobookshelf/.felhom.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] bentopdf/docker-compose.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] bentopdf/.felhom.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] bookstack/docker-compose.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] bookstack/.felhom.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] calcom/docker-compose.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] calcom/.felhom.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] calibre-web/docker-compose.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] calibre-web/.felhom.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] claper/docker-compose.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] claper/.felhom.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] code-server/docker-compose.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] code-server/.felhom.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] crafty-controller/docker-compose.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] crafty-controller/.felhom.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] dawarich/docker-compose.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] dawarich/.felhom.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] docmost/docker-compose.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] docmost/.felhom.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] emby/docker-compose.yml: hash match, skipped +2026/10/29 03:35:24 sync.go:559: [DEBUG] [sync] emby/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] ghost/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] ghost/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] gitea/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] gitea/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] glance/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] glance/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] gokapi/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] gokapi/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] grafana/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] grafana/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] gramps-web/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] gramps-web/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] grimmory/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] grimmory/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] grocy/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] grocy/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] home-assistant/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] home-assistant/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] homebox/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] homebox/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] homepage/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] homepage/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] immich/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] immich/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] jellyfin/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] jellyfin/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] karakeep/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] karakeep/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] kimai/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] kimai/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] komga/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] komga/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] lubelogger/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] lubelogger/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] mealie/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] mealie/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] metube/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] metube/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] n8n/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] n8n/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] navidrome/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] navidrome/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] nextcloud/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] nextcloud/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] nzbget/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] nzbget/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] onlyoffice/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] onlyoffice/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] opengist/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] opengist/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] outline/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] outline/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] paperless-ngx/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] paperless-ngx/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] papra/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] papra/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] plant-it/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] plant-it/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] privatebin/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] privatebin/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] prowlarr/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] prowlarr/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] qbittorrent/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] qbittorrent/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] radarr/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] radarr/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] radicale/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] radicale/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] rallly/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] rallly/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] recipe-importer/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] recipe-importer/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] romm/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] romm/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] sabnzbd/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] sabnzbd/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] seerr/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] seerr/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] sonarr/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] sonarr/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] sparkyfitness/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] sparkyfitness/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] tandoor/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] tandoor/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] termix/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] termix/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] transmission/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] transmission/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] uptime-kuma/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] uptime-kuma/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] vaultwarden/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] vaultwarden/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] vikunja/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] vikunja/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] wanderer/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] wanderer/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] wger/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] wger/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] wishlist/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] wishlist/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] zipline/docker-compose.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:559: [DEBUG] [sync] zipline/.felhom.yml: hash match, skipped +2026/10/29 03:35:25 sync.go:299: [INFO] [sync] Catalog sync complete +2026/10/29 03:35:25 sync.go:142: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +2026/10/29 03:35:25 manager.go:258: [DEBUG] [integrations] ReapplyConfigForTarget: target=filebrowser integrations=0 +2026/10/29 03:35:25 handlers.go:3956: [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (11 storage path(s)) +2026/10/29 03:35:29 info.go:13: [DEBUG] [system] CPUCollector: first sample — cpu=36.2% (idle=2489 total=3901) +2026/10/29 03:35:29 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:35:29 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event controller_started (severity info) — further controller_started events are logged at DEBUG only +2026/10/29 03:35:34 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:35:34 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:35:34 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:35:34 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:35:34 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (0 skipped not due, 0 skipped no container) +2026/10/29 03:35:39 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:35:39 main.go:2871: [INFO] [bootrecon] boot window: fleet settled after 10s (3 identical samples 5s apart) — sweeping +2026/10/29 03:35:39 bootrecon.go:255: [INFO] [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start) +2026/10/29 03:35:39 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:35:44 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:35:44 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (0 skipped not due, 0 skipped no container) +2026/10/29 03:35:44 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:35:44 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:35:54 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:35:54 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (0 skipped not due, 0 skipped no container) +2026/10/29 03:35:54 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:35:54 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/29 03:35:54 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/29 03:35:54 crashboot.go:147: [INFO] [deadapp] boot grace 1m30s: no agent to ask — a normal boot (R-856) +2026/10/29 03:35:54 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:35:54 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/29 03:35:54 aftercrash.go:106: [INFO] [aftercrash] a normal (or unknown) boot — no app repair after an unclean boot (R-1007) +2026/10/29 03:35:54 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:36:04 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:36:04 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (0 skipped not due, 0 skipped no container) +2026/10/29 03:36:04 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:36:04 healthprobe.go:187: [DEBUG] Health probe paperless-ngx: HTTP GET :8000/ → 302 (4ms) +2026/10/29 03:36:04 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:36:04 healthprobe.go:167: [INFO] Health probes: 1 ok (of 1 probed) +2026/10/29 03:36:09 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:36:14 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:36:14 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:36:14 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:36:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 10s ago, effective interval 5m0s, healthy=true +2026/10/29 03:36:14 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:36:24 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/10/29 03:36:24 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈12516MB avail≈13382MB +2026/10/29 03:36:24 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=15870844 → total=25898MB avail=13382MB used=12516MB (48.3%) +2026/10/29 03:36:24 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=9.8GB avail=55.0GB (15.2%) +2026/10/29 03:36:24 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=235.7GB avail=654.4GB (26.5%) +2026/10/29 03:36:24 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.75 1.46 1.56 6/1592 552" → 1m=1.75 5m=1.46 15m=1.56 +2026/10/29 03:36:24 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/29 03:36:24 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.0°C (hwmon0) +2026/10/29 03:36:24 info.go:13: [DEBUG] [system] GetInfo done in 69ms — mem=12516MB/25898MB (48.3%), rootDisk=9.8GB/68.4GB (15.2%), load=1.75/1.46/1.56, temp=49.0°C (hwmon0), cpu=5.0% +2026/10/29 03:36:24 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:36:24 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/29 03:36:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 20s ago, effective interval 5m0s, healthy=true +2026/10/29 03:36:24 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/29 03:36:24 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/29 03:36:24 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/29 03:36:24 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:36:24 holdreport.go:169: [DEBUG] [holdreport] full re-send: 0 hold(s) sent, 0 cleared +2026/10/29 03:36:24 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 03:36:24 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:36:24 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:36:24 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/29 03:36:24 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:36:34 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:36:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/29 03:36:34 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:36:34 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:36:34 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:36:39 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:36:44 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:36:44 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:36:44 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:36:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/29 03:36:44 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:36:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=9.8 GB avail=55.0GB (15.2%) +2026/10/29 03:36:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:36:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:36:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/calibre-web" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:36:54 info.go:13: [DEBUG] [system] GetDiskUsage: statfs("/mnt/felhom-drives/scratch_hdd/userdata/grimmory") failed: no such file or directory +2026/10/29 03:36:54 fillwatch.go:233: [DEBUG] [fillwatch] /mnt/felhom-drives/scratch_hdd/userdata/grimmory: usage unreadable — skipped (an unreadable filesystem is not a full one) +2026/10/29 03:36:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/immich" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:36:54 info.go:13: [DEBUG] [system] GetDiskUsage: statfs("/mnt/felhom-drives/scratch_hdd/userdata/metube") failed: no such file or directory +2026/10/29 03:36:54 fillwatch.go:233: [DEBUG] [fillwatch] /mnt/felhom-drives/scratch_hdd/userdata/metube: usage unreadable — skipped (an unreadable filesystem is not a full one) +2026/10/29 03:36:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:36:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:36:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:36:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:36:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sizing-bench" total=31.2 GB used=1.8 GB avail=27.7GB (6.2%) +2026/10/29 03:36:54 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=9.8 GB avail=55.0GB (15.2%) +2026/10/29 03:36:54 fillwatch.go:284: [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: all ok +2026/10/29 03:36:54 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:36:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/29 03:36:54 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:36:54 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/29 03:36:54 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/29 03:36:54 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/29 03:36:54 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:36:54 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:36:54 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:37:04 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:37:04 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:37:04 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:37:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/29 03:37:04 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:37:09 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:37:14 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:37:14 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:37:14 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:37:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/10/29 03:37:14 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:37:24 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/10/29 03:37:24 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈12521MB avail≈13377MB +2026/10/29 03:37:24 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=15864500 → total=25898MB avail=13377MB used=12521MB (48.3%) +2026/10/29 03:37:24 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=9.8GB avail=55.0GB (15.2%) +2026/10/29 03:37:24 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=235.7GB avail=654.4GB (26.5%) +2026/10/29 03:37:24 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.05 1.34 1.52 7/1592 685" → 1m=1.05 5m=1.34 15m=1.52 +2026/10/29 03:37:24 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/29 03:37:24 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.0°C (hwmon0) +2026/10/29 03:37:24 info.go:13: [DEBUG] [system] GetInfo done in 61ms — mem=12521MB/25898MB (48.3%), rootDisk=9.8GB/68.4GB (15.2%), load=1.05/1.34/1.52, temp=49.0°C (hwmon0), cpu=5.1% +2026/10/29 03:37:24 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/29 03:37:24 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/29 03:37:24 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/29 03:37:24 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/29 03:37:24 scheduler.go:450: [INFO] [scheduler] Running job: stack-scan +2026/10/29 03:37:24 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:37:24 scheduler.go:85: [DEBUG] [scheduler] job stack-scan: execution starting +2026/10/29 03:37:24 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:37:24 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/29 03:37:24 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 03:37:24 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=false composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=false composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "dawarich" deployed=false composePath=/opt/docker/stacks/dawarich/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grimmory" deployed=false composePath=/opt/docker/stacks/grimmory/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grocy" deployed=false composePath=/opt/docker/stacks/grocy/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "karakeep" deployed=false composePath=/opt/docker/stacks/karakeep/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "lubelogger" deployed=false composePath=/opt/docker/stacks/lubelogger/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "metube" deployed=false composePath=/opt/docker/stacks/metube/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=false composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=false composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "nzbget" deployed=false composePath=/opt/docker/stacks/nzbget/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=false composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "prowlarr" deployed=false composePath=/opt/docker/stacks/prowlarr/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "qbittorrent" deployed=false composePath=/opt/docker/stacks/qbittorrent/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "radicale" deployed=false composePath=/opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=false composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sabnzbd" deployed=false composePath=/opt/docker/stacks/sabnzbd/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "transmission" deployed=false composePath=/opt/docker/stacks/transmission/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=false composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/10/29 03:37:24 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/10/29 03:37:24 manager.go:1964: [DEBUG] [stacks] getCatalogTemplateSlugs: found 65 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/10/29 03:37:24 manager.go:761: [DEBUG] [stacks] ScanStacks: catalog has 65 template slugs for orphan detection +2026/10/29 03:37:24 manager.go:792: [INFO] [stacks] ScanStacks complete: 70 stacks found (1 deployed, 69 available) +2026/10/29 03:37:24 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:37:24 scheduler.go:467: [INFO] [scheduler] Job stack-scan completed (took 109ms) +2026/10/29 03:37:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/10/29 03:37:24 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:37:24 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks diff --git a/documentation/audits/night-2026-10-28/part5-claims/selfheal/ps-after-reboot.txt b/documentation/audits/night-2026-10-28/part5-claims/selfheal/ps-after-reboot.txt new file mode 100644 index 00000000..bdada7c7 --- /dev/null +++ b/documentation/audits/night-2026-10-28/part5-claims/selfheal/ps-after-reboot.txt @@ -0,0 +1,6 @@ +felhom-controller Up 32 seconds (healthy) +paperless-webserver Up 33 seconds (healthy) +paperless-redis Up 33 seconds (healthy) +paperless-postgres Up 33 seconds (healthy) +filebrowser Up 33 seconds (healthy) +traefik Up 33 seconds diff --git a/documentation/audits/night-2026-10-28/part5-claims/selfheal/t0-kill.txt b/documentation/audits/night-2026-10-28/part5-claims/selfheal/t0-kill.txt new file mode 100644 index 00000000..b86570bd --- /dev/null +++ b/documentation/audits/night-2026-10-28/part5-claims/selfheal/t0-kill.txt @@ -0,0 +1 @@ +2026-10-29T03:31:55Z diff --git a/documentation/audits/night-2026-10-28/part5-claims/selfheal/t0-reboot.txt b/documentation/audits/night-2026-10-28/part5-claims/selfheal/t0-reboot.txt new file mode 100644 index 00000000..55ac04f7 --- /dev/null +++ b/documentation/audits/night-2026-10-28/part5-claims/selfheal/t0-reboot.txt @@ -0,0 +1 @@ +2026-10-29T03:34:11Z diff --git a/documentation/audits/night-2026-10-28/r1256-restore/h8-live/after-hand-cleanup.txt b/documentation/audits/night-2026-10-28/r1256-restore/h8-live/after-hand-cleanup.txt new file mode 100644 index 00000000..bea86b3f --- /dev/null +++ b/documentation/audits/night-2026-10-28/r1256-restore/h8-live/after-hand-cleanup.txt @@ -0,0 +1,2 @@ +/mnt/felhom-drives/scratch_hdd/backups/restore-undo +/mnt/sys_drive/felhom-data/backups/restore-undo diff --git a/documentation/audits/night-2026-10-28/r1256-restore/h8-live/box-state.txt b/documentation/audits/night-2026-10-28/r1256-restore/h8-live/box-state.txt new file mode 100644 index 00000000..1bd7e132 --- /dev/null +++ b/documentation/audits/night-2026-10-28/r1256-restore/h8-live/box-state.txt @@ -0,0 +1,7 @@ +gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.n1028 +paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 2 hours (healthy) +paperless-redis redis:7-alpine Up 2 hours (healthy) +paperless-postgres postgres:18-alpine Up 2 hours (healthy) +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.n1028 Up 6 hours (healthy) +filebrowser gtstef/filebrowser:1.5.8-stable Up 29 hours (healthy) +traefik traefik:v3.7.16 Up 37 hours diff --git a/documentation/audits/night-2026-10-28/r1256-restore/h8-live/controller-0329-0330.log b/documentation/audits/night-2026-10-28/r1256-restore/h8-live/controller-0329-0330.log new file mode 100644 index 00000000..5a4aac76 --- /dev/null +++ b/documentation/audits/night-2026-10-28/r1256-restore/h8-live/controller-0329-0330.log @@ -0,0 +1,358 @@ +2026/10/29 03:29:04 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:29:04 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:29:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 30s ago, effective interval 5m0s, healthy=true +2026/10/29 03:29:04 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:29:04 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:29:14 sync.go:228: [INFO] [sync] Starting catalog sync +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/10/29 03:29:14 sync.go:358: [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (branch: main) +2026/10/29 03:29:14 sync.go:360: [DEBUG] [sync] git fetch --depth 1 origin main in /opt/docker/felhom-controller/data/catalog-cache +2026/10/29 03:29:14 sync.go:787: [DEBUG] [sync] Running: git fetch --depth 1 origin main +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈12546MB avail≈13352MB +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=15834148 → total=25898MB avail=13352MB used=12546MB (48.4%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=9.8GB avail=55.0GB (15.2%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=235.7GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.83 1.56 1.75 12/1650 85998" → 1m=0.83 5m=1.56 15m=1.75 +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetInfo done in 83ms — mem=12546MB/25898MB (48.4%), rootDisk=9.8GB/68.4GB (15.2%), load=0.83/1.56/1.75, temp=49.9°C (hwmon1), cpu=6.4% +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:29:14 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 40s ago, effective interval 5m0s, healthy=true +2026/10/29 03:29:14 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:29:14 scheduler.go:450: [INFO] [scheduler] Running job: stack-scan +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job stack-scan: execution starting +2026/10/29 03:29:14 scheduler.go:450: [INFO] [scheduler] Running job: offsite-credential-retry +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job offsite-credential-retry: execution starting +2026/10/29 03:29:14 scheduler.go:467: [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/10/29 03:29:14 scheduler.go:450: [INFO] [scheduler] Running job: hold-report +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job hold-report: execution starting +2026/10/29 03:29:14 scheduler.go:467: [INFO] [scheduler] Job hold-report completed (took 0s) +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/29 03:29:14 scheduler.go:450: [INFO] [scheduler] Running job: conversion-copy-release +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job conversion-copy-release: execution starting +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/29 03:29:14 scheduler.go:450: [INFO] [scheduler] Running job: restore-copy-sweep +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job restore-copy-sweep: execution starting +2026/10/29 03:29:14 scheduler.go:450: [INFO] [scheduler] Running job: docker-socket-users +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-users: execution starting +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:29:14 scheduler.go:450: [INFO] [scheduler] Running job: fill-watch-interval +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/29 03:29:14 scheduler.go:450: [INFO] [scheduler] Running job: system-health +2026/10/29 03:29:14 scheduler.go:450: [INFO] [scheduler] Running job: backup-cache +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job fill-watch-interval: execution starting +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job system-health: execution starting +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job backup-cache: execution starting +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=9.8 GB avail=55.0GB (15.2%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/calibre-web" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: statfs("/mnt/felhom-drives/scratch_hdd/userdata/grimmory") failed: no such file or directory +2026/10/29 03:29:14 fillwatch.go:233: [DEBUG] [fillwatch] /mnt/felhom-drives/scratch_hdd/userdata/grimmory: usage unreadable — skipped (an unreadable filesystem is not a full one) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/immich" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: statfs("/mnt/felhom-drives/scratch_hdd/userdata/metube") failed: no such file or directory +2026/10/29 03:29:14 fillwatch.go:233: [DEBUG] [fillwatch] /mnt/felhom-drives/scratch_hdd/userdata/metube: usage unreadable — skipped (an unreadable filesystem is not a full one) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sizing-bench" total=31.2 GB used=1.8 GB avail=27.7GB (6.2%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=9.8 GB avail=55.0GB (15.2%) +2026/10/29 03:29:14 fillwatch.go:284: [INFO] [fillwatch] checked 11 filesystem(s), 2 unreadable/skipped, 0 notification(s); bands: all ok +2026/10/29 03:29:14 scheduler.go:467: [INFO] [scheduler] Job fill-watch-interval completed (took 5ms) +2026/10/29 03:29:14 restore_copy_sweep.go:92: [INFO] [backup] paperless: the unused restore copy /mnt/felhom-drives/scratch_hdd/backups/restore-undo/paperless/20261005T010000Z (from 2026-10-05T01:00:00Z) is removed — no hold, no pending put-back, older than 336h0m0s (R-1256) +2026/10/29 03:29:14 restore_copy_sweep.go:92: [INFO] [backup] r1256probe-old: the unused restore copy /mnt/felhom-drives/scratch_hdd/backups/restore-undo/r1256probe-old/20261001T000000Z (from 2026-10-01T00:00:00Z) is removed — no hold, no pending put-back, older than 336h0m0s (R-1256) +2026/10/29 03:29:14 scheduler.go:467: [INFO] [scheduler] Job restore-copy-sweep completed (took 8ms) +2026/10/29 03:29:14 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:29:14 scheduler.go:467: [INFO] [scheduler] Job conversion-copy-release completed (took 39ms) +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=false composePath=/opt/docker/stacks/chaosoom/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=false composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "dawarich" deployed=false composePath=/opt/docker/stacks/dawarich/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grimmory" deployed=false composePath=/opt/docker/stacks/grimmory/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "grocy" deployed=false composePath=/opt/docker/stacks/grocy/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "karakeep" deployed=false composePath=/opt/docker/stacks/karakeep/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "lubelogger" deployed=false composePath=/opt/docker/stacks/lubelogger/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "metube" deployed=false composePath=/opt/docker/stacks/metube/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=false composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=false composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "nzbget" deployed=false composePath=/opt/docker/stacks/nzbget/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714380KB) → used≈12544MB avail≈13354MB +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714380 availKB=15836540 → total=25898MB avail=13354MB used=12544MB (48.4%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=9.8GB avail=55.0GB (15.2%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=235.7GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.83 1.56 1.75 12/1643 86058" → 1m=0.83 5m=1.56 15m=1.75 +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 53.2°C (hwmon2) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetInfo done in 83ms — mem=12544MB/25898MB (48.4%), rootDisk=9.8GB/68.4GB (15.2%), load=0.83/1.56/1.75, temp=53.2°C (hwmon2), cpu=6.4% +2026/10/29 03:29:14 healthcheck.go:128: [DEBUG] [monitor] Raw values: disk=15.2%, hdd=26.5% (configured=true), mem=48.4% (12544MB/25898MB), cpu=6.4%, temp=53.2°C (hwmon2) +2026/10/29 03:29:14 healthcheck.go:409: [DEBUG] [monitor] SSD disk: OK (15%) +2026/10/29 03:29:14 healthcheck.go:445: [DEBUG] [monitor] Memory: OK (48%) +2026/10/29 03:29:14 healthcheck.go:464: [DEBUG] [monitor] CPU: OK (6%) +2026/10/29 03:29:14 healthcheck.go:483: [DEBUG] [monitor] Temperature: OK (53°C) +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=false composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "prowlarr" deployed=false composePath=/opt/docker/stacks/prowlarr/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "qbittorrent" deployed=false composePath=/opt/docker/stacks/qbittorrent/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "radicale" deployed=false composePath=/opt/docker/stacks/radicale/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=false composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sabnzbd" deployed=false composePath=/opt/docker/stacks/sabnzbd/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "transmission" deployed=false composePath=/opt/docker/stacks/transmission/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=false composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/10/29 03:29:14 manager.go:680: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/10/29 03:29:14 manager.go:1964: [DEBUG] [stacks] getCatalogTemplateSlugs: found 65 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/10/29 03:29:14 manager.go:761: [DEBUG] [stacks] ScanStacks: catalog has 65 template slugs for orphan detection +2026/10/29 03:29:14 manager.go:792: [INFO] [stacks] ScanStacks complete: 70 stacks found (1 deployed, 69 available) +2026/10/29 03:29:14 sync.go:787: [DEBUG] [sync] Running: git reset --hard origin/main +2026/10/29 03:29:14 healthcheck.go:145: [DEBUG] [monitor] Docker daemon: OK +2026/10/29 03:29:14 healthcheck.go:153: [DEBUG] [monitor] Checking 1 protected containers: [traefik] +2026/10/29 03:29:14 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:29:14 scheduler.go:467: [INFO] [scheduler] Job stack-scan completed (took 148ms) +2026/10/29 03:29:14 backup.go:540: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx → [paperless-ngx] +2026/10/29 03:29:14 backup.go:1246: [INFO] [backup] Found 2 DB dump files across drives +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] actualbudget/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] actualbudget/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] adventurelog/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] adventurelog/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] audiobookshelf/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] audiobookshelf/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] bentopdf/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] bentopdf/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] bookstack/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] bookstack/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] calcom/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] calcom/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] calibre-web/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] calibre-web/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] claper/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] claper/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] code-server/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] code-server/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] crafty-controller/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 scheduler.go:467: [INFO] [scheduler] Job docker-socket-users completed (took 176ms) +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] crafty-controller/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] dawarich/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] dawarich/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] docmost/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] docmost/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 healthcheck.go:161: [DEBUG] [monitor] All protected containers running +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/immich" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/calibre-web" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sizing-bench" total=31.2 GB used=1.8 GB avail=27.7GB (6.2%) +2026/10/29 03:29:14 healthcheck.go:176: [INFO] [monitor] Health check: status=warn +2026/10/29 03:29:14 healthcheck.go:180: [DEBUG] [monitor] Final status: warn (issues=0, warnings=10, info=5) +2026/10/29 03:29:14 notifier.go:1467: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/10/29 03:29:14 scheduler.go:467: [INFO] [scheduler] Job system-health completed (took 186ms) +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] emby/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] emby/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] ghost/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] ghost/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] gitea/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] gitea/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 817bf55354d0 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +95bdf0a25695 paperless-redis paperless-ngx redis:7-alpine +b85f77872d9e paperless-postgres paperless-ngx postgres:18-alpine +210f32433ef7 felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.n1028 +ee3380605608 filebrowser filebrowser gtstef/filebrowser:1.5.8-stable +e27bdddefe5e traefik traefik traefik:v3.7.16 +2026/10/29 03:29:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/10/29 03:29:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/10/29 03:29:14 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=b85f77872d9e) +2026/10/29 03:29:14 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] glance/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] glance/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] gokapi/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] gokapi/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] grafana/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] grafana/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] gramps-web/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] gramps-web/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] grimmory/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] grimmory/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] grocy/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] grocy/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] home-assistant/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] home-assistant/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] homebox/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] homebox/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] homepage/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] homepage/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] immich/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] immich/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] jellyfin/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] jellyfin/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] karakeep/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/10/29 03:29:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.n1028, not a database) +2026/10/29 03:29:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.8-stable, not a database) +2026/10/29 03:29:14 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.16, not a database) +2026/10/29 03:29:14 dbdump.go:205: [DEBUG] DiscoverDatabases: found 1 database(s), skipped 5 non-DB container(s) +2026/10/29 03:29:14 dbdump.go:208: [INFO] [backup] Discovered 1 databases +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] karakeep/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] kimai/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] kimai/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] komga/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] komga/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=235.7 GB avail=654.4GB (26.5%) +2026/10/29 03:29:14 [INFO] [backup] Discovered app data: 1 apps +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] lubelogger/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] lubelogger/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] mealie/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] mealie/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] metube/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] metube/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 backup.go:1317: [INFO] [backup] Backup status cache refreshed +2026/10/29 03:29:14 scheduler.go:467: [INFO] [scheduler] Job backup-cache completed (took 246ms) +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] n8n/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] n8n/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] navidrome/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] navidrome/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] nextcloud/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] nextcloud/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] nzbget/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] nzbget/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] onlyoffice/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] onlyoffice/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] opengist/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] opengist/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] outline/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] outline/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] paperless-ngx/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] paperless-ngx/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] papra/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] papra/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] plant-it/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] plant-it/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] privatebin/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] privatebin/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] prowlarr/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] prowlarr/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] qbittorrent/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] qbittorrent/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] radarr/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] radarr/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] radicale/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] radicale/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] rallly/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] rallly/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] recipe-importer/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] recipe-importer/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] romm/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] romm/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] sabnzbd/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] sabnzbd/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] seerr/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] seerr/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] sonarr/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] sonarr/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] sparkyfitness/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] sparkyfitness/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] tandoor/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] tandoor/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] termix/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] termix/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] transmission/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] transmission/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] uptime-kuma/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] uptime-kuma/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] vaultwarden/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] vaultwarden/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] vikunja/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] vikunja/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] wanderer/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] wanderer/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] wger/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] wger/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] wishlist/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] wishlist/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] zipline/docker-compose.yml: hash match, skipped +2026/10/29 03:29:14 sync.go:559: [DEBUG] [sync] zipline/.felhom.yml: hash match, skipped +2026/10/29 03:29:14 infra.go:97: [INFO] [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node) +2026/10/29 03:29:14 sync.go:299: [INFO] [sync] Catalog sync complete +2026/10/29 03:29:14 sync.go:155: [INFO] [sync] Periodic sync: Sablonok naprakészek — nincs változás +2026/10/29 03:29:24 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:29:24 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 50s ago, effective interval 5m0s, healthy=true +2026/10/29 03:29:24 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:29:24 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:29:24 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:29:29 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:29:34 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:29:34 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/29 03:29:34 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:29:34 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:29:34 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:29:44 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:29:44 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/29 03:29:44 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/29 03:29:44 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/29 03:29:44 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/29 03:29:44 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:29:44 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:29:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/10/29 03:29:44 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:29:54 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/29 03:29:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/10/29 03:29:54 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (1 skipped not due, 0 skipped no container) +2026/10/29 03:29:54 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/29 03:29:54 manager.go:843: [INFO] [stacks] Status refresh: 5 containers across 70 stacks +2026/10/29 03:29:59 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting diff --git a/documentation/audits/night-2026-10-28/r1256-restore/h8-live/sweep-0329.log b/documentation/audits/night-2026-10-28/r1256-restore/h8-live/sweep-0329.log new file mode 100644 index 00000000..7a7366f5 --- /dev/null +++ b/documentation/audits/night-2026-10-28/r1256-restore/h8-live/sweep-0329.log @@ -0,0 +1,12 @@ +03:30:05 +2026/10/29 03:29:14 scheduler.go:450: [INFO] [scheduler] Running job: restore-copy-sweep +2026/10/29 03:29:14 scheduler.go:85: [DEBUG] [scheduler] job restore-copy-sweep: execution starting +2026/10/29 03:29:14 restore_copy_sweep.go:92: [INFO] [backup] paperless: the unused restore copy /mnt/felhom-drives/scratch_hdd/backups/restore-undo/paperless/20261005T010000Z (from 2026-10-05T01:00:00Z) is removed — no hold, no pending put-back, older than 336h0m0s (R-1256) +2026/10/29 03:29:14 restore_copy_sweep.go:92: [INFO] [backup] r1256probe-old: the unused restore copy /mnt/felhom-drives/scratch_hdd/backups/restore-undo/r1256probe-old/20261001T000000Z (from 2026-10-01T00:00:00Z) is removed — no hold, no pending put-back, older than 336h0m0s (R-1256) +2026/10/29 03:29:14 scheduler.go:467: [INFO] [scheduler] Job restore-copy-sweep completed (took 8ms) +/mnt/felhom-drives/scratch_hdd/backups/restore-undo +/mnt/felhom-drives/scratch_hdd/backups/restore-undo/r1256probe-young +/mnt/felhom-drives/scratch_hdd/backups/restore-undo/r1256probe-young/20261028T213014Z +/mnt/sys_drive/felhom-data/backups/restore-undo + +[exited with code 0]