R-356 docs: correct R-107 in the architecture, record the design, refresh STATUS, compress the register
gates / gates (push) Failing after 17s

07-backup-architecture.md: three places said no offsite action unpacks the
named-volume tars. R-107 closed in controller v0.218.0; all three corrected with
a dated [FACT], the old sentence kept in the past tense. R-102 is NOT closed and
the correction says so explicitly.

New [DESIGN] paragraph in 6.3: the restore destination is resolved by the same
rule as the capture destination, and the wrong-disk refusal applies to apps that
have a drive to get wrong. Carries the 13/40 measurement.

STATUS.md was internally contradictory - nothing waiting, and one decision
waiting, for something the same page recorded as shipped. 218 -> 102 lines; the
deciding section now says what happens if nothing is done.

R-356 compressed into CLOSED-ITEMS.md; OPEN-ITEMS 327109 -> 325236 bytes.

Drill record and 16 evidence files for the live walk on demo-hp.
This commit is contained in:
2026-08-22 13:26:10 +02:00
parent e18668f9e1
commit c297b9f85e
22 changed files with 1316 additions and 486 deletions
@@ -334,12 +334,43 @@ refuses **before** stopping the app and names the action that works.
|---|---|---|---|
| Tier-1 | unit incl. volume tars + DB dumps | all of it | none |
| Tier-2 | unit mirror **+** file legs | `hdd/` and `userdata/` **only** (`tier2_restore.go:101-104`) | **the unit mirror is read by nothing** — `RecoveryUnitPath` resolves to `backups/primary/` (`appbackup/paths.go:46-48`) → **R-102** |
| Tier-3 | unit (incl. volume tars) + mandatory legs | files + DB replay; the unit is **skipped** on the way to live (`offbox_reconstitute.go:284-289`; placed only if the live unit is absent, `offbox_restore.go:352-356`) | **no offsite action unpacks the named-volume tars it captures** → **R-107** |
| Tier-3 | unit (incl. volume tars) + mandatory legs | files + DB replay **+ the named-volume tars, replayed from the scratch unit** (`offbox_reconstitute.go` `volReplay`, controller **v0.218.0**); the unit itself is still **skipped** on the way to live (`offbox_reconstitute.go:284-289`; placed only if the live unit is absent, `offbox_restore.go:352-356`) | **CLOSED — R-107** |
**[FACT] 2026-08-22 — the Tier-3 row above was corrected; the Tier-2 row was NOT.** Until controller
**v0.218.0** this table said *"no offsite action unpacks the named-volume tars it captures"*, and that
was true from the day Tier-3 shipped until 2026-08-21. **R-107 closed in v0.218.0**: `volReplay`
(`offbox_reconstitute.go`) replays the scratch unit's `volume-dumps/` into the live named volumes,
proven live on `demo-hp`. The old sentence is kept here, in the past tense, because a correction that
erases what was believed leaves the next reader no way to tell a fixed gap from one that was never
noticed.
**R-102 — the Tier-2 half — is NOT closed and nothing in this correction touches it.** The Tier-2 row
above stands exactly as written: the secondary unit mirror is still read by nothing. Do not read
"R-107 closed" as covering both; they were always two register rows, and only one of them moved.
**[FACT]** Tier-2's gap is the sharper one because of *when* it bites: Tier-2 exists for the case
where the primary drive is lost — and in exactly that case the primary unit is gone while this
mirror survives on the second drive, unreachable by any customer action.
**[DESIGN] 2026-08-22 — the restore destination is resolved by the same rule as the capture
destination.** The drive if the app declares one (`HDD_PATH`), the system data path otherwise —
`Manager.GetAppDrivePath`, one expression, used by `CaptureRecoveryUnit` and, since controller
**v0.219.0**, by `ReconstituteFromOffsite` and `PlaceOffsiteRestore` too.
The refusal that protects a drive app from being restored onto the wrong disk (R-253, R-351) applies
to apps that **have a drive to get wrong**. It used to be reached by testing `HDD_PATH == ""`, which
also answered "is this app installed?" — one predicate for two questions. Measured in the catalogue at
`459766cb1639`: **53 templates, 13 declare `needs_hdd: true`, 40 declare `false`**, so for 40 apps that
test was permanently true and the off-site restore refused them forever, while they were running,
with a message telling the customer to reinstall them "in the same place" — a place those apps never
offer. **An app with no drive is not misconfigured** (§8 of `01-topology-and-trust.md` carries the
`[DESIGN]` marker); it is the majority case.
Since v0.219.0 the two questions are asked separately: *installed?* of `ListDeployedStacks()`, failing
CLOSED when there is no provider to ask; *where?* of `GetAppDrivePath`. A third refusal, with its own
sentence, covers installed-but-no-resolvable-data-root. **R-356**; reasoning also recorded in
`felhom-controller/CONTEXT.md`.
---
## 7. The recovery chain (D3) — the reason this document exists
@@ -525,10 +556,14 @@ anywhere under the backup namespace. Full record: `audits/D5-drive-alone-restore
drive. In that failure the primary recovery unit is gone; the surviving mirror on the second drive
is `backups/secondary/<app>/recovery-unit/`, which **no code path reads** (§6.3). For the 45-or-43
class-B apps the restore is a guaranteed no-op in exactly its designed scenario. → **R-102**
- **Tier-3 vs guest loss.** Tier-3 holds the volume tars and the DB dump. Reconstitution requires
the app to be deployed and skips the unit; the tars are unpacked only by the Tier-1 path, which
requires the guest's secrets. So offsite alone cannot rebuild an app onto a fresh guest.
→ **R-107**
- **Tier-3 vs guest loss — HALF of this closed.** Tier-3 holds the volume tars and the DB dump. It
was true until controller v0.218.0 that the tars were unpacked only by the Tier-1 path; since
v0.218.0 the reconstitution replays them itself (`volReplay`) → **R-107 CLOSED 2026-08-22**. What
is **still** true, and is the part that was never R-107: reconstitution requires the app to be
**deployed** and skips the unit, so offsite alone cannot rebuild an app onto a fresh guest. That
requirement is a deliberate decision (R-253) — the restore does not choose a customer's drive for
them — and since **v0.219.0** it means only what it says: an app that is not deployed. It no longer
catches the 40 driveless apps → **R-356**.
### 7.3 ~~What D5 would change — and why it was blocked~~ — **D5 SHIPPED 2026-07-30 (v0.188.0)**
@@ -737,7 +772,7 @@ crosses the line — **R-158**.
| 3 | **An app's DB and named volumes are lost** | the guest, the unit | „Visszaállítás indítása" — Tier-1 unit restore (the **only** path that unpacks volume tars) | **customer** | **18.25 s** (path execution) · **27.6 s** (D5 drill, guest `app.yaml` absent) | 24 h | **PROVEN** (content recovery proven 2026-07-30) | CAMPAIGN-9 A2 proved the path executes; **D5 v0.188.0 closed the content gap** — after a restore with the guest's `app.yaml` moved aside, the app read the seeded row **over TCP with its own credential**, the pre-backup row returned and a post-backup row was gone (so the tar was really restored). No `.sql` dump in the unit ⇒ the DB came back from the volume tar |
| 3c | *same, with the GUEST GONE (secrets unavailable)* | the drive | Tier-1 unit restore — **the unit carries the portable secrets** | **customer** | **27.6 s** | 24 h | **PROVEN** | D5, §7.4. Before v0.188.0 this row was **NONE**: the data-key gate refused and the customer's own copy of their own data was not a recovery |
| 3b | *same, for a class-B app via Tier-2* | — | **no route** — Tier-2 never reads the unit mirror | — | | | **NONE** | §6.3; **R-102** |
| 4 | **Primary drive dies** | Tier-2 copy on the second drive; Tier-3 offsite; the guest | Tier-2 for **file legs** (7 or 9 of 53 apps); Tier-3 reconstitute for files + DB; **the volume tars in either copy are unreachable** | **customer** (both) | | 24 h | **PARTIAL** | §7.2; **R-102**, **R-107** |
| 4 | **Primary drive dies** | Tier-2 copy on the second drive; Tier-3 offsite; the guest | Tier-2 for **file legs** (7 or 9 of 53 apps); Tier-3 reconstitute for files + DB **+ the named-volume tars since controller v0.218.0** (`volReplay`); **the Tier-2 copy's volume tars remain unreachable** | **customer** (both) | | 24 h | **PARTIAL** | §7.2; **R-102** (open); **R-107 CLOSED 2026-08-22, v0.218.0** |
| 5 | **Secondary drive dies** | everything the customer uses | none needed — Tier-2 is a derived copy, rebuilt on the next run (`07` §8 migration rule: *"Migration = rebuild, not preserve"*) | automatic | | 24 h | **PROVEN** (by construction) | tier2 v2 layout marker + rebuild, `internal/backup/tier2.go:359-393` |
| 6 | **Guest lost or corrupted** | the host, both whole-guest tiers, the data drives (they are host binds) | `pct restore` from `local:` or `felhom-pbs:` | **operator** (SSH) | **84–112 s** local · **1101 s** PBS (both = restore-test into a scratch guest, boot + verify + teardown) | 24 h local · 7 d offsite | **PROVEN** | CAMPAIGN-2 T-P9; CAMPAIGN-8 Phase C (exact mount parity, `unprivileged: 1` preserved); LIVE restore-tests on both boxes this session |
| 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human |
@@ -899,7 +934,8 @@ does **not** hold as written. → **R-108**
| **R-104** | An interrupted offsite run leaves an exclusive restic lock the existing self-heal cannot reach, reported as *„ismeretlen okból"* | the offsite tier stays dead until a human unlocks. Was C9-F3 |
| **R-105** | Three hub-held DR records are empty on the whole live fleet: `hosts.dr_record_json`, `host_escrow.directive_json`, `dr_recipe.host_half.drives` | the Recipe (§4) is incomplete in exactly the fields host-loss recovery reads. Causes may differ per field |
| **R-106** | `dr_recipe.host_half.pbs.namespace` records `"root"` on every box | the recorded restore coordinate is wrong; real namespaces are per-customer |
| **R-107** | No offsite action unpacks the named-volume tars Tier-3 captures on every run | offsite alone cannot rebuild a named-volume app (§7.2) |
| **R-107** | ~~No offsite action unpacks the named-volume tars Tier-3 captures on every run~~ — **CLOSED, controller v0.218.0, 2026-08-22** (`volReplay`, proven live on `demo-hp`). True from the day Tier-3 shipped until 2026-08-21. | was: offsite alone cannot rebuild a named-volume app (§7.2). Now: the tars replay; what remains is that the app must be **deployed** (R-253), which is not R-107 |
| **R-356** | The off-site restore resolved its destination with the raw `HDD_PATH` and read an empty answer as "not installed" — **CLOSED, controller v0.219.0, 2026-08-22** | 40 of 53 apps were refused permanently while running (§6.3 `[DESIGN]`) |
| ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` |
| **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) |
| R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) |
@@ -0,0 +1,78 @@
# DRILL — R-356: the off-site restore for an app with no data drive (2026-08-22)
**Subject:** `demo-hp` (Tier 0, disposable), guest 9201, controller **v0.219.0**.
**Method:** endpoint-level. `claude-in-chrome` is not available on DooPlex, so every action was the
exact HTTP endpoint the UI's own form posts to — `POST /backup/offbox/run`,
`POST /backup/offbox/restore`, `POST /backup/offbox/reconstitute` — driven with the session cookie and
the session CSRF token scraped from the page. No server logic was skipped; only rendering.
## What this drill was for
Nobody had ever driven an off-site reconstitution to completion for an app whose snapshot contains
**only** the recovery unit. The R-356 refusal always fired first. So the downstream legs — placement
mapping, the stat pre-pass, `CheckPlacement`, the safety dump, the DB-service-only window, `volReplay`
— had never run in that shape. **This was a first-run measurement, not a confirmation.**
**Result: every downstream leg held.** No finding was raised against them.
## Phase 1 — the driveless class (`privatebin`, `needs_hdd: false`)
| step | what happened |
|---|---|
| plant | 2 pastes created through PrivateBin's **own JSON API** (`X-Requested-With: JSONHttpRequest`) + 4 files planted directly into the app's named volume, one with a Hungarian accented name |
| findability control | planted a control string, **found** it, removed it, **failed to find** it — `02` in the evidence dir |
| off-site | `POST /backup/offbox/run` → snapshot at 11:16:34Z, 2m20s, `status: ok`, store 44.5 → 47.0 MB |
| destroy | 11 files deleted outright (`rm -rf`, no trash) — `03-destruction.txt` names every one |
| prepare | `POST /backup/offbox/restore mode=full` → snapshot `306accff` into `/mnt/felhom-drives/hdd_1/backups/offsite-restore/privatebin` |
| **the scratch shape** | **unit only** — `.../mnt/sys_drive/felhom-data/backups/primary/privatebin/` with `volume-dumps/privatebin_privatebin_data.tar`. The whole dataset is that one tar. This is the shape that had never been driven through. |
| restore | `POST /backup/offbox/reconstitute` — **no refusal**. Volume replayed, app restarted, healthy. |
| verify | **15/15 files byte-for-byte identical**, `diff` of the two sha256 manifests is empty |
**The accented filename, as explicit hex bytes:**
`c3a17276c3ad7a74c5b172c5912d74c3bc6bc3b67266c3ba72c3b367c3a9702d723335362e747874`
(`árvíztűrő-tükörfúrógép-r356.txt`). Non-ASCII never crossed a shell chain: the planting script was
base64-encoded on DooPlex and decoded inside the guest.
**The customer-facing message, verbatim (160 bytes):**
> A(z) privatebin: 0 fájl és 1 adatkötet visszaállítva (mentés: 2026-08-22 13:14) — az alkalmazás
> újraindult. Ennek az alkalmazásnak nincs adatbázisa.
**Judged, not merely recorded.** It **names the volume** beside the file count, so it is not the R-354
shape wearing a new hat: a customer reading it is told what actually came back. There is no warning
sitting beside a success. The one honest criticism — the first number a customer reads is `0 fájl`,
and for all 40 driveless apps it will always be 0 — is a wording matter, filed as an observation and
**not acted on here**.
## Phase 2 — the drive class (`calibre-web`, `needs_hdd: true`, `HDD_PATH=/mnt/felhom-drives/hdd_1`)
The same walk, to prove the 13-class did not move. The scratch resolved under
`/mnt/felhom-drives/hdd_1/...` — the app's own drive, not the system data path — and the restore
placed there. See `11`–`14` in the evidence dir.
## Evidence
`evidence/` holds every artefact, **copied off `demo-hp` at the end of each phase, before any revert**.
Nothing was reverted or torn down during this drill, so no intermediate teardown could have taken it.
## Deliberately NOT addressed here
The message that calls an empty restore a success; the delete guard on verification copies; the
absence of any readability check on the off-site store (**R-359**). Separate rows, untouched.
### Phase 2 result
**3/3 files back byte for byte**, accented name included (`Örkény-egyperces-r356.txt`, hex
`c396726bc3a96e792d6567797065726365732d723335362e747874`). The restore placed them under
`/mnt/felhom-drives/hdd_1/userdata/media/books/` — the app's own drive.
`/mnt/sys_drive/felhom-data/userdata/media` **does not exist**, so there was no silent misplacement
onto the system disk.
Message, verbatim (161 bytes):
> A(z) calibre-web: 3 fájl és 1 adatkötet visszaállítva (mentés: 2026-08-22 13:20) — az alkalmazás
> újraindult. Ennek az alkalmazásnak nincs adatbázisa.
**No behaviour change for the 13-class was observed.** That is an absence claim, so it does not stand
on the live walk alone: the unit-test red-proof for it plants the system-data fallback into the drive
path and three tests convict it (`TestR356_ScenarioB_*`, `TestR356_ScenarioE_PlaceDriveApp*`).
@@ -0,0 +1,15 @@
fa28d421a28ed9b6f4b997d5119c2b8adbacff6e2dd119a179b71230f9664934 ./.htaccess
0e9aeb5dd4334bfaacfd51fd5fa5905ea6856b63c375a842bedc1620ce7c16ab ./38/8e/388e3bd405a76985.php
0c23c8531214fe20cbc1ed177da22f51d65570c5aaedb39e380aa8d4d3e62991 ./DRILL-2026-08-21/SENTINEL.txt
725763bbe679b22d5c231a14083ff13155f475c028c22ade4623955b50a2a84a ./DRILL-2026-08-21/binary-1mb.bin
a39ad6f623da67ac72f2e62a24245eef46c722c279ae89cd6b0c7c45afa6a4c4 ./DRILL-2026-08-21/nested/őszibarack.md
07e91a985809fc96752f97cddcf6523b211bc607c81a5808befabe35573926f1 ./DRILL-2026-08-21/plain.txt
0d6a22ec56acf61b8a80e73eb12cb223d607acaa3684acfcfa7c3e34ecdfabea ./DRILL-2026-08-21/árvíztűrő-tükörfúrógép.txt
eb8c7a45d319701dc42c13b3175babc17367ce0f6e945386b11d99a4f9ac1d54 ./DRILL-r356-2026-08-22/SENTINEL.txt
4bcf89782290f2057c85ab875b912be47d5d46732f01f09e35221a25401d74f7 ./DRILL-r356-2026-08-22/binary-1mb.bin
576ff7dcc11b2b980bff935b9c35132bca72fbd8155d8e7c1c716d77b2546740 ./DRILL-r356-2026-08-22/nested/plain.txt
2bfa318e42a4514ebd00da28fac8e1ca5f44a674dd72c30e087c168688503835 ./DRILL-r356-2026-08-22/árvíztűrő-tükörfúrógép-r356.txt
f450cdc502162f20c7444576dec29d22140d009845c43e824cda5c3d3b84e320 ./ff/a1/ffa12cc12cd4fa71.php
985ef4b2fca0b01bbbe5bcf25125293926e4d58053a2737ec9c470b0e74c397d ./purge_limiter.php
b5106bc17cb0ea56e7dcf69cc5480def0668670204c24aa8c7ee6db3f8e569eb ./salt.php
9c94379f2de428183db3364c23af55f1dba88c43af95f737ad5926be26e49713 ./traffic_limiter.php
@@ -0,0 +1,18 @@
Off-site runs during this drill, read from POST /backup/offbox/run + GET /backup/offbox/status.
(The earlier 02-offsite-run-after-plant.json captured empty — the status GET raced the phase change.
Recorded here from the same endpoint plus the controller log, rather than left as a 1-byte file.)
RUN 1 — after the privatebin plant
last_run : 2026-08-22T11:16:34Z
last_duration: 2m20s
last_error : "" status: ok active: false
repo_size : 44.5 MB -> 47.0 MB
snapshots : 27
RUN 2 — after the calibre-web plant
last_run : 2026-08-22T11:22:13Z
last_error : "" status: ok active: false
snapshots : 27
log excerpt : "backed up privatebin (/mnt/sys_drive/felhom-data/backups/primary/privatebin, 0 mandatory path(s))"
"backed up calibre-web (/mnt/felhom-drives/hdd_1/backups/primary/calibre-web, 1 mandatory path(s))"
^ the two classes, visible in one pair of lines: system data path vs the app's own drive.
@@ -0,0 +1,17 @@
== deleting:
/var/lib/docker/volumes/privatebin_privatebin_data/_data/38/8e/388e3bd405a76985.php
/var/lib/docker/volumes/privatebin_privatebin_data/_data/DRILL-2026-08-21/SENTINEL.txt
/var/lib/docker/volumes/privatebin_privatebin_data/_data/DRILL-2026-08-21/binary-1mb.bin
/var/lib/docker/volumes/privatebin_privatebin_data/_data/DRILL-2026-08-21/nested/őszibarack.md
/var/lib/docker/volumes/privatebin_privatebin_data/_data/DRILL-2026-08-21/plain.txt
/var/lib/docker/volumes/privatebin_privatebin_data/_data/DRILL-2026-08-21/árvíztűrő-tükörfúrógép.txt
/var/lib/docker/volumes/privatebin_privatebin_data/_data/DRILL-r356-2026-08-22/SENTINEL.txt
/var/lib/docker/volumes/privatebin_privatebin_data/_data/DRILL-r356-2026-08-22/binary-1mb.bin
/var/lib/docker/volumes/privatebin_privatebin_data/_data/DRILL-r356-2026-08-22/nested/plain.txt
/var/lib/docker/volumes/privatebin_privatebin_data/_data/DRILL-r356-2026-08-22/árvíztűrő-tükörfúrógép-r356.txt
/var/lib/docker/volumes/privatebin_privatebin_data/_data/ff/a1/ffa12cc12cd4fa71.php
== remaining files:
/var/lib/docker/volumes/privatebin_privatebin_data/_data/.htaccess
/var/lib/docker/volumes/privatebin_privatebin_data/_data/purge_limiter.php
/var/lib/docker/volumes/privatebin_privatebin_data/_data/salt.php
/var/lib/docker/volumes/privatebin_privatebin_data/_data/traffic_limiter.php
@@ -0,0 +1,3 @@
csrf_len=64
HTTP/2 302
location: /backups/restore/app?name=privatebin&flash=A+t%C3%A1voli+vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
@@ -0,0 +1,5 @@
/mnt/felhom-drives/hdd_1/backups/offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/compose/.felhom.yml
/mnt/felhom-drives/hdd_1/backups/offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/compose/app.yaml
/mnt/felhom-drives/hdd_1/backups/offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/compose/docker-compose.yml
/mnt/felhom-drives/hdd_1/backups/offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/manifest.json
/mnt/felhom-drives/hdd_1/backups/offsite-restore/privatebin/mnt/sys_drive/felhom-data/backups/primary/privatebin/volume-dumps/privatebin_privatebin_data.tar
@@ -0,0 +1,3 @@
csrf_len=64
HTTP/2 302
location: /backups/restore/app?name=privatebin&flash=A+teljes+vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
@@ -0,0 +1,30 @@
2026/08/22 11:14:26 backup.go:750: [INFO] [backup] Volume dump: opengist/opengist_opengist_data → 177.0 KB
2026/08/22 11:14:26 backup.go:859: [INFO] [backup] Restarting opengist after volume dump
2026/08/22 11:14:27 backup.go:849: [INFO] [backup] Stopping paperless-ngx for safe volume dump
2026/08/22 11:14:34 backup.go:725: [DEBUG] [backup] Dumping volume paperless-ngx_paperless_postgres_data for paperless-ngx
2026/08/22 11:14:35 backup.go:750: [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 68.9 MB
2026/08/22 11:14:35 backup.go:725: [DEBUG] [backup] Dumping volume paperless-ngx_paperless_redis_data for paperless-ngx
2026/08/22 11:14:35 backup.go:750: [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_redis_data → 383.0 KB
2026/08/22 11:14:35 backup.go:725: [DEBUG] [backup] Dumping volume paperless-ngx_paperless_data for paperless-ngx
2026/08/22 11:14:36 backup.go:750: [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 488.5 KB
2026/08/22 11:14:36 backup.go:859: [INFO] [backup] Restarting paperless-ngx after volume dump
2026/08/22 11:14:47 backup.go:849: [INFO] [backup] Stopping privatebin for safe volume dump
2026/08/22 11:14:47 backup.go:725: [DEBUG] [backup] Dumping volume privatebin_privatebin_data for privatebin
2026/08/22 11:14:47 backup.go:750: [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 2.0 MB
2026/08/22 11:14:47 backup.go:859: [INFO] [backup] Restarting privatebin after volume dump
2026/08/22 11:14:48 backup.go:849: [INFO] [backup] Stopping romm for safe volume dump
2026/08/22 11:14:52 backup.go:725: [DEBUG] [backup] Dumping volume romm_romm_config for romm
2026/08/22 11:14:52 backup.go:750: [INFO] [backup] Volume dump: romm/romm_romm_config → 2.5 KB
2026/08/22 11:14:52 backup.go:725: [DEBUG] [backup] Dumping volume romm_romm_db_data for romm
2026/08/22 11:14:53 backup.go:750: [INFO] [backup] Volume dump: romm/romm_romm_db_data → 152.8 MB
2026/08/22 11:14:53 backup.go:725: [DEBUG] [backup] Dumping volume romm_romm_redis_data for romm
2026/08/22 11:14:53 backup.go:750: [INFO] [backup] Volume dump: romm/romm_romm_redis_data → 14.5 MB
2026/08/22 11:14:53 backup.go:859: [INFO] [backup] Restarting romm after volume dump
2026/08/22 11:15:05 backup.go:584: [INFO] [backup] App-data backup completed: 3 databases (422.0 KB total), 6 volume dump(s) (55.518s)
2026/08/22 11:18:22 auth.go:134: [DEBUG] [web] auth: valid session for POST /backup/offbox/reconstitute
2026/08/22 11:18:22 server.go:393: [DEBUG] [web] ServeHTTP: POST /backup/offbox/reconstitute from 172.18.0.4:60064
2026/08/22 11:18:25 restore.go:140: [INFO] [backup] Restoring Docker volume privatebin_privatebin_data for privatebin
2026/08/22 11:18:26 restore.go:169: [DEBUG] [backup] Volume privatebin_privatebin_data restored successfully
2026/08/22 11:18:26 restore.go:174: [INFO] [backup] Restored 1 Docker volume(s) for privatebin
2026/08/22 11:18:34 offbox_reconstitute.go:452: [INFO] [offbox] reconstituted privatebin from snapshot 306accff: 0 file(s) placed, 0 DB dump(s) replayed, safety dump=., skewed=false
2026/08/22 11:18:34 offbox_handlers.go:455: [INFO] [web] off-box reconstitute privatebin completed (async): files=0 dbs=0 snapshot=306accff
@@ -0,0 +1,15 @@
fa28d421a28ed9b6f4b997d5119c2b8adbacff6e2dd119a179b71230f9664934 ./.htaccess
0e9aeb5dd4334bfaacfd51fd5fa5905ea6856b63c375a842bedc1620ce7c16ab ./38/8e/388e3bd405a76985.php
0c23c8531214fe20cbc1ed177da22f51d65570c5aaedb39e380aa8d4d3e62991 ./DRILL-2026-08-21/SENTINEL.txt
725763bbe679b22d5c231a14083ff13155f475c028c22ade4623955b50a2a84a ./DRILL-2026-08-21/binary-1mb.bin
a39ad6f623da67ac72f2e62a24245eef46c722c279ae89cd6b0c7c45afa6a4c4 ./DRILL-2026-08-21/nested/őszibarack.md
07e91a985809fc96752f97cddcf6523b211bc607c81a5808befabe35573926f1 ./DRILL-2026-08-21/plain.txt
0d6a22ec56acf61b8a80e73eb12cb223d607acaa3684acfcfa7c3e34ecdfabea ./DRILL-2026-08-21/árvíztűrő-tükörfúrógép.txt
eb8c7a45d319701dc42c13b3175babc17367ce0f6e945386b11d99a4f9ac1d54 ./DRILL-r356-2026-08-22/SENTINEL.txt
4bcf89782290f2057c85ab875b912be47d5d46732f01f09e35221a25401d74f7 ./DRILL-r356-2026-08-22/binary-1mb.bin
576ff7dcc11b2b980bff935b9c35132bca72fbd8155d8e7c1c716d77b2546740 ./DRILL-r356-2026-08-22/nested/plain.txt
2bfa318e42a4514ebd00da28fac8e1ca5f44a674dd72c30e087c168688503835 ./DRILL-r356-2026-08-22/árvíztűrő-tükörfúrógép-r356.txt
f450cdc502162f20c7444576dec29d22140d009845c43e824cda5c3d3b84e320 ./ff/a1/ffa12cc12cd4fa71.php
985ef4b2fca0b01bbbe5bcf25125293926e4d58053a2737ec9c470b0e74c397d ./purge_limiter.php
b5106bc17cb0ea56e7dcf69cc5480def0668670204c24aa8c7ee6db3f8e569eb ./salt.php
9c94379f2de428183db3364c23af55f1dba88c43af95f737ad5926be26e49713 ./traffic_limiter.php
@@ -0,0 +1,7 @@
=== CUSTOMER-FACING OUTCOME MESSAGE, VERBATIM ===
A(z) privatebin: 0 fájl és 1 adatkötet visszaállítva (mentés: 2026-08-22 13:14) — az alkalmazás újraindult. Ennek az alkalmazásnak nincs adatbázisa.
=== the same string as UTF-8 hex bytes ===
41287a29207072697661746562696e3a20302066c3a16a6c20c3a973203120616461746bc3b674657420766973737a61c3a16c6cc3ad74766120286d656e74c3a9733a20323032362d30382d32322031333a31342920e2809420617a20616c6b616c6d617ac3a17320c3ba6a7261696e64756c742e20456e6e656b20617a20616c6b616c6d617ac3a1736e616b206e696e6373206164617462c3a17a6973612e
=== byte length: 160
@@ -0,0 +1,400 @@
2026/08/22 11:17:27 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:17:27 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:17:27 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:17:27 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:17:27 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:17:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:17:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:17:32 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:17:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 3m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 2m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 2m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:32 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:17:38 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:17:38 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:17:38 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:17:38Z, active_sessions=24
2026/08/22 11:17:38 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:17:38 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:17:38 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:17:38 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:17:38 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:17:38 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:17:38 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:17:38 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:17:38 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:17:38 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:17:38Z, active_sessions=25
2026/08/22 11:17:38 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:17:38 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:17:38 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:17:38 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:17:38 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:17:38 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:17:38 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:17:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:17:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 3m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 2m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 3m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:42 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:17:42 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/08/22 11:17:42 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/08/22 11:17:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:17:42 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:17:43 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (602ms)
2026/08/22 11:17:44 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (623ms)
2026/08/22 11:17:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:17:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 3m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 2m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 3m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:17:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:17:52 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:17:52 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:17:54 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:17:54 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:17:54 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:17:54Z, active_sessions=26
2026/08/22 11:17:54 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:17:54 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:17:54 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:17:54 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:17:54 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:17:54 auth.go:134: [DEBUG] [web] auth: valid session for POST /backup/offbox/restore
2026/08/22 11:17:54 server.go:393: [DEBUG] [web] ServeHTTP: POST /backup/offbox/restore from 172.18.0.4:60064
2026/08/22 11:17:59 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:17:59 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:17:59 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:17:59Z, active_sessions=27
2026/08/22 11:17:59 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:17:59 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:17:59 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:17:59 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:17:59 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:18:00 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:18:00 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:18:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:18:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:18:02 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:18:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 3m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 3m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 3m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:02 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:18:03 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:18:03 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:18:03 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:18:03Z, active_sessions=28
2026/08/22 11:18:03 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:18:03 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:18:03 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:18:03 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:18:03 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:18:03 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/restore/status
2026/08/22 11:18:03 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/restore/status from 172.18.0.4:60064
2026/08/22 11:18:03 server.go:636: [WARN] [web] 404 Not Found: GET /backup/restore/status
2026/08/22 11:18:05 offbox_restore.go:270: [INFO] [offbox] restored privatebin (306accff, full=true) → /mnt/felhom-drives/hdd_1/backups/offsite-restore/privatebin
2026/08/22 11:18:05 offbox_handlers.go:372: [INFO] [web] off-box restore privatebin completed (full=true, async)
2026/08/22 11:18:11 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/hdd_1", hasCPUCollector=true)
2026/08/22 11:18:11 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈3517MB avail≈22381MB
2026/08/22 11:18:11 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=26542728 → total=25898MB avail=22381MB used=3517MB (13.6%)
2026/08/22 11:18:11 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.7GB used=8.0GB avail=57.2GB (11.6%)
2026/08/22 11:18:11 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/hdd_1" bsize=4096 total=937.8GB used=5.5GB avail=884.7GB (0.6%)
2026/08/22 11:18:11 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.08 1.32 0.93 3/765 2831" → 1m=1.08 5m=1.32 15m=0.93
2026/08/22 11:18:11 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
2026/08/22 11:18:11 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 52.9°C (hwmon1)
2026/08/22 11:18:11 info.go:13: [DEBUG] [system] GetInfo done in 59ms — mem=3517MB/25898MB (13.6%), rootDisk=8.0GB/68.7GB (11.6%), load=1.08/1.32/0.93, temp=52.9°C (hwmon1), cpu=10.2%
2026/08/22 11:18:12 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan
2026/08/22 11:18:12 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting
2026/08/22 11:18:12 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/08/22 11:18:12 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/08/22 11:18:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:18:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=true composePath=/opt/docker/stacks/calibre-web/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "cloudflared" deployed=false composePath=/opt/docker/stacks/cloudflared/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=true composePath=/opt/docker/stacks/kimai/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml
2026/08/22 11:18:12 scheduler.go:346: [INFO] [scheduler] Running job: agent-channel-health
2026/08/22 11:18:12 scheduler.go:67: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=false composePath=/opt/docker/stacks/navidrome/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=false composePath=/opt/docker/stacks/nextcloud/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=true composePath=/opt/docker/stacks/opengist/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=false composePath=/opt/docker/stacks/vikunja/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml
2026/08/22 11:18:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml
2026/08/22 11:18:12 manager.go:1497: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates
2026/08/22 11:18:12 manager.go:539: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection
2026/08/22 11:18:12 manager.go:570: [INFO] [stacks] ScanStacks complete: 56 stacks found (6 deployed, 50 available)
2026/08/22 11:18:12 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:18:12 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 47ms)
2026/08/22 11:18:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 3m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 3m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 3m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:12 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:18:12 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:18:12 client.go:1142: [DEBUG] [agentapi] GET /storage -> 200 (577ms)
2026/08/22 11:18:12 scheduler.go:363: [INFO] [scheduler] Job agent-channel-health completed (took 577ms)
2026/08/22 11:18:13 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (607ms)
2026/08/22 11:18:14 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (676ms)
2026/08/22 11:18:21 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:18:21 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:18:21 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:18:21Z, active_sessions=29
2026/08/22 11:18:21 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:18:21 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:18:21 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:18:21 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:18:21 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:18:21 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:18:21 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:18:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:18:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:18:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 3m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 3m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 3m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:22 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:18:22 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:18:22 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:18:22 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:18:22 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:18:22Z, active_sessions=30
2026/08/22 11:18:22 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:18:22 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:18:22 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:18:22 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:18:22 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:18:22 auth.go:134: [DEBUG] [web] auth: valid session for POST /backup/offbox/reconstitute
2026/08/22 11:18:22 server.go:393: [DEBUG] [web] ServeHTTP: POST /backup/offbox/reconstitute from 172.18.0.4:60064
2026/08/22 11:18:25 dbdump.go:92: [DEBUG] DiscoverDatabases: running docker ps to find database containers
2026/08/22 11:18:25 dbdump.go:111: [DEBUG] DiscoverDatabases: docker ps output: 4c72492af5ca romm romm rommapp/romm:5.0.0
b261d717618d romm-db romm mariadb:11.4
9010a9d7927f romm-redis romm redis:7-alpine
83d6e8adc484 privatebin privatebin privatebin/pdo:2.0.5
80fc6005741e paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
ae015380d719 paperless-postgres paperless-ngx postgres:16-alpine
fddae44de81e paperless-redis paperless-ngx redis:7-alpine
3aae6ced0216 opengist opengist ghcr.io/thomiceli/opengist:1.13
d0c84add6adf kimai kimai kimai/kimai2:apac...
2026/08/22 11:18:25 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.0.0, not a database)
2026/08/22 11:18:25 dbdump.go:140: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=b261d717618d)
2026/08/22 11:18:25 dbdump.go:160: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm
2026/08/22 11:18:25 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database)
2026/08/22 11:18:25 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.5, not a database)
2026/08/22 11:18:25 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
2026/08/22 11:18:25 dbdump.go:140: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=ae015380d719)
2026/08/22 11:18:25 dbdump.go:805: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
2026/08/22 11:18:25 dbdump.go:160: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
2026/08/22 11:18:25 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
2026/08/22 11:18:25 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container opengist (image=ghcr.io/thomiceli/opengist:1.13, not a database)
2026/08/22 11:18:25 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container kimai (image=kimai/kimai2:apache-2.57.0, not a database)
2026/08/22 11:18:25 dbdump.go:140: [DEBUG] DiscoverDatabases: found mariadb container: kimai-db (id=fb9ebd95951c)
2026/08/22 11:18:25 dbdump.go:160: [DEBUG] DiscoverDatabases: kimai-db → stack=kimai, dbUser=root, dbName=kimai
2026/08/22 11:18:25 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container calibre-web (image=crocodilestick/calibre-web-automated:v4.0.6, not a database)
2026/08/22 11:18:25 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.219.0, not a database)
2026/08/22 11:18:25 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database)
2026/08/22 11:18:25 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container cloudflared (image=cloudflare/cloudflared:2026.6.0, not a database)
2026/08/22 11:18:25 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database)
2026/08/22 11:18:25 dbdump.go:167: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 12 non-DB container(s)
2026/08/22 11:18:25 dbdump.go:170: [INFO] [backup] Discovered 3 databases
2026/08/22 11:18:25 manager.go:1067: [DEBUG] [stacks] StopStack privatebin: current state=running deployed=true containers=1
2026/08/22 11:18:25 manager.go:1070: [INFO] [stacks] Stopping stack: privatebin
2026/08/22 11:18:25 manager.go:1261: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/08/22 11:18:25 manager.go:1270: [DEBUG] Running: docker compose down (in /opt/docker/stacks/privatebin)
2026/08/22 11:18:25 manager.go:1290: [DEBUG] Command completed: docker compose down (took 0.3s)
2026/08/22 11:18:25 manager.go:1079: [INFO] [stacks] Stack privatebin stopped successfully (took 0.3s)
2026/08/22 11:18:25 manager.go:621: [INFO] [stacks] Status refresh: 13 containers across 56 stacks
2026/08/22 11:18:25 restore.go:140: [INFO] [backup] Restoring Docker volume privatebin_privatebin_data for privatebin
2026/08/22 11:18:26 restore.go:169: [DEBUG] [backup] Volume privatebin_privatebin_data restored successfully
2026/08/22 11:18:26 restore.go:174: [INFO] [backup] Restored 1 Docker volume(s) for privatebin
2026/08/22 11:18:26 manager.go:986: [DEBUG] [stacks] StartStack privatebin: current state=stopped deployed=true
2026/08/22 11:18:26 manager.go:989: [INFO] [stacks] Starting stack: privatebin
2026/08/22 11:18:26 manager.go:996: [DEBUG] [stacks] StartStack privatebin: prepared 8 env vars for compose
2026/08/22 11:18:26 manager.go:1261: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/08/22 11:18:26 manager.go:1270: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/privatebin)
2026/08/22 11:18:26 manager.go:1290: [DEBUG] Command completed: docker compose up -d (took 0.3s)
2026/08/22 11:18:26 manager.go:1004: [INFO] [stacks] Stack privatebin started successfully (took 0.3s)
2026/08/22 11:18:26 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:18:29 manager.go:1261: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
2026/08/22 11:18:29 manager.go:1270: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/privatebin)
2026/08/22 11:18:29 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:18:29 restore.go:204: [DEBUG] [backup] Post-restore health check: privatebin not yet running, waiting...
2026/08/22 11:18:29 manager.go:1290: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s)
2026/08/22 11:18:29 manager.go:1350: [INFO] [stacks] Stack privatebin post-start status:
2026/08/22 11:18:29 manager.go:1353: [INFO] [stacks] privatebin privatebin/pdo:2.0.5 running Up 3 seconds (health: starting)
2026/08/22 11:18:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:18:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 3m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 3m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 4m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:32 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (4 skipped not due, 1 skipped no container)
2026/08/22 11:18:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:18:32 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:18:34 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:18:34 restore.go:199: [DEBUG] [backup] Post-restore health check: privatebin is running
2026/08/22 11:18:34 offbox_reconstitute.go:452: [INFO] [offbox] reconstituted privatebin from snapshot 306accff: 0 file(s) placed, 0 DB dump(s) replayed, safety dump=., skewed=false
2026/08/22 11:18:34 offbox_handlers.go:455: [INFO] [web] off-box reconstitute privatebin completed (async): files=0 dbs=0 snapshot=306accff
2026/08/22 11:18:42 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/08/22 11:18:42 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/08/22 11:18:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:18:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 4m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 4m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 3m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:42 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (4 skipped not due, 1 skipped no container)
2026/08/22 11:18:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:18:42 healthprobe.go:153: [DEBUG] Health probe privatebin: HTTP GET :8080/ → 200 (5ms)
2026/08/22 11:18:42 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:18:42 healthprobe.go:133: [INFO] Health probes: 1 ok (of 1 probed)
2026/08/22 11:18:42 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:18:42 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:18:42 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:18:42Z, active_sessions=31
2026/08/22 11:18:42 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:18:42 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:18:42 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:18:42 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:18:42 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:18:42 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:18:42 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:18:43 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (589ms)
2026/08/22 11:18:44 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (601ms)
2026/08/22 11:18:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:18:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:18:52 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:18:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 4m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 3m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 4m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:18:52 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:19:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:19:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 4m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 4m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 4m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:02 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:19:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:19:02 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:19:04 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:19:04 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:19:04 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:19:04Z, active_sessions=32
2026/08/22 11:19:04 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:19:04 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:19:04 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:19:04 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:19:04 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:19:04 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:19:04 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:19:06 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:19:06 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:19:06 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:19:06Z, active_sessions=33
2026/08/22 11:19:06 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:19:06 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:19:06 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:19:06 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:19:06 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:19:06 auth.go:134: [DEBUG] [web] auth: valid session for GET /backups/restore/app
2026/08/22 11:19:06 server.go:393: [DEBUG] [web] ServeHTTP: GET /backups/restore/app from 172.18.0.4:60064
2026/08/22 11:19:11 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/hdd_1", hasCPUCollector=true)
2026/08/22 11:19:11 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈3600MB avail≈22298MB
2026/08/22 11:19:11 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=26444532 → total=25898MB avail=22298MB used=3600MB (13.9%)
2026/08/22 11:19:11 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.7GB used=8.0GB avail=57.2GB (11.6%)
2026/08/22 11:19:11 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/hdd_1" bsize=4096 total=937.8GB used=5.5GB avail=884.7GB (0.6%)
2026/08/22 11:19:11 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.95 1.25 0.92 6/825 3250" → 1m=0.95 5m=1.25 15m=0.92
2026/08/22 11:19:11 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
2026/08/22 11:19:11 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 57.8°C (hwmon2)
2026/08/22 11:19:11 info.go:13: [DEBUG] [system] GetInfo done in 59ms — mem=3600MB/25898MB (13.9%), rootDisk=8.0GB/68.7GB (11.6%), load=0.95/1.25/0.92, temp=57.8°C (hwmon2), cpu=8.4%
2026/08/22 11:19:12 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/08/22 11:19:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:19:12 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/08/22 11:19:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:19:12 scheduler.go:346: [INFO] [scheduler] Running job: agent-channel-health
2026/08/22 11:19:12 scheduler.go:67: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026/08/22 11:19:12 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:19:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 4m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 4m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 4m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:12 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:19:12 client.go:1142: [DEBUG] [agentapi] GET /storage -> 200 (561ms)
2026/08/22 11:19:12 scheduler.go:363: [INFO] [scheduler] Job agent-channel-health completed (took 561ms)
2026/08/22 11:19:13 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (660ms)
2026/08/22 11:19:14 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (674ms)
2026/08/22 11:19:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:19:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 4m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 4m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 4m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:22 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:19:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:19:22 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:19:25 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:19:25 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:19:25 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:19:25Z, active_sessions=34
2026/08/22 11:19:25 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:19:25 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:19:25 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:19:25 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:19:25 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:19:25 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:19:25 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:19:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:19:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 5m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 4m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 4m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:19:32 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:19:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:19:32 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
@@ -0,0 +1,3 @@
bc6d35efa862246b543a1e66b2a2ab4fa889bbe28a827cc30018593ca026561a /mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-r356-13class-2026-08-22/book.bin
93defa514998c95ec4e1483b7b4935166ab8f9c6d850b464c39114118360f82d /mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-r356-13class-2026-08-22/Örkény-egyperces-r356.txt
85c29e22129c14d03815e107fc1fceab004bf682dc030d927f24fdd68fc6ceea /mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-r356-13class-2026-08-22/SENTINEL.txt
@@ -0,0 +1,13 @@
/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-r356-13class-2026-08-22/SENTINEL.txt
/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-r356-13class-2026-08-22/book.bin
/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-r356-13class-2026-08-22/Örkény-egyperces-r356.txt
-- after:
total 456
drwxrwsr-x 4 1000 1000 4096 Aug 22 11:22 .
drwxrwsr-x 10 root 1000 4096 Jul 26 06:17 ..
drwxrwxr-x 3 1000 1000 4096 Aug 21 20:09 DRILL-2026-08-21
-rw-r--r-- 1 1000 1000 181 Aug 4 12:53 DRILL-SENTINEL.txt
-rw-r--r-- 1 1000 1000 413696 Aug 4 12:52 metadata.db
-rw-r--r-- 1 1000 1000 32768 Aug 22 11:20 metadata.db-shm
-rw-r--r-- 1 1000 1000 0 Aug 9 02:15 metadata.db-wal
drwxrwxr-x 3 1000 1000 4096 Aug 9 07:59 rehearsal-2026-08-09
@@ -0,0 +1,20 @@
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/compose/.felhom.yml
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/compose/app.yaml
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/compose/docker-compose.yml
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/manifest.json
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/backups/primary/calibre-web/volume-dumps/calibre-web_calibre_web_config.tar
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/POST-SNAPSHOT.txt
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/SENTINEL.txt
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/binary-1mb.bin
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/nested/őszibarack.md
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/plain.txt
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-2026-08-21/árvíztűrő-tükörfúrógép.txt
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-SENTINEL.txt
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-r356-13class-2026-08-22/SENTINEL.txt
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-r356-13class-2026-08-22/book.bin
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/DRILL-r356-13class-2026-08-22/Örkény-egyperces-r356.txt
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/metadata.db
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/metadata.db-shm
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/metadata.db-wal
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/binary-3mb.bin
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web/mnt/felhom-drives/hdd_1/userdata/media/books/rehearsal-2026-08-09/nested/őszibarack.md
@@ -0,0 +1,3 @@
85c29e22129c14d03815e107fc1fceab004bf682dc030d927f24fdd68fc6ceea ./SENTINEL.txt
bc6d35efa862246b543a1e66b2a2ab4fa889bbe28a827cc30018593ca026561a ./book.bin
93defa514998c95ec4e1483b7b4935166ab8f9c6d850b464c39114118360f82d ./Örkény-egyperces-r356.txt
@@ -0,0 +1,7 @@
=== 13-CLASS OUTCOME MESSAGE, VERBATIM ===
A(z) calibre-web: 3 fájl és 1 adatkötet visszaállítva (mentés: 2026-08-22 13:20) — az alkalmazás újraindult. Ennek az alkalmazásnak nincs adatbázisa.
=== UTF-8 hex ===
41287a292063616c696272652d7765623a20332066c3a16a6c20c3a973203120616461746bc3b674657420766973737a61c3a16c6cc3ad74766120286d656e74c3a9733a20323032362d30382d32322031333a32302920e2809420617a20616c6b616c6d617ac3a17320c3ba6a7261696e64756c742e20456e6e656b20617a20616c6b616c6d617ac3a1736e616b206e696e6373206164617462c3a17a6973612e
=== byte length: 161
@@ -0,0 +1,500 @@
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=true composePath=/opt/docker/stacks/kimai/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml
2026/08/22 11:22:12 scheduler.go:346: [INFO] [scheduler] Running job: agent-channel-health
2026/08/22 11:22:12 scheduler.go:67: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=false composePath=/opt/docker/stacks/navidrome/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=false composePath=/opt/docker/stacks/nextcloud/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=true composePath=/opt/docker/stacks/opengist/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=false composePath=/opt/docker/stacks/vikunja/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml
2026/08/22 11:22:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml
2026/08/22 11:22:12 manager.go:1497: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates
2026/08/22 11:22:12 manager.go:539: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection
2026/08/22 11:22:12 manager.go:570: [INFO] [stacks] ScanStacks complete: 56 stacks found (6 deployed, 50 available)
2026/08/22 11:22:12 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:22:12 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 44ms)
2026/08/22 11:22:12 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:22:12 client.go:1142: [DEBUG] [agentapi] GET /storage -> 200 (557ms)
2026/08/22 11:22:12 scheduler.go:363: [INFO] [scheduler] Job agent-channel-health completed (took 557ms)
2026/08/22 11:22:13 settings.go:697: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (5736 bytes)
2026/08/22 11:22:13 settings.go:700: [INFO] [settings] Settings saved
2026/08/22 11:22:13 settings.go:697: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (5730 bytes)
2026/08/22 11:22:13 settings.go:700: [INFO] [settings] Settings saved
2026/08/22 11:22:13 offbox.go:1129: [INFO] [offbox] backup OK: 6 app(s) backed up, 27 snapshot(s), 2m8s
2026/08/22 11:22:13 offbox_progress.go:342: [INFO] [offbox] manual run progress reporting ended after 2m14s
2026/08/22 11:22:13 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (679ms)
2026/08/22 11:22:14 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:36778 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:22:14 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:22:14 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:22:14 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:22:14Z, active_sessions=54
2026/08/22 11:22:14 auth.go:214: [INFO] [web] Login from 172.18.0.4:36778
2026/08/22 11:22:14 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:22:14 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:36778
2026/08/22 11:22:14 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:22:14 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:36778
2026/08/22 11:22:14 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:22:14 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:22:14Z, active_sessions=55
2026/08/22 11:22:14 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:22:14 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:22:14 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:22:14 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:22:14 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:22:14 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:22:14 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:22:14 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:22:14 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:22:14 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (681ms)
2026/08/22 11:22:14 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:22:15 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:22:15 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:22:15Z, active_sessions=56
2026/08/22 11:22:15 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:22:15 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:22:15 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:22:15 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:22:15 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:22:15 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:22:15 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:22:15 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:22:15 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:22:15 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:22:15Z, active_sessions=57
2026/08/22 11:22:15 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:22:15 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:22:15 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:22:15 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:22:15 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:22:15 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:22:15 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:22:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:22:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:22:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:22 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:22:22 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:22:28 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:22:28 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:22:28 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:22:28Z, active_sessions=58
2026/08/22 11:22:28 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:22:28 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:22:28 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:22:28 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:22:28 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:22:28 auth.go:134: [DEBUG] [web] auth: valid session for POST /backup/offbox/restore
2026/08/22 11:22:28 server.go:393: [DEBUG] [web] ServeHTTP: POST /backup/offbox/restore from 172.18.0.4:60064
2026/08/22 11:22:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:22:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:22:32 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:22:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 2m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 2m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:32 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:22:35 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:22:35 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:22:35 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:22:35Z, active_sessions=59
2026/08/22 11:22:35 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:22:35 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:22:35 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:22:35 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:22:35 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:22:35 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:22:35 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:22:36 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:22:36 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:22:36 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:22:36Z, active_sessions=60
2026/08/22 11:22:36 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:22:36 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:22:36 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:22:36 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:22:36 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:22:36 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:22:36 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:22:39 offbox_restore.go:270: [INFO] [offbox] restored calibre-web (b921bae3, full=true) → /mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web
2026/08/22 11:22:39 offbox_handlers.go:372: [INFO] [web] off-box restore calibre-web completed (full=true, async)
2026/08/22 11:22:42 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/08/22 11:22:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:22:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:22:42 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/08/22 11:22:42 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:22:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 2m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 1m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 2m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:42 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:22:43 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (605ms)
2026/08/22 11:22:44 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (596ms)
2026/08/22 11:22:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:22:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:22:52 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:22:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 2m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 2m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:22:52 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:22:56 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:22:56 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:22:56 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:22:56Z, active_sessions=61
2026/08/22 11:22:56 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:22:56 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:22:56 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:22:56 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:22:56 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:22:56 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:22:56 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:22:57 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:22:57 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:22:57 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:22:57Z, active_sessions=62
2026/08/22 11:22:57 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:22:57 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:22:57 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:22:57 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:22:57 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:22:57 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:22:57 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:23:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:23:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 2m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 2m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 2m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:02 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:23:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:23:02 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:23:02 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:23:02 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:23:02 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:23:02Z, active_sessions=63
2026/08/22 11:23:02 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:23:02 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:23:02 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:23:02 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:23:02 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:23:02 auth.go:134: [DEBUG] [web] auth: valid session for POST /backup/offbox/reconstitute
2026/08/22 11:23:02 server.go:393: [DEBUG] [web] ServeHTTP: POST /backup/offbox/reconstitute from 172.18.0.4:60064
2026/08/22 11:23:04 dbdump.go:92: [DEBUG] DiscoverDatabases: running docker ps to find database containers
2026/08/22 11:23:04 dbdump.go:111: [DEBUG] DiscoverDatabases: docker ps output: 5afaefbd4e91 romm romm rommapp/romm:5.0.0
79cae59f7986 romm-db romm mariadb:11.4
c7c079ae23af romm-redis romm redis:7-alpine
adca811fb993 privatebin privatebin privatebin/pdo:2.0.5
9fe7dc9f36ae paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
4a432b653108 paperless-postgres paperless-ngx postgres:16-alpine
950113e71f40 paperless-redis paperless-ngx redis:7-alpine
4d81ef1c6cde opengist opengist ghcr.io/thomiceli/opengist:1.13
4df3efdb8366 kimai kimai kimai/kimai2:apac...
2026/08/22 11:23:04 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.0.0, not a database)
2026/08/22 11:23:04 dbdump.go:140: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=79cae59f7986)
2026/08/22 11:23:04 dbdump.go:160: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm
2026/08/22 11:23:04 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database)
2026/08/22 11:23:04 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.5, not a database)
2026/08/22 11:23:04 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
2026/08/22 11:23:04 dbdump.go:140: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=4a432b653108)
2026/08/22 11:23:04 dbdump.go:805: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
2026/08/22 11:23:05 dbdump.go:160: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
2026/08/22 11:23:05 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
2026/08/22 11:23:05 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container opengist (image=ghcr.io/thomiceli/opengist:1.13, not a database)
2026/08/22 11:23:05 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container kimai (image=kimai/kimai2:apache-2.57.0, not a database)
2026/08/22 11:23:05 dbdump.go:140: [DEBUG] DiscoverDatabases: found mariadb container: kimai-db (id=d3e87cbf6539)
2026/08/22 11:23:05 dbdump.go:160: [DEBUG] DiscoverDatabases: kimai-db → stack=kimai, dbUser=root, dbName=kimai
2026/08/22 11:23:05 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container calibre-web (image=crocodilestick/calibre-web-automated:v4.0.6, not a database)
2026/08/22 11:23:05 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.219.0, not a database)
2026/08/22 11:23:05 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database)
2026/08/22 11:23:05 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container cloudflared (image=cloudflare/cloudflared:2026.6.0, not a database)
2026/08/22 11:23:05 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database)
2026/08/22 11:23:05 dbdump.go:167: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 12 non-DB container(s)
2026/08/22 11:23:05 dbdump.go:170: [INFO] [backup] Discovered 3 databases
2026/08/22 11:23:05 manager.go:1067: [DEBUG] [stacks] StopStack calibre-web: current state=running deployed=true containers=1
2026/08/22 11:23:05 manager.go:1070: [INFO] [stacks] Stopping stack: calibre-web
2026/08/22 11:23:05 manager.go:1261: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, HDD_PATH, SUBDOMAIN, DOMAIN, USERDATA_PATH, IMPORT_PATH] (10 app + 0 system)
2026/08/22 11:23:05 manager.go:1270: [DEBUG] Running: docker compose down (in /opt/docker/stacks/calibre-web)
2026/08/22 11:23:09 manager.go:1290: [DEBUG] Command completed: docker compose down (took 4.0s)
2026/08/22 11:23:09 manager.go:1079: [INFO] [stacks] Stack calibre-web stopped successfully (took 4.0s)
2026/08/22 11:23:09 manager.go:621: [INFO] [stacks] Status refresh: 13 containers across 56 stacks
2026/08/22 11:23:09 restore.go:140: [INFO] [backup] Restoring Docker volume calibre-web_calibre_web_config for calibre-web
2026/08/22 11:23:09 restore.go:169: [DEBUG] [backup] Volume calibre-web_calibre_web_config restored successfully
2026/08/22 11:23:09 restore.go:174: [INFO] [backup] Restored 1 Docker volume(s) for calibre-web
2026/08/22 11:23:09 manager.go:986: [DEBUG] [stacks] StartStack calibre-web: current state=stopped deployed=true
2026/08/22 11:23:09 manager.go:989: [INFO] [stacks] Starting stack: calibre-web
2026/08/22 11:23:09 manager.go:996: [DEBUG] [stacks] StartStack calibre-web: prepared 10 env vars for compose
2026/08/22 11:23:09 manager.go:1261: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, HDD_PATH, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (10 app + 0 system)
2026/08/22 11:23:09 manager.go:1270: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/calibre-web)
2026/08/22 11:23:09 manager.go:1290: [DEBUG] Command completed: docker compose up -d (took 0.3s)
2026/08/22 11:23:09 manager.go:1004: [INFO] [stacks] Stack calibre-web started successfully (took 0.3s)
2026/08/22 11:23:09 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:23:11 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/hdd_1", hasCPUCollector=true)
2026/08/22 11:23:11 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈3326MB avail≈22572MB
2026/08/22 11:23:11 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=26768972 → total=25898MB avail=22572MB used=3326MB (12.8%)
2026/08/22 11:23:11 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.7GB used=8.0GB avail=57.3GB (11.6%)
2026/08/22 11:23:11 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/hdd_1" bsize=4096 total=937.8GB used=5.5GB avail=884.7GB (0.6%)
2026/08/22 11:23:11 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.11 1.44 1.09 3/776 5238" → 1m=1.11 5m=1.44 15m=1.09
2026/08/22 11:23:11 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
2026/08/22 11:23:11 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 55.9°C (hwmon2)
2026/08/22 11:23:11 info.go:13: [DEBUG] [system] GetInfo done in 62ms — mem=3326MB/25898MB (12.8%), rootDisk=8.0GB/68.7GB (11.6%), load=1.11/1.44/1.09, temp=55.9°C (hwmon2), cpu=5.8%
2026/08/22 11:23:12 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/08/22 11:23:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:23:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:12 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/08/22 11:23:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:23:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 2m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 2m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:12 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (4 skipped not due, 1 skipped no container)
2026/08/22 11:23:12 scheduler.go:346: [INFO] [scheduler] Running job: agent-channel-health
2026/08/22 11:23:12 scheduler.go:67: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026/08/22 11:23:12 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:23:12 client.go:1142: [DEBUG] [agentapi] GET /storage -> 200 (580ms)
2026/08/22 11:23:12 scheduler.go:363: [INFO] [scheduler] Job agent-channel-health completed (took 580ms)
2026/08/22 11:23:12 manager.go:1261: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, HDD_PATH, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (10 app + 0 system)
2026/08/22 11:23:12 manager.go:1270: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/calibre-web)
2026/08/22 11:23:12 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:23:12 restore.go:204: [DEBUG] [backup] Post-restore health check: calibre-web not yet running, waiting...
2026/08/22 11:23:12 manager.go:1290: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s)
2026/08/22 11:23:12 manager.go:1350: [INFO] [stacks] Stack calibre-web post-start status:
2026/08/22 11:23:12 manager.go:1353: [INFO] [stacks] calibre-web crocodilestick/calibre-web-automated:v4.0.6 running Up 3 seconds (health: starting)
2026/08/22 11:23:14 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (713ms)
2026/08/22 11:23:14 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (637ms)
2026/08/22 11:23:17 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:23:17 restore.go:204: [DEBUG] [backup] Post-restore health check: calibre-web not yet running, waiting...
2026/08/22 11:23:18 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:23:18 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:23:18 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:23:18Z, active_sessions=64
2026/08/22 11:23:18 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:23:18 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:23:18 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:23:18 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:23:18 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:23:18 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:23:18 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:23:18 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:23:18 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:23:18 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:23:18Z, active_sessions=65
2026/08/22 11:23:18 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:23:19 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:23:19 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:23:19 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:23:19 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:23:19 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:23:19 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:23:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:23:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:23:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 2m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 2m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:22 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (4 skipped not due, 1 skipped no container)
2026/08/22 11:23:22 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:23:22 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:23:22 restore.go:204: [DEBUG] [backup] Post-restore health check: calibre-web not yet running, waiting...
2026/08/22 11:23:27 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:23:27 restore.go:199: [DEBUG] [backup] Post-restore health check: calibre-web is running
2026/08/22 11:23:27 offbox_reconstitute.go:452: [INFO] [offbox] reconstituted calibre-web from snapshot b921bae3: 3 file(s) placed, 0 DB dump(s) replayed, safety dump=., skewed=false
2026/08/22 11:23:27 offbox_handlers.go:455: [INFO] [web] off-box reconstitute calibre-web completed (async): files=3 dbs=0 snapshot=b921bae3
2026/08/22 11:23:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:23:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:23:32 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:23:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 2m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 3m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:32 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (4 skipped not due, 1 skipped no container)
2026/08/22 11:23:32 healthprobe.go:153: [DEBUG] Health probe calibre-web: HTTP GET :8083/ → 302 (4ms)
2026/08/22 11:23:32 healthprobe.go:133: [INFO] Health probes: 1 ok (of 1 probed)
2026/08/22 11:23:39 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:23:39 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:23:39 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:23:39Z, active_sessions=66
2026/08/22 11:23:39 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:23:39 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:23:39 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:23:39 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:23:39 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:23:39 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:23:39 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:23:40 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:23:40 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:23:40 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:23:40Z, active_sessions=67
2026/08/22 11:23:40 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:23:40 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:23:40 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:23:40 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:23:40 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:23:40 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:23:40 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:23:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:23:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:23:42 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/08/22 11:23:42 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/08/22 11:23:42 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:23:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 3m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 2m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:42 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:23:43 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (622ms)
2026/08/22 11:23:44 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (609ms)
2026/08/22 11:23:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:23:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:23:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 2m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 3m20s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:23:52 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:23:52 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:24:00 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:24:00 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:24:00 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:24:00Z, active_sessions=68
2026/08/22 11:24:00 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:24:00 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:24:00 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:24:00 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:24:00 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:24:00 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:24:00 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:24:01 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:24:01 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:24:01 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:24:01Z, active_sessions=69
2026/08/22 11:24:01 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:24:01 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:24:01 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:24:01 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:24:01 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:24:01 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
2026/08/22 11:24:01 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:60064
2026/08/22 11:24:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:24:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:24:02 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:24:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 3m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:24:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 3m30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:24:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m0s ago, effective interval 5m0s, healthy=true
2026/08/22 11:24:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:24:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 30s ago, effective interval 5m0s, healthy=true
2026/08/22 11:24:02 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:24:09 auth.go:146: [DEBUG] [web] login attempt from 172.18.0.4:60064 (X-Forwarded-For: 172.18.0.1)
2026/08/22 11:24:09 auth.go:194: [DEBUG] [web] login successful from 172.18.0.1, creating session
2026/08/22 11:24:09 auth.go:260: [DEBUG] [web] session created, expires=2026-08-29T11:24:09Z, active_sessions=70
2026/08/22 11:24:09 auth.go:214: [INFO] [web] Login from 172.18.0.4:60064
2026/08/22 11:24:09 auth.go:134: [DEBUG] [web] auth: valid session for GET /
2026/08/22 11:24:09 server.go:393: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.4:60064
2026/08/22 11:24:09 auth.go:134: [DEBUG] [web] auth: valid session for GET /launcher
2026/08/22 11:24:09 server.go:393: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.4:60064
2026/08/22 11:24:09 auth.go:134: [DEBUG] [web] auth: valid session for GET /backups/restore/app
2026/08/22 11:24:09 server.go:393: [DEBUG] [web] ServeHTTP: GET /backups/restore/app from 172.18.0.4:60064
2026/08/22 11:24:11 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/hdd_1", hasCPUCollector=true)
2026/08/22 11:24:11 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈3555MB avail≈22343MB
2026/08/22 11:24:11 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=26497444 → total=25898MB avail=22343MB used=3555MB (13.7%)
2026/08/22 11:24:11 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.7GB used=8.0GB avail=57.2GB (11.6%)
2026/08/22 11:24:11 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/hdd_1" bsize=4096 total=937.8GB used=5.5GB avail=884.7GB (0.6%)
2026/08/22 11:24:11 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.99 1.36 1.08 3/810 5442" → 1m=0.99 5m=1.36 15m=1.08
2026/08/22 11:24:11 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
2026/08/22 11:24:11 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 53.9°C (hwmon2)
2026/08/22 11:24:11 info.go:13: [DEBUG] [system] GetInfo done in 61ms — mem=3555MB/25898MB (13.7%), rootDisk=8.0GB/68.7GB (11.6%), load=0.99/1.36/1.08, temp=53.9°C (hwmon2), cpu=1.4%
2026/08/22 11:24:12 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan
2026/08/22 11:24:12 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting
2026/08/22 11:24:12 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting
2026/08/22 11:24:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting
2026/08/22 11:24:12 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting
2026/08/22 11:24:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=true composePath=/opt/docker/stacks/calibre-web/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "cloudflared" deployed=false composePath=/opt/docker/stacks/cloudflared/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=true composePath=/opt/docker/stacks/kimai/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=false composePath=/opt/docker/stacks/navidrome/docker-compose.yml
2026/08/22 11:24:12 scheduler.go:346: [INFO] [scheduler] Running job: agent-channel-health
2026/08/22 11:24:12 scheduler.go:67: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=false composePath=/opt/docker/stacks/nextcloud/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=true composePath=/opt/docker/stacks/opengist/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=false composePath=/opt/docker/stacks/vikunja/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml
2026/08/22 11:24:12 manager.go:502: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml
2026/08/22 11:24:12 manager.go:1497: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates
2026/08/22 11:24:12 manager.go:539: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection
2026/08/22 11:24:12 manager.go:570: [INFO] [stacks] ScanStacks complete: 56 stacks found (6 deployed, 50 available)
2026/08/22 11:24:12 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:24:12 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 47ms)
2026/08/22 11:24:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m50s ago, effective interval 5m0s, healthy=true
2026/08/22 11:24:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping calibre-web — last check 40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:24:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping kimai — last check 3m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:24:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping opengist — last check 3m40s ago, effective interval 5m0s, healthy=true
2026/08/22 11:24:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m10s ago, effective interval 5m0s, healthy=true
2026/08/22 11:24:12 healthprobe.go:76: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (5 skipped not due, 1 skipped no container)
2026/08/22 11:24:12 manager.go:621: [INFO] [stacks] Status refresh: 14 containers across 56 stacks
2026/08/22 11:24:12 client.go:1142: [DEBUG] [agentapi] GET /storage -> 200 (559ms)
2026/08/22 11:24:12 scheduler.go:363: [INFO] [scheduler] Job agent-channel-health completed (took 559ms)
2026/08/22 11:24:13 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (660ms)
2026/08/22 11:24:14 client.go:1142: [DEBUG] [agentapi] GET /disks -> 200 (679ms)
+1
View File
@@ -26,6 +26,7 @@
---
| **R-356** | **The off-site restore refused every app that has no data drive — it asked "does this app have an HDD path?" to answer "is this app installed?", and for 40 of 53 catalogue apps the honest answer to the first is permanently no.** Shipped in controller v0.219.0. Evidence: `audits/DRILL-r356-hot-only-restore-2026-08-22/evidence/`. **Reasoning kept:** *the restore destination is resolved by the SAME rule as the capture destination — the drive if the app has one, the system data path otherwise (`Manager.GetAppDrivePath`, one expression). The refusal that protects a drive app from being restored onto the wrong disk applies to apps that HAVE a drive to get wrong.* **An app with no drive is not misconfigured** — `01-topology-and-trust.md` §8 carries the `[DESIGN]` marker; between 19 and 22 August that design was called a defect four times. **Deployment is asked of `ListDeployedStacks()` and FAILS CLOSED on a nil provider:** "cannot tell" must not become "go ahead" when the caller's next act is a write. **Two different failures get two different sentences** — installed-but-no-resolvable-data-root has its own refusal and its own route; widening `nincs telepítve` to cover it would send a customer to reinstall a running app and hide the real fault. **Measured, and load-bearing: 53 templates, 13 `needs_hdd: true`, 40 `false`** (catalogue @ `459766cb1639`). **The capture side's raw `GetStackHDDPath` is FENCED and was not changed** — capture resolves an app's declared `userdata`/`import` file legs against that value, and a system-data fallback there would write a snapshot claiming to hold files it does not. | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.219.0, 2026-08-22; `privatebin` on `demo-hp`: planted, backed up, deleted, restored, 15/15 files byte-identical including two Hungarian accented names) | full text: `git show e18668f9e19f:documentation/backlog/OPEN-ITEMS.md` |
| **R-216** | **A correct recovery code was reported to the customer as wrong.** Shipped in 0.120.0, v0.125.0. | **SHIPPED** (controller v0.201.0 + hub v0.97.0/0.97.1) — **but see R-223**: the feature does not work on a NEW box until the manifest vouches agent 0.125.0. Until then such a box is correctly HELD, not lied to | full text: `git show fddfe00ce268:documentation/backlog/OPEN-ITEMS.md` |
| **R-218** | **Succeeding at recovery stopped the box asking for what it still needed.** Shipped in v0.203.0. Evidence: `documentation/tests/part4-rewalk-2026-08-06/journal.md`. | **CLOSED 2026-08-06 — controller v0.203.0, proven live.** *(State corrected 2026-08-06: this field read REOPENED while the body below already recorded the fix shipped and proven. The history of the over-claim is kept deliberately — it is why the row is worded as it is.)* **The over-claim, as it stood: the fix covered the DECLARATION half only.** Measured on the R-201 re-walk: the box declared, and **`offsiteheal` re-staged the secret at 11:44:57** saying *"the box re-consumes on its next cycle"* — **the next cycle came and went** (`host-report` 11:55:46, `Received report` 11:55:54, a full cycle **with a positive control that it ran**) **and the credential was still not consumed.** 23 minutes after the re-stage the box's last off-site-apply attempt was still the pre-re-stage one. A census of the customer-reachable actions on `/backups/remote` (`config`, `reset`, `run`, `toggle`) found **none that fetches a staged credential**, and the only lever is `systemctl restart felhom-controller-bootstrap.service` **inside the guest** — which worked in **18 s** (Campaign 11 measured 17), confirming nothing was wrong with the credential, the target or the key: **the only thing missing is anything at all to trigger a retry.** **This is the FIRST of the two dead ends that keep the recovery journey failing** | full text: `git show fddfe00ce268:documentation/backlog/OPEN-ITEMS.md` |
| **R-219** | **The listing the screen promises could never render on the shape it exists for.** | **SHIPPED** (controller v0.201.0) — the unlock now places the key, brings the tier up, then lists | full text: `git show fddfe00ce268:documentation/backlog/OPEN-ITEMS.md` |
-1
View File
@@ -525,7 +525,6 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
| **R-349** | **"Prove it by hand, then publish" leaves the fleet running a DIFFERENT binary under the SAME version name — and self-update cannot notice.** Hit on 2026-08-20 during the R-344 train, caught and corrected the same hour, filed because the next prove-then-publish train will hit it identically. **The mechanism:** a proof deploy is a hand build (`go build -ldflags "-X main.version=0.130.0"`), while `scripts/release-agent.sh` deliberately builds with **`-trimpath -buildvcs=false`** so the published artifact is reproducible (R-186). Same source, same version string, **different bytes**: `256e0829...` on the boxes vs **`a56a92a7...`** published and vouched. **Nothing corrects it automatically**, and that is the sharp edge: the boxes already report `0.130.0`, so the self-update path sees the vouched version as already installed and does nothing, **forever**. The divergence is invisible to every version check in the system — the hub, `--version`, and the artifact manifest all agree, because they all compare the version STRING. **Consequence if unnoticed:** the binary a customer box runs is not the binary the operator vouched, and not the one a reinstall would fetch — so a bug reproduced on the fleet may not exist in the published artifact, or vice versa. It is the same "one version name, two binaries" hazard `publish-agent.sh` already carries a comment about for `CGO_ENABLED`; that comment fixed the two ENTRY POINTS and does not cover a hand build during a proof. **Corrected here** by downloading the published artifact from the registry (not rebuilding it locally — the boxes get the bytes a fresh install would get) and installing it on both; both now report `sha256 a56a92a7...`, matching the vouch. | **READY (S) — NEW 2026-08-20** | — | Make the reconciliation a step, not a memory: the honest fix is for the agent to REPORT the sha256 of its own binary in the host report, so the hub can compare it against the vouched `agent_sha256` and flag drift — **exactly the mechanism `wrapper_sha256` already implements for the PBS wrapper** (R-50b), whose manifest help text says it *"makes host drift visible: agents report the installed file's hash and a mismatch is surfaced on the host page"*. The pattern exists and is proven; it simply was never extended to the agent's own binary. Cheaper interim: end every prove-then-publish train by installing the DOWNLOADED artifact. | CC |
| **R-350** | **SECURITY — the hub operator password was printed in cleartext into a session transcript by CC, 2026-08-20. Rotation recommended.** **What happened:** vouching the artifact manifest used `curl -w '%{redirect_url}'` for confirmation. The hub answers `POST /configuration/artifacts` with a **303**, and curl renders the redirect target **with the basic-auth credentials re-attached** — so the URL it printed contained `http://:<HUB_PW>@10.43.52.34:8080/configuration?flash=artifacts_set`. The password was never read aloud from the credentials file, never echoed deliberately, and every other call in the session correctly printed only `${#HUB_PW}`; it arrived through curl's own output formatting, which is why the usual discipline did not catch it. **Blast radius, stated precisely rather than minimised:** the value is **not** in git, not in `CHANGELOG.md`/`REPORT*.md`/any committed file (checked), and not in the evidence directory — it is in the Claude Code session transcript under `~/.claude/projects/` on DooPlex, which is operator-readable and persists across sessions. The hub UI is reachable only on the k3s ClusterIP and via the operator's own routes, not from the internet. **The value is deliberately not recorded here; it is stored out-of-band in the usual credentials file.** | **READY (S) — NEW 2026-08-20** | — | **Operator decides whether to rotate.** The hub's own `/configuration` password form does it (`current_password`/`new_password`/`confirm_password`), and per `hub-password-ui-2026-07-13` the DB override wins over the ConfigMap, which stays break-glass. CC can perform the rotation **file-to-file without printing the new value** (the `operator-present-one-time-secrets` convention) if asked — it did not do so unilaterally, because rotating a credential the operator holds in their own head or notes is their call, not CC's. **The reusable half, which matters more than this one password:** never use curl's `%{redirect_url}` (or `-v`, or `--libcurl`) against a basic-auth endpoint — all three re-render the credential. Confirm a redirect with `%{http_code}` and read the flash from a follow-up GET. | **Viktor decides**, CC executes |
| **R-353** | **A restore reported success having returned configuration and no data — and no screen could have told the customer.** `demo-hp`, 2026-08-21, OpenGist. The off-site reconstitution refused at 16:37:14 (not installed); the person reinstalled and ran the local unit restore, which reported `Restore-from-unit completed: opengist in 8.666896042s`. **The unit it restored from contains `manifest.json` + `compose/{app.yaml,.felhom.yml,docker-compose.yml}` and NOTHING else — `volume_dumps: None`, `db_dumps: None`** — and the off-site snapshot was **182.3 KB**. So the restore returned the app's configuration; there was no data leg in the unit to return, and the outcome said only that it had completed. **A warning beside a success is read as a success, and an unknown must never be drawn as healthy.** **Compounding, and recorded as UNKNOWN rather than fine:** whether the 40-class reaches the off-site tier at all has **not been observed** — `runVolumeDumps` (`backup/backup.go:607+`) covers them on paper, but every unit on the box reported `volume_dumps: None`, including `calibre-web` on the data drive, because no nightly dump run had happened on a one-hour-old box. | **OPEN — NEXT SESSION'S FIRST ITEM** | — | **Two things, in order. (1)** A restore whose unit carries no `db_dumps` and no `volume_dumps` must **say so in its outcome** — „a mentés csak a beállításokat tartalmazta, adatot nem" — instead of reporting a bare completion. The verdict must consult what was actually placed, not merely that the operation ended. **(2)** Then *prove* the off-site coverage of a named-volume app by running a dump cycle and reading the resulting manifest, rather than inferring it from the gate order. Do not close (1) on the strength of (2) being likely. **(2) IS NOW SATISFIED — drill 2026-08-21.** A dump cycle was run and the manifests read: `privatebin volume_dumps=[privatebin_privatebin_data.tar]`, `opengist volume_dumps=[opengist_opengist_data.tar]`, `kimai volume_dumps=[kimai_kimai_db_data.tar, kimai_kimai_var.tar]` — the 40-class DOES reach the off-site tier, and PrivateBin's planted 1 MB came back byte-identical from its off-site snapshot into the checking folder. **(1) stands and is now strictly larger than when written:** R-354 shows the bare completion is also reported over a unit that DID carry a data leg, because the off-site restore never replays volume dumps at all. | CC |
| **R-356** | **The off-site restore refuses for all 40 no-drive apps, says the app "is not installed" when it is running, and then gives an instruction those apps make impossible.** `ReconstituteFromOffsite` refuses when `GetStackHDDPath(stack)` is empty (`offbox_reconstitute.go:208-227`); for a 40-class app that is ALWAYS empty, because they are offered no storage field at deploy time (R-352's own measurement). Observed 2026-08-21 22:21 on `privatebin` while it was `deployed=true, state=running, healthy`: „a(z) privatebin nincs telepítve, ezért nincs hová visszaállítani az adatait. A mentése szerint az adatai itt voltak: /mnt/sys_drive. Telepítsd újra az alkalmazást ugyanerre a helyre…". **The predicate is "has an HDD path"; the sentence says "is not installed"; for this class they are different things**, and the remedy offered cannot be carried out. This is also what the 2026-08-21 afternoon OpenGist journey hit before falling back to the local restore (R-353). | **OPEN — HIGH** | — | Separate the two questions. A 40-class app has a destination — the system data path — and the restore already knows it. **RE-CHECKED 2026-08-22 against the corrected placement framing: this row SURVIVES UNCHANGED and is strengthened by it.** The 40-class has no `HDD_PATH` *because the architecture puts its hot data inside the guest by design* (`01-topology-and-trust.md:150-152`) — so an absent HDD path is the NORMAL state for most of the catalogue, and a restore that reads it as "the app is not installed" is misreading a correct configuration, not reporting a misconfiguration. The one sentence in this row that leaned on the old framing — "because they are offered no storage field at deploy time (R-352's own measurement)" — should read: *because these apps have no bulk volume to place, so the field correctly does not exist.* | CC |
| **R-357** | **The DESTRUCTIVE restore has no free-space gate; the three that exist are all on non-destructive paths.** `offbox_reconstitute.go` contains **zero** references to `offboxFree`; the gates sit at `offbox_restore.go:231` (scratch restore), `:297` (prepare-full) and `:423` (place-to-live). Proven 2026-08-21 23:11 with 300 KB free and 1 MB to write: it stopped `paperless-ngx`, failed halfway (`rsync … No space left on device (28)`), left the data directory holding **2 of 5** planted entries, and restarted the app. The message is honest but is raw rsync output. | **OPEN — MEDIUM** | — | Same gate, same wording as `:297`, before the stop. | CC |
| **R-358** | **A FAILED scratch restore leaves a partial copy that the product then offers as a full restore source — and the destructive restore runs from it and reports success.** `OffboxFullScratchReady` (`offbox_restore.go:305`) asks only whether the directory exists and is non-empty; its comment defers completeness to `PlaceOffsiteRestore`, which stats top-level placements, not files. Proven 2026-08-21 22:54-22:56 against a deliberately corrupted store: the restore failed honestly (`ciphertext verification failed`, 54 files, 15 of 16 originals), the wizard then offered „Teljes visszaállítás indítása", and pressing it reported `ok=true`. **The failure is detected and then forgotten.** | **OPEN — MEDIUM** | — | Record the failure against the scratch and refuse to place from it until it is re-prepared. | CC |
| **R-359** | **The off-site restic store is never verified by anything, ever.** The complete set of restic verbs in the controller is `restore, snapshots, backup, unlock, stats, init, forget, prune, cat` — **no `check`**. The agent's `RestoreTest` is PBS-tier only. Established 2026-08-21 by deliberately corrupting one pack: `restic check` catches it immediately („ciphertext verification failed", „Fatal: repository contains errors"), and the product only meets the damage when a customer is already trying to recover. | **OPEN — MEDIUM** | — | A periodic `restic check` (structure) with an occasional `--read-data`, reported like any other backup verdict. Note PBS already has verify jobs; this is the tier that does not. | CC |