rehearsal 2026-10-20: B6 #8 wrong clock — R-1201 (ep0 tunnel dead after a clock ahead, P2), R-1202 (clock behind: silent app stop, blames the hub)
This commit is contained in:
@@ -0,0 +1,19 @@
|
||||
minus2y=2024-10-21T07:13:35
|
||||
down 07:15:30
|
||||
update VM 9320: -startdate 2024-10-21T07:13:35
|
||||
started with RTC 2024-10-21T07:13:35 at real 07:15:33
|
||||
real 2026-10-21T07:16:16Z box Mon, 13 Apr 2026 19:38:33 GMT
|
||||
real 2026-10-21T07:25:31Z box Mon, 13 Apr 2026 19:47:48 GMT
|
||||
plus2y=2028-10-21T07:28:25
|
||||
down 07:30:20
|
||||
update VM 9320: -startdate 2028-10-21T07:28:25
|
||||
started with RTC 2028-10-21T07:28:25 at real 07:30:23
|
||||
real 2026-10-21T07:31:03Z box Sat, 21 Oct 2028 07:29:05 GMT
|
||||
real 2026-10-21T07:39:57Z box Sat, 21 Oct 2028 07:38:00 GMT
|
||||
iifname "tap9320i0" udp dport 123 counter packets 314 bytes 23864 drop
|
||||
iifname "tap9320i0" tcp dport 4460 counter packets 0 bytes 0 drop
|
||||
down 07:43:44
|
||||
update VM 9320: -delete startdate
|
||||
started with real RTC at 07:43:47
|
||||
ntp on 07:43:48
|
||||
real 2026-10-21T07:44:31Z box Wed, 21 Oct 2026 07:44:31 GMT
|
||||
+200
@@ -0,0 +1,200 @@
|
||||
2026-04-13T19:38:28Z INFO [infra] filebrowser compose exists — starting without regenerating
|
||||
2026-04-13T19:38:29Z INFO [web] sessions: restored 124 dashboard session(s) from disk (dropped 0 expired, 0 malformed)
|
||||
2026-04-13T19:38:29Z INFO [web] Auth: using password from settings.json
|
||||
2026-04-13T19:38:29Z INFO [scheduler] Registered periodic job: disk-health-check (every 1h0m0s)
|
||||
2026-04-13T19:38:29Z INFO [scheduler] Registered periodic job: agent-channel-health (every 1m0s)
|
||||
2026-04-13T19:38:29Z INFO Web UI listening on :8080
|
||||
2026-04-13T19:38:29Z INFO [web] FileBrowser: drive /mnt/felhom-drives/hdd_2 not mounted — skipping userdata skeleton
|
||||
2026-04-13T19:38:29Z INFO [backup] Discovered 4 databases
|
||||
2026-04-13T19:38:30Z INFO [monitor] Health check: status=fail
|
||||
2026-04-13T19:38:30Z INFO [web] FileBrowser sync — no config/compose change, ensured running without recreate (3 storage path(s))
|
||||
2026-04-13T19:38:30Z INFO [infra] connected felhom-controller to felhom-filebrowser (the FileBrowser default-login check, R-1126)
|
||||
2026-04-13T19:38:31Z INFO [backup] Backup status cache refreshed
|
||||
2026-04-13T19:38:32Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:33Z INFO [report] Building system report
|
||||
2026-04-13T19:38:35Z INFO [samba] applying samba stack: shares=2 config_changed=false
|
||||
2026-04-13T19:38:36Z INFO [gate] boot 1776109082-1457: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/hdd_1] before recreating drive-backed apps
|
||||
2026-04-13T19:38:36Z INFO [gate] boot 1776109082-1457: live bind confirmed — recreating drive-backed app immich (state=degraded) onto /mnt/felhom-drives/hdd_1
|
||||
2026-04-13T19:38:36Z INFO [stacks] Stopping stack: immich
|
||||
2026-04-13T19:38:37Z INFO [stacks] Stack immich stopped successfully (took 1.0s)
|
||||
2026-04-13T19:38:37Z INFO [stacks] Status refresh: 15 containers across 68 stacks
|
||||
2026-04-13T19:38:37Z INFO [stacks] Starting stack: immich
|
||||
2026-04-13T19:38:37Z INFO [stacks] Status refresh: 15 containers across 68 stacks
|
||||
2026-04-13T19:38:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:38Z INFO Health probes: 2 ok (of 2 probed)
|
||||
2026-04-13T19:38:39Z WARN Event push failed after 3 attempts (controller_started/info): Post "https://hub.felhom.eu/api/v1/event": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:38:39Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:38:42Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:47Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:48Z INFO [monitor] Health check: status=ok
|
||||
2026-04-13T19:38:48Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:48Z INFO Health probes: 1 ok (of 1 probed)
|
||||
2026-04-13T19:38:48Z INFO [stacks] Stack immich started successfully (took 11.1s)
|
||||
2026-04-13T19:38:49Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:49Z INFO [gate] boot 1776109082-1457: live bind confirmed — recreating drive-backed app nextcloud (state=degraded) onto /mnt/felhom-drives/hdd_1
|
||||
2026-04-13T19:38:49Z INFO [stacks] Stopping stack: nextcloud
|
||||
2026-04-13T19:38:49Z INFO [stacks] Stack nextcloud stopped successfully (took 0.4s)
|
||||
2026-04-13T19:38:49Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:38:49Z INFO [stacks] Starting stack: nextcloud
|
||||
2026-04-13T19:38:52Z INFO [stacks] Stack immich post-start status:
|
||||
2026-04-13T19:38:52Z INFO [stacks] immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4@sha256:e16c2f166a8174901959fdf85e2e4c7bd1ebc4b37e0b6655de97c41408a260c4 running Up 13 seconds (healthy)
|
||||
2026-04-13T19:38:52Z INFO [stacks] immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0@sha256:1a078b237c1d9b420b0ee59147386b4aa60d3a07a8e6a402fc84a57e41b043a4 running Up 13 seconds (healthy)
|
||||
2026-04-13T19:38:52Z INFO [stacks] immich-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 13 seconds (healthy)
|
||||
2026-04-13T19:38:52Z INFO [stacks] immich-server ghcr.io/immich-app/immich-server:v3.2.4@sha256:d317916b28090c33eb36b308464ea391f8b7df1d850fcfea227a39ec879718c2 running Up 3 seconds (health: starting)
|
||||
2026-04-13T19:38:52Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:56Z INFO [stacks] Stack nextcloud started successfully (took 6.5s)
|
||||
2026-04-13T19:38:56Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:56Z INFO [gate] boot 1776109082-1457: live bind confirmed — recreating drive-backed app paperless-ngx (state=degraded) onto /mnt/felhom-drives/hdd_1
|
||||
2026-04-13T19:38:56Z INFO [stacks] Stopping stack: paperless-ngx
|
||||
2026-04-13T19:38:56Z INFO [stacks] Stack paperless-ngx stopped successfully (took 0.5s)
|
||||
2026-04-13T19:38:56Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:38:56Z INFO [stacks] Starting stack: paperless-ngx
|
||||
2026-04-13T19:38:58Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:58Z INFO Health probes: 1 ok (of 1 probed)
|
||||
2026-04-13T19:38:58Z INFO [deadapp] boot grace 1m30s: the host's last boot was clean (R-856)
|
||||
2026-04-13T19:38:58Z INFO [aftercrash] a normal (or unknown) boot — no app repair after an unclean boot (R-1007)
|
||||
2026-04-13T19:38:58Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:59Z WARN [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:38:59Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:38:59Z WARN Startup hub report attempt 1/3 failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:38:59Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:38:59Z INFO [stacks] Stack nextcloud post-start status:
|
||||
2026-04-13T19:38:59Z INFO [stacks] nextcloud nextcloud:34.0.4-apache@sha256:37b109885aa3cba3e056362a899556a625c986f2c17cd2c0cc157d21c789f53b running Up 3 seconds (health: starting)
|
||||
2026-04-13T19:38:59Z INFO [stacks] nextcloud-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy)
|
||||
2026-04-13T19:38:59Z INFO [stacks] nextcloud-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 9 seconds (healthy)
|
||||
2026-04-13T19:39:03Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:07Z INFO [stacks] Stack paperless-ngx started successfully (took 10.9s)
|
||||
2026-04-13T19:39:07Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:07Z INFO [gate] boot 1776109082-1457: re-syncing FileBrowser mounts against the live binds
|
||||
2026-04-13T19:39:07Z INFO [settings] Settings saved
|
||||
2026-04-13T19:39:07Z INFO [web] FileBrowser: drive /mnt/felhom-drives/hdd_2 not mounted — skipping userdata skeleton
|
||||
2026-04-13T19:39:08Z INFO [web] FileBrowser sync — no config/compose change, ensured running without recreate (3 storage path(s))
|
||||
2026-04-13T19:39:08Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:08Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:08Z INFO Health probes: 3 ok (of 3 probed)
|
||||
2026-04-13T19:39:11Z INFO [stacks] Stack paperless-ngx post-start status:
|
||||
2026-04-13T19:39:11Z INFO [stacks] paperless-postgres postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 running Up 13 seconds (healthy)
|
||||
2026-04-13T19:39:11Z INFO [stacks] paperless-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 13 seconds (healthy)
|
||||
2026-04-13T19:39:11Z INFO [stacks] paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15@sha256:6c86cad803970ea782683a8e80e7403444c5bf3cf70de63b4d3c8e87500db92f running Up 3 seconds (health: starting)
|
||||
2026-04-13T19:39:13Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:18Z INFO [bootrecon] boot window: fleet settled after 46s (3 identical samples 5s apart) — sweeping
|
||||
2026-04-13T19:39:18Z INFO [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start)
|
||||
2026-04-13T19:39:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:24Z WARN [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:39:24Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:39:24Z WARN Startup hub report attempt 2/3 failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:39:24Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:39:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:39:36Z INFO [scheduler] Job agent-channel-health completed (took 7.226s)
|
||||
2026-04-13T19:39:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:48Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:49Z WARN [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:39:49Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:39:49Z WARN Startup hub report attempt 3/3 failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:39:49Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:39:49Z WARN Startup hub report failed after 3 attempts — next scheduled push in 15m
|
||||
2026-04-13T19:39:57Z INFO [fillwatch] checked 4 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026-04-13T19:39:57Z WARN [offsite-apply] settle-gate: GO — floor still unknown after 1m30s; a hub that cannot serve a floor cannot serve a consume (no burn risk)
|
||||
2026-04-13T19:39:58Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:08Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:14Z WARN [selfupdate] Registry check failed: HTTP request failed: Get "https://gitea.dooplex.hu/v2/admin/felhom-controller/tags/list": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:40:14Z is before 2026-10-05T03:52:51Z
|
||||
2026-04-13T19:40:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:28Z INFO [scheduler] Running job: stack-scan
|
||||
2026-04-13T19:40:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:28Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2026-04-13T19:40:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:28Z INFO [scheduler] Job stack-scan completed (took 419ms)
|
||||
2026-04-13T19:40:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:40:36Z INFO [scheduler] Job agent-channel-health completed (took 7.212s)
|
||||
2026-04-13T19:40:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:48Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:58Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:41:08Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:41:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:41:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:41:28Z INFO [stacks] controller image retention: pass over 2 controller image(s) — running 0.336.0, previous "0.331.0" (the self-update's record), 0 candidate(s), 0 deleted, the rest kept
|
||||
2026-04-13T19:41:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:41:36Z INFO [scheduler] Job agent-channel-health completed (took 7.217s)
|
||||
2026-04-13T19:41:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:41:48Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:41:58Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:42:08Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:42:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:42:28Z INFO [scheduler] Running job: stack-scan
|
||||
2026-04-13T19:42:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:42:28Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2026-04-13T19:42:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:42:28Z INFO [scheduler] Job stack-scan completed (took 209ms)
|
||||
2026-04-13T19:42:28Z WARN [deadapp] paperless-ngx: crash_loop — 7 in 10m0s; STOPPING it (decision 28)
|
||||
2026-04-13T19:42:28Z INFO [stacks] Stopping stack: paperless-ngx
|
||||
2026-04-13T19:42:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:42:36Z INFO [scheduler] Job agent-channel-health completed (took 7.22s)
|
||||
2026-04-13T19:42:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:42:39Z INFO [stacks] Stack paperless-ngx stopped successfully (took 10.8s)
|
||||
2026-04-13T19:42:39Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:42:39Z WARN [settings] restore hold SET for paperless-ngx — the app stays stopped until it is cleared
|
||||
2026-04-13T19:42:39Z INFO [settings] Settings saved
|
||||
2026-04-13T19:42:39Z INFO [settings] Settings saved
|
||||
2026-04-13T19:42:39Z WARN [backup] paperless-ngx is STOPPED by the box: crash_loop (trip 1 within 24h0m0s) — Start gives it one more try (decision 28)
|
||||
2026-04-13T19:42:45Z WARN Event push failed after 3 attempts (app_stopped_unhealthy/warning): Post "https://hub.felhom.eu/api/v1/event": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:42:45Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:42:48Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:42:58Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:08Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:18Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Running job: backup-cache
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Running job: system-health
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Running job: docker-socket-users
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Running job: offsite-credential-retry
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Job offsite-credential-retry completed (took 1ms)
|
||||
2026-04-13T19:43:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:28Z INFO [backup] Found 7 DB dump files across drives
|
||||
2026-04-13T19:43:28Z INFO [backup] Discovered 3 databases
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Job docker-socket-users completed (took 271ms)
|
||||
2026-04-13T19:43:28Z INFO [monitor] Health check: status=ok
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Job system-health completed (took 356ms)
|
||||
2026-04-13T19:43:28Z INFO [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node)
|
||||
2026-04-13T19:43:29Z INFO [backup] Backup status cache refreshed
|
||||
2026-04-13T19:43:29Z INFO [scheduler] Job backup-cache completed (took 675ms)
|
||||
2026-04-13T19:43:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:43:37Z INFO [scheduler] Job agent-channel-health completed (took 7.986s)
|
||||
2026-04-13T19:43:38Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:38Z INFO Health probes: 2 ok (of 2 probed)
|
||||
2026-04-13T19:43:48Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:58Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:58Z INFO Health probes: 2 ok (of 2 probed)
|
||||
2026-04-13T19:44:08Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:44:18Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:44:18Z INFO Health probes: 3 ok (of 3 probed)
|
||||
2026-04-13T19:44:28Z INFO [scheduler] Running job: stack-scan
|
||||
2026-04-13T19:44:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:44:28Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2026-04-13T19:44:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:44:28Z INFO [scheduler] Job stack-scan completed (took 177ms)
|
||||
2026-04-13T19:44:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:44:36Z INFO [scheduler] Job agent-channel-health completed (took 7.199s)
|
||||
2026-04-13T19:44:38Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:44:48Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:44:58Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:45:08Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:45:18Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:45:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:45:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:45:36Z INFO [scheduler] Job agent-channel-health completed (took 7.219s)
|
||||
2026-04-13T19:45:38Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:45:48Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:45:58Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:08Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:18Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:28Z INFO [scheduler] Running job: stack-scan
|
||||
2026-04-13T19:46:28Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2026-04-13T19:46:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:28Z INFO [scheduler] Job stack-scan completed (took 137ms)
|
||||
2026-04-13T19:46:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:46:36Z INFO [scheduler] Job agent-channel-health completed (took 7.226s)
|
||||
2026-04-13T19:46:38Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:48Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:58Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:47:08Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:47:18Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:47:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:47:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:47:36Z INFO [scheduler] Job agent-channel-health completed (took 7.246s)
|
||||
2026-04-13T19:47:38Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:47:48Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:47:49Z INFO [web] Login from 192.168.0.180
|
||||
+1
@@ -0,0 +1 @@
|
||||
{"ok": true, "data": {"logs": ""}}
|
||||
+200
@@ -0,0 +1,200 @@
|
||||
2028-10-21T07:29:31Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:31Z INFO [gate] boot 1855726117-1561: re-syncing FileBrowser mounts against the live binds
|
||||
2028-10-21T07:29:31Z INFO [gate] boot 1855726117-1561: 1 drive-backed app(s) left stopped on purpose (recorded Stop, or a legacy app.yaml at zero containers)
|
||||
2028-10-21T07:29:31Z INFO [settings] Settings saved
|
||||
2028-10-21T07:29:31Z INFO [web] FileBrowser: drive /mnt/felhom-drives/hdd_2 not mounted — skipping userdata skeleton
|
||||
2028-10-21T07:29:32Z INFO [web] FileBrowser mounts synced (recreated) — 3 storage path(s), config updated
|
||||
2028-10-21T07:29:32Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:33Z INFO [deadapp] boot grace 1m30s: the host's last boot was clean (R-856)
|
||||
2028-10-21T07:29:33Z INFO [aftercrash] a normal (or unknown) boot — no app repair after an unclean boot (R-1007)
|
||||
2028-10-21T07:29:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:34Z INFO [infra] filebrowser: default login refused for this run — joined traefik-public, files.<domain> served (R-1126)
|
||||
2028-10-21T07:29:34Z INFO [stacks] Stack nextcloud post-start status:
|
||||
2028-10-21T07:29:34Z INFO [stacks] nextcloud nextcloud:34.0.4-apache@sha256:37b109885aa3cba3e056362a899556a625c986f2c17cd2c0cc157d21c789f53b running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:29:34Z INFO [stacks] nextcloud-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy)
|
||||
2028-10-21T07:29:34Z INFO [stacks] nextcloud-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 9 seconds (healthy)
|
||||
2028-10-21T07:29:37Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:42Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:43Z INFO Health probes: 3 ok (of 3 probed)
|
||||
2028-10-21T07:29:45Z WARN [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T07:29:45Z is after 2027-01-12T07:36:57Z
|
||||
2028-10-21T07:29:45Z WARN Startup hub report attempt 2/3 failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T07:29:45Z is after 2027-01-12T07:36:57Z
|
||||
2028-10-21T07:29:47Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:52Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:52Z INFO [bootrecon] boot window: fleet settled after 45s (3 identical samples 5s apart) — sweeping
|
||||
2028-10-21T07:29:52Z INFO [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start)
|
||||
2028-10-21T07:29:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:30:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:30:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:30:03Z INFO [scheduler] Job agent-channel-health completed (took 476ms)
|
||||
2028-10-21T07:30:11Z WARN [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T07:30:11Z is after 2027-01-12T07:36:57Z
|
||||
2028-10-21T07:30:11Z WARN Startup hub report attempt 3/3 failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T07:30:11Z is after 2027-01-12T07:36:57Z
|
||||
2028-10-21T07:30:11Z WARN Startup hub report failed after 3 attempts — next scheduled push in 15m
|
||||
2028-10-21T07:30:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:30:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:30:32Z INFO [fillwatch] checked 4 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2028-10-21T07:30:32Z WARN [offsite-apply] settle-gate: GO — floor still unknown after 1m30s; a hub that cannot serve a floor cannot serve a consume (no burn risk)
|
||||
2028-10-21T07:30:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:30:36Z WARN [selfupdate] Registry check failed: HTTP request failed: Get "https://gitea.dooplex.hu/v2/admin/felhom-controller/tags/list": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T07:30:36Z is after 2027-01-03T03:52:50Z
|
||||
2028-10-21T07:30:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:30:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:02Z INFO [scheduler] Running job: stack-scan
|
||||
2028-10-21T07:31:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:03Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2028-10-21T07:31:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:03Z INFO [scheduler] Job stack-scan completed (took 160ms)
|
||||
2028-10-21T07:31:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:31:03Z INFO [scheduler] Job agent-channel-health completed (took 586ms)
|
||||
2028-10-21T07:31:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:32:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:32:03Z INFO [stacks] controller image retention: pass over 2 controller image(s) — running 0.336.0, previous "0.331.0" (the self-update's record), 0 candidate(s), 0 deleted, the rest kept
|
||||
2028-10-21T07:32:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:32:03Z INFO [scheduler] Job agent-channel-health completed (took 471ms)
|
||||
2028-10-21T07:32:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:32:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:32:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:32:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:32:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:02Z INFO [scheduler] Running job: stack-scan
|
||||
2028-10-21T07:33:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:03Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2028-10-21T07:33:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:03Z INFO [scheduler] Job stack-scan completed (took 158ms)
|
||||
2028-10-21T07:33:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:33:03Z INFO [scheduler] Job agent-channel-health completed (took 477ms)
|
||||
2028-10-21T07:33:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:34:02Z INFO [scheduler] Running job: backup-cache
|
||||
2028-10-21T07:34:02Z INFO [scheduler] Running job: system-health
|
||||
2028-10-21T07:34:02Z INFO [scheduler] Running job: docker-socket-users
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Running job: offsite-credential-retry
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Job offsite-credential-retry completed (took 0s)
|
||||
2028-10-21T07:34:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:34:03Z INFO [backup] Found 7 DB dump files across drives
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Job docker-socket-users completed (took 243ms)
|
||||
2028-10-21T07:34:03Z INFO [backup] Discovered 3 databases
|
||||
2028-10-21T07:34:03Z INFO [monitor] Health check: status=ok
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Job system-health completed (took 329ms)
|
||||
2028-10-21T07:34:03Z INFO [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node)
|
||||
2028-10-21T07:34:03Z INFO [samba] applying samba stack: shares=2 config_changed=true
|
||||
2028-10-21T07:34:03Z INFO [backup] Backup status cache refreshed
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Job backup-cache completed (took 710ms)
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Job agent-channel-health completed (took 494ms)
|
||||
2028-10-21T07:34:07Z INFO [quiesce] backup due on 2 tier(s) — quiescing 7 stack(s): [actualbudget bookstack immich mealie nextcloud radicale vaultwarden]
|
||||
2028-10-21T07:34:07Z INFO [stacks] Stopping stack: actualbudget
|
||||
2028-10-21T07:34:07Z INFO [stacks] Stack actualbudget stopped successfully (took 0.3s)
|
||||
2028-10-21T07:34:07Z INFO [stacks] Status refresh: 15 containers across 68 stacks
|
||||
2028-10-21T07:34:07Z INFO [stacks] Stopping stack: bookstack
|
||||
2028-10-21T07:34:11Z INFO [stacks] Stack bookstack stopped successfully (took 4.1s)
|
||||
2028-10-21T07:34:11Z INFO [stacks] Status refresh: 13 containers across 68 stacks
|
||||
2028-10-21T07:34:11Z INFO [stacks] Stopping stack: immich
|
||||
2028-10-21T07:34:12Z INFO [stacks] Stack immich stopped successfully (took 0.7s)
|
||||
2028-10-21T07:34:12Z INFO [stacks] Status refresh: 9 containers across 68 stacks
|
||||
2028-10-21T07:34:12Z INFO [stacks] Stopping stack: mealie
|
||||
2028-10-21T07:34:13Z INFO [stacks] Status refresh: 9 containers across 68 stacks
|
||||
2028-10-21T07:34:13Z INFO [stacks] Stack mealie stopped successfully (took 1.7s)
|
||||
2028-10-21T07:34:14Z INFO [stacks] Status refresh: 8 containers across 68 stacks
|
||||
2028-10-21T07:34:14Z INFO [stacks] Stopping stack: nextcloud
|
||||
2028-10-21T07:34:15Z INFO [stacks] Stack nextcloud stopped successfully (took 1.7s)
|
||||
2028-10-21T07:34:15Z INFO [stacks] Status refresh: 5 containers across 68 stacks
|
||||
2028-10-21T07:34:15Z INFO [stacks] Stopping stack: radicale
|
||||
2028-10-21T07:34:16Z INFO [stacks] Stack radicale stopped successfully (took 0.4s)
|
||||
2028-10-21T07:34:16Z INFO [stacks] Status refresh: 4 containers across 68 stacks
|
||||
2028-10-21T07:34:16Z INFO [stacks] Stopping stack: vaultwarden
|
||||
2028-10-21T07:34:16Z INFO [stacks] Stack vaultwarden stopped successfully (took 0.2s)
|
||||
2028-10-21T07:34:16Z INFO [stacks] Status refresh: 3 containers across 68 stacks
|
||||
2028-10-21T07:34:16Z INFO [quiesce] tier local: backup job backup-9201-1855726456347094750 started — polling
|
||||
2028-10-21T07:34:23Z INFO [stacks] Status refresh: 3 containers across 68 stacks
|
||||
2028-10-21T07:34:26Z INFO [quiesce] tier local: job backup-9201-1855726456347094750 snapshotted — resuming app early (8B.2); the upload continues with the apps running
|
||||
2028-10-21T07:34:26Z INFO [quiesce] unquiescing (snapshotted (early resume)): restarting 7 stack(s)
|
||||
2028-10-21T07:34:26Z INFO [stacks] Starting stack: actualbudget
|
||||
2028-10-21T07:34:26Z INFO [stacks] Stack actualbudget started successfully (took 0.5s)
|
||||
2028-10-21T07:34:27Z INFO [stacks] Status refresh: 4 containers across 68 stacks
|
||||
2028-10-21T07:34:27Z INFO [stacks] Starting stack: bookstack
|
||||
2028-10-21T07:34:30Z INFO [stacks] Stack actualbudget post-start status:
|
||||
2028-10-21T07:34:30Z INFO [stacks] actualbudget actualbudget/actual-server:26.9.0@sha256:552beab3dec8c93d46b8b9245612d63c3f123b8a45063a474f53e229b17621d3 running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:34:33Z INFO [stacks] Status refresh: 6 containers across 68 stacks
|
||||
2028-10-21T07:34:33Z WARN Health probe bookstack: HTTP GET :80/ → Get "http://bookstack-db:80/": dial tcp: lookup bookstack-db on 127.0.0.11:53: no such host
|
||||
2028-10-21T07:34:33Z WARN Health probes: 1 ok, 1 unhealthy (of 2 probed)
|
||||
2028-10-21T07:34:33Z INFO [stacks] Stack bookstack started successfully (took 6.6s)
|
||||
2028-10-21T07:34:33Z INFO [stacks] Status refresh: 6 containers across 68 stacks
|
||||
2028-10-21T07:34:33Z INFO [stacks] Starting stack: immich
|
||||
2028-10-21T07:34:36Z INFO [stacks] Stack bookstack post-start status:
|
||||
2028-10-21T07:34:36Z INFO [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:34:36Z INFO [stacks] bookstack-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy)
|
||||
2028-10-21T07:34:43Z INFO [stacks] Status refresh: 10 containers across 68 stacks
|
||||
2028-10-21T07:34:45Z INFO [stacks] Stack immich started successfully (took 11.3s)
|
||||
2028-10-21T07:34:45Z INFO [stacks] Status refresh: 10 containers across 68 stacks
|
||||
2028-10-21T07:34:45Z INFO [stacks] Starting stack: mealie
|
||||
2028-10-21T07:34:46Z INFO [stacks] Stack mealie started successfully (took 0.6s)
|
||||
2028-10-21T07:34:46Z INFO [stacks] Status refresh: 11 containers across 68 stacks
|
||||
2028-10-21T07:34:46Z INFO [stacks] Starting stack: nextcloud
|
||||
2028-10-21T07:34:48Z INFO [stacks] Stack immich post-start status:
|
||||
2028-10-21T07:34:48Z INFO [stacks] immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4@sha256:e16c2f166a8174901959fdf85e2e4c7bd1ebc4b37e0b6655de97c41408a260c4 running Up 13 seconds (healthy)
|
||||
2028-10-21T07:34:48Z INFO [stacks] immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0@sha256:1a078b237c1d9b420b0ee59147386b4aa60d3a07a8e6a402fc84a57e41b043a4 running Up 14 seconds (healthy)
|
||||
2028-10-21T07:34:48Z INFO [stacks] immich-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy)
|
||||
2028-10-21T07:34:48Z INFO [stacks] immich-server ghcr.io/immich-app/immich-server:v3.2.4@sha256:d317916b28090c33eb36b308464ea391f8b7df1d850fcfea227a39ec879718c2 running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:34:49Z INFO [stacks] Stack mealie post-start status:
|
||||
2028-10-21T07:34:49Z INFO [stacks] mealie ghcr.io/mealie-recipes/mealie:v3.28.0@sha256:8b02290f4d1806f02acac6f25f6d48a3c965612fda1f8e914d5af533276f8688 running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:34:53Z INFO [stacks] Status refresh: 14 containers across 68 stacks
|
||||
2028-10-21T07:34:53Z WARN Health probe nextcloud: API GET :80/status.php → Get "http://nextcloud-redis:80/status.php": dial tcp: lookup nextcloud-redis on 127.0.0.11:53: no such host
|
||||
2028-10-21T07:34:53Z WARN Health probes: 1 ok, 1 unhealthy (of 2 probed)
|
||||
2028-10-21T07:34:58Z INFO [stacks] Stack nextcloud started successfully (took 12.2s)
|
||||
2028-10-21T07:34:59Z INFO [stacks] Status refresh: 14 containers across 68 stacks
|
||||
2028-10-21T07:34:59Z INFO [stacks] Starting stack: radicale
|
||||
2028-10-21T07:34:59Z INFO [stacks] Stack radicale started successfully (took 0.7s)
|
||||
2028-10-21T07:35:00Z INFO [stacks] Status refresh: 15 containers across 68 stacks
|
||||
2028-10-21T07:35:00Z INFO [stacks] Starting stack: vaultwarden
|
||||
2028-10-21T07:35:01Z INFO [stacks] Stack vaultwarden started successfully (took 1.1s)
|
||||
2028-10-21T07:35:01Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:02Z INFO [stacks] Stack nextcloud post-start status:
|
||||
2028-10-21T07:35:02Z INFO [stacks] nextcloud nextcloud:34.0.4-apache@sha256:37b109885aa3cba3e056362a899556a625c986f2c17cd2c0cc157d21c789f53b running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:35:02Z INFO [stacks] nextcloud-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 14 seconds (healthy)
|
||||
2028-10-21T07:35:02Z INFO [stacks] nextcloud-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 15 seconds (healthy)
|
||||
2028-10-21T07:35:02Z INFO [scheduler] Running job: stack-scan
|
||||
2028-10-21T07:35:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:35:03Z INFO [stacks] Stack radicale post-start status:
|
||||
2028-10-21T07:35:03Z INFO [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:35:03Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2028-10-21T07:35:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:03Z INFO [scheduler] Job stack-scan completed (took 654ms)
|
||||
2028-10-21T07:35:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:03Z INFO [scheduler] Job agent-channel-health completed (took 565ms)
|
||||
2028-10-21T07:35:04Z INFO [stacks] Stack vaultwarden post-start status:
|
||||
2028-10-21T07:35:04Z INFO [stacks] vaultwarden vaultwarden/server:1.37.4-alpine@sha256:19ea2b669945d842dadc3d500f2d6a795866a003330f59a47b9634658bae98bb running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:35:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:13Z INFO Health probes: 1 ok (of 1 probed)
|
||||
2028-10-21T07:35:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:23Z INFO Health probes: 4 ok (of 4 probed)
|
||||
2028-10-21T07:35:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:36:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:36:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:36:04Z INFO [scheduler] Job agent-channel-health completed (took 1.082s)
|
||||
2028-10-21T07:36:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:36:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:36:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:36:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:36:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:03Z INFO [scheduler] Running job: stack-scan
|
||||
2028-10-21T07:37:03Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2028-10-21T07:37:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:03Z INFO [scheduler] Job stack-scan completed (took 140ms)
|
||||
2028-10-21T07:37:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:37:03Z INFO [scheduler] Job agent-channel-health completed (took 463ms)
|
||||
2028-10-21T07:37:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:54Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:38:00Z INFO [web] Login from 192.168.0.180
|
||||
+335
@@ -0,0 +1,335 @@
|
||||
2028-10-21T09:28:47.937+02:00 INFO msg="felhom-agent daemon starting" version=0.161.0 host_id=Tester4-816e21 hub_url=https://hub.felhom.eu interval_s=900
|
||||
2028-10-21T09:28:47.957+02:00 INFO msg="daemon: operator signers pinned" count=2
|
||||
2028-10-21T09:28:47.965+02:00 INFO msg="local-api leaf LOADED" fingerprint_sha256=d0e3c957e507ec37772877c71f1d6ad79e01c50177e12c556328aa24d7fcb0bc cert=/var/lib/felhom-agent/local-api.crt
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="lanresolver: enabled" host_ip=192.168.0.134 upstreams="[1.1.1.1 8.8.8.8]" interval_s=300
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="wgtunnel: enabled" interval_s=60 state_dir=/var/lib/felhom-agent
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="poke: agent-plane sync listener enabled" port=51822
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="felhomsshd (OOB): enabled" interval_s=60 state_dir=/var/lib/felhom-agent
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
|
||||
2028-10-21T09:28:48.98+02:00 INFO msg="capabilities self-check" ok=76 total=76 degraded=0 inactive=0
|
||||
2028-10-21T09:28:48.98+02:00 INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
|
||||
2028-10-21T09:28:48.981+02:00 INFO msg="pbs: verify loop starting" cadence=6h0m0s
|
||||
2028-10-21T09:28:48.982+02:00 INFO msg="storage: watchdog starting" interval=5s debounce=15s
|
||||
2028-10-21T09:28:48.982+02:00 INFO msg="fast-tick armed: 30s out-of-band cadence while desired-state is unapplied" interval=30s
|
||||
2028-10-21T09:28:48.983+02:00 INFO msg="poke: listening for hub sync-pokes (WG-confined, contentless)" addr=10.77.0.6:51822
|
||||
2028-10-21T09:28:49.023+02:00 DEBUG msg="reconcile: no drift, no actions" desired_guests=0 actual_guests=1
|
||||
2028-10-21T09:28:49.604+02:00 INFO msg="local-api: drive intent recorded" where=/mnt/belso action=enrolled durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:28:49.604+02:00 INFO msg="internal place: mounted and enrolled" where=/mnt/belso fs_uuid=5aa4f4f8-2402-4d8c-841a-53bd4858e35d gib=56
|
||||
2028-10-21T09:28:49.74+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:28:49.784+02:00 DEBUG msg="hub: tunnel probe could not decide" detail="no running customer guest"
|
||||
2028-10-21T09:28:50.118+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:28:50.179+02:00 INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/belso stable=/mnt/felhom-drives/belso prior_binds=0
|
||||
2028-10-21T09:28:50.179+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:28:50.238+02:00 INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/hdd_1 stable=/mnt/felhom-drives/hdd_1 prior_binds=0
|
||||
2028-10-21T09:28:50.238+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:28:50.239+02:00 INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
|
||||
2028-10-21T09:28:50.239+02:00 INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
|
||||
2028-10-21T09:28:50.268+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:28:50.273+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:28:50.284+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:28:50.299+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:28:50.305+02:00 INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s guests_dir=/var/lib/felhom-agent/guests
|
||||
2028-10-21T09:28:50.305+02:00 INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
|
||||
2028-10-21T09:28:50.305+02:00 INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
|
||||
2028-10-21T09:28:50.306+02:00 INFO msg="local-api server listening" addr=169.254.253.1:8443
|
||||
2028-10-21T09:28:50.306+02:00 INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
2028-10-21T09:28:50.74+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:28:50.977+02:00 INFO msg="felhomsshd: config applied" port=8822 action=reload
|
||||
2028-10-21T09:28:51.069+02:00 INFO msg="felhomsshd belt: set synced" set=ssh_port elements=8822
|
||||
2028-10-21T09:28:51.232+02:00 INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=1
|
||||
2028-10-21T09:28:51.591+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:28:51+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:28:54.453+02:00 INFO msg="lanresolver: guest IP not yet leased — skipping (will retry)" vmid=9201
|
||||
2028-10-21T09:29:00.307+02:00 WARN msg="local-api: storage view unavailable for class hints" err="storage: NodeStorage: proxmox: GET /nodes/felhom/storage: Get \"https://127.0.0.1:8006/api2/json/nodes/felhom/storage\": context canceled"
|
||||
2028-10-21T09:29:00.308+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=368
|
||||
2028-10-21T09:29:01.55+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=511
|
||||
2028-10-21T09:29:03.995+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=629
|
||||
2028-10-21T09:29:08.673+02:00 INFO msg="local-api: same-disk" vmid=9201 a=/mnt/sys_drive b=/mnt/felhom-drives/belso different=false proven=true reason=same_disk
|
||||
2028-10-21T09:29:08.673+02:00 DEBUG msg="local-api: request" method=POST path=/disks/same-disk status=200 duration_ms=510
|
||||
2028-10-21T09:29:09.223+02:00 INFO msg="local-api: same-disk" vmid=9201 a=/mnt/sys_drive b=/mnt/felhom-drives/hdd_1 different=true proven=false reason=vm
|
||||
2028-10-21T09:29:09.223+02:00 DEBUG msg="local-api: request" method=POST path=/disks/same-disk status=200 duration_ms=549
|
||||
2028-10-21T09:29:10.82+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:29:10.935+02:00 INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/belso stable=/mnt/felhom-drives/belso prior_binds=1
|
||||
2028-10-21T09:29:10.936+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:29:11.056+02:00 INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/hdd_1 stable=/mnt/felhom-drives/hdd_1 prior_binds=1
|
||||
2028-10-21T09:29:11.057+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:29:14.138+02:00 DEBUG msg="local-api: request" method=GET path=/debug/logs status=200 duration_ms=0
|
||||
2028-10-21T09:29:18.982+02:00 INFO msg="fast-tick: desired-state unapplied — pulsing out-of-band reports" reason="desired-state not yet fetched" cadence=30s
|
||||
2028-10-21T09:29:18.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:29:19.647+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:29:20.351+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:29:21.604+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:29:21.737+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:29:21+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:29:30.71+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:29:30.776+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:29:30.86+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:29:31.931+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=598
|
||||
2028-10-21T09:29:33.005+02:00 DEBUG msg="local-api: /host/crash-guard served" vmid=9201 last_boot_at=2028-10-21T07:28:37Z last_boot_unclean=false tripped=true
|
||||
2028-10-21T09:29:33.005+02:00 DEBUG msg="local-api: request" method=GET path=/host/crash-guard status=200 duration_ms=0
|
||||
2028-10-21T09:29:48.985+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:29:49.663+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:29:50.291+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:29:50.301+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:29:50.369+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:29:50.385+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:29:50.797+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:29:50.868+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:29:50.935+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:29:51.732+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:29:51.864+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:29:51+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:29:54.285+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:30:02.546+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=613
|
||||
2028-10-21T09:30:03.15+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=603
|
||||
2028-10-21T09:30:03.805+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=475
|
||||
2028-10-21T09:30:10.75+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:30:10.81+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:30:10.882+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:30:18.986+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:30:19.627+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:30:20.354+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:30:21.662+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:30:21.791+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:30:21+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:30:30.73+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:30:30.807+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:30:30.876+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:30:32.543+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=610
|
||||
2028-10-21T09:30:33.155+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=612
|
||||
2028-10-21T09:30:48.983+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:30:49.812+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:30:50.284+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:30:50.29+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:30:50.369+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:30:50.373+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:30:50.889+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:30:50.984+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:30:51.058+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:30:51.906+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:30:52.059+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:30:52+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:30:54.279+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:31:02.727+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=790
|
||||
2028-10-21T09:31:03.5+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=772
|
||||
2028-10-21T09:31:03.916+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=582
|
||||
2028-10-21T09:31:10.727+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:31:10.789+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:31:10.85+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:31:18.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:31:19.618+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:31:20.346+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:31:21.609+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:31:21.756+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:31:21+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:31:30.715+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:31:30.792+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:31:30.858+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:31:32.534+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=598
|
||||
2028-10-21T09:31:33.158+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=623
|
||||
2028-10-21T09:31:48.985+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:31:49.628+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:31:50.29+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:31:50.313+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:31:50.365+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:31:50.38+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:31:50.8+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:31:50.89+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:31:50.972+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:31:51.807+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:31:51.946+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:31:51+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:31:54.382+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:32:02.56+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=624
|
||||
2028-10-21T09:32:03.311+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=750
|
||||
2028-10-21T09:32:03.8+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=470
|
||||
2028-10-21T09:32:10.756+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:32:10.837+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:32:10.906+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:32:18.985+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:32:19.666+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:32:20.353+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:32:21.721+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:32:21.869+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:32:21+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:32:30.79+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:32:30.849+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:32:30.916+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:32:32.55+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=614
|
||||
2028-10-21T09:32:33.155+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=603
|
||||
2028-10-21T09:32:48.985+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:32:49.644+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:32:50.283+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:32:50.296+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:32:50.37+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:32:50.38+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:32:50.8+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:32:50.943+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:32:51.025+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:32:52.006+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:32:52.149+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:32:52+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:32:54.587+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:33:02.55+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=617
|
||||
2028-10-21T09:33:03.175+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=624
|
||||
2028-10-21T09:33:03.807+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=476
|
||||
2028-10-21T09:33:10.755+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:33:10.819+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:33:10.88+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:33:18.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:33:19.637+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:33:20.353+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:33:21.644+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:33:21.797+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:33:21+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:33:30.766+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:33:30.837+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:33:30.903+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:33:32.575+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=639
|
||||
2028-10-21T09:33:33.189+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=613
|
||||
2028-10-21T09:33:48.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:33:49.684+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:33:50.285+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:33:50.299+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:33:50.311+02:00 DEBUG msg="fstrim: outside the daytime window — not looking" local="Sat 09:33"
|
||||
2028-10-21T09:33:50.364+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:33:50.371+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:33:50.77+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:33:50.855+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:33:50.935+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:33:51.821+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:33:51.968+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:33:51+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:33:54.451+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:34:02.61+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=674
|
||||
2028-10-21T09:34:03.211+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=599
|
||||
2028-10-21T09:34:03.436+02:00 DEBUG msg="local-api: request" method=GET path=/backup/tiers status=200 duration_ms=965
|
||||
2028-10-21T09:34:03.823+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=491
|
||||
2028-10-21T09:34:03.952+02:00 DEBUG msg="local-api: request" method=GET path=/backup/due status=200 duration_ms=514
|
||||
2028-10-21T09:34:04.714+02:00 DEBUG msg="local-api: request" method=GET path=/backup/due status=200 duration_ms=761
|
||||
2028-10-21T09:34:05.689+02:00 DEBUG msg="local-api: request" method=GET path=/backup/tiers status=200 duration_ms=973
|
||||
2028-10-21T09:34:05.689+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:34:05.69+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:34:07.036+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=1345
|
||||
2028-10-21T09:34:10.741+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:34:10.815+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:34:10.877+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:34:16.347+02:00 DEBUG msg="local-api: request" method=POST path=/backup status=202 duration_ms=0
|
||||
2028-10-21T09:34:16.348+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:34:16.779+02:00 INFO msg="backup: space preflight passed" vmid=9201 target=local last_archive_bytes=4628112507 need_bytes=6858882457 avail_bytes=62883717120
|
||||
2028-10-21T09:34:17.837+02:00 INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=local job=backup-9201-1855726456347094750
|
||||
2028-10-21T09:34:18.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:34:19.653+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:34:20.362+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:34:21.881+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:34:22.029+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:34:22+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:34:26.351+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:34:30.783+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:34:30.861+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:34:30.952+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:34:32.544+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=610
|
||||
2028-10-21T09:34:33.116+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=571
|
||||
2028-10-21T09:34:48.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:34:50.059+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:34:50.287+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:34:50.288+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:34:50.442+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:34:50.447+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:34:51.194+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:34:51.363+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:34:51.56+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:34:54.668+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:34:54.9+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:34:54+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:34:57.62+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:35:02.781+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=847
|
||||
2028-10-21T09:35:03.719+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=908
|
||||
2028-10-21T09:35:03.894+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=561
|
||||
2028-10-21T09:35:10.868+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:35:10.99+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:35:11.067+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:35:11.737+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:35:18.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:35:19.666+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:35:20.355+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:35:21.739+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:35:22.088+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:35:22.252+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:35:22+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:35:30.74+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:35:30.805+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:35:30.873+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:35:31.741+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:35:38.914+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=6981
|
||||
2028-10-21T09:35:39.508+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=592
|
||||
2028-10-21T09:35:41.742+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:35:48.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:35:49.698+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:35:50.29+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:35:50.301+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:35:50.387+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:35:50.579+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:35:50.991+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:35:51.107+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:35:51.23+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:35:51.743+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:35:52.187+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:35:52.346+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:35:52+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:35:55.134+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:36:01.747+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:36:02.555+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=622
|
||||
2028-10-21T09:36:03.183+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=627
|
||||
2028-10-21T09:36:04.41+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=1080
|
||||
2028-10-21T09:36:10.743+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:36:10.818+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:36:10.889+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:36:11.751+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:36:18.983+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:36:19.919+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:36:20.348+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:36:21.755+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:36:22.04+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:36:22.188+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:36:22+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:36:30.781+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:36:30.862+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:36:30.934+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:36:31.762+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:36:32.608+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=675
|
||||
2028-10-21T09:36:33.245+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=635
|
||||
2028-10-21T09:36:41.766+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:36:48.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:36:49.643+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:36:50.281+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:36:50.292+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:36:50.373+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:36:50.378+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:36:51.025+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:36:51.091+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:36:51.156+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:36:51.767+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:36:52.129+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:36:52.291+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:36:52+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:36:54.925+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:37:01.771+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:37:02.862+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=929
|
||||
2028-10-21T09:37:03.465+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=601
|
||||
2028-10-21T09:37:03.792+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=460
|
||||
2028-10-21T09:37:11.775+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:37:14.77+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:37:14.834+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:37:14.902+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:37:18.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:37:19.687+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:37:20.349+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:37:21.777+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:37:21.974+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:37:22.107+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:37:22+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:37:30.738+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:37:30.817+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:37:30.896+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:37:31.781+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:37:32.614+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=681
|
||||
2028-10-21T09:37:33.308+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=692
|
||||
2028-10-21T09:37:41.785+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:37:48.983+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:37:50.28+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:37:50.291+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:37:50.37+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:37:50.402+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:37:51.787+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:37:54.611+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:37:54.639+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:37:54.706+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:37:54.8+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:37:56.656+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:37:56.971+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:37:57.118+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:37:57+02:00 is after 2027-01-12T07:36:57Z"
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
apps: {'actualbudget': 'running', 'bookstack': 'running', 'immich': 'running', 'mealie': 'running', 'nextcloud': 'running', 'paperless-ngx': 'stopped', 'radicale': 'running', 'vaultwarden': 'running'}
|
||||
PAPERLESS - ess-ngx Digitális irattár - szkennelés, OCR és automatikus rendszerezés ● Leállítva Naprakész paperless.tester4.test Fájlok ↗ Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Ind
|
||||
PAPERLESS - ítás gombbal újra megpróbálhatod. Beállítások Adatok és mentés Útmutató Napló Működés Leállítva Indítás Belépés az alkalmazásba Az első bejelentkezéshez. Ha
|
||||
07:46:49Z Start paperless 200
|
||||
07:47:51Z paperless running
|
||||
BACKUPS: Teljes rendszermentés: kb. 04:30–08:30 között ✓ Adatmentés aktív – Távoli rendszermentés nincs beállítva 1 Adatbázis mentve
|
||||
teljes ment -> B (28%) Belső tárhely 4,23 GB / 54,8 GB (8%) Családi adatlemez 4,24 GB / 196 GB (2%) DB mentések 4 fájl Rendszermentés (teljes mentés) A teljes szerver — alkalmazások, beállítások és adatbázisok együtt — időszakos mentése, amelyből az egész készülék visszaállítható. Ezt a host-üg
|
||||
Rendszermentés (teljes mentés) A teljes szerver — alkalmazások, beállítások és adatbázisok együtt — időszakos mentése, amelyből az egész készülék visszaállítható. Ezt a host-ügynök készíti és kezeli. A host-ügynök jelenleg nem elérhető. Mentési időablak A mentések egymás után futnak: adatbázis-mentés, helyi másolat, távoli mentés, majd a teljes rendszermentés. Mentési időablak kezdete Mentés Adatbázis-mentés: 02:30 Helyi másolat: 03:30 Távoli mentés: 04:15 Teljes rendszermentés: kb. 04:30–08:30 között ✓ Adatmentés aktív – Távoli rendszermentés nincs beállítva 4 Adatbázis mentve
|
||||
07:54:05Z Rendszermentés (teljes mentés) A teljes szerver — alkalmazások, beállítások és adatbázisok együtt — időszakos mentése, amelyből az egész készülék visszaállítható. Ezt a host-ügynök készíti és kezeli. A host-ügynök jelenleg nem elérhető. Mentési időablak A mentések egymás után futnak: adatbázis-mentés, helyi másolat, távoli mentés, majd a teljes rendszermentés. Mentési időablak kezdete Mentés Adatbázis-mentés: 02:30 Helyi másolat: 03:30 Távoli mentés: 04:15 Teljes rendszermentés: kb. 04:30–08:30 között ✓ Adatmentés aktív – Távoli rendszermentés nincs beállítva 4 Adatbázis mentve
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
2026-10-21T07:45:25Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-10-21T07:45:32Z INFO [scheduler] Job agent-channel-health completed (took 7.206s)
|
||||
2026-10-21T07:46:32Z INFO [scheduler] Job agent-channel-health completed (took 7.239s)
|
||||
2026-10-21T07:47:33Z INFO [scheduler] Job agent-channel-health completed (took 7.55s)
|
||||
2026-10-21T07:48:34Z INFO [scheduler] Job agent-channel-health completed (took 8.814s)
|
||||
2026-10-21T07:49:34Z INFO [scheduler] Job agent-channel-health completed (took 8.3s)
|
||||
2026-10-21T07:49:39Z ERROR [quiesce] cycle error: check due: agentapi: GET /backup/tiers: Get "https://169.254.253.1:8443/backup/tiers": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
||||
2026-10-21T07:50:33Z INFO [scheduler] Job agent-channel-health completed (took 7.25s)
|
||||
2026-10-21T07:51:32Z INFO [scheduler] Job agent-channel-health completed (took 7.24s)
|
||||
2026-10-21T07:52:32Z INFO [scheduler] Job agent-channel-health completed (took 7.249s)
|
||||
2026-10-21T07:53:32Z INFO [scheduler] Job agent-channel-health completed (took 7.229s)
|
||||
agent-logs http 200 {'data': {'entries': [{'timestamp': '2026-10-21T09:44:10.609+02:00', 'level': 'INFO', 'message': 'msg="felhom-agent daemon starting" version=0.161.0 host_id=Tester4-816e21 hub_url=https://hub.felhom.eu interval_s=900'}, {'timestamp': '2026-10-21T09:44:10.615+02:00', 'level': 'INFO', 'message': 'msg=
|
||||
09:44:26.268 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
09:44:26.403 INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/belso stable=/mnt/felhom-drives/belso prior_binds=0
|
||||
09:44:26.403 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
09:44:26.558 INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
|
||||
09:44:26.565 INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
|
||||
09:44:26.593 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
09:44:26.676 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
09:44:26.794 INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s guests_dir=/var/lib/felhom-agent/guests
|
||||
09:44:26.794 INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
|
||||
09:44:26.794 INFO msg="local-api server listening" addr=169.254.253.1:8443
|
||||
09:44:26.795 INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
|
||||
09:44:26.795 INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
09:44:26.85+ DEBUG msg="pbs: credential probe inconclusive (not a rejection)" storage=felhom-pbs err="pbs: GET /version: Get \"https://10.77.0.1:8007/api2/json/version\": context deadline exceeded"
|
||||
09:44:26.85+ DEBUG msg="pbs: live list failed; using last-known-good" datastore=felhom-offsite err="pbs: GET /admin/datastore/felhom-offsite/snapshots?ns=Tester4: Get \"https://10.77.0.1:8007/api2/json/admin/datastore/felhom-offsite/snapshots?ns=Tes
|
||||
09:44:27.752 INFO msg="felhomsshd: config applied" port=8822 action=reload
|
||||
09:44:27.853 INFO msg="felhomsshd belt: set synced" set=ssh_port elements=8822
|
||||
09:44:33.405 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
09:44:34.913 DEBUG msg="hub: report sent" guests=1 duration_ms=23389 blocked=false desired_generation=2 has_signed_ops=false
|
||||
09:44:34.913 DEBUG msg="desired: generation advanced — fetching desired-state" have_generation=0 envelope_generation=2
|
||||
09:44:34.926 DEBUG msg="desired: fetched" generation=2 duration_ms=13
|
||||
09:44:34.926 INFO msg="desired: updated from hub" generation=2 guests=0
|
||||
09:44:35.02+ INFO msg="felhomsshd: operator authorized_keys updated" user=felhom-op present=true
|
||||
09:44:35.115 INFO msg="felhomsshd belt: set synced" set=operator_ips elements=10.77.0.250
|
||||
09:44:37.818 INFO msg="local-api: same-disk" vmid=9201 a=/mnt/sys_drive b=/mnt/felhom-drives/belso different=false proven=true reason=same_disk
|
||||
09:44:37.818 DEBUG msg="local-api: request" method=POST path=/disks/same-disk status=200 duration_ms=7228
|
||||
09:44:41.573 WARN msg="pbs: verify trigger failed; reporting current snapshot state" datastore=felhom-offsite err="pbs: POST /admin/datastore/felhom-offsite/verify: Post \"https://10.77.0.1:8007/api2/json/admin/datastore/felhom-offsite/verify\": co
|
||||
09:44:55.34+ DEBUG msg="local-api: /host/crash-guard served" vmid=9201 last_boot_at=2026-10-21T07:43:58Z last_boot_unclean=false tripped=true
|
||||
09:44:55.34+ DEBUG msg="local-api: request" method=GET path=/host/crash-guard status=200 duration_ms=0
|
||||
09:45:03.144 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=7397
|
||||
09:45:11.574 WARN msg="pbs: snapshot list failed" datastore=felhom-offsite err="pbs: GET /admin/datastore/felhom-offsite/snapshots?ns=Tester4: Get \"https://10.77.0.1:8007/api2/json/admin/datastore/felhom-offsite/snapshots?ns=Tester4\": context dea
|
||||
09:45:32.956 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=7204
|
||||
09:46:11.039 WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=felhom-pbs err="proxmox: GET /nodes/felhom/storage/felhom-pbs/content: Get \"https://127.0.0.1:8006/api2/
|
||||
09:46:11.039 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=15001
|
||||
09:46:25.467 DEBUG msg="local-api: request" method=GET path=/backup/tiers status=200 duration_ms=14427
|
||||
09:46:40.054 INFO msg="local-api: same-disk" vmid=9201 a=/mnt/felhom-drives/hdd_1 b=/mnt/sys_drive different=true proven=false reason=vm
|
||||
09:49:26.796 DEBUG msg="fstrim: outside the daytime window — not looking" local="Wed 09:49"
|
||||
09:49:39.71+ WARN msg="local-api: storage view unavailable for the tier presence report" target=felhom-pbs err="storage: NodeStorage: proxmox: GET /nodes/felhom/storage: Get \"https://127.0.0.1:8006/api2/json/nodes/felhom/storage\": context cancele
|
||||
09:51:12.002 DEBUG msg="local-api: request" method=GET path=/debug/logs status=200 duration_ms=1
|
||||
09:54:39.068 INFO msg="guest-power: watchdog alive" sweeps_since_boot=10 guests_evaluated=1 currently_stopped=0
|
||||
09:54:40.09+ INFO msg="controller-supervisor: alive" sweeps_since_boot=20 guests_evaluated=1 controllers_not_running=0
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
# ep0 wg0 peers, handshake age in seconds, read-only, 2026-10-21T07:56:57Z
|
||||
10.77.0.2/32 handshake_age_s=7
|
||||
10.77.0.250/32 handshake_age_s=116
|
||||
10.77.0.3/32 handshake_age_s=110
|
||||
10.77.0.4/32 handshake_age_s=65
|
||||
10.77.0.5/32 handshake_age_s=1432134
|
||||
10.77.0.6/32 handshake_age_s=970
|
||||
@@ -0,0 +1,37 @@
|
||||
# B6 ★8 — the clock wrong at boot: −2 years, then +2 years, then the real time (2026-10-21 07:15–07:58 real UTC)
|
||||
|
||||
Method (`00-expected.txt`): NTP dropped at the VM's tap (314 packets dropped in all — positive control); clean restarts
|
||||
with the RTC set; clock control = the box's HTTP Date header (`01-run.txt`).
|
||||
|
||||
## −2 years (RTC 2024-10-21)
|
||||
|
||||
- **The box came up at 2026-04-13 19:38, not 2024** — the operating system will not run before its own build date and
|
||||
raised the clock to it. So the real test was „6 months behind".
|
||||
- The hub and the registry refuse the box: „x509: certificate … not yet valid". The dashboard says only **„Hub nem
|
||||
elérhető"** — not that the clock is wrong, although the log knows the exact reason.
|
||||
- **Paperless crash-looped and the box stopped it** with a restore hold („crash_loop — 7 in 10m0s; STOPPING it"); its
|
||||
warning event could not reach the hub, so nobody was told. Its log was empty when read (`03-…`).
|
||||
- Nothing was deleted; no backup ran.
|
||||
|
||||
## +2 years (RTC 2028-10-21)
|
||||
|
||||
- Every tier looked 2 years old: a local whole-box catch-up ran at once (`vzdump-lxc-9201-2028_10_21-09_34_16`, 4.6 GB).
|
||||
The weekly PBS tier had not started when the clock was put back. Hub and registry refused the box („certificate has
|
||||
expired"). No deletion: off-site retention is the hub's (decision 68), local keep-daily 3 kept all three archives.
|
||||
|
||||
## Back to the real time (07:43:47 real) — two lasting effects
|
||||
|
||||
1. **The box's WireGuard tunnel to ep0 is dead** (`07-…`, `08-ep0-wg-handshakes.txt`): every PBS call from the agent
|
||||
times out; the agent answers the controller so slowly that the backups page says „A host-ügynök jelenleg nem
|
||||
elérhető" 10+ minutes after the return. ep0's side, read-only: Tester4's last handshake was 07:40:47 — while its
|
||||
clock said 2028 — and none since, while the other peers shook hands 21–96 s ago. WireGuard refuses a handshake whose
|
||||
timestamp is older than the last one it accepted, so the tunnel stays dead until the real date passes 2028 or the
|
||||
peer is reset on ep0. **→ R-1201.** Not reset (ep0 is read-only in this run).
|
||||
2. **The local whole-box tier's newest archive is dated 2028-10-21.** By the code (`localapi/server.go:1187`), the due
|
||||
rule answers „within cadence" until then — the D3/T11 shape (R-1189 a). The live answer was not read: the controller
|
||||
asks only inside its night window, and the box dies in B7 before tonight.
|
||||
- Paperless stayed stopped after the return with a clear sentence („leállítottuk, mert újra és újra összeomlott … Az
|
||||
Indítás gombbal újra megpróbálhatod"); one Start → running in 1 minute. **→ R-1202** (with the „Hub nem elérhető" text).
|
||||
|
||||
**Verdict: FINDINGS.** A box that runs once with its clock ahead loses its DR tunnel (R-1201, P2) and, by the code, its
|
||||
whole-box schedule (R-1189 a); a box with its clock behind stops an app silently and blames the hub (R-1202, P3).
|
||||
@@ -84,3 +84,4 @@
|
||||
| 05:27–05:45 | due checks | R-1019 closed (demo-felhom read-only, two channels), R-895 first verdict recorded, R-435 moved to 10-27 (logs gone) |
|
||||
| 05:40 | mail | the nightly DooPlex secret sweep named three rehearsal secrets in session records: OLD_BIND_TOKEN, BIND_TOKEN, SETUP_CODE — my mail reads (full text of the bind and setup mails) carried them in; all three are one-time and already spent. From now on mail is read names-only |
|
||||
| 05:58–07:13 | B6 ★7 | DST live: the first 02:30 skipped, the second ran once (01:30:00Z box), PASS; side effects of the 4-day jump: a local whole-box archive and off-site snapshots dated 2026-10-25; W30 |
|
||||
| 07:15–07:58 | B6 ★8 | −2 y (OS raised it to 2026-04-13): „Hub nem elérhető" only, Paperless stopped silently → R-1202; +2 y: whole-box archive dated 2028; back to real time: ep0 WireGuard tunnel dead (handshake replay) → R-1201 (P2); R-1189 (a) half-live |
|
||||
|
||||
@@ -195,10 +195,11 @@ stopping line that lies.
|
||||
| **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC |
|
||||
| **R-1187** | Backup & restore | P3 | **`.fab` export/import candidates from the data-loss hunt, not yet proven:** (a) an export silently drops a drive folder that holds a symlink (`appexport/fabplan.go:292`, `export.go:714`); (b) the import never sets file owners and drops symlinks (`restore.go:1157-1176`); (c) import into a STOPPED app merges into the existing volumes, and the default export (stop ticked) carries no SQL dump (`restore.go:361-374`, `export.go:262,366`, `appbackup/dbdump.go:142`); (d) no concurrency guard, no undo, no record (`restore.go:181-186`, `web/handler_export.go:312-353`); (e) PROVEN and split out as R-1191 (the import drops the family gate); (f) it trusts the bundle's HDD_PATH (`restore.go:589-631`); (g) the DB import runs psql without ON_ERROR_STOP and misses postgis (`restore.go:1052-1053, 928-937`). Read by reviewer 1; main session spot-checked (a); (e) proven live → R-1191. Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Prove (c) first on 9202 with a throwaway app; split into rows only what reproduces | CC |
|
||||
| **R-1188** | Backup & restore | P3 | **Restore-path candidates from the data-loss hunt, not yet proven (one confirmed in code):** (a) PROVEN LIVE → R-1195 (a power cut mid-restore restarts the app on the backup's data; the undo copy is not used); (b) a failed `volume rm` is ignored before the untar (`restore.go:218`); (c) PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-R12-leftover-temp/`): an interrupted second-drive whole restore leaves `<file>.felhom-restoring`, and EVERY retry fails „fájlmásolás sikertelen: … file exists" with the file never back (`tier2_whole.go:86-87`, O_EXCL); (d) the full off-site download has a 30-min cap per press and the household is not told to retry (`web/offbox_handlers.go:455`); (e) place and full restore re-read the LATEST snapshot instead of the downloaded one (`offbox_restore.go:675-679`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Fix (c) now (remove a stale temp of the same name before the copy); prove (a) with the B6 power-cut-mid-restore scenario | CC |
|
||||
| **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt, not yet proven:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it, and the hub reads the same clock (agent `localapi/server.go:1178-1192`, `backup/runner.go:607-629`, hub `monitor/deadline.go:243`; R-973 cap counts future runs `backup_cap.go:61`); (b) the whole-guest start creeps ~5 min/night until a night is skipped silently (`server.go:1184`, `quiesce.go:150, 807-818`) — PARTLY SUPPORTED by demo-hp's agent journal 2026-09-30…10-20: completions drift 04:36→04:50 between resets, and THREE nights had no local whole-box backup (Oct 2, 6, 9), each right after an off-cycle daytime/evening run (Oct 1 22:15, Oct 5 11:24, Oct 8 22:25) — the 24-hour „not due yet" rule skips the next night; nothing tells anyone; (c) the whole-guest quiesce stops apps mid-restore or mid-update (`quiesce.go:213-275, 565`); (d) CONFIRMED in code: the off-site proof and integrity check are plain daily jobs outside the catch-up (`nightchain.go` header: the catch-up runs only the three backup legs), so they never run on a box that is off at 05:30–06:00, while the comment at `cmd/controller/main.go:1575-1577` says the opposite; (e) CONFIRMED in code: the hub's 05:00 check wants a dump „since Budapest midnight" (`hub/internal/monitor/deadline.go:416-419, 556-565`), so an evening backup time (which the banner suggests) raises a false `expected_dbdump_missed` error every morning — and on 2026-10-25 the check also runs at 04:00 (R-1183); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — design question; (g) spring forward 2027-03-28 fires dump and second copy together; (h) self-update busy check before the pull only (`updater.go:532-545, 764`); (i) a double-clicked restore shows failed while it runs (`web/handlers.go:1853-1874`). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`. | VERIFY | — | (a) is B6 #8's live test; (b) read the agent journal on demo-hp; the rest by unit tests | CC |
|
||||
| **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt, not yet proven:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it, and the hub reads the same clock (agent `localapi/server.go:1178-1192`, `backup/runner.go:607-629`, hub `monitor/deadline.go:243`; R-973 cap counts future runs `backup_cap.go:61`); (b) the whole-guest start creeps ~5 min/night until a night is skipped silently (`server.go:1184`, `quiesce.go:150, 807-818`) — PARTLY SUPPORTED by demo-hp's agent journal 2026-09-30…10-20: completions drift 04:36→04:50 between resets, and THREE nights had no local whole-box backup (Oct 2, 6, 9), each right after an off-cycle daytime/evening run (Oct 1 22:15, Oct 5 11:24, Oct 8 22:25) — the 24-hour „not due yet" rule skips the next night; nothing tells anyone; (c) the whole-guest quiesce stops apps mid-restore or mid-update (`quiesce.go:213-275, 565`); (d) CONFIRMED in code: the off-site proof and integrity check are plain daily jobs outside the catch-up (`nightchain.go` header: the catch-up runs only the three backup legs), so they never run on a box that is off at 05:30–06:00, while the comment at `cmd/controller/main.go:1575-1577` says the opposite; (e) CONFIRMED in code: the hub's 05:00 check wants a dump „since Budapest midnight" (`hub/internal/monitor/deadline.go:416-419, 556-565`), so an evening backup time (which the banner suggests) raises a false `expected_dbdump_missed` error every morning — and on 2026-10-25 the check also runs at 04:00 (R-1183); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — design question; (g) spring forward 2027-03-28 fires dump and second copy together; (h) self-update busy check before the pull only (`updater.go:532-545, 764`); (i) a double-clicked restore shows failed while it runs (`web/handlers.go:1853-1874`). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`. | VERIFY **(a) half-live 2026-10-21 (B6 #8):** the box wrote a local whole-box archive dated 2028-10-21 while its clock was 2 years ahead (`s8-wrong-clock/README.md`); by `localapi/server.go:1187` the local tier is „within cadence" until then. The due answer itself was not read live (the box died in B7 before its night window). The same run found R-1201 (the ep0 tunnel). | — | (a) is B6 #8's live test; (b) read the agent journal on demo-hp; the rest by unit tests | CC |
|
||||
| **R-1190** | Backup & restore | P3 | **Key and retention candidates from the data-loss hunt, not yet proven:** (a) a recovery ceremony while off-site is not configured seals a package with no repo password and supersedes the good one (`escrow_handlers.go:217`, agent `main.go:2975-2981`, hub `store.go:3392`); (b) the global „create your recovery code" bar links to the ceremony the backups page hides during a recovery offer (`escrow_banner.go:23-33` vs `backups_remote.html:144-160`); (c) after a reuse-path reinstall the new PBS key K is never escrowed and nothing compares fingerprints (`felhom-pbs-apply:99`, `escrow_confirm.go:182-205`); (d) whole-guest off-site copies run before any escrow exists (design question vs 07 §6.1's Tier-3 gate); (e) the escrow PUT is two statements; (f) the operator's off-site re-issue resets the provider password before saving it (hub `offsite.go:217-245`); (g) „Megőrzött adatok" can list a running app's live folder when its app.yaml is unreadable (`kept.go:220-235, 330-402`); (h) a unit restore can point an app back at an old drive and F5 then prunes the current drive's unit (`backup.go:1517-1574`). Evidence `audits/dataloss-hunt-2026-10-20/C1-keys-candidates.md`, `C1-delete-retention-candidates.md`. | VERIFY | — | Unit tests first for (a), (e), (g) | CC |
|
||||
| **R-1198** | Backup & restore | P3 | **A backup disk the family adds later is never offered as the place for the whole-box backup; the page keeps offering the disk that holds the apps' own data.** Rehearsal 2026-10-20 B6 #13 (`audits/rehearsal-2026-10-20/B6-failures/s13-second-drive/`): a 50 GB disk plugged in and initialised as „Mentolemez" → the backups page still offers only „Családi adatlemez — Kijelölöm"; `internal/web/backup_target_offer.go:101` `firstOfferableDrive` offers the FIRST eligible drive, never a choice, and the drive's card has no „use for backups". | OPEN | — | Offer every eligible drive (largest empty first) and let the household pick | CC |
|
||||
| **R-1200** | Backup & restore | P3 | **A box installed in the afternoon gets no whole-box backup on its first night: the install's own backup starts the 24 h clock, so the family's whole first day reaches a whole-box copy only on the SECOND night (~38 h after install).** Rehearsal 2026-10-20 B5 (`audits/rehearsal-2026-10-20/B5-night/README.md`): Tester4's install backup at 12:09Z; at the 02:30–06:30Z window the agent answered „within cadence window" (`felhom-agent internal/localapi/server.go:1187`, `age >= tier.Cadence`, default 24 h). R-899 (closed) fixed the same shape for a daytime PRESS only. The other tiers did run that night (database copy, second drive, off-site — all PASS), so app data was covered; the whole-box (DR) tier was not. | **OPEN** — a design question: should the first scheduled night always run (like a press), or is one tier enough on night one? | — | decide; if „always run": treat the install backup like a press in the due rule, with a test on the first night | CC |
|
||||
| **R-1201** | Backup & restore | P2 | **A box that runs once with its clock ahead loses its WireGuard tunnel to ep0 — the PBS whole-box copies and the operator's door — until the real date passes the wrong one, and nothing says why.** Rehearsal 2026-10-20 B6 #8 (`audits/rehearsal-2026-10-20/B6-failures/s8-wrong-clock/`): Tester4 booted with its clock at 2028-10-21 for 13 min, then the real time. ep0's `wg show` (read-only): Tester4's last handshake 07:40:47Z (clock at 2028), none in the 14 min after the return, the other peers 21–96 s. WireGuard refuses a handshake initiation whose TAI64N timestamp is not newer than the last one accepted from that peer, so the peer stays dead until 2028 or until ep0 resets it. Effects seen: every agent PBS call times out (15 s), the controller's due-check errors, the backups page says „A host-ügynök jelenleg nem elérhető". Likely triggers in the field: a dead CMOS battery, a hand-set BIOS clock, a VM restored with a wrong RTC. | **OPEN** — reproduced once live; not reset on ep0 (read-only in this run; the peer still shows the stale handshake). The box is destroyed in B7, which ends it for Tester4. | — | design: detect it (the hub sees the peer's handshake age) and heal it (re-key the peer, or ep0 resets the stored timestamp on a hub-signed request); a test with a fake clock | CC |
|
||||
|
||||
## Storage & devices — 10 rows (P3 4, P4 6)
|
||||
|
||||
@@ -266,6 +267,7 @@ stopping line that lies.
|
||||
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
|
||||
| **R-1199** | Monitoring & notifications | P2 | **The household's notification page shows disk-warning, drive-disconnected and backup-failed mails switched ON, but on the rehearsal day only the OPERATOR got them — the household's mailbox received one alarm (a failed off-site run) out of nine.** Rehearsal 2026-10-20, read from the Gmail catch-all (the second channel): to `tester4@` only the setup code and „A biztonsági mentés sikertelen!" (17:01); to `admin@` also `disk_warning` 89 % (18:36 — the box logged „notifying the customer"), `storage_disconnected` ×2, `storage_reconnected` ×2, `backup_run_failures` (3 of 8 apps), `app_oom`, `host/node_stale`. The box's page (`/settings/notifications`) has `event_disk_warning`, `event_storage_disconnected`, `event_backup_failed`, `event_crossdrive_failed` … ticked; the hub sends household mail only by ITS OWN stored prefs (`hub/internal/notify/dispatcher.go` `processCustomer`: `isEventEnabled(prefs.EnabledEvents, …)`) — whether the two lists disagree is not read (the token cannot read the customer record). Evidence `audits/rehearsal-2026-10-20/B6-failures/notifications-defaults.txt`, `journal.md`. | VERIFY | an operator read of Tester4's hub notification prefs and notification log | Compare the hub's `notification_prefs` / `notification_log` for Tester4 with the box's page; if they differ, make one the source of truth | CC |
|
||||
| **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC |
|
||||
| **R-1202** | Monitoring & notifications | P3 | **A box whose clock is behind blames the hub and stops an app without telling anyone.** Rehearsal 2026-10-20 B6 #8: RTC set 2 years back (the OS raised it to 2026-04-13): the dashboard shows only „Hub nem elérhető" while the log says „x509: certificate … not yet valid: current time 2026-04-13…"; Paperless crash-looped and the box stopped it with a restore hold; the `app_stopped_unhealthy` event failed (hub unreachable), so neither the household nor the operator heard. After the real time came back the app stayed stopped until a Start (clear sentence, 1 min). | **OPEN** | — | say „the box's clock is wrong (it says <date>)" when TLS fails with not-yet-valid/expired against a hub that answers; keep the stop event queued until the hub is reachable | CC |
|
||||
| **R-340** | Monitoring & notifications | P3 | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice** **REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc/<pid>/fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC |
|
||||
| **R-388** | Monitoring & notifications | P3 | **PRODUCT DECISION (not a defect): the customer notification model is the wrong shape, and the settings page grows by one toggle per detector.** The operator's framing, recorded verbatim 2026-08-23: *"A customer should be notified only about things they can act on or are responsible for — the drive they unplugged, the storage they filled. **A failed backup is our incident, not theirs.** The intended shape is that we detect it, we tell them we noticed and are dealing with it, and they are not handed an error they cannot solve. The subscription should feel like being looked after, not like being on call."* Today's page is the opposite shape — one switch per detector, and it **grew from 12 to 15 in a single session** (one new alarm plus two compound toggles split into four). That growth is the argument, not an aside: a page that grows per detector keeps asking a household to make engineering decisions. | **READY** — **2026-10-12 evening: slice 1 delivered** (operator Q1/Q2 „as picked", `09` §3 decisions 234–235): the page shows three group switches; the per-detector switches and the cooldown sit under „Részletek"; an „egyedi" group is saved back unchanged; the default list fills groups 1 and 3 (households with no saved list), group 2 stays „egyedi" (`app_start_failed` off, R-329). Tests red-proved. controller 0.309.0, delivered 2026-10-12 18:08 to demo-hp, demo-felhom and Tester 1 (`controller-0.309.0-delivery.txt`). **READY** — **2026-10-12: the design is written** (`08` §8.1): three group switches („A gépem és a meghajtóim", „Az alkalmazásaim", „Amit mi intézünk") mapped onto today's 18 detectors; option A picked; slice 1 changes no wire. Two questions for the operator (group 3 default; the cooldown field). **READY** — operator ruling 2026-10-12 (S5, `09` §3 decision 200): design now (one page: options, a pick, the first slice), build in slices later. **OPEN — DIRECTION, operator's call** | a decision on scope; nothing here is a bug | Slice 2 (hub): group 3's mails say „we noticed X and are handling it" instead of the raw error | CC (+ operator: two answers) |
|
||||
| **R-435** | Monitoring & notifications | P3 | **The snapshot-drop detector is blind to a single-app deletion — the exact shape the box can produce.** `snapshotDropFraction = 0.5` and `snapshotDropFloor = 5` (`hub/internal/monitor/offsite.go`) require a fall of MORE than half the previous count. demo-hp's baseline is **69** across 9 apps, so ~35 snapshots must go before it speaks; **one app's tag is ~9 and is invisible.** `offbox.go:1388` runs `forget --prune` **grouped by host,tags** — a per-tag wipe is precisely the shape a faulty retention or a targeted deletion produces. **This is deliberate, not accidental:** the constant's own comment argues the insensitivity, and *"a detector that cries wolf is switched off within a fortnight"* is a lesson this project paid for. **So this row is NOT a demand to lower the threshold.** It is a demand that the blind spot be written where the operator reads it, because "an unexplained fall is noticed within a day" (R-431, and `STATUS.md`) is true only of falls above half. **Discovered by arithmetic while planning the drill's Phase 2/3 pairing, which could not have worked:** Phase 2 deletes one app and Phase 3 expects the alarm to fire. **LIMITATION NOW WRITTEN INTO THE ALARM'S OWN DOCUMENTATION, hub v0.111.1** — the comment above `snapshotDropFraction` in `hub/internal/monitor/offsite.go` now states what the detector does NOT see, with the demo-hp arithmetic and the `forget --prune` grouping that makes the blind spot sit on the most likely single-app failure. **It also says explicitly that the numbers must NOT be lowered to "fix" this** and that per-app detection needs a SECOND signal keyed on the per-tag count. **The row stays OPEN because documenting a blind spot is not covering it** — and because `STATUS.md` and R-431 both still say "noticed within a day", which is true only of falls above half. **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-435.md` — the box's key can no longer delete (decisions 68–69); deletion through any other credential still needs >50 % to alarm. Question D7 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D7 (`09` §3 decision 191):** yes — option C, an error mail when even one snapshot disappears outside a hub-opened clean-up window (pinned tiers). | **VERIFY** — **2026-10-12 read: not provable today.** No mail for a snapshot drop reached the operator since 2026-10-11 (Gmail), but the window itself could not be read: no hub page shows `offsite_windows`, Tester 1's controller log restarted with 0.308.0 at 13:49Z, and the guest-agent path to its VM did not answer. Re-dated after the next weekly clean-up. **VERIFY — 2026-10-09: DELIVERED in hub 0.144.0. Live proof NOT run: a window is granted only when the box asks for one, and Tester 1's weekly clean-up is next due ~2026-10-12 (last window 2026-10-05); prove it then (no alarm inside the window; the hub's window row as control).** **VERIFY — built on hub main 2026-10-08 (D7, decision 191), ships tomorrow; closes after the 9202 proof (a hand-granted window removes N, no alarm; the hub's window row as the control).** **OPEN — documented in code v0.111.1; the coverage gap itself is unclosed** **2026-10-05 (burn-down night): NEEDS A DESIGN.** Covering one-app deletions needs per-app snapshot counts from the controller and a per-tag threshold measured on real counts — two repos, a mechanism nobody has measured. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the night session that reached it was not briefed for this read (a Tester 1 log and the mailbox); the window row and the mails stay readable a day later. **2026-10-21: due date moved 10-21 → 10-27** — Tester 1's weekly window was 2026-10-19 02:15:53Z (its box log, 2026-10-21 02:16), but both channels for its counts are gone: Tester 1's controller restarted 2026-10-20 08:51Z (the floor) and the hub pod 2026-10-20 06:03Z, so the before/after lines no longer exist. The next window is ~2026-10-26 02:15Z; read the box log and the hub log the next morning, BEFORE any restart. | — | 2026-10-20: read Tester 1's controller log (window opened/closed, count before/after) and the operator mailbox (no drop alarm) the morning after its weekly clean-up | CC |
|
||||
|
||||
Reference in New Issue
Block a user