rehearsal 2026-10-20: B6 #8 wrong clock — R-1201 (ep0 tunnel dead after a clock ahead, P2), R-1202 (clock behind: silent app stop, blames the hub)
This commit is contained in:
@@ -0,0 +1,19 @@
|
||||
minus2y=2024-10-21T07:13:35
|
||||
down 07:15:30
|
||||
update VM 9320: -startdate 2024-10-21T07:13:35
|
||||
started with RTC 2024-10-21T07:13:35 at real 07:15:33
|
||||
real 2026-10-21T07:16:16Z box Mon, 13 Apr 2026 19:38:33 GMT
|
||||
real 2026-10-21T07:25:31Z box Mon, 13 Apr 2026 19:47:48 GMT
|
||||
plus2y=2028-10-21T07:28:25
|
||||
down 07:30:20
|
||||
update VM 9320: -startdate 2028-10-21T07:28:25
|
||||
started with RTC 2028-10-21T07:28:25 at real 07:30:23
|
||||
real 2026-10-21T07:31:03Z box Sat, 21 Oct 2028 07:29:05 GMT
|
||||
real 2026-10-21T07:39:57Z box Sat, 21 Oct 2028 07:38:00 GMT
|
||||
iifname "tap9320i0" udp dport 123 counter packets 314 bytes 23864 drop
|
||||
iifname "tap9320i0" tcp dport 4460 counter packets 0 bytes 0 drop
|
||||
down 07:43:44
|
||||
update VM 9320: -delete startdate
|
||||
started with real RTC at 07:43:47
|
||||
ntp on 07:43:48
|
||||
real 2026-10-21T07:44:31Z box Wed, 21 Oct 2026 07:44:31 GMT
|
||||
+200
@@ -0,0 +1,200 @@
|
||||
2026-04-13T19:38:28Z INFO [infra] filebrowser compose exists — starting without regenerating
|
||||
2026-04-13T19:38:29Z INFO [web] sessions: restored 124 dashboard session(s) from disk (dropped 0 expired, 0 malformed)
|
||||
2026-04-13T19:38:29Z INFO [web] Auth: using password from settings.json
|
||||
2026-04-13T19:38:29Z INFO [scheduler] Registered periodic job: disk-health-check (every 1h0m0s)
|
||||
2026-04-13T19:38:29Z INFO [scheduler] Registered periodic job: agent-channel-health (every 1m0s)
|
||||
2026-04-13T19:38:29Z INFO Web UI listening on :8080
|
||||
2026-04-13T19:38:29Z INFO [web] FileBrowser: drive /mnt/felhom-drives/hdd_2 not mounted — skipping userdata skeleton
|
||||
2026-04-13T19:38:29Z INFO [backup] Discovered 4 databases
|
||||
2026-04-13T19:38:30Z INFO [monitor] Health check: status=fail
|
||||
2026-04-13T19:38:30Z INFO [web] FileBrowser sync — no config/compose change, ensured running without recreate (3 storage path(s))
|
||||
2026-04-13T19:38:30Z INFO [infra] connected felhom-controller to felhom-filebrowser (the FileBrowser default-login check, R-1126)
|
||||
2026-04-13T19:38:31Z INFO [backup] Backup status cache refreshed
|
||||
2026-04-13T19:38:32Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:33Z INFO [report] Building system report
|
||||
2026-04-13T19:38:35Z INFO [samba] applying samba stack: shares=2 config_changed=false
|
||||
2026-04-13T19:38:36Z INFO [gate] boot 1776109082-1457: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/hdd_1] before recreating drive-backed apps
|
||||
2026-04-13T19:38:36Z INFO [gate] boot 1776109082-1457: live bind confirmed — recreating drive-backed app immich (state=degraded) onto /mnt/felhom-drives/hdd_1
|
||||
2026-04-13T19:38:36Z INFO [stacks] Stopping stack: immich
|
||||
2026-04-13T19:38:37Z INFO [stacks] Stack immich stopped successfully (took 1.0s)
|
||||
2026-04-13T19:38:37Z INFO [stacks] Status refresh: 15 containers across 68 stacks
|
||||
2026-04-13T19:38:37Z INFO [stacks] Starting stack: immich
|
||||
2026-04-13T19:38:37Z INFO [stacks] Status refresh: 15 containers across 68 stacks
|
||||
2026-04-13T19:38:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:38Z INFO Health probes: 2 ok (of 2 probed)
|
||||
2026-04-13T19:38:39Z WARN Event push failed after 3 attempts (controller_started/info): Post "https://hub.felhom.eu/api/v1/event": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:38:39Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:38:42Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:47Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:48Z INFO [monitor] Health check: status=ok
|
||||
2026-04-13T19:38:48Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:48Z INFO Health probes: 1 ok (of 1 probed)
|
||||
2026-04-13T19:38:48Z INFO [stacks] Stack immich started successfully (took 11.1s)
|
||||
2026-04-13T19:38:49Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:49Z INFO [gate] boot 1776109082-1457: live bind confirmed — recreating drive-backed app nextcloud (state=degraded) onto /mnt/felhom-drives/hdd_1
|
||||
2026-04-13T19:38:49Z INFO [stacks] Stopping stack: nextcloud
|
||||
2026-04-13T19:38:49Z INFO [stacks] Stack nextcloud stopped successfully (took 0.4s)
|
||||
2026-04-13T19:38:49Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:38:49Z INFO [stacks] Starting stack: nextcloud
|
||||
2026-04-13T19:38:52Z INFO [stacks] Stack immich post-start status:
|
||||
2026-04-13T19:38:52Z INFO [stacks] immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4@sha256:e16c2f166a8174901959fdf85e2e4c7bd1ebc4b37e0b6655de97c41408a260c4 running Up 13 seconds (healthy)
|
||||
2026-04-13T19:38:52Z INFO [stacks] immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0@sha256:1a078b237c1d9b420b0ee59147386b4aa60d3a07a8e6a402fc84a57e41b043a4 running Up 13 seconds (healthy)
|
||||
2026-04-13T19:38:52Z INFO [stacks] immich-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 13 seconds (healthy)
|
||||
2026-04-13T19:38:52Z INFO [stacks] immich-server ghcr.io/immich-app/immich-server:v3.2.4@sha256:d317916b28090c33eb36b308464ea391f8b7df1d850fcfea227a39ec879718c2 running Up 3 seconds (health: starting)
|
||||
2026-04-13T19:38:52Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:56Z INFO [stacks] Stack nextcloud started successfully (took 6.5s)
|
||||
2026-04-13T19:38:56Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:56Z INFO [gate] boot 1776109082-1457: live bind confirmed — recreating drive-backed app paperless-ngx (state=degraded) onto /mnt/felhom-drives/hdd_1
|
||||
2026-04-13T19:38:56Z INFO [stacks] Stopping stack: paperless-ngx
|
||||
2026-04-13T19:38:56Z INFO [stacks] Stack paperless-ngx stopped successfully (took 0.5s)
|
||||
2026-04-13T19:38:56Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:38:56Z INFO [stacks] Starting stack: paperless-ngx
|
||||
2026-04-13T19:38:58Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:58Z INFO Health probes: 1 ok (of 1 probed)
|
||||
2026-04-13T19:38:58Z INFO [deadapp] boot grace 1m30s: the host's last boot was clean (R-856)
|
||||
2026-04-13T19:38:58Z INFO [aftercrash] a normal (or unknown) boot — no app repair after an unclean boot (R-1007)
|
||||
2026-04-13T19:38:58Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:38:59Z WARN [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:38:59Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:38:59Z WARN Startup hub report attempt 1/3 failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:38:59Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:38:59Z INFO [stacks] Stack nextcloud post-start status:
|
||||
2026-04-13T19:38:59Z INFO [stacks] nextcloud nextcloud:34.0.4-apache@sha256:37b109885aa3cba3e056362a899556a625c986f2c17cd2c0cc157d21c789f53b running Up 3 seconds (health: starting)
|
||||
2026-04-13T19:38:59Z INFO [stacks] nextcloud-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy)
|
||||
2026-04-13T19:38:59Z INFO [stacks] nextcloud-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 9 seconds (healthy)
|
||||
2026-04-13T19:39:03Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:07Z INFO [stacks] Stack paperless-ngx started successfully (took 10.9s)
|
||||
2026-04-13T19:39:07Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:07Z INFO [gate] boot 1776109082-1457: re-syncing FileBrowser mounts against the live binds
|
||||
2026-04-13T19:39:07Z INFO [settings] Settings saved
|
||||
2026-04-13T19:39:07Z INFO [web] FileBrowser: drive /mnt/felhom-drives/hdd_2 not mounted — skipping userdata skeleton
|
||||
2026-04-13T19:39:08Z INFO [web] FileBrowser sync — no config/compose change, ensured running without recreate (3 storage path(s))
|
||||
2026-04-13T19:39:08Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:08Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:08Z INFO Health probes: 3 ok (of 3 probed)
|
||||
2026-04-13T19:39:11Z INFO [stacks] Stack paperless-ngx post-start status:
|
||||
2026-04-13T19:39:11Z INFO [stacks] paperless-postgres postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 running Up 13 seconds (healthy)
|
||||
2026-04-13T19:39:11Z INFO [stacks] paperless-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 13 seconds (healthy)
|
||||
2026-04-13T19:39:11Z INFO [stacks] paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15@sha256:6c86cad803970ea782683a8e80e7403444c5bf3cf70de63b4d3c8e87500db92f running Up 3 seconds (health: starting)
|
||||
2026-04-13T19:39:13Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:18Z INFO [bootrecon] boot window: fleet settled after 46s (3 identical samples 5s apart) — sweeping
|
||||
2026-04-13T19:39:18Z INFO [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start)
|
||||
2026-04-13T19:39:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:24Z WARN [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:39:24Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:39:24Z WARN Startup hub report attempt 2/3 failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:39:24Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:39:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:39:36Z INFO [scheduler] Job agent-channel-health completed (took 7.226s)
|
||||
2026-04-13T19:39:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:48Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:39:49Z WARN [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:39:49Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:39:49Z WARN Startup hub report attempt 3/3 failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:39:49Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:39:49Z WARN Startup hub report failed after 3 attempts — next scheduled push in 15m
|
||||
2026-04-13T19:39:57Z INFO [fillwatch] checked 4 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2026-04-13T19:39:57Z WARN [offsite-apply] settle-gate: GO — floor still unknown after 1m30s; a hub that cannot serve a floor cannot serve a consume (no burn risk)
|
||||
2026-04-13T19:39:58Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:08Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:14Z WARN [selfupdate] Registry check failed: HTTP request failed: Get "https://gitea.dooplex.hu/v2/admin/felhom-controller/tags/list": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:40:14Z is before 2026-10-05T03:52:51Z
|
||||
2026-04-13T19:40:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:28Z INFO [scheduler] Running job: stack-scan
|
||||
2026-04-13T19:40:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:28Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2026-04-13T19:40:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:28Z INFO [scheduler] Job stack-scan completed (took 419ms)
|
||||
2026-04-13T19:40:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:40:36Z INFO [scheduler] Job agent-channel-health completed (took 7.212s)
|
||||
2026-04-13T19:40:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:48Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:40:58Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:41:08Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:41:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:41:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:41:28Z INFO [stacks] controller image retention: pass over 2 controller image(s) — running 0.336.0, previous "0.331.0" (the self-update's record), 0 candidate(s), 0 deleted, the rest kept
|
||||
2026-04-13T19:41:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:41:36Z INFO [scheduler] Job agent-channel-health completed (took 7.217s)
|
||||
2026-04-13T19:41:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:41:48Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:41:58Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:42:08Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:42:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:42:28Z INFO [scheduler] Running job: stack-scan
|
||||
2026-04-13T19:42:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:42:28Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2026-04-13T19:42:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:42:28Z INFO [scheduler] Job stack-scan completed (took 209ms)
|
||||
2026-04-13T19:42:28Z WARN [deadapp] paperless-ngx: crash_loop — 7 in 10m0s; STOPPING it (decision 28)
|
||||
2026-04-13T19:42:28Z INFO [stacks] Stopping stack: paperless-ngx
|
||||
2026-04-13T19:42:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:42:36Z INFO [scheduler] Job agent-channel-health completed (took 7.22s)
|
||||
2026-04-13T19:42:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks
|
||||
2026-04-13T19:42:39Z INFO [stacks] Stack paperless-ngx stopped successfully (took 10.8s)
|
||||
2026-04-13T19:42:39Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:42:39Z WARN [settings] restore hold SET for paperless-ngx — the app stays stopped until it is cleared
|
||||
2026-04-13T19:42:39Z INFO [settings] Settings saved
|
||||
2026-04-13T19:42:39Z INFO [settings] Settings saved
|
||||
2026-04-13T19:42:39Z WARN [backup] paperless-ngx is STOPPED by the box: crash_loop (trip 1 within 24h0m0s) — Start gives it one more try (decision 28)
|
||||
2026-04-13T19:42:45Z WARN Event push failed after 3 attempts (app_stopped_unhealthy/warning): Post "https://hub.felhom.eu/api/v1/event": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-04-13T19:42:45Z is before 2026-10-14T07:36:58Z
|
||||
2026-04-13T19:42:48Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:42:58Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:08Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:18Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Running job: backup-cache
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Running job: system-health
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Running job: docker-socket-users
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Running job: offsite-credential-retry
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Job offsite-credential-retry completed (took 1ms)
|
||||
2026-04-13T19:43:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:28Z INFO [backup] Found 7 DB dump files across drives
|
||||
2026-04-13T19:43:28Z INFO [backup] Discovered 3 databases
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Job docker-socket-users completed (took 271ms)
|
||||
2026-04-13T19:43:28Z INFO [monitor] Health check: status=ok
|
||||
2026-04-13T19:43:28Z INFO [scheduler] Job system-health completed (took 356ms)
|
||||
2026-04-13T19:43:28Z INFO [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node)
|
||||
2026-04-13T19:43:29Z INFO [backup] Backup status cache refreshed
|
||||
2026-04-13T19:43:29Z INFO [scheduler] Job backup-cache completed (took 675ms)
|
||||
2026-04-13T19:43:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:43:37Z INFO [scheduler] Job agent-channel-health completed (took 7.986s)
|
||||
2026-04-13T19:43:38Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:38Z INFO Health probes: 2 ok (of 2 probed)
|
||||
2026-04-13T19:43:48Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:58Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:43:58Z INFO Health probes: 2 ok (of 2 probed)
|
||||
2026-04-13T19:44:08Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:44:18Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:44:18Z INFO Health probes: 3 ok (of 3 probed)
|
||||
2026-04-13T19:44:28Z INFO [scheduler] Running job: stack-scan
|
||||
2026-04-13T19:44:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:44:28Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2026-04-13T19:44:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:44:28Z INFO [scheduler] Job stack-scan completed (took 177ms)
|
||||
2026-04-13T19:44:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:44:36Z INFO [scheduler] Job agent-channel-health completed (took 7.199s)
|
||||
2026-04-13T19:44:38Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:44:48Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:44:58Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:45:08Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:45:18Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:45:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:45:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:45:36Z INFO [scheduler] Job agent-channel-health completed (took 7.219s)
|
||||
2026-04-13T19:45:38Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:45:48Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:45:58Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:08Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:18Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:28Z INFO [scheduler] Running job: stack-scan
|
||||
2026-04-13T19:46:28Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2026-04-13T19:46:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:28Z INFO [scheduler] Job stack-scan completed (took 137ms)
|
||||
2026-04-13T19:46:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:46:36Z INFO [scheduler] Job agent-channel-health completed (took 7.226s)
|
||||
2026-04-13T19:46:38Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:48Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:46:58Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:47:08Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:47:18Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:47:28Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:47:29Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-04-13T19:47:36Z INFO [scheduler] Job agent-channel-health completed (took 7.246s)
|
||||
2026-04-13T19:47:38Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:47:48Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2026-04-13T19:47:49Z INFO [web] Login from 192.168.0.180
|
||||
+1
@@ -0,0 +1 @@
|
||||
{"ok": true, "data": {"logs": ""}}
|
||||
+200
@@ -0,0 +1,200 @@
|
||||
2028-10-21T07:29:31Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:31Z INFO [gate] boot 1855726117-1561: re-syncing FileBrowser mounts against the live binds
|
||||
2028-10-21T07:29:31Z INFO [gate] boot 1855726117-1561: 1 drive-backed app(s) left stopped on purpose (recorded Stop, or a legacy app.yaml at zero containers)
|
||||
2028-10-21T07:29:31Z INFO [settings] Settings saved
|
||||
2028-10-21T07:29:31Z INFO [web] FileBrowser: drive /mnt/felhom-drives/hdd_2 not mounted — skipping userdata skeleton
|
||||
2028-10-21T07:29:32Z INFO [web] FileBrowser mounts synced (recreated) — 3 storage path(s), config updated
|
||||
2028-10-21T07:29:32Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:33Z INFO [deadapp] boot grace 1m30s: the host's last boot was clean (R-856)
|
||||
2028-10-21T07:29:33Z INFO [aftercrash] a normal (or unknown) boot — no app repair after an unclean boot (R-1007)
|
||||
2028-10-21T07:29:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:34Z INFO [infra] filebrowser: default login refused for this run — joined traefik-public, files.<domain> served (R-1126)
|
||||
2028-10-21T07:29:34Z INFO [stacks] Stack nextcloud post-start status:
|
||||
2028-10-21T07:29:34Z INFO [stacks] nextcloud nextcloud:34.0.4-apache@sha256:37b109885aa3cba3e056362a899556a625c986f2c17cd2c0cc157d21c789f53b running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:29:34Z INFO [stacks] nextcloud-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy)
|
||||
2028-10-21T07:29:34Z INFO [stacks] nextcloud-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 9 seconds (healthy)
|
||||
2028-10-21T07:29:37Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:42Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:43Z INFO Health probes: 3 ok (of 3 probed)
|
||||
2028-10-21T07:29:45Z WARN [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T07:29:45Z is after 2027-01-12T07:36:57Z
|
||||
2028-10-21T07:29:45Z WARN Startup hub report attempt 2/3 failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T07:29:45Z is after 2027-01-12T07:36:57Z
|
||||
2028-10-21T07:29:47Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:52Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:29:52Z INFO [bootrecon] boot window: fleet settled after 45s (3 identical samples 5s apart) — sweeping
|
||||
2028-10-21T07:29:52Z INFO [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start)
|
||||
2028-10-21T07:29:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:30:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:30:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:30:03Z INFO [scheduler] Job agent-channel-health completed (took 476ms)
|
||||
2028-10-21T07:30:11Z WARN [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T07:30:11Z is after 2027-01-12T07:36:57Z
|
||||
2028-10-21T07:30:11Z WARN Startup hub report attempt 3/3 failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T07:30:11Z is after 2027-01-12T07:36:57Z
|
||||
2028-10-21T07:30:11Z WARN Startup hub report failed after 3 attempts — next scheduled push in 15m
|
||||
2028-10-21T07:30:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:30:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:30:32Z INFO [fillwatch] checked 4 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||
2028-10-21T07:30:32Z WARN [offsite-apply] settle-gate: GO — floor still unknown after 1m30s; a hub that cannot serve a floor cannot serve a consume (no burn risk)
|
||||
2028-10-21T07:30:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:30:36Z WARN [selfupdate] Registry check failed: HTTP request failed: Get "https://gitea.dooplex.hu/v2/admin/felhom-controller/tags/list": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T07:30:36Z is after 2027-01-03T03:52:50Z
|
||||
2028-10-21T07:30:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:30:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:02Z INFO [scheduler] Running job: stack-scan
|
||||
2028-10-21T07:31:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:03Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2028-10-21T07:31:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:03Z INFO [scheduler] Job stack-scan completed (took 160ms)
|
||||
2028-10-21T07:31:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:31:03Z INFO [scheduler] Job agent-channel-health completed (took 586ms)
|
||||
2028-10-21T07:31:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:31:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:32:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:32:03Z INFO [stacks] controller image retention: pass over 2 controller image(s) — running 0.336.0, previous "0.331.0" (the self-update's record), 0 candidate(s), 0 deleted, the rest kept
|
||||
2028-10-21T07:32:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:32:03Z INFO [scheduler] Job agent-channel-health completed (took 471ms)
|
||||
2028-10-21T07:32:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:32:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:32:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:32:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:32:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:02Z INFO [scheduler] Running job: stack-scan
|
||||
2028-10-21T07:33:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:03Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2028-10-21T07:33:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:03Z INFO [scheduler] Job stack-scan completed (took 158ms)
|
||||
2028-10-21T07:33:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:33:03Z INFO [scheduler] Job agent-channel-health completed (took 477ms)
|
||||
2028-10-21T07:33:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:33:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:34:02Z INFO [scheduler] Running job: backup-cache
|
||||
2028-10-21T07:34:02Z INFO [scheduler] Running job: system-health
|
||||
2028-10-21T07:34:02Z INFO [scheduler] Running job: docker-socket-users
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Running job: offsite-credential-retry
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Job offsite-credential-retry completed (took 0s)
|
||||
2028-10-21T07:34:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:34:03Z INFO [backup] Found 7 DB dump files across drives
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Job docker-socket-users completed (took 243ms)
|
||||
2028-10-21T07:34:03Z INFO [backup] Discovered 3 databases
|
||||
2028-10-21T07:34:03Z INFO [monitor] Health check: status=ok
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Job system-health completed (took 329ms)
|
||||
2028-10-21T07:34:03Z INFO [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node)
|
||||
2028-10-21T07:34:03Z INFO [samba] applying samba stack: shares=2 config_changed=true
|
||||
2028-10-21T07:34:03Z INFO [backup] Backup status cache refreshed
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Job backup-cache completed (took 710ms)
|
||||
2028-10-21T07:34:03Z INFO [scheduler] Job agent-channel-health completed (took 494ms)
|
||||
2028-10-21T07:34:07Z INFO [quiesce] backup due on 2 tier(s) — quiescing 7 stack(s): [actualbudget bookstack immich mealie nextcloud radicale vaultwarden]
|
||||
2028-10-21T07:34:07Z INFO [stacks] Stopping stack: actualbudget
|
||||
2028-10-21T07:34:07Z INFO [stacks] Stack actualbudget stopped successfully (took 0.3s)
|
||||
2028-10-21T07:34:07Z INFO [stacks] Status refresh: 15 containers across 68 stacks
|
||||
2028-10-21T07:34:07Z INFO [stacks] Stopping stack: bookstack
|
||||
2028-10-21T07:34:11Z INFO [stacks] Stack bookstack stopped successfully (took 4.1s)
|
||||
2028-10-21T07:34:11Z INFO [stacks] Status refresh: 13 containers across 68 stacks
|
||||
2028-10-21T07:34:11Z INFO [stacks] Stopping stack: immich
|
||||
2028-10-21T07:34:12Z INFO [stacks] Stack immich stopped successfully (took 0.7s)
|
||||
2028-10-21T07:34:12Z INFO [stacks] Status refresh: 9 containers across 68 stacks
|
||||
2028-10-21T07:34:12Z INFO [stacks] Stopping stack: mealie
|
||||
2028-10-21T07:34:13Z INFO [stacks] Status refresh: 9 containers across 68 stacks
|
||||
2028-10-21T07:34:13Z INFO [stacks] Stack mealie stopped successfully (took 1.7s)
|
||||
2028-10-21T07:34:14Z INFO [stacks] Status refresh: 8 containers across 68 stacks
|
||||
2028-10-21T07:34:14Z INFO [stacks] Stopping stack: nextcloud
|
||||
2028-10-21T07:34:15Z INFO [stacks] Stack nextcloud stopped successfully (took 1.7s)
|
||||
2028-10-21T07:34:15Z INFO [stacks] Status refresh: 5 containers across 68 stacks
|
||||
2028-10-21T07:34:15Z INFO [stacks] Stopping stack: radicale
|
||||
2028-10-21T07:34:16Z INFO [stacks] Stack radicale stopped successfully (took 0.4s)
|
||||
2028-10-21T07:34:16Z INFO [stacks] Status refresh: 4 containers across 68 stacks
|
||||
2028-10-21T07:34:16Z INFO [stacks] Stopping stack: vaultwarden
|
||||
2028-10-21T07:34:16Z INFO [stacks] Stack vaultwarden stopped successfully (took 0.2s)
|
||||
2028-10-21T07:34:16Z INFO [stacks] Status refresh: 3 containers across 68 stacks
|
||||
2028-10-21T07:34:16Z INFO [quiesce] tier local: backup job backup-9201-1855726456347094750 started — polling
|
||||
2028-10-21T07:34:23Z INFO [stacks] Status refresh: 3 containers across 68 stacks
|
||||
2028-10-21T07:34:26Z INFO [quiesce] tier local: job backup-9201-1855726456347094750 snapshotted — resuming app early (8B.2); the upload continues with the apps running
|
||||
2028-10-21T07:34:26Z INFO [quiesce] unquiescing (snapshotted (early resume)): restarting 7 stack(s)
|
||||
2028-10-21T07:34:26Z INFO [stacks] Starting stack: actualbudget
|
||||
2028-10-21T07:34:26Z INFO [stacks] Stack actualbudget started successfully (took 0.5s)
|
||||
2028-10-21T07:34:27Z INFO [stacks] Status refresh: 4 containers across 68 stacks
|
||||
2028-10-21T07:34:27Z INFO [stacks] Starting stack: bookstack
|
||||
2028-10-21T07:34:30Z INFO [stacks] Stack actualbudget post-start status:
|
||||
2028-10-21T07:34:30Z INFO [stacks] actualbudget actualbudget/actual-server:26.9.0@sha256:552beab3dec8c93d46b8b9245612d63c3f123b8a45063a474f53e229b17621d3 running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:34:33Z INFO [stacks] Status refresh: 6 containers across 68 stacks
|
||||
2028-10-21T07:34:33Z WARN Health probe bookstack: HTTP GET :80/ → Get "http://bookstack-db:80/": dial tcp: lookup bookstack-db on 127.0.0.11:53: no such host
|
||||
2028-10-21T07:34:33Z WARN Health probes: 1 ok, 1 unhealthy (of 2 probed)
|
||||
2028-10-21T07:34:33Z INFO [stacks] Stack bookstack started successfully (took 6.6s)
|
||||
2028-10-21T07:34:33Z INFO [stacks] Status refresh: 6 containers across 68 stacks
|
||||
2028-10-21T07:34:33Z INFO [stacks] Starting stack: immich
|
||||
2028-10-21T07:34:36Z INFO [stacks] Stack bookstack post-start status:
|
||||
2028-10-21T07:34:36Z INFO [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:34:36Z INFO [stacks] bookstack-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy)
|
||||
2028-10-21T07:34:43Z INFO [stacks] Status refresh: 10 containers across 68 stacks
|
||||
2028-10-21T07:34:45Z INFO [stacks] Stack immich started successfully (took 11.3s)
|
||||
2028-10-21T07:34:45Z INFO [stacks] Status refresh: 10 containers across 68 stacks
|
||||
2028-10-21T07:34:45Z INFO [stacks] Starting stack: mealie
|
||||
2028-10-21T07:34:46Z INFO [stacks] Stack mealie started successfully (took 0.6s)
|
||||
2028-10-21T07:34:46Z INFO [stacks] Status refresh: 11 containers across 68 stacks
|
||||
2028-10-21T07:34:46Z INFO [stacks] Starting stack: nextcloud
|
||||
2028-10-21T07:34:48Z INFO [stacks] Stack immich post-start status:
|
||||
2028-10-21T07:34:48Z INFO [stacks] immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4@sha256:e16c2f166a8174901959fdf85e2e4c7bd1ebc4b37e0b6655de97c41408a260c4 running Up 13 seconds (healthy)
|
||||
2028-10-21T07:34:48Z INFO [stacks] immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0@sha256:1a078b237c1d9b420b0ee59147386b4aa60d3a07a8e6a402fc84a57e41b043a4 running Up 14 seconds (healthy)
|
||||
2028-10-21T07:34:48Z INFO [stacks] immich-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy)
|
||||
2028-10-21T07:34:48Z INFO [stacks] immich-server ghcr.io/immich-app/immich-server:v3.2.4@sha256:d317916b28090c33eb36b308464ea391f8b7df1d850fcfea227a39ec879718c2 running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:34:49Z INFO [stacks] Stack mealie post-start status:
|
||||
2028-10-21T07:34:49Z INFO [stacks] mealie ghcr.io/mealie-recipes/mealie:v3.28.0@sha256:8b02290f4d1806f02acac6f25f6d48a3c965612fda1f8e914d5af533276f8688 running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:34:53Z INFO [stacks] Status refresh: 14 containers across 68 stacks
|
||||
2028-10-21T07:34:53Z WARN Health probe nextcloud: API GET :80/status.php → Get "http://nextcloud-redis:80/status.php": dial tcp: lookup nextcloud-redis on 127.0.0.11:53: no such host
|
||||
2028-10-21T07:34:53Z WARN Health probes: 1 ok, 1 unhealthy (of 2 probed)
|
||||
2028-10-21T07:34:58Z INFO [stacks] Stack nextcloud started successfully (took 12.2s)
|
||||
2028-10-21T07:34:59Z INFO [stacks] Status refresh: 14 containers across 68 stacks
|
||||
2028-10-21T07:34:59Z INFO [stacks] Starting stack: radicale
|
||||
2028-10-21T07:34:59Z INFO [stacks] Stack radicale started successfully (took 0.7s)
|
||||
2028-10-21T07:35:00Z INFO [stacks] Status refresh: 15 containers across 68 stacks
|
||||
2028-10-21T07:35:00Z INFO [stacks] Starting stack: vaultwarden
|
||||
2028-10-21T07:35:01Z INFO [stacks] Stack vaultwarden started successfully (took 1.1s)
|
||||
2028-10-21T07:35:01Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:02Z INFO [stacks] Stack nextcloud post-start status:
|
||||
2028-10-21T07:35:02Z INFO [stacks] nextcloud nextcloud:34.0.4-apache@sha256:37b109885aa3cba3e056362a899556a625c986f2c17cd2c0cc157d21c789f53b running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:35:02Z INFO [stacks] nextcloud-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 14 seconds (healthy)
|
||||
2028-10-21T07:35:02Z INFO [stacks] nextcloud-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 15 seconds (healthy)
|
||||
2028-10-21T07:35:02Z INFO [scheduler] Running job: stack-scan
|
||||
2028-10-21T07:35:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:35:03Z INFO [stacks] Stack radicale post-start status:
|
||||
2028-10-21T07:35:03Z INFO [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:35:03Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2028-10-21T07:35:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:03Z INFO [scheduler] Job stack-scan completed (took 654ms)
|
||||
2028-10-21T07:35:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:03Z INFO [scheduler] Job agent-channel-health completed (took 565ms)
|
||||
2028-10-21T07:35:04Z INFO [stacks] Stack vaultwarden post-start status:
|
||||
2028-10-21T07:35:04Z INFO [stacks] vaultwarden vaultwarden/server:1.37.4-alpine@sha256:19ea2b669945d842dadc3d500f2d6a795866a003330f59a47b9634658bae98bb running Up 3 seconds (health: starting)
|
||||
2028-10-21T07:35:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:13Z INFO Health probes: 1 ok (of 1 probed)
|
||||
2028-10-21T07:35:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:23Z INFO Health probes: 4 ok (of 4 probed)
|
||||
2028-10-21T07:35:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:35:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:36:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:36:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:36:04Z INFO [scheduler] Job agent-channel-health completed (took 1.082s)
|
||||
2028-10-21T07:36:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:36:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:36:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:36:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:36:53Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:03Z INFO [scheduler] Running job: stack-scan
|
||||
2028-10-21T07:37:03Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available)
|
||||
2028-10-21T07:37:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:03Z INFO [scheduler] Job stack-scan completed (took 140ms)
|
||||
2028-10-21T07:37:03Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:03Z INFO [scheduler] Running job: agent-channel-health
|
||||
2028-10-21T07:37:03Z INFO [scheduler] Job agent-channel-health completed (took 463ms)
|
||||
2028-10-21T07:37:13Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:23Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:33Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:43Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:37:54Z INFO [stacks] Status refresh: 16 containers across 68 stacks
|
||||
2028-10-21T07:38:00Z INFO [web] Login from 192.168.0.180
|
||||
+335
@@ -0,0 +1,335 @@
|
||||
2028-10-21T09:28:47.937+02:00 INFO msg="felhom-agent daemon starting" version=0.161.0 host_id=Tester4-816e21 hub_url=https://hub.felhom.eu interval_s=900
|
||||
2028-10-21T09:28:47.957+02:00 INFO msg="daemon: operator signers pinned" count=2
|
||||
2028-10-21T09:28:47.965+02:00 INFO msg="local-api leaf LOADED" fingerprint_sha256=d0e3c957e507ec37772877c71f1d6ad79e01c50177e12c556328aa24d7fcb0bc cert=/var/lib/felhom-agent/local-api.crt
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=3 prune="keep-daily=3" wait_timeout=30m0s prune_pbs_allowed=false primary=true
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 prune="" wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="lanresolver: enabled" host_ip=192.168.0.134 upstreams="[1.1.1.1 8.8.8.8]" interval_s=300
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="wgtunnel: enabled" interval_s=60 state_dir=/var/lib/felhom-agent
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="poke: agent-plane sync listener enabled" port=51822
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="felhomsshd (OOB): enabled" interval_s=60 state_dir=/var/lib/felhom-agent
|
||||
2028-10-21T09:28:47.966+02:00 INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
|
||||
2028-10-21T09:28:48.98+02:00 INFO msg="capabilities self-check" ok=76 total=76 degraded=0 inactive=0
|
||||
2028-10-21T09:28:48.98+02:00 INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
|
||||
2028-10-21T09:28:48.981+02:00 INFO msg="pbs: verify loop starting" cadence=6h0m0s
|
||||
2028-10-21T09:28:48.982+02:00 INFO msg="storage: watchdog starting" interval=5s debounce=15s
|
||||
2028-10-21T09:28:48.982+02:00 INFO msg="fast-tick armed: 30s out-of-band cadence while desired-state is unapplied" interval=30s
|
||||
2028-10-21T09:28:48.983+02:00 INFO msg="poke: listening for hub sync-pokes (WG-confined, contentless)" addr=10.77.0.6:51822
|
||||
2028-10-21T09:28:49.023+02:00 DEBUG msg="reconcile: no drift, no actions" desired_guests=0 actual_guests=1
|
||||
2028-10-21T09:28:49.604+02:00 INFO msg="local-api: drive intent recorded" where=/mnt/belso action=enrolled durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:28:49.604+02:00 INFO msg="internal place: mounted and enrolled" where=/mnt/belso fs_uuid=5aa4f4f8-2402-4d8c-841a-53bd4858e35d gib=56
|
||||
2028-10-21T09:28:49.74+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:28:49.784+02:00 DEBUG msg="hub: tunnel probe could not decide" detail="no running customer guest"
|
||||
2028-10-21T09:28:50.118+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:28:50.179+02:00 INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/belso stable=/mnt/felhom-drives/belso prior_binds=0
|
||||
2028-10-21T09:28:50.179+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:28:50.238+02:00 INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/hdd_1 stable=/mnt/felhom-drives/hdd_1 prior_binds=0
|
||||
2028-10-21T09:28:50.238+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:28:50.239+02:00 INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
|
||||
2028-10-21T09:28:50.239+02:00 INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
|
||||
2028-10-21T09:28:50.268+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:28:50.273+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:28:50.284+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:28:50.299+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:28:50.305+02:00 INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s guests_dir=/var/lib/felhom-agent/guests
|
||||
2028-10-21T09:28:50.305+02:00 INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
|
||||
2028-10-21T09:28:50.305+02:00 INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
|
||||
2028-10-21T09:28:50.306+02:00 INFO msg="local-api server listening" addr=169.254.253.1:8443
|
||||
2028-10-21T09:28:50.306+02:00 INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
2028-10-21T09:28:50.74+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:28:50.977+02:00 INFO msg="felhomsshd: config applied" port=8822 action=reload
|
||||
2028-10-21T09:28:51.069+02:00 INFO msg="felhomsshd belt: set synced" set=ssh_port elements=8822
|
||||
2028-10-21T09:28:51.232+02:00 INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=1
|
||||
2028-10-21T09:28:51.591+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:28:51+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:28:54.453+02:00 INFO msg="lanresolver: guest IP not yet leased — skipping (will retry)" vmid=9201
|
||||
2028-10-21T09:29:00.307+02:00 WARN msg="local-api: storage view unavailable for class hints" err="storage: NodeStorage: proxmox: GET /nodes/felhom/storage: Get \"https://127.0.0.1:8006/api2/json/nodes/felhom/storage\": context canceled"
|
||||
2028-10-21T09:29:00.308+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=368
|
||||
2028-10-21T09:29:01.55+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=511
|
||||
2028-10-21T09:29:03.995+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=629
|
||||
2028-10-21T09:29:08.673+02:00 INFO msg="local-api: same-disk" vmid=9201 a=/mnt/sys_drive b=/mnt/felhom-drives/belso different=false proven=true reason=same_disk
|
||||
2028-10-21T09:29:08.673+02:00 DEBUG msg="local-api: request" method=POST path=/disks/same-disk status=200 duration_ms=510
|
||||
2028-10-21T09:29:09.223+02:00 INFO msg="local-api: same-disk" vmid=9201 a=/mnt/sys_drive b=/mnt/felhom-drives/hdd_1 different=true proven=false reason=vm
|
||||
2028-10-21T09:29:09.223+02:00 DEBUG msg="local-api: request" method=POST path=/disks/same-disk status=200 duration_ms=549
|
||||
2028-10-21T09:29:10.82+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:29:10.935+02:00 INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/belso stable=/mnt/felhom-drives/belso prior_binds=1
|
||||
2028-10-21T09:29:10.936+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:29:11.056+02:00 INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/hdd_1 stable=/mnt/felhom-drives/hdd_1 prior_binds=1
|
||||
2028-10-21T09:29:11.057+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:29:14.138+02:00 DEBUG msg="local-api: request" method=GET path=/debug/logs status=200 duration_ms=0
|
||||
2028-10-21T09:29:18.982+02:00 INFO msg="fast-tick: desired-state unapplied — pulsing out-of-band reports" reason="desired-state not yet fetched" cadence=30s
|
||||
2028-10-21T09:29:18.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:29:19.647+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:29:20.351+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:29:21.604+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:29:21.737+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:29:21+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:29:30.71+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:29:30.776+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:29:30.86+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:29:31.931+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=598
|
||||
2028-10-21T09:29:33.005+02:00 DEBUG msg="local-api: /host/crash-guard served" vmid=9201 last_boot_at=2028-10-21T07:28:37Z last_boot_unclean=false tripped=true
|
||||
2028-10-21T09:29:33.005+02:00 DEBUG msg="local-api: request" method=GET path=/host/crash-guard status=200 duration_ms=0
|
||||
2028-10-21T09:29:48.985+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:29:49.663+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:29:50.291+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:29:50.301+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:29:50.369+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:29:50.385+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:29:50.797+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:29:50.868+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:29:50.935+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:29:51.732+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:29:51.864+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:29:51+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:29:54.285+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:30:02.546+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=613
|
||||
2028-10-21T09:30:03.15+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=603
|
||||
2028-10-21T09:30:03.805+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=475
|
||||
2028-10-21T09:30:10.75+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:30:10.81+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:30:10.882+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:30:18.986+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:30:19.627+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:30:20.354+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:30:21.662+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:30:21.791+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:30:21+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:30:30.73+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:30:30.807+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:30:30.876+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:30:32.543+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=610
|
||||
2028-10-21T09:30:33.155+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=612
|
||||
2028-10-21T09:30:48.983+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:30:49.812+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:30:50.284+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:30:50.29+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:30:50.369+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:30:50.373+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:30:50.889+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:30:50.984+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:30:51.058+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:30:51.906+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:30:52.059+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:30:52+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:30:54.279+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:31:02.727+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=790
|
||||
2028-10-21T09:31:03.5+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=772
|
||||
2028-10-21T09:31:03.916+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=582
|
||||
2028-10-21T09:31:10.727+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:31:10.789+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:31:10.85+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:31:18.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:31:19.618+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:31:20.346+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:31:21.609+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:31:21.756+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:31:21+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:31:30.715+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:31:30.792+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:31:30.858+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:31:32.534+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=598
|
||||
2028-10-21T09:31:33.158+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=623
|
||||
2028-10-21T09:31:48.985+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:31:49.628+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:31:50.29+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:31:50.313+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:31:50.365+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:31:50.38+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:31:50.8+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:31:50.89+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:31:50.972+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:31:51.807+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:31:51.946+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:31:51+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:31:54.382+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:32:02.56+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=624
|
||||
2028-10-21T09:32:03.311+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=750
|
||||
2028-10-21T09:32:03.8+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=470
|
||||
2028-10-21T09:32:10.756+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:32:10.837+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:32:10.906+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:32:18.985+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:32:19.666+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:32:20.353+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:32:21.721+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:32:21.869+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:32:21+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:32:30.79+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:32:30.849+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:32:30.916+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:32:32.55+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=614
|
||||
2028-10-21T09:32:33.155+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=603
|
||||
2028-10-21T09:32:48.985+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:32:49.644+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:32:50.283+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:32:50.296+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:32:50.37+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:32:50.38+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:32:50.8+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:32:50.943+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:32:51.025+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:32:52.006+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:32:52.149+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:32:52+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:32:54.587+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:33:02.55+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=617
|
||||
2028-10-21T09:33:03.175+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=624
|
||||
2028-10-21T09:33:03.807+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=476
|
||||
2028-10-21T09:33:10.755+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:33:10.819+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:33:10.88+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:33:18.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:33:19.637+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:33:20.353+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:33:21.644+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:33:21.797+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:33:21+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:33:30.766+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:33:30.837+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:33:30.903+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:33:32.575+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=639
|
||||
2028-10-21T09:33:33.189+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=613
|
||||
2028-10-21T09:33:48.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:33:49.684+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:33:50.285+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:33:50.299+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:33:50.311+02:00 DEBUG msg="fstrim: outside the daytime window — not looking" local="Sat 09:33"
|
||||
2028-10-21T09:33:50.364+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:33:50.371+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:33:50.77+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:33:50.855+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:33:50.935+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:33:51.821+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:33:51.968+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:33:51+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:33:54.451+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:34:02.61+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=674
|
||||
2028-10-21T09:34:03.211+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=599
|
||||
2028-10-21T09:34:03.436+02:00 DEBUG msg="local-api: request" method=GET path=/backup/tiers status=200 duration_ms=965
|
||||
2028-10-21T09:34:03.823+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=491
|
||||
2028-10-21T09:34:03.952+02:00 DEBUG msg="local-api: request" method=GET path=/backup/due status=200 duration_ms=514
|
||||
2028-10-21T09:34:04.714+02:00 DEBUG msg="local-api: request" method=GET path=/backup/due status=200 duration_ms=761
|
||||
2028-10-21T09:34:05.689+02:00 DEBUG msg="local-api: request" method=GET path=/backup/tiers status=200 duration_ms=973
|
||||
2028-10-21T09:34:05.689+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:34:05.69+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:34:07.036+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=1345
|
||||
2028-10-21T09:34:10.741+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:34:10.815+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:34:10.877+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:34:16.347+02:00 DEBUG msg="local-api: request" method=POST path=/backup status=202 duration_ms=0
|
||||
2028-10-21T09:34:16.348+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:34:16.779+02:00 INFO msg="backup: space preflight passed" vmid=9201 target=local last_archive_bytes=4628112507 need_bytes=6858882457 avail_bytes=62883717120
|
||||
2028-10-21T09:34:17.837+02:00 INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=local job=backup-9201-1855726456347094750
|
||||
2028-10-21T09:34:18.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:34:19.653+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:34:20.362+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:34:21.881+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:34:22.029+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:34:22+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:34:26.351+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:34:30.783+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:34:30.861+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:34:30.952+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:34:32.544+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=610
|
||||
2028-10-21T09:34:33.116+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=571
|
||||
2028-10-21T09:34:48.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:34:50.059+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:34:50.287+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:34:50.288+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:34:50.442+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:34:50.447+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:34:51.194+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:34:51.363+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:34:51.56+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:34:54.668+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:34:54.9+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:34:54+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:34:57.62+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:35:02.781+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=847
|
||||
2028-10-21T09:35:03.719+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=908
|
||||
2028-10-21T09:35:03.894+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=561
|
||||
2028-10-21T09:35:10.868+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:35:10.99+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:35:11.067+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:35:11.737+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:35:18.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:35:19.666+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:35:20.355+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:35:21.739+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:35:22.088+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:35:22.252+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:35:22+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:35:30.74+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:35:30.805+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:35:30.873+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:35:31.741+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:35:38.914+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=6981
|
||||
2028-10-21T09:35:39.508+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=592
|
||||
2028-10-21T09:35:41.742+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:35:48.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:35:49.698+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:35:50.29+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:35:50.301+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:35:50.387+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:35:50.579+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:35:50.991+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:35:51.107+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:35:51.23+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:35:51.743+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:35:52.187+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:35:52.346+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:35:52+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:35:55.134+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:36:01.747+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:36:02.555+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=622
|
||||
2028-10-21T09:36:03.183+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=627
|
||||
2028-10-21T09:36:04.41+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=1080
|
||||
2028-10-21T09:36:10.743+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:36:10.818+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:36:10.889+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:36:11.751+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:36:18.983+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:36:19.919+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:36:20.348+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:36:21.755+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:36:22.04+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:36:22.188+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:36:22+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:36:30.781+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:36:30.862+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:36:30.934+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:36:31.762+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:36:32.608+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=675
|
||||
2028-10-21T09:36:33.245+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=635
|
||||
2028-10-21T09:36:41.766+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:36:48.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:36:49.643+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:36:50.281+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:36:50.292+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:36:50.373+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:36:50.378+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:36:51.025+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:36:51.091+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:36:51.156+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:36:51.767+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:36:52.129+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:36:52.291+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:36:52+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:36:54.925+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:37:01.771+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:37:02.862+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=929
|
||||
2028-10-21T09:37:03.465+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=601
|
||||
2028-10-21T09:37:03.792+02:00 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=460
|
||||
2028-10-21T09:37:11.775+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:37:14.77+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:37:14.834+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:37:14.902+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:37:18.982+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:37:19.687+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:37:20.349+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:37:21.777+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:37:21.974+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:37:22.107+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:37:22+02:00 is after 2027-01-12T07:36:57Z"
|
||||
2028-10-21T09:37:30.738+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:37:30.817+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:37:30.896+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:37:31.781+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:37:32.614+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=681
|
||||
2028-10-21T09:37:33.308+02:00 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=692
|
||||
2028-10-21T09:37:41.785+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:37:48.983+02:00 INFO msg="hub: out-of-band report triggered (watchdog/poke)"
|
||||
2028-10-21T09:37:50.28+02:00 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
2028-10-21T09:37:50.291+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:37:50.37+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:37:50.402+02:00 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
2028-10-21T09:37:51.787+02:00 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=0
|
||||
2028-10-21T09:37:54.611+02:00 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
2028-10-21T09:37:54.639+02:00 DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
|
||||
2028-10-21T09:37:54.706+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
2028-10-21T09:37:54.8+02:00 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:92d44e80-fbcd-4ed6-bb24-45ad01f69692
|
||||
2028-10-21T09:37:56.656+02:00 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
2028-10-21T09:37:56.971+02:00 DEBUG msg="hub: system facts unavailable" err="no running customer guest"
|
||||
2028-10-21T09:37:57.118+02:00 WARN msg="hub: report failed; keeping current interval" err="hub: transport error: Post \"https://hub.felhom.eu/api/v1/host-report\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2028-10-21T09:37:57+02:00 is after 2027-01-12T07:36:57Z"
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
apps: {'actualbudget': 'running', 'bookstack': 'running', 'immich': 'running', 'mealie': 'running', 'nextcloud': 'running', 'paperless-ngx': 'stopped', 'radicale': 'running', 'vaultwarden': 'running'}
|
||||
PAPERLESS - ess-ngx Digitális irattár - szkennelés, OCR és automatikus rendszerezés ● Leállítva Naprakész paperless.tester4.test Fájlok ↗ Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Ind
|
||||
PAPERLESS - ítás gombbal újra megpróbálhatod. Beállítások Adatok és mentés Útmutató Napló Működés Leállítva Indítás Belépés az alkalmazásba Az első bejelentkezéshez. Ha
|
||||
07:46:49Z Start paperless 200
|
||||
07:47:51Z paperless running
|
||||
BACKUPS: Teljes rendszermentés: kb. 04:30–08:30 között ✓ Adatmentés aktív – Távoli rendszermentés nincs beállítva 1 Adatbázis mentve
|
||||
teljes ment -> B (28%) Belső tárhely 4,23 GB / 54,8 GB (8%) Családi adatlemez 4,24 GB / 196 GB (2%) DB mentések 4 fájl Rendszermentés (teljes mentés) A teljes szerver — alkalmazások, beállítások és adatbázisok együtt — időszakos mentése, amelyből az egész készülék visszaállítható. Ezt a host-üg
|
||||
Rendszermentés (teljes mentés) A teljes szerver — alkalmazások, beállítások és adatbázisok együtt — időszakos mentése, amelyből az egész készülék visszaállítható. Ezt a host-ügynök készíti és kezeli. A host-ügynök jelenleg nem elérhető. Mentési időablak A mentések egymás után futnak: adatbázis-mentés, helyi másolat, távoli mentés, majd a teljes rendszermentés. Mentési időablak kezdete Mentés Adatbázis-mentés: 02:30 Helyi másolat: 03:30 Távoli mentés: 04:15 Teljes rendszermentés: kb. 04:30–08:30 között ✓ Adatmentés aktív – Távoli rendszermentés nincs beállítva 4 Adatbázis mentve
|
||||
07:54:05Z Rendszermentés (teljes mentés) A teljes szerver — alkalmazások, beállítások és adatbázisok együtt — időszakos mentése, amelyből az egész készülék visszaállítható. Ezt a host-ügynök készíti és kezeli. A host-ügynök jelenleg nem elérhető. Mentési időablak A mentések egymás után futnak: adatbázis-mentés, helyi másolat, távoli mentés, majd a teljes rendszermentés. Mentési időablak kezdete Mentés Adatbázis-mentés: 02:30 Helyi másolat: 03:30 Távoli mentés: 04:15 Teljes rendszermentés: kb. 04:30–08:30 között ✓ Adatmentés aktív – Távoli rendszermentés nincs beállítva 4 Adatbázis mentve
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
2026-10-21T07:45:25Z INFO [scheduler] Running job: agent-channel-health
|
||||
2026-10-21T07:45:32Z INFO [scheduler] Job agent-channel-health completed (took 7.206s)
|
||||
2026-10-21T07:46:32Z INFO [scheduler] Job agent-channel-health completed (took 7.239s)
|
||||
2026-10-21T07:47:33Z INFO [scheduler] Job agent-channel-health completed (took 7.55s)
|
||||
2026-10-21T07:48:34Z INFO [scheduler] Job agent-channel-health completed (took 8.814s)
|
||||
2026-10-21T07:49:34Z INFO [scheduler] Job agent-channel-health completed (took 8.3s)
|
||||
2026-10-21T07:49:39Z ERROR [quiesce] cycle error: check due: agentapi: GET /backup/tiers: Get "https://169.254.253.1:8443/backup/tiers": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
||||
2026-10-21T07:50:33Z INFO [scheduler] Job agent-channel-health completed (took 7.25s)
|
||||
2026-10-21T07:51:32Z INFO [scheduler] Job agent-channel-health completed (took 7.24s)
|
||||
2026-10-21T07:52:32Z INFO [scheduler] Job agent-channel-health completed (took 7.249s)
|
||||
2026-10-21T07:53:32Z INFO [scheduler] Job agent-channel-health completed (took 7.229s)
|
||||
agent-logs http 200 {'data': {'entries': [{'timestamp': '2026-10-21T09:44:10.609+02:00', 'level': 'INFO', 'message': 'msg="felhom-agent daemon starting" version=0.161.0 host_id=Tester4-816e21 hub_url=https://hub.felhom.eu interval_s=900'}, {'timestamp': '2026-10-21T09:44:10.615+02:00', 'level': 'INFO', 'message': 'msg=
|
||||
09:44:26.268 DEBUG msg="reconcile: guest-bind re-assert pass" guests=1 resolved_mounts=3
|
||||
09:44:26.403 INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/belso stable=/mnt/felhom-drives/belso prior_binds=0
|
||||
09:44:26.403 INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/belso guest_path=/mnt/felhom-drives/belso durable_id=uuid:5aa4f4f8-2402-4d8c-841a-53bd4858e35d
|
||||
09:44:26.558 INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
|
||||
09:44:26.565 INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
|
||||
09:44:26.593 DEBUG msg="selfheal: node check healthy" heal=networking
|
||||
09:44:26.676 INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
|
||||
09:44:26.794 INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s guests_dir=/var/lib/felhom-agent/guests
|
||||
09:44:26.794 INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
|
||||
09:44:26.794 INFO msg="local-api server listening" addr=169.254.253.1:8443
|
||||
09:44:26.795 INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
|
||||
09:44:26.795 INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
|
||||
09:44:26.85+ DEBUG msg="pbs: credential probe inconclusive (not a rejection)" storage=felhom-pbs err="pbs: GET /version: Get \"https://10.77.0.1:8007/api2/json/version\": context deadline exceeded"
|
||||
09:44:26.85+ DEBUG msg="pbs: live list failed; using last-known-good" datastore=felhom-offsite err="pbs: GET /admin/datastore/felhom-offsite/snapshots?ns=Tester4: Get \"https://10.77.0.1:8007/api2/json/admin/datastore/felhom-offsite/snapshots?ns=Tes
|
||||
09:44:27.752 INFO msg="felhomsshd: config applied" port=8822 action=reload
|
||||
09:44:27.853 INFO msg="felhomsshd belt: set synced" set=ssh_port elements=8822
|
||||
09:44:33.405 DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true
|
||||
09:44:34.913 DEBUG msg="hub: report sent" guests=1 duration_ms=23389 blocked=false desired_generation=2 has_signed_ops=false
|
||||
09:44:34.913 DEBUG msg="desired: generation advanced — fetching desired-state" have_generation=0 envelope_generation=2
|
||||
09:44:34.926 DEBUG msg="desired: fetched" generation=2 duration_ms=13
|
||||
09:44:34.926 INFO msg="desired: updated from hub" generation=2 guests=0
|
||||
09:44:35.02+ INFO msg="felhomsshd: operator authorized_keys updated" user=felhom-op present=true
|
||||
09:44:35.115 INFO msg="felhomsshd belt: set synced" set=operator_ips elements=10.77.0.250
|
||||
09:44:37.818 INFO msg="local-api: same-disk" vmid=9201 a=/mnt/sys_drive b=/mnt/felhom-drives/belso different=false proven=true reason=same_disk
|
||||
09:44:37.818 DEBUG msg="local-api: request" method=POST path=/disks/same-disk status=200 duration_ms=7228
|
||||
09:44:41.573 WARN msg="pbs: verify trigger failed; reporting current snapshot state" datastore=felhom-offsite err="pbs: POST /admin/datastore/felhom-offsite/verify: Post \"https://10.77.0.1:8007/api2/json/admin/datastore/felhom-offsite/verify\": co
|
||||
09:44:55.34+ DEBUG msg="local-api: /host/crash-guard served" vmid=9201 last_boot_at=2026-10-21T07:43:58Z last_boot_unclean=false tripped=true
|
||||
09:44:55.34+ DEBUG msg="local-api: request" method=GET path=/host/crash-guard status=200 duration_ms=0
|
||||
09:45:03.144 DEBUG msg="local-api: request" method=GET path=/disks status=200 duration_ms=7397
|
||||
09:45:11.574 WARN msg="pbs: snapshot list failed" datastore=felhom-offsite err="pbs: GET /admin/datastore/felhom-offsite/snapshots?ns=Tester4: Get \"https://10.77.0.1:8007/api2/json/admin/datastore/felhom-offsite/snapshots?ns=Tester4\": context dea
|
||||
09:45:32.956 DEBUG msg="local-api: request" method=GET path=/storage status=200 duration_ms=7204
|
||||
09:46:11.039 WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=felhom-pbs err="proxmox: GET /nodes/felhom/storage/felhom-pbs/content: Get \"https://127.0.0.1:8006/api2/
|
||||
09:46:11.039 DEBUG msg="local-api: request" method=GET path=/backup/status status=200 duration_ms=15001
|
||||
09:46:25.467 DEBUG msg="local-api: request" method=GET path=/backup/tiers status=200 duration_ms=14427
|
||||
09:46:40.054 INFO msg="local-api: same-disk" vmid=9201 a=/mnt/felhom-drives/hdd_1 b=/mnt/sys_drive different=true proven=false reason=vm
|
||||
09:49:26.796 DEBUG msg="fstrim: outside the daytime window — not looking" local="Wed 09:49"
|
||||
09:49:39.71+ WARN msg="local-api: storage view unavailable for the tier presence report" target=felhom-pbs err="storage: NodeStorage: proxmox: GET /nodes/felhom/storage: Get \"https://127.0.0.1:8006/api2/json/nodes/felhom/storage\": context cancele
|
||||
09:51:12.002 DEBUG msg="local-api: request" method=GET path=/debug/logs status=200 duration_ms=1
|
||||
09:54:39.068 INFO msg="guest-power: watchdog alive" sweeps_since_boot=10 guests_evaluated=1 currently_stopped=0
|
||||
09:54:40.09+ INFO msg="controller-supervisor: alive" sweeps_since_boot=20 guests_evaluated=1 controllers_not_running=0
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
# ep0 wg0 peers, handshake age in seconds, read-only, 2026-10-21T07:56:57Z
|
||||
10.77.0.2/32 handshake_age_s=7
|
||||
10.77.0.250/32 handshake_age_s=116
|
||||
10.77.0.3/32 handshake_age_s=110
|
||||
10.77.0.4/32 handshake_age_s=65
|
||||
10.77.0.5/32 handshake_age_s=1432134
|
||||
10.77.0.6/32 handshake_age_s=970
|
||||
@@ -0,0 +1,37 @@
|
||||
# B6 ★8 — the clock wrong at boot: −2 years, then +2 years, then the real time (2026-10-21 07:15–07:58 real UTC)
|
||||
|
||||
Method (`00-expected.txt`): NTP dropped at the VM's tap (314 packets dropped in all — positive control); clean restarts
|
||||
with the RTC set; clock control = the box's HTTP Date header (`01-run.txt`).
|
||||
|
||||
## −2 years (RTC 2024-10-21)
|
||||
|
||||
- **The box came up at 2026-04-13 19:38, not 2024** — the operating system will not run before its own build date and
|
||||
raised the clock to it. So the real test was „6 months behind".
|
||||
- The hub and the registry refuse the box: „x509: certificate … not yet valid". The dashboard says only **„Hub nem
|
||||
elérhető"** — not that the clock is wrong, although the log knows the exact reason.
|
||||
- **Paperless crash-looped and the box stopped it** with a restore hold („crash_loop — 7 in 10m0s; STOPPING it"); its
|
||||
warning event could not reach the hub, so nobody was told. Its log was empty when read (`03-…`).
|
||||
- Nothing was deleted; no backup ran.
|
||||
|
||||
## +2 years (RTC 2028-10-21)
|
||||
|
||||
- Every tier looked 2 years old: a local whole-box catch-up ran at once (`vzdump-lxc-9201-2028_10_21-09_34_16`, 4.6 GB).
|
||||
The weekly PBS tier had not started when the clock was put back. Hub and registry refused the box („certificate has
|
||||
expired"). No deletion: off-site retention is the hub's (decision 68), local keep-daily 3 kept all three archives.
|
||||
|
||||
## Back to the real time (07:43:47 real) — two lasting effects
|
||||
|
||||
1. **The box's WireGuard tunnel to ep0 is dead** (`07-…`, `08-ep0-wg-handshakes.txt`): every PBS call from the agent
|
||||
times out; the agent answers the controller so slowly that the backups page says „A host-ügynök jelenleg nem
|
||||
elérhető" 10+ minutes after the return. ep0's side, read-only: Tester4's last handshake was 07:40:47 — while its
|
||||
clock said 2028 — and none since, while the other peers shook hands 21–96 s ago. WireGuard refuses a handshake whose
|
||||
timestamp is older than the last one it accepted, so the tunnel stays dead until the real date passes 2028 or the
|
||||
peer is reset on ep0. **→ R-1201.** Not reset (ep0 is read-only in this run).
|
||||
2. **The local whole-box tier's newest archive is dated 2028-10-21.** By the code (`localapi/server.go:1187`), the due
|
||||
rule answers „within cadence" until then — the D3/T11 shape (R-1189 a). The live answer was not read: the controller
|
||||
asks only inside its night window, and the box dies in B7 before tonight.
|
||||
- Paperless stayed stopped after the return with a clear sentence („leállítottuk, mert újra és újra összeomlott … Az
|
||||
Indítás gombbal újra megpróbálhatod"); one Start → running in 1 minute. **→ R-1202** (with the „Hub nem elérhető" text).
|
||||
|
||||
**Verdict: FINDINGS.** A box that runs once with its clock ahead loses its DR tunnel (R-1201, P2) and, by the code, its
|
||||
whole-box schedule (R-1189 a); a box with its clock behind stops an app silently and blames the hub (R-1202, P3).
|
||||
@@ -84,3 +84,4 @@
|
||||
| 05:27–05:45 | due checks | R-1019 closed (demo-felhom read-only, two channels), R-895 first verdict recorded, R-435 moved to 10-27 (logs gone) |
|
||||
| 05:40 | mail | the nightly DooPlex secret sweep named three rehearsal secrets in session records: OLD_BIND_TOKEN, BIND_TOKEN, SETUP_CODE — my mail reads (full text of the bind and setup mails) carried them in; all three are one-time and already spent. From now on mail is read names-only |
|
||||
| 05:58–07:13 | B6 ★7 | DST live: the first 02:30 skipped, the second ran once (01:30:00Z box), PASS; side effects of the 4-day jump: a local whole-box archive and off-site snapshots dated 2026-10-25; W30 |
|
||||
| 07:15–07:58 | B6 ★8 | −2 y (OS raised it to 2026-04-13): „Hub nem elérhető" only, Paperless stopped silently → R-1202; +2 y: whole-box archive dated 2028; back to real time: ep0 WireGuard tunnel dead (handshake replay) → R-1201 (P2); R-1189 (a) half-live |
|
||||
|
||||
Reference in New Issue
Block a user