diff --git a/documentation/audits/BIGNIGHT-household-month-2026-09-14.md b/documentation/audits/BIGNIGHT-household-month-2026-09-14.md index 138df241..107ae384 100644 --- a/documentation/audits/BIGNIGHT-household-month-2026-09-14.md +++ b/documentation/audits/BIGNIGHT-household-month-2026-09-14.md @@ -71,3 +71,4 @@ reached the box 14 m 25 s after the push, „Frissítés elérhető — ma", **D | F5 | drive out 30 min, then back | badge „Aktív" at once; stale banner cleared within 10 min | re-bound as `sdc` by UUID, ext4 recovery, restarted the 4 apps | **91 s** | `storage_reconnected (info)` / true | — | | F6 | drive unplugged during a backup | nothing about skipped apps | skipped 4 volume dumps but `success:true`; torn `.tmp` not promoted; next run honest and complete | 122 s after re-attach | `storage_disconnected` + 4, **all mails suppressed by cooldown** | the second drive loss (R-521); the incomplete run (R-519) | | F7 | system disk to 95 % | dashboard „90 % · Kritikusan kevés hely"; banner **English** „SSD disk usage high: 90%" | stayed reachable; uploads and a backup still succeeded; warnings cleared 3½ min after cleanup | — | `health_degraded` / true, **mail suppressed by cooldown**; no disk event | operator told nothing (R-521) | +| F8 | internet gone 17½ min (venue: hub on the LAN, so the hub's ingress was blocked too) | LAN dashboard 200 throughout, **no offline notice, tunnel tile „Fut"** (R-522) | report push failed and backed off; on return pushed at once, tunnel re-registered +9 s (all 4 +58 s) | ≈ 1 min | `node_stale` + mail at 31 min since last report, `node_recovered` + mail / true | customer notice (R-522) | diff --git a/documentation/audits/evidence-bignight-2026-09-14/alarm-truth-table.md b/documentation/audits/evidence-bignight-2026-09-14/alarm-truth-table.md index 110d1372..97e2752b 100644 --- a/documentation/audits/evidence-bignight-2026-09-14/alarm-truth-table.md +++ b/documentation/audits/evidence-bignight-2026-09-14/alarm-truth-table.md @@ -28,6 +28,8 @@ Source: hub log (`alarms.sh`, CEST times), the operator mailbox and the customer | 22:38:45 | `app_start_failed (warning)` × 4 | F6 consequence | **suppressed — cooldown** × 4 | true, not delivered | | 22:39:45 | `health_degraded (warning)` „Rendszer állapot romlott (volt: ok)" | F6 | operator | true | | 22:54:46 | `health_degraded (warning)` | F7 system disk 95 % | **suppressed — cooldown** | true, not delivered; no disk-specific event exists (R-521) | +| 23:25:43 | `node_stale` (staleness ok → stale) | F8 internet cut (31 min after the last report) | operator | true | +| 23:26:43 | `node_recovered` | F8 internet back | operator | true | Hub WARN lines without an event or mail: `host tester-1-a61396 backup FAILED: target=felhom-pbs … does not exist` at 21:12:43 and 21:27:43 (the agent's own PBS attempts). @@ -45,3 +47,4 @@ Hub WARN lines without an event or mail: `host tester-1-a61396 backup FAILED: ta | F5 drive back after 30 min | drive back | `storage_reconnected (info)` | correct; customer banner stale 2–10 min | | F6 drive lost during backup | drive lost; backup incomplete | `storage_disconnected` logged, **mail suppressed by cooldown**; the run reported `success:true` | **MISSED** on both counts (R-521, R-519) | | F7 system disk 95 % | disk nearly full | `health_degraded` only, mail suppressed; customer banner in English | **MISSED** for the operator (R-521); customer told, in English (R-516) | +| F8 internet gone 17½ min | box offline | `node_stale` + mail, then `node_recovered` + mail | correct for the operator; the customer's dashboard says „Fut" (R-522) | diff --git a/documentation/audits/evidence-bignight-2026-09-14/journal.md b/documentation/audits/evidence-bignight-2026-09-14/journal.md index 74d7ebe8..b7bce26a 100644 --- a/documentation/audits/evidence-bignight-2026-09-14/journal.md +++ b/documentation/audits/evidence-bignight-2026-09-14/journal.md @@ -608,3 +608,22 @@ probe file is the observable.) Hub at 21:08:42Z (just after the cut): „Last report: 14 min ago · **warn**" — **not a stall**: the controller's `hub-report` job runs every 15 min (pushed 20:39:46Z and 20:54:46Z, `Hub report pushed successfully`); the „warn" is the state that last report carried (F7's disk). The next push, due 21:09:46Z, fell inside the cut. + +**Phase 6 pre-note (written 21:13Z, before the phase):** the brief asks to restore one DB-backed app **from off-site +onto 9202**. On this record there is no off-site copy to restore from — restic Tier 3 was off (and not ticked: money), +and the DR tier never provisioned (R-511); `/backup/offbox/status` → `snapshots 0`. **Not followed, one line why:** the +promise „restore from off-site" cannot be walked without an off-site tier, and faking one (copying a unit by hand onto +9202) would prove a path no customer has. Phase 6 instead restores a DB-backed app from its **local** copy on the box +itself, through the product's restore endpoint, and reads the data back — recorded as such. + +**F8 result (attempt 3)** (`phase5/F8-internet-gone.txt`, `phase5/F8/`): + +| | | +|---|---| +| how | bridge rule on demo-hp, `tap333i0` only: 21:08:36 → **21:26:07Z** — **17 m 31 s**, not the brief's 20 (the runner's loop count was short — harness). Dropped at removal: 236 packets to the hub, 1 393 to the internet, 54 IPv6 | +| LAN dashboard still works? | **yes** — 200 on every probe from demo-hp (every 26 s); polled pages (every 2 min) showed **no banner at all** and the tile „Cloudflare Tunnel … **Fut** · Védett" the whole time → **R-522 (P3)** | +| what the box did | cloudflared ≈ 20 errors per 2 min (`failed to dial to edge with quic: timeout`), retrying; controller `[report] Push failed … context deadline exceeded` 21:11:26Z, `Job hub-report failed … after 3 attempts`, wait channel backing off; public name 502 → **530** (no connector) | +| on return | report pushed **at 21:26:07Z, the same second** the rule went; tunnel re-registered **+9 s** (21:26:16Z), all four connections **+58 s** (21:27:05Z) — **reconnected by itself** ✓; public name back to 502 (R-510) | +| hub shows DOWN, and when | never DOWN (threshold 1 h). Status „warn · Last report N min ago" climbing 14 → 31 min; **`node_stale` 21:25:43Z** (31 min after the last report, 17 min into the cut) + operator mail; **`node_recovered` 21:26:43Z** + mail | +| alarms true? | `node_stale` true; `node_recovered` true — a cut shorter than ≈ 13 min after a report would raise nothing, by design | +| should have fired, did not | a customer-side notice that the box is offline (R-522) | diff --git a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8-internet-gone.txt b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8-internet-gone.txt index 62be4019..73df2c07 100644 --- a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8-internet-gone.txt +++ b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8-internet-gone.txt @@ -29,3 +29,117 @@ bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{ bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 21:12:36 +240s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:13:02 +266s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:13:28 +292s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:13:55 +319s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:14:21 +345s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:14:47 +371s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:15:13 +397s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:15:39 +423s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:16:06 +450s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:16:32 +476s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:16:58 +502s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:17:24 +528s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:17:50 +554s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:18:17 +581s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:18:43 +607s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:19:09 +633s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:19:35 +659s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:20:01 +685s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:20:28 +712s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:20:54 +738s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:21:20 +764s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:21:46 +790s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:22:12 +816s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:22:38 +842s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:23:05 +869s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:23:30 +894s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:23:57 +921s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:24:23 +947s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:24:49 +973s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:25:15 +999s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 +21:25:41 +1025s LAN-dashboard=200 guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' public-name=530 + iifname "tap333i0" ip daddr 192.168.0.192 counter packets 236 bytes 14376 drop + iifname "tap333i0" ip daddr 192.168.0.0/24 counter packets 1604 bytes 1112016 accept + iifname "tap333i0" ether type ip counter packets 1393 bytes 932292 drop + iifname "tap333i0" ether type ip6 counter packets 54 bytes 6264 drop +0 +F8 internet back at 21:26:07 (cut 1051 s) +21:26:09 +2s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:26:30 +23s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=1 +21:26:52 +45s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=1 +21:27:14 +67s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=4 +21:27:35 +88s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=3 +21:27:57 +110s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=3 +21:28:19 +132s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:28:40 +153s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:29:02 +175s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:29:24 +197s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:29:46 +219s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:30:07 +240s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:30:29 +262s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:30:51 +284s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:31:12 +305s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:31:34 +327s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:31:56 +349s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:32:18 +371s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:32:39 +392s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:33:01 +414s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:33:23 +436s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:33:44 +457s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:34:06 +479s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 +21:34:28 +501s guest->internet=bash: -c: line 1: syntax error near unexpected token `(' +bash: -c: line 1: `pct exec 9201 -- sh -c \"a=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://1.1.1.1/ || echo fail); b=\$(curl -s -o /dev/null -m 5 -w %{http_code} https://hub.felhom.eu/ || echo fail); echo 1.1.1.1:\$a hub:\$b\"' cloudflared-registered-last-60s=0 diff --git a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/cloudflared-reconnect.txt b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/cloudflared-reconnect.txt new file mode 100644 index 00000000..fe30fad8 --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/cloudflared-reconnect.txt @@ -0,0 +1,9 @@ +2026-09-14T21:25:39Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 +2026-09-14T21:25:39Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:25:39Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:25:49Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 +grep: write error: Broken pipe +2026-09-14T21:26:16Z INF Registered tunnel connection connIndex=0 connection=38a +2026-09-14T21:27:04Z INF Registered tunnel connection connIndex=2 connection=71f +2026-09-14T21:27:05Z INF Registered tunnel connection connIndex=1 connection=a6e +2026-09-14T21:27:05Z INF Registered tunnel connection connIndex=3 connection=961 diff --git a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/cloudflared.log b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/cloudflared.log new file mode 100644 index 00000000..1dcc3aed --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/cloudflared.log @@ -0,0 +1,299 @@ +2026-09-14T21:08:00Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=3 event=0 ip=198.41.200.73 +2026-09-14T21:08:00Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=2 event=0 ip=198.41.192.67 +2026-09-14T21:08:00Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=1 event=0 ip=198.41.192.107 +2026-09-14T21:08:00Z ERR failed to run the datagram handler error="timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:08:00Z ERR failed to accept incoming stream requests error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:08:00Z WRN failed to serve tunnel connection error="datagram manager encountered a failure while serving" connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:08:00Z ERR Serve tunnel error error="datagram manager encountered a failure while serving" connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:08:00Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:08:00Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:08:05Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 event=0 ip=198.41.192.67 +2026-09-14T21:08:05Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 event=0 ip=198.41.200.73 +2026-09-14T21:08:05Z INF Retrying connection in up to 8s connIndex=2 event=0 ip=198.41.192.67 +2026-09-14T21:08:05Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 event=0 ip=198.41.192.107 +2026-09-14T21:08:05Z INF Retrying connection in up to 8s connIndex=1 event=0 ip=198.41.192.107 +2026-09-14T21:08:05Z INF Retrying connection in up to 8s connIndex=3 event=0 ip=198.41.200.73 +2026-09-14T21:08:05Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:08:05Z INF Retrying connection in up to 4s connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:08:07Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 +2026-09-14T21:08:08Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:08:08Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 +2026-09-14T21:08:11Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 +2026-09-14T21:08:13Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:08:13Z INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:08:20Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:08:21Z INF Registered tunnel connection connIndex=0 connection=5c678f49-4083-40b2-b8ac-760ac9a61a16 event=0 ip=198.41.200.73 location=vie05 protocol=quic +2026-09-14T21:08:37Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=0 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:08:37Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=0 dest=https://felhom.enkicsifelhom.hu/ event=0 ip=198.41.200.73 type=http +2026-09-14T21:08:45Z ERR failed to accept incoming stream requests error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:08:45Z ERR failed to run the datagram handler error="timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:08:45Z ERR failed to serve tunnel connection error="accept stream listener encountered a failure while serving" connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:08:45Z ERR Serve tunnel error error="accept stream listener encountered a failure while serving" connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:08:45Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:08:47Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:08:52Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:08:52Z INF Retrying connection in up to 4s connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:08:54Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:08:59Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:08:59Z INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:09:00Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=3 event=0 ip=198.41.200.113 +2026-09-14T21:09:00Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=1 event=0 ip=198.41.192.47 +2026-09-14T21:09:00Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=2 event=0 ip=198.41.192.167 +2026-09-14T21:09:00Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:09:05Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 event=0 ip=198.41.192.167 +2026-09-14T21:09:05Z INF Retrying connection in up to 16s connIndex=2 event=0 ip=198.41.192.167 +2026-09-14T21:09:05Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 event=0 ip=198.41.200.113 +2026-09-14T21:09:05Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 event=0 ip=198.41.192.47 +2026-09-14T21:09:05Z INF Retrying connection in up to 16s connIndex=3 event=0 ip=198.41.200.113 +2026-09-14T21:09:05Z INF Retrying connection in up to 16s connIndex=1 event=0 ip=198.41.192.47 +2026-09-14T21:09:05Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 +2026-09-14T21:09:05Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:09:05Z INF Retrying connection in up to 16s connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:09:07Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:09:12Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:09:12Z INF Retrying connection in up to 32s connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:09:13Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 +2026-09-14T21:09:19Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 +2026-09-14T21:09:30Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:09:35Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:09:35Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:09:47Z ERR Failed to refresh DNS local resolver error="lookup region1.v2.argotunnel.com: i/o timeout" +2026-09-14T21:09:56Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.113 +2026-09-14T21:10:01Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.113 +2026-09-14T21:10:01Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.113 +2026-09-14T21:10:10Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:10:15Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:10:15Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:10:20Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.33 +2026-09-14T21:10:25Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.33 +2026-09-14T21:10:25Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.33 +2026-09-14T21:10:41Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.193 +2026-09-14T21:10:46Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.193 +2026-09-14T21:10:46Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.193 +2026-09-14T21:10:51Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.43 +2026-09-14T21:10:56Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.43 +2026-09-14T21:10:56Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.43 +2026-09-14T21:11:24Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:11:29Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:11:29Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:11:52Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.233 +2026-09-14T21:11:53Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=2 event=0 ip=198.41.192.27 +2026-09-14T21:11:53Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=1 event=0 ip=198.41.192.67 +2026-09-14T21:11:53Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=3 event=0 ip=198.41.200.63 +2026-09-14T21:11:57Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.233 +2026-09-14T21:11:57Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.233 +2026-09-14T21:11:58Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 event=0 ip=198.41.192.67 +2026-09-14T21:11:58Z INF Retrying connection in up to 32s connIndex=1 event=0 ip=198.41.192.67 +2026-09-14T21:11:58Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 event=0 ip=198.41.192.27 +2026-09-14T21:11:58Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 event=0 ip=198.41.200.63 +2026-09-14T21:11:58Z INF Retrying connection in up to 32s connIndex=3 event=0 ip=198.41.200.63 +2026-09-14T21:11:58Z INF Retrying connection in up to 32s connIndex=2 event=0 ip=198.41.192.27 +2026-09-14T21:12:02Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 +2026-09-14T21:12:05Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:12:10Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:12:10Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:12:13Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 +2026-09-14T21:12:24Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 +2026-09-14T21:12:27Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:12:32Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:12:32Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:12:33Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:12:38Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:12:38Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:12:44Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:12:49Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:12:49Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:13:18Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:13:23Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:13:23Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:13:47Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:13:52Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:13:52Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:14:22Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.43 +2026-09-14T21:14:27Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.43 +2026-09-14T21:14:27Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.43 +2026-09-14T21:14:46Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:14:51Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:14:51Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:14:52Z ERR Failed to refresh DNS local resolver error="lookup region1.v2.argotunnel.com: i/o timeout" +2026-09-14T21:15:16Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.193 +2026-09-14T21:15:21Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.193 +2026-09-14T21:15:21Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.193 +2026-09-14T21:15:51Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:15:56Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:15:56Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:16:27Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:16:32Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:16:32Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:16:34Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=1 event=0 ip=198.41.192.77 +2026-09-14T21:16:34Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=2 event=0 ip=198.41.192.47 +2026-09-14T21:16:34Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=3 event=0 ip=198.41.200.23 +2026-09-14T21:16:39Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 event=0 ip=198.41.192.77 +2026-09-14T21:16:39Z INF Retrying connection in up to 1m4s connIndex=1 event=0 ip=198.41.192.77 +2026-09-14T21:16:39Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 event=0 ip=198.41.192.47 +2026-09-14T21:16:39Z INF Retrying connection in up to 1m4s connIndex=2 event=0 ip=198.41.192.47 +2026-09-14T21:16:39Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 event=0 ip=198.41.200.23 +2026-09-14T21:16:39Z INF Retrying connection in up to 1m4s connIndex=3 event=0 ip=198.41.200.23 +2026-09-14T21:16:43Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:16:48Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:16:48Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:16:55Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:16:55Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 +2026-09-14T21:16:56Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 +2026-09-14T21:17:00Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:17:00Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:17:02Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 +2026-09-14T21:17:08Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:17:14Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:17:14Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:17:37Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:17:42Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:17:42Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:18:00Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.193 +2026-09-14T21:18:05Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.193 +2026-09-14T21:18:05Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.193 +2026-09-14T21:18:28Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:18:33Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:18:33Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:18:34Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=1 event=0 ip=198.41.192.57 +2026-09-14T21:18:34Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=3 event=0 ip=198.41.200.113 +2026-09-14T21:18:34Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=2 event=0 ip=198.41.192.67 +2026-09-14T21:18:36Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:18:39Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 event=0 ip=198.41.192.57 +2026-09-14T21:18:39Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 event=0 ip=198.41.192.67 +2026-09-14T21:18:39Z INF Retrying connection in up to 1m4s connIndex=1 event=0 ip=198.41.192.57 +2026-09-14T21:18:39Z INF Retrying connection in up to 1m4s connIndex=2 event=0 ip=198.41.192.67 +2026-09-14T21:18:39Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 event=0 ip=198.41.200.113 +2026-09-14T21:18:39Z INF Retrying connection in up to 1m4s connIndex=3 event=0 ip=198.41.200.113 +2026-09-14T21:18:41Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:18:41Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:18:53Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 +2026-09-14T21:19:05Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 +2026-09-14T21:19:09Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 +2026-09-14T21:19:10Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.113 +2026-09-14T21:19:15Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.113 +2026-09-14T21:19:15Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.113 +2026-09-14T21:19:19Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:19:24Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:19:24Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:19:25Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.113 +2026-09-14T21:19:30Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.113 +2026-09-14T21:19:30Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.113 +2026-09-14T21:19:48Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=2 event=0 ip=198.41.192.77 +2026-09-14T21:19:48Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=3 event=0 ip=198.41.200.43 +2026-09-14T21:19:48Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=1 event=0 ip=198.41.192.107 +2026-09-14T21:19:50Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:19:53Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 event=0 ip=198.41.192.107 +2026-09-14T21:19:53Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 event=0 ip=198.41.192.77 +2026-09-14T21:19:53Z INF Retrying connection in up to 1m4s connIndex=1 event=0 ip=198.41.192.107 +2026-09-14T21:19:53Z INF Retrying connection in up to 1m4s connIndex=2 event=0 ip=198.41.192.77 +2026-09-14T21:19:53Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 event=0 ip=198.41.200.43 +2026-09-14T21:19:53Z INF Retrying connection in up to 1m4s connIndex=3 event=0 ip=198.41.200.43 +2026-09-14T21:19:55Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:19:55Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:19:56Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:19:57Z ERR Failed to refresh DNS local resolver error="lookup region1.v2.argotunnel.com: i/o timeout" +2026-09-14T21:20:01Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:20:01Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.23 +2026-09-14T21:20:09Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 +2026-09-14T21:20:16Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 +2026-09-14T21:20:16Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 +2026-09-14T21:20:17Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:20:22Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:20:22Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:20:50Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:20:55Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:20:55Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:21:12Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:21:17Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:21:17Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:21:40Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:21:45Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:21:45Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:21:55Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:22:00Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:22:00Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:22:19Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:22:24Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:22:24Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:22:55Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.113 +2026-09-14T21:23:00Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.113 +2026-09-14T21:23:00Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.113 +2026-09-14T21:23:04Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:23:09Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:23:09Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:23:27Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:23:32Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:23:32Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:23:45Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:23:50Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:23:50Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:24:19Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.233 +2026-09-14T21:24:24Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.233 +2026-09-14T21:24:24Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.233 +2026-09-14T21:24:38Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:24:43Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:24:43Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:25:02Z ERR Failed to refresh DNS local resolver error="lookup region1.v2.argotunnel.com: i/o timeout" +2026-09-14T21:25:04Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:25:09Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:25:09Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.13 +2026-09-14T21:25:16Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=2 event=0 ip=198.41.192.47 +2026-09-14T21:25:16Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=3 event=0 ip=198.41.200.193 +2026-09-14T21:25:16Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=1 event=0 ip=198.41.192.57 +2026-09-14T21:25:21Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 event=0 ip=198.41.192.57 +2026-09-14T21:25:21Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 event=0 ip=198.41.192.47 +2026-09-14T21:25:21Z INF Retrying connection in up to 1m4s connIndex=1 event=0 ip=198.41.192.57 +2026-09-14T21:25:21Z INF Retrying connection in up to 1m4s connIndex=2 event=0 ip=198.41.192.47 +2026-09-14T21:25:21Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 event=0 ip=198.41.200.193 +2026-09-14T21:25:21Z INF Retrying connection in up to 1m4s connIndex=3 event=0 ip=198.41.200.193 +2026-09-14T21:25:34Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:25:39Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=2 +2026-09-14T21:25:39Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:25:39Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:25:49Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=1 +2026-09-14T21:25:51Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:25:53Z ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3 +2026-09-14T21:25:56Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:25:56Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.73 +2026-09-14T21:25:59Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:26:04Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:26:04Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.63 +2026-09-14T21:26:16Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=0 event=0 ip=198.41.200.53 +2026-09-14T21:26:16Z INF Registered tunnel connection connIndex=0 connection=38a36b5b-8947-44c7-8f6f-f4ec4c77ebfa event=0 ip=198.41.200.53 location=vie06 protocol=quic +2026-09-14T21:27:01Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=0 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:27:01Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=0 dest=https://felhom.enkicsifelhom.hu/api/health event=0 ip=198.41.200.53 type=http +2026-09-14T21:27:02Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=0 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:27:02Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=0 dest=https://wiki.enkicsifelhom.hu/ event=0 ip=198.41.200.53 type=http +2026-09-14T21:27:04Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=3 event=0 ip=198.41.200.13 +2026-09-14T21:27:04Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=2 event=0 ip=198.41.192.227 +2026-09-14T21:27:04Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=1 event=0 ip=198.41.192.27 +2026-09-14T21:27:04Z INF Registered tunnel connection connIndex=2 connection=71f90c5a-39fb-4fcb-9afe-0cac3c86a736 event=0 ip=198.41.192.227 location=bud01 protocol=quic +2026-09-14T21:27:05Z INF Registered tunnel connection connIndex=1 connection=a6ef9710-fe33-45bb-92dd-c910a50adb12 event=0 ip=198.41.192.27 location=bud01 protocol=quic +2026-09-14T21:27:05Z INF Registered tunnel connection connIndex=3 connection=961ee7f4-56f1-4342-ae49-4fafe2794365 event=0 ip=198.41.200.13 location=vie06 protocol=quic +2026-09-14T21:28:01Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:28:01Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 dest=https://felhom.enkicsifelhom.hu/api/health event=0 ip=198.41.200.13 type=http +2026-09-14T21:28:02Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:28:02Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 dest=https://wiki.enkicsifelhom.hu/ event=0 ip=198.41.200.13 type=http +2026-09-14T21:29:01Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:29:01Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 dest=https://felhom.enkicsifelhom.hu/api/health event=0 ip=198.41.200.13 type=http +2026-09-14T21:29:02Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:29:02Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 dest=https://wiki.enkicsifelhom.hu/ event=0 ip=198.41.200.13 type=http +2026-09-14T21:30:01Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:30:01Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 dest=https://felhom.enkicsifelhom.hu/api/health event=0 ip=198.41.200.13 type=http +2026-09-14T21:30:02Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:30:02Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 dest=https://wiki.enkicsifelhom.hu/ event=0 ip=198.41.200.13 type=http +2026-09-14T21:31:01Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:31:01Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 dest=https://felhom.enkicsifelhom.hu/api/health event=0 ip=198.41.200.13 type=http +2026-09-14T21:31:02Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:31:02Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 dest=https://wiki.enkicsifelhom.hu/ event=0 ip=198.41.200.13 type=http +2026-09-14T21:32:01Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:32:01Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 dest=https://felhom.enkicsifelhom.hu/api/health event=0 ip=198.41.200.13 type=http +2026-09-14T21:32:02Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:32:02Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 dest=https://wiki.enkicsifelhom.hu/ event=0 ip=198.41.200.13 type=http +2026-09-14T21:33:01Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:33:01Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 dest=https://felhom.enkicsifelhom.hu/api/health event=0 ip=198.41.200.13 type=http +2026-09-14T21:33:02Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:33:02Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 dest=https://wiki.enkicsifelhom.hu/ event=0 ip=198.41.200.13 type=http +2026-09-14T21:33:31Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 event=1 ingressRule=0 originService=https://traefik +2026-09-14T21:33:31Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: failed to verify certificate: x509: certificate is valid for a581424e2258c779d00f472f76e75f7d.df397237e7abc5030bf59ec00fc297ba.traefik.default, not traefik" connIndex=3 dest=https://felhom.enkicsifelhom.hu/ event=0 ip=198.41.200.13 type=http diff --git a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/controller.log b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/controller.log new file mode 100644 index 00000000..98dfc493 --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/controller.log @@ -0,0 +1,436 @@ +2026/09/14 21:08:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:08:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:08:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:08:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:08:45 [INFO] [scheduler] Running job: stack-scan +2026/09/14 21:08:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:08:45 [INFO] [stacks] ScanStacks complete: 57 stacks found (12 deployed, 45 available) +2026/09/14 21:08:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:08:45 [INFO] [scheduler] Job stack-scan completed (took 88ms) +2026/09/14 21:08:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:08:45 [INFO] [scheduler] Job agent-channel-health completed (took 179ms) +2026/09/14 21:08:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:09:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:09:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:09:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:09:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:09:35 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:09:44 [INFO] [sync] Starting catalog sync +2026/09/14 21:09:44 [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (branch: main) +2026/09/14 21:09:45 [INFO] [scheduler] Running job: hub-report +2026/09/14 21:09:45 [INFO] [report] Building system report +2026/09/14 21:09:45 [INFO] [scheduler] Running job: offsite-credential-retry +2026/09/14 21:09:45 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/09/14 21:09:45 [INFO] [scheduler] Running job: backup-cache +2026/09/14 21:09:45 [INFO] [scheduler] Running job: system-health +2026/09/14 21:09:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:09:45 [INFO] [backup] Found 6 DB dump files across drives +2026/09/14 21:09:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:09:45 [INFO] [backup] Discovered 6 databases +2026/09/14 21:09:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/14 21:09:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/09/14 21:09:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/14 21:09:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/14 21:09:45 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml +2026/09/14 21:09:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/14 21:09:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/mealie/docker-compose.yml +2026/09/14 21:09:45 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/14 21:09:45 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/14 21:09:45 [INFO] [monitor] Health check: status=ok +2026/09/14 21:09:45 [INFO] [scheduler] Job system-health completed (took 309ms) +2026/09/14 21:09:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/14 21:09:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/14 21:09:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/14 21:09:45 [INFO] [backup] Discovered app data: 12 apps +2026/09/14 21:09:45 [INFO] [backup] Backup status cache refreshed +2026/09/14 21:09:45 [INFO] [scheduler] Job backup-cache completed (took 346ms) +2026/09/14 21:09:45 [INFO] [monitor] Health check: status=ok +2026/09/14 21:09:45 [INFO] [scheduler] Job agent-channel-health completed (took 210ms) +2026/09/14 21:09:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:09:55 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:10:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:10:05 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:10:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:10:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:10:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:10:45 [INFO] [scheduler] Running job: stack-scan +2026/09/14 21:10:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:10:45 [INFO] [stacks] ScanStacks complete: 57 stacks found (12 deployed, 45 available) +2026/09/14 21:10:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:10:45 [INFO] [scheduler] Job stack-scan completed (took 104ms) +2026/09/14 21:10:45 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:10:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:10:45 [INFO] [scheduler] Job agent-channel-health completed (took 182ms) +2026/09/14 21:10:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:11:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:11:05 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:11:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:11:15 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:11:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:11:26 [WARN] [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": context deadline exceeded (Client.Timeout exceeded while awaiting headers) +2026/09/14 21:11:26 [ERROR] [scheduler] Job hub-report failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": context deadline exceeded (Client.Timeout exceeded while awaiting headers) (took 1m40.88s) +2026/09/14 21:11:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:11:35 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:11:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:11:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:11:45 [INFO] [scheduler] Job agent-channel-health completed (took 185ms) +2026/09/14 21:11:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:11:59 [ERROR] [sync] Catalog sync failed: git fetch: git fetch --depth 1 origin main: context deadline exceeded (subprocess killed at the 2m0s deadline): signal: killed +stderr: fatal: unable to access 'https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git/': Failed to connect to gitea.dooplex.hu port 443 after 135199 ms: Couldn't connect to server +2026/09/14 21:11:59 [INFO] [sync] Periodic sync: Git hiba: git fetch: git fetch --depth 1 origin main: context deadline exceeded (subprocess killed at the 2m0s deadline): signal: killed +stderr: fatal: unable to access 'https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git/': Failed to connect to gitea.dooplex.hu port 443 after 135199 ms: Couldn't connect to server +2026/09/14 21:12:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:12:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:12:24 [WARN] [report] wait channel error: context deadline exceeded — backing off (the 15-min cycle still reconciles) +2026/09/14 21:12:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:12:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:12:45 [INFO] [scheduler] Running job: stack-scan +2026/09/14 21:12:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:12:45 [INFO] [stacks] ScanStacks complete: 57 stacks found (12 deployed, 45 available) +2026/09/14 21:12:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:12:45 [INFO] [scheduler] Job stack-scan completed (took 90ms) +2026/09/14 21:12:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:12:45 [INFO] [scheduler] Job agent-channel-health completed (took 185ms) +2026/09/14 21:12:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:13:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:13:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:13:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:13:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:13:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:13:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:13:45 [INFO] [scheduler] Job agent-channel-health completed (took 192ms) +2026/09/14 21:13:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:14:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:14:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:14:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:14:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:14:35 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:14:45 [INFO] [scheduler] Running job: system-health +2026/09/14 21:14:45 [INFO] [scheduler] Running job: offsite-credential-retry +2026/09/14 21:14:45 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/09/14 21:14:45 [INFO] [scheduler] Running job: backup-cache +2026/09/14 21:14:45 [INFO] [scheduler] Running job: stack-scan +2026/09/14 21:14:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:14:45 [INFO] [stacks] ScanStacks complete: 57 stacks found (12 deployed, 45 available) +2026/09/14 21:14:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:14:45 [INFO] [scheduler] Job stack-scan completed (took 105ms) +2026/09/14 21:14:45 [INFO] [backup] Found 6 DB dump files across drives +2026/09/14 21:14:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:14:45 [INFO] [monitor] Health check: status=ok +2026/09/14 21:14:45 [INFO] [scheduler] Job system-health completed (took 268ms) +2026/09/14 21:14:45 [INFO] [backup] Discovered 6 databases +2026/09/14 21:14:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/14 21:14:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/09/14 21:14:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/14 21:14:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/14 21:14:45 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml +2026/09/14 21:14:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/14 21:14:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/mealie/docker-compose.yml +2026/09/14 21:14:45 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/14 21:14:45 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/14 21:14:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/14 21:14:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/14 21:14:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/14 21:14:45 [INFO] [backup] Discovered app data: 12 apps +2026/09/14 21:14:45 [INFO] [backup] Backup status cache refreshed +2026/09/14 21:14:45 [INFO] [scheduler] Job backup-cache completed (took 355ms) +2026/09/14 21:14:45 [INFO] [scheduler] Job agent-channel-health completed (took 215ms) +2026/09/14 21:14:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:15:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:15:05 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:15:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:15:15 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:15:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:15:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:15:45 [INFO] [deadapp] check alive: 160 scans since boot, 12 deployed app(s) evaluated, 0 currently down +2026/09/14 21:15:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:15:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:15:45 [INFO] [scheduler] Job agent-channel-health completed (took 180ms) +2026/09/14 21:15:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:15:55 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:16:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:16:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:16:15 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:16:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:16:25 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:16:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:16:45 [INFO] [scheduler] Running job: stack-scan +2026/09/14 21:16:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:16:45 [INFO] [stacks] ScanStacks complete: 57 stacks found (12 deployed, 45 available) +2026/09/14 21:16:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:16:45 [INFO] [scheduler] Job stack-scan completed (took 86ms) +2026/09/14 21:16:45 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:16:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:16:45 [INFO] [scheduler] Job agent-channel-health completed (took 213ms) +2026/09/14 21:16:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:17:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:17:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:17:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:17:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:17:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:17:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:17:45 [INFO] [scheduler] Job agent-channel-health completed (took 203ms) +2026/09/14 21:17:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:18:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:18:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:18:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:18:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:18:45 [INFO] [scheduler] Running job: stack-scan +2026/09/14 21:18:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:18:45 [INFO] [stacks] ScanStacks complete: 57 stacks found (12 deployed, 45 available) +2026/09/14 21:18:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:18:45 [INFO] [scheduler] Job stack-scan completed (took 92ms) +2026/09/14 21:18:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:18:45 [INFO] [scheduler] Job agent-channel-health completed (took 171ms) +2026/09/14 21:18:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:19:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:19:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:19:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:19:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:19:45 [INFO] [scheduler] Running job: offsite-credential-retry +2026/09/14 21:19:45 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/09/14 21:19:45 [INFO] [scheduler] Running job: backup-cache +2026/09/14 21:19:45 [INFO] [scheduler] Running job: system-health +2026/09/14 21:19:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:19:45 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:19:45 [INFO] [backup] Found 6 DB dump files across drives +2026/09/14 21:19:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:19:45 [INFO] [backup] Discovered 6 databases +2026/09/14 21:19:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/14 21:19:45 [INFO] [monitor] Health check: status=ok +2026/09/14 21:19:45 [INFO] [scheduler] Job system-health completed (took 249ms) +2026/09/14 21:19:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/09/14 21:19:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/14 21:19:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/14 21:19:45 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml +2026/09/14 21:19:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/14 21:19:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/mealie/docker-compose.yml +2026/09/14 21:19:45 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/14 21:19:45 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/14 21:19:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/14 21:19:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/14 21:19:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/14 21:19:45 [INFO] [backup] Discovered app data: 12 apps +2026/09/14 21:19:45 [INFO] [backup] Backup status cache refreshed +2026/09/14 21:19:45 [INFO] [scheduler] Job backup-cache completed (took 297ms) +2026/09/14 21:19:45 [INFO] [scheduler] Job agent-channel-health completed (took 189ms) +2026/09/14 21:19:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:20:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:20:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:20:15 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:20:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:20:25 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:20:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:20:45 [INFO] [scheduler] Running job: stack-scan +2026/09/14 21:20:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:20:45 [INFO] [stacks] ScanStacks complete: 57 stacks found (12 deployed, 45 available) +2026/09/14 21:20:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:20:45 [INFO] [scheduler] Job stack-scan completed (took 95ms) +2026/09/14 21:20:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:20:45 [INFO] [scheduler] Job agent-channel-health completed (took 187ms) +2026/09/14 21:20:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:21:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:21:05 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:21:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:21:15 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:21:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:21:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:21:35 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:21:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:21:45 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:21:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:21:45 [INFO] [scheduler] Job agent-channel-health completed (took 196ms) +2026/09/14 21:21:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:21:55 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:22:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:22:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:22:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:22:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:22:45 [INFO] [scheduler] Running job: stack-scan +2026/09/14 21:22:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:22:45 [INFO] [stacks] ScanStacks complete: 57 stacks found (12 deployed, 45 available) +2026/09/14 21:22:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:22:45 [INFO] [scheduler] Job stack-scan completed (took 104ms) +2026/09/14 21:22:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:22:45 [INFO] [scheduler] Job agent-channel-health completed (took 194ms) +2026/09/14 21:22:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:23:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:23:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:23:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:23:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:23:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:23:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:23:45 [INFO] [scheduler] Job agent-channel-health completed (took 188ms) +2026/09/14 21:23:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:24:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:24:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:24:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:24:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:24:44 [INFO] [sync] Starting catalog sync +2026/09/14 21:24:44 [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (branch: main) +2026/09/14 21:24:45 [INFO] [scheduler] Running job: offsite-credential-retry +2026/09/14 21:24:45 [INFO] [scheduler] Running job: hub-report +2026/09/14 21:24:45 [INFO] [report] Building system report +2026/09/14 21:24:45 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/09/14 21:24:45 [INFO] [scheduler] Running job: stack-scan +2026/09/14 21:24:45 [INFO] [scheduler] Running job: backup-cache +2026/09/14 21:24:45 [INFO] [scheduler] Running job: system-health +2026/09/14 21:24:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:24:45 [INFO] [stacks] ScanStacks complete: 57 stacks found (12 deployed, 45 available) +2026/09/14 21:24:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:24:45 [INFO] [scheduler] Job stack-scan completed (took 156ms) +2026/09/14 21:24:45 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:24:45 [INFO] [backup] Found 6 DB dump files across drives +2026/09/14 21:24:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:24:45 [INFO] [monitor] Health check: status=ok +2026/09/14 21:24:45 [INFO] [scheduler] Job system-health completed (took 320ms) +2026/09/14 21:24:45 [INFO] [scheduler] Job agent-channel-health completed (took 188ms) +2026/09/14 21:24:45 [INFO] [backup] Discovered 6 databases +2026/09/14 21:24:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/14 21:24:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/09/14 21:24:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/14 21:24:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/14 21:24:45 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml +2026/09/14 21:24:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/14 21:24:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/mealie/docker-compose.yml +2026/09/14 21:24:45 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/14 21:24:45 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/14 21:24:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/14 21:24:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/14 21:24:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/14 21:24:45 [INFO] [backup] Discovered app data: 12 apps +2026/09/14 21:24:46 [INFO] [backup] Backup status cache refreshed +2026/09/14 21:24:46 [INFO] [scheduler] Job backup-cache completed (took 477ms) +2026/09/14 21:24:46 [INFO] [monitor] Health check: status=ok +2026/09/14 21:24:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:25:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:25:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:25:15 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:25:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:25:25 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:25:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:25:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:25:45 [INFO] [deadapp] check alive: 180 scans since boot, 12 deployed app(s) evaluated, 0 currently down +2026/09/14 21:25:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:25:45 [INFO] [scheduler] Job agent-channel-health completed (took 199ms) +2026/09/14 21:25:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:26:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:26:07 [INFO] [report] Hub report pushed successfully (14763 bytes) +2026/09/14 21:26:07 [INFO] [settings] Settings saved +2026/09/14 21:26:07 [INFO] [scheduler] Job hub-report completed (took 1m22.314s) +2026/09/14 21:26:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:26:15 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:26:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:26:25 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:26:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:26:45 [INFO] [scheduler] Running job: stack-scan +2026/09/14 21:26:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:26:45 [INFO] [stacks] ScanStacks complete: 57 stacks found (12 deployed, 45 available) +2026/09/14 21:26:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:26:45 [INFO] [scheduler] Job stack-scan completed (took 98ms) +2026/09/14 21:26:45 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:26:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:26:45 [INFO] [scheduler] Job agent-channel-health completed (took 184ms) +2026/09/14 21:26:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:26:55 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:27:00 [ERROR] [sync] Catalog sync failed: git fetch: git fetch --depth 1 origin main: context deadline exceeded (subprocess killed at the 2m0s deadline): signal: killed +stderr: fatal: unable to access 'https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git/': Failed to connect to gitea.dooplex.hu port 443 after 136321 ms: Couldn't connect to server +2026/09/14 21:27:00 [INFO] [sync] Periodic sync: Git hiba: git fetch: git fetch --depth 1 origin main: context deadline exceeded (subprocess killed at the 2m0s deadline): signal: killed +stderr: fatal: unable to access 'https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git/': Failed to connect to gitea.dooplex.hu port 443 after 136321 ms: Couldn't connect to server +2026/09/14 21:27:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:27:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:27:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:27:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:27:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:27:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:27:45 [INFO] [scheduler] Job agent-channel-health completed (took 191ms) +2026/09/14 21:27:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:28:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:28:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:28:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:28:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:28:45 [INFO] [scheduler] Running job: stack-scan +2026/09/14 21:28:45 [INFO] [stacks] ScanStacks complete: 57 stacks found (12 deployed, 45 available) +2026/09/14 21:28:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:28:45 [INFO] [scheduler] Job stack-scan completed (took 61ms) +2026/09/14 21:28:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:28:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:28:45 [INFO] [scheduler] Job agent-channel-health completed (took 194ms) +2026/09/14 21:28:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:29:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:29:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:29:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:29:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:29:45 [INFO] [scheduler] Running job: offsite-credential-retry +2026/09/14 21:29:45 [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/09/14 21:29:45 [INFO] [scheduler] Running job: system-health +2026/09/14 21:29:45 [INFO] [scheduler] Running job: backup-cache +2026/09/14 21:29:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:29:45 [INFO] [backup] Found 6 DB dump files across drives +2026/09/14 21:29:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:29:45 [INFO] [backup] Discovered 6 databases +2026/09/14 21:29:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/14 21:29:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/bookstack/docker-compose.yml +2026/09/14 21:29:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/14 21:29:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/14 21:29:45 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml +2026/09/14 21:29:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/14 21:29:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/mealie/docker-compose.yml +2026/09/14 21:29:45 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/14 21:29:45 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/14 21:29:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/14 21:29:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/14 21:29:45 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/14 21:29:45 [INFO] [backup] Discovered app data: 12 apps +2026/09/14 21:29:45 [INFO] [backup] Backup status cache refreshed +2026/09/14 21:29:45 [INFO] [scheduler] Job backup-cache completed (took 314ms) +2026/09/14 21:29:45 [INFO] [monitor] Health check: status=ok +2026/09/14 21:29:45 [INFO] [scheduler] Job system-health completed (took 322ms) +2026/09/14 21:29:45 [INFO] [scheduler] Job agent-channel-health completed (took 157ms) +2026/09/14 21:29:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:29:55 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:30:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:30:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:30:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:30:25 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:30:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:30:35 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:30:45 [INFO] [scheduler] Running job: stack-scan +2026/09/14 21:30:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:30:45 [INFO] [stacks] ScanStacks complete: 57 stacks found (12 deployed, 45 available) +2026/09/14 21:30:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:30:45 [INFO] [scheduler] Job stack-scan completed (took 89ms) +2026/09/14 21:30:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:30:45 [INFO] [scheduler] Job agent-channel-health completed (took 187ms) +2026/09/14 21:30:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:31:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:31:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:31:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:31:25 [INFO] Health probes: 4 ok (of 4 probed) +2026/09/14 21:31:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:31:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:31:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:31:45 [INFO] [scheduler] Job agent-channel-health completed (took 184ms) +2026/09/14 21:31:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:31:55 [INFO] Health probes: 2 ok (of 2 probed) +2026/09/14 21:32:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:32:05 [INFO] Health probes: 1 ok (of 1 probed) +2026/09/14 21:32:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:32:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:32:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:32:45 [INFO] [scheduler] Running job: stack-scan +2026/09/14 21:32:45 [INFO] [stacks] ScanStacks complete: 57 stacks found (12 deployed, 45 available) +2026/09/14 21:32:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:32:45 [INFO] [scheduler] Job stack-scan completed (took 64ms) +2026/09/14 21:32:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:32:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:32:45 [INFO] [scheduler] Job agent-channel-health completed (took 198ms) +2026/09/14 21:32:49 [INFO] [report] wait channel recovered +2026/09/14 21:32:55 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:33:05 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:33:15 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:33:25 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:33:35 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:33:45 [INFO] [stacks] Status refresh: 28 containers across 57 stacks +2026/09/14 21:33:45 [INFO] [scheduler] Running job: agent-channel-health +2026/09/14 21:33:45 [INFO] [scheduler] Job agent-channel-health completed (took 189ms) diff --git a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/guest-wan-probe.txt b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/guest-wan-probe.txt index 43673c44..9251f616 100644 --- a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/guest-wan-probe.txt +++ b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/guest-wan-probe.txt @@ -1,2 +1,13 @@ 21:09:30 21:11:20 guest->1.1.1.1=fail guest->hub=fail cloudflared(last120s registered=0 errors=15) +21:13:21 guest->1.1.1.1=fail guest->hub=fail cloudflared(last120s registered=0 errors=21) +21:15:22 guest->1.1.1.1=fail guest->hub=fail cloudflared(last120s registered=0 errors=9) +21:17:24 guest->1.1.1.1=fail guest->hub=fail cloudflared(last120s registered=0 errors=19) +21:19:25 guest->1.1.1.1=fail guest->hub=fail cloudflared(last120s registered=0 errors=23) +21:21:27 guest->1.1.1.1=fail guest->hub=fail cloudflared(last120s registered=0 errors=20) +21:23:28 guest->1.1.1.1=fail guest->hub=fail cloudflared(last120s registered=0 errors=12) +21:25:29 guest->1.1.1.1=fail guest->hub=fail cloudflared(last120s registered=0 errors=18) +21:27:31 guest->1.1.1.1=301 guest->hub=302 cloudflared(last120s registered=4 errors=13) +21:29:23 guest->1.1.1.1=301 guest->hub=302 cloudflared(last120s registered=0 errors=8) +21:31:14 guest->1.1.1.1=301 guest->hub=302 cloudflared(last120s registered=0 errors=8) +21:33:05 guest->1.1.1.1=301 guest->hub=302 cloudflared(last120s registered=0 errors=8) diff --git a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/hub-alarms-during-cut.txt b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/hub-alarms-during-cut.txt new file mode 100644 index 00000000..d3c39981 --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/hub-alarms-during-cut.txt @@ -0,0 +1,8 @@ +=== hub alarms during the cut, read 21:26:37 +2026/09/14 23:04:45 [INFO] Event from tester-1: health_recovered (info) — Rendszer állapot helyreállt: ok (volt: warn) +2026/09/14 23:25:43 [INFO] Staleness: tester-1 ok → stale (node_stale) +2026/09/14 23:25:44 [INFO] Operator email sent for tester-1/node_stale +2026/09/14 23:25:43 [INFO] Staleness: tester-1 ok → stale (node_stale) +2026/09/14 23:25:44 [INFO] Operator email sent for tester-1/node_stale +2026/09/14 23:26:43 [INFO] Staleness: tester-1 stale → ok (node_recovered) +2026/09/14 23:26:43 [INFO] Operator email sent for tester-1/node_recovered diff --git a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/hub-and-dashboard-poll.txt b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/hub-and-dashboard-poll.txt index 0ae3d113..d3856a06 100644 --- a/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/hub-and-dashboard-poll.txt +++ b/documentation/audits/evidence-bignight-2026-09-14/phase5/F8/hub-and-dashboard-poll.txt @@ -10,3 +10,47 @@ LAN dashboard banners: ↗ | hub: Last report: 18 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | warn | Controller | 0.242. LAN dashboard banners: ↗ | tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc +=== 21:14:14 +hub: Last report: 19 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | warn | Controller | 0.242. +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc +=== 21:16:05 +hub: Last report: 21 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | warn | Controller | 0.242. +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc +=== 21:17:55 +hub: Last report: 23 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | warn | Controller | 0.242. +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc +=== 21:19:46 +hub: Last report: 25 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | warn | Controller | 0.242. +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc +=== 21:21:37 +hub: Last report: 27 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | warn | Controller | 0.242. +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc +=== 21:23:28 +hub: Last report: 29 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | warn | Controller | 0.242. +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc +=== 21:25:19 +hub: Last report: 31 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | warn | Controller | 0.242. +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc +=== 21:27:09 +hub: Last report: 1 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | ok | Controller | 0.242.0 | +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc +=== 21:29:00 +hub: Last report: 3 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | ok | Controller | 0.242.0 | +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc +=== 21:30:51 +hub: Last report: 5 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | ok | Controller | 0.242.0 | +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc +=== 21:32:41 +hub: Last report: 7 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | ok | Controller | 0.242.0 | +LAN dashboard banners: ↗ | + tunnel tile: Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Doc diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 94887002..1f4adc2a 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -725,6 +725,7 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-519** | **[P2-MEDIUM] After a backup torn by a power cut, an app's restore point carries the new database dump's time while its files are from the previous run — and no customer screen says the run was interrupted.** MEASURED 2026-09-14 (BIGNIGHT F2, VM 333): „Mentés most" 19:40:03Z; the power was cut 19:40:08Z while adventurelog was stopped for its volume dump. On disk afterwards, `backups/primary/adventurelog`: `db-dumps/adventurelog-postgres.sql` **19:40:07**, `volume-dumps/*.tar` **19:00:35**, `manifest.json created_at 19:02:34Z`; bookstack the same shape (sql 19:40:07, tars 19:00:48). `GET /api/backup/snapshots` for both → `time 2026-09-14T19:40:07Z, helyi` (`phase5/F2/units-on-disk.txt`, `backup-honesty.txt`). `/backups/apps` shows „Utolsó adatbázis mentés 2026-09-14 21:40 · … OK" and every app „Utolsó: 5 perce"; `/backups` and `/dashboard` contain no word of an interruption (fragments `megszakad|sikertelen|nem sikerült` = 0; control „hiba" appears in the standing warning text). The controller itself knew: `[appstop] crash recovery: an app-data backup (volume dump) … was interrupted … restarting them: [adventurelog]` and pushed `backup_failed (error)` to the hub. A household restoring „the 21:40 backup" gets 21:00 files for BookStack's uploads. **Fix shape:** date a point by the oldest part it contains (or mark it partial) and show the interrupted run on the backups page until the next complete one. **F6 (drive unplugged 1 s into a backup, 20:37:59Z) adds three facts:** the run skipped four apps' volume dumps („Skipping volume dump for immich — drive disconnected", also jellyfin, nextcloud, paperless-ngx) and still reported `db_dump {"count":6, "success":true}`; nextcloud's point is dated 20:37:59Z (its SQL finished before the unplug) beside 19:01 volume tars; and a torn `immich-postgres.sql.tmp` (20:38:00) plus F3's `pre-restore-…-nextcloud-mariadb.sql.tmp` are left in the units on the drive. Immich's point correctly stayed at 19:02:34Z (the `.tmp` was not promoted). `/backups/apps` fragments `kihagy|sikertelen|részleges` = 0 (`phase5/F6/`). | **READY — rank P2-MEDIUM; owner: CC (controller)** | | **R-520** | **[P3-LOW] A power cut during a guarded Update leaves no record that an update was running, so whether the journal resumes or aborts honestly is unmeasured for a real version change.** MEASURED 2026-09-14 (BIGNIGHT F3, VM 333): the only Update the catalog allowed after the Phase 4 revert was a same-version one on nextcloud; power was cut 2 s after `update nextcloud: phase pulling` (safety dump written, pin advanced to the unchanged definition). After boot: no log line resumes, aborts or names the interrupted update; `app.yaml` `pinned_images` = `installed_images`, no hold, no verdict; page „Fut · Naprakész". Nothing wrong was produced — and nothing could have been, with identical images. **What it needs:** the same cut on a real bump (a throwaway app with a one-step catalog move on a scratch branch or the scratch guest), asserting the page's message and the pin after boot. | **READY — rank P3-LOW; owner: CC (drill)** | | **R-521** | **[P3-LOW] One unplugged drive sends the operator five e-mails and the household none.** MEASURED 2026-09-14 (BIGNIGHT F4, VM 333): `storage_disconnected (error)` at 21:58:02 CEST plus `app_start_failed (warning)` for each of the four apps the drive carries at 21:58:15, each with its own operator mail (hub log: five `Operator email sent`). The customer's mailbox (`tester1@felhom.eu`, read through the connector) received nothing; the household learns of it only on the dashboard, which is honest and says what to do. The apps' stop is a consequence of the drive event, so the four warnings add no information. **Fix shape:** suppress `app_start_failed` for apps stopped by a `storage_disconnected` (the dead-app check already knows the reason — „Hiányzó tárhely"), and decide whether a household gets a mail for a lost drive. **F6, 40 min later, the opposite failure:** a second, separate drive loss (20:38:33Z) produced `storage_disconnected (error)` and four `app_start_failed`, and the hub logged `Operator email suppressed … cooldown` for all five — **no mail at all for the second unplug**; only `health_degraded (warning)` mailed. A per-key cooldown that outlives the recovery (`storage_reconnected` came between them) silences a new incident. **F7:** the system disk at 95 % produced only `health_degraded (warning)`, whose operator mail was **suppressed by the cooldown** left by F6's `health_degraded` 15 minutes earlier; no disk-specific event reached the hub at all — the operator was not told the disk was nearly full. | **READY — rank P3-LOW; owner: CC (controller) · operator (customer mail policy)** | +| **R-522** | **[P3-LOW] While the box has no internet, the dashboard's „Cloudflare Tunnel" tile keeps saying „Fut", and no page tells the household the box is offline.** MEASURED 2026-09-14 (BIGNIGHT F8, VM 333): VM 333's traffic off the LAN and to the hub was dropped at demo-hp's bridge 21:08:36 → 21:26:07Z. Throughout, the LAN dashboard (probed every 26 s from demo-hp) answered 200 and, polled every 2 min, showed no banner and the tile „Cloudflare Tunnel — Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. · **Fut** · Védett"; meanwhile cloudflared logged ≈ 20 errors every 2 minutes, the public name answered 530, and the controller logged `[report] Push failed … context deadline exceeded` and `Job hub-report failed: hub push failed after 3 attempts`. The tile reports the container, not the connection. A household whose remote access is gone sees „Fut". **Fix shape:** the tile reads the tunnel's connection state (cloudflared's registered connections or the report push result) and says „Nincs internetkapcsolat" when either fails. | **READY — rank P3-LOW; owner: CC (controller)** |