diff --git a/documentation/architecture/03-host-agent.md b/documentation/architecture/03-host-agent.md index 34c41b68..2dadaad8 100644 --- a/documentation/architecture/03-host-agent.md +++ b/documentation/architecture/03-host-agent.md @@ -199,6 +199,22 @@ by verb**: neither lose nor invent one. New goldens run the controller with `--restart always`, which covers only a Docker daemon restart. +- **`[DESIGN]` 2026-10-29 — the operator's „Restart controller" (agent `637c92c`, released v0.169.0; `04` §6.1).** The + host report's reply carries the AGENT's own closed list — `restart_controller ` only; the agent acts once per id + with the supervisor's act and guards (provisioned, running, not parked, no swap, not locked, no vzdump; an unreadable + guest list or vzdump state refuses), the brake counts these restarts with the supervisor's (3 in 15 minutes → refused, + then 30 minutes refused, `opBrakeUntil`), and the result rides the next report, requested at once. It works with the + controller stopped or stuck. **[FACT] Tester 1, 2026-10-29 13:01–13:02 CET:** four presses through the host page's + endpoint — poke to the agent in ~1 s, the controller healthy again after 9 s each, the result on the hub 15–17 s after + the press; the second press with the controller stopped by hand (`docker stop`) restarted it 5 s after the stop; the + fourth was refused („3 restarts in 15 minutes — presses are refused until 12:32 UTC") (`audits/release-2026-10-29b/`). +- **`[DESIGN]` 2026-10-29 — a controller that RUNS but does not answer (R-1344): ALARM ONLY.** Each sweep also reads + Docker's health status (the controller's own HEALTHCHECK, `curl -f /api/health` every 30 s, timeout 5 s); 6 + consecutive „unhealthy" sweeps — on top of Docker's 3 failed probes, ~4.5 minutes in all — start an episode: + `unresponsive_since` in the stanza, the hub's operator-only `controller_unresponsive`. Not counted during a swap, a + parked or locked guest, or a vzdump. **It does not restart for it** until the R-1344 measurement (every probe, both + test boxes, through the 2026-10-29/30 night chain) shows a busy night never reads unhealthy that long. + **[RULING 2026-09-16, operator] The brake catches a FAST crash loop and is blind to a SLOW one — a second, slower counter is owed (R-539).** MEASURED the same day on the drill box: the controller was killed four times, each kill 20 minutes after the last. Every one was restarted (dashboard back in diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index d9f2f1f3..e8cf4464 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -816,3 +816,12 @@ the DB override. Now `main` always mounts `webServer.Handler()` (`TestWebMountAl answers **503** on every operator route; `/bind/` stays public, `/login` renders but cannot succeed (`TestRequireAuth_NoPasswordRefusesEveryOperatorRoute`). **The live hub was never open:** 2026-10-15, hub 0.150.0, an unauthenticated `GET /`, `/configuration`, `/hosts` each answered 302 → `/login`, and the ConfigMap carries a hash. + + +**`[DESIGN]` 2026-10-29 — two buttons on the host page (hub v0.176.0/0.176.1; `04` §6.1, D64 = B).** „Restart controller" +sits in the guest table, one per guest of a customer host (0.176.0 gated it on a reported controller version, which the +agent does not report — found live, fixed in 0.176.1); it stores an AGENT row (`operator_actions.host_id`), pokes the box +and shows the result under Operator Actions. „Clear hold" sits on each Held apps row; its confirm step names the app, +the reason and since when, and for a restore or put-back hold warns that the app may start on half-restored data; the hub +refuses an app the box does not report held now. Both are `hosts:write`, CSRF like every form (a token press needs none +and is recorded as `access token ""`), and neither is reachable without the confirm fold-out in the browser. \ No newline at end of file diff --git a/documentation/architecture/08-alarm-ladder.md b/documentation/architecture/08-alarm-ladder.md index bf1456fb..59c3d4db 100644 --- a/documentation/architecture/08-alarm-ladder.md +++ b/documentation/architecture/08-alarm-ladder.md @@ -307,6 +307,8 @@ bands. |---|---|---|---|---| | `controller_slow_crashloop` | warning | **operator only** | hub, from the agent's `slow_crashloop_since` moving (agent v0.132.0, hub v0.117.0) | host ids and vmids; the household's side of it is the dashboard coming back each time | | `restore_interrupted` | warning | **the household** (and the operator) | controller v0.246.0 at startup, once per interruption | they pressed restore, were told it started, and can run it again — Hungarian `customerMessages` entry | +| `controller_unresponsive` (2026-10-29, R-1344) | warning | **operator only** | hub, from the agent's `unresponsive_since` moving (agent v0.169.0, hub v0.176.0) | the controller runs but does not answer; the agent only alarms — the operator decides (Restart controller) | +| `restore_copy_deleted` (2026-10-29, D62) | info | **operator only** | controller v0.347.0, per „Törlés most" | the household did it after the warning; the record (who, app, date, size) is support's | | Family | Grain | Key carries | Why | |---|---|---|---| diff --git a/documentation/audits/release-2026-10-29b/p1-restart/agent-journal-presses.txt b/documentation/audits/release-2026-10-29b/p1-restart/agent-journal-presses.txt new file mode 100644 index 00000000..1bf55c0b --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p1-restart/agent-journal-presses.txt @@ -0,0 +1,948 @@ +Oct 29 12:59:02 felhom sudo[701763]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 12:59:02 felhom sudo[701763]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:02 felhom sudo[701763]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:02 felhom sudo[701766]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 12:59:02 felhom sudo[701766]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:02 felhom sudo[701766]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:03 felhom sudo[701803]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 12:59:03 felhom sudo[701803]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:03 felhom sudo[701803]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:03 felhom sudo[701806]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 12:59:03 felhom sudo[701806]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:03 felhom sudo[701806]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:04 felhom sudo[701809]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 12:59:04 felhom sudo[701809]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:04 felhom sudo[701809]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:04 felhom sudo[701812]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 12:59:04 felhom sudo[701812]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:04 felhom sudo[701812]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:04 felhom sudo[701815]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 12:59:04 felhom sudo[701815]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:04 felhom sudo[701815]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:04 felhom sudo[701827]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 12:59:04 felhom sudo[701827]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:04 felhom sudo[701827]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:04 felhom sudo[701830]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 12:59:04 felhom sudo[701830]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:04 felhom sudo[701830]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:04 felhom sudo[701833]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 12:59:04 felhom sudo[701833]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:04 felhom sudo[701833]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:04 felhom sudo[701836]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 12:59:04 felhom sudo[701836]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:04 felhom sudo[701836]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:04 felhom sudo[701839]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 12:59:04 felhom sudo[701839]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:04 felhom sudo[701839]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:06 felhom sudo[701863]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives +Oct 29 12:59:06 felhom sudo[701863]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:06 felhom sudo[701863]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:07 felhom sudo[701875]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 12:59:07 felhom sudo[701875]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:07 felhom sudo[701875]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:07 felhom sudo[701878]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 12:59:07 felhom sudo[701878]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:07 felhom sudo[701878]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:07 felhom sudo[701881]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/adatlemez/felhom-data +Oct 29 12:59:07 felhom sudo[701881]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:07 felhom sudo[701881]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:07 felhom sudo[701884]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/chown 100000:100000 /mnt/adatlemez/felhom-data +Oct 29 12:59:07 felhom sudo[701884]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:07 felhom sudo[701884]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:07 felhom sudo[701887]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives/adatlemez +Oct 29 12:59:07 felhom sudo[701887]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:07 felhom sudo[701887]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:07 felhom sudo[701890]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 12:59:07 felhom sudo[701890]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:07 felhom sudo[701890]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:07 felhom felhom-agent[697615]: time=2026-10-29T12:59:07.139+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/adatlemez guest_path=/mnt/felhom-drives/adatlemez durable_id=uuid:863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 29 12:59:16 felhom felhom-agent[697615]: time=2026-10-29T12:59:16.722+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 12:59:16 felhom sudo[702039]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f {{.State.Status}} felhom-controller +Oct 29 12:59:16 felhom sudo[702039]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:17 felhom sudo[702039]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:17 felhom sudo[702052]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 12:59:17 felhom sudo[702052]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:18 felhom sudo[702052]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:26 felhom sudo[702272]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives +Oct 29 12:59:26 felhom sudo[702272]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:26 felhom sudo[702272]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:27 felhom sudo[702284]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 12:59:27 felhom sudo[702284]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:27 felhom sudo[702284]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:27 felhom sudo[702287]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 12:59:27 felhom sudo[702287]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:27 felhom sudo[702287]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:27 felhom sudo[702290]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/adatlemez/felhom-data +Oct 29 12:59:27 felhom sudo[702290]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:27 felhom sudo[702290]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:27 felhom sudo[702293]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/chown 100000:100000 /mnt/adatlemez/felhom-data +Oct 29 12:59:27 felhom sudo[702293]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:27 felhom sudo[702293]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:27 felhom sudo[702296]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives/adatlemez +Oct 29 12:59:27 felhom sudo[702296]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:27 felhom sudo[702296]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:27 felhom sudo[702299]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 12:59:27 felhom sudo[702299]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:27 felhom sudo[702299]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:27 felhom felhom-agent[697615]: time=2026-10-29T12:59:27.175+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/adatlemez guest_path=/mnt/felhom-drives/adatlemez durable_id=uuid:863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 29 12:59:33 felhom sudo[702434]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 12:59:33 felhom sudo[702434]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:33 felhom sudo[702434]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:33 felhom sudo[702437]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 12:59:33 felhom sudo[702437]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:33 felhom sudo[702437]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:34 felhom sudo[702440]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 12:59:34 felhom sudo[702440]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:34 felhom sudo[702440]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:34 felhom sudo[702443]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 12:59:34 felhom sudo[702443]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:34 felhom sudo[702443]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:34 felhom sudo[702446]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 12:59:34 felhom sudo[702446]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:34 felhom sudo[702446]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:34 felhom sudo[702473]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 12:59:34 felhom sudo[702473]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:34 felhom sudo[702473]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:34 felhom sudo[702476]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 12:59:34 felhom sudo[702476]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:34 felhom sudo[702476]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:34 felhom sudo[702479]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 12:59:34 felhom sudo[702479]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:34 felhom sudo[702479]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:34 felhom sudo[702482]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 12:59:34 felhom sudo[702482]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:34 felhom sudo[702482]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:34 felhom sudo[702485]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 12:59:34 felhom sudo[702485]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:34 felhom sudo[702485]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:46 felhom sudo[702654]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives +Oct 29 12:59:46 felhom sudo[702654]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:46 felhom sudo[702654]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:46 felhom felhom-agent[697615]: time=2026-10-29T12:59:46.617+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 12:59:46 felhom sudo[702662]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- ip -4 -o addr show dev eth0 +Oct 29 12:59:46 felhom sudo[702662]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:46 felhom sudo[702673]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 12:59:46 felhom sudo[702673]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:46 felhom sudo[702673]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:46 felhom felhom-agent[697615]: time=2026-10-29T12:59:46.749+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 12:59:46 felhom felhom-agent[697615]: time=2026-10-29T12:59:46.753+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 12:59:46 felhom sudo[702679]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 12:59:46 felhom sudo[702679]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:46 felhom sudo[702679]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:46 felhom sudo[702680]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f {{.State.Status}} felhom-controller +Oct 29 12:59:46 felhom sudo[702680]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:47 felhom sudo[702690]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 12:59:47 felhom sudo[702690]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:47 felhom sudo[702690]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:47 felhom sudo[702693]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 12:59:47 felhom sudo[702693]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:47 felhom sudo[702693]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:47 felhom sudo[702696]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/adatlemez/felhom-data +Oct 29 12:59:47 felhom sudo[702696]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:47 felhom sudo[702696]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:47 felhom sudo[702699]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/chown 100000:100000 /mnt/adatlemez/felhom-data +Oct 29 12:59:47 felhom sudo[702699]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:47 felhom sudo[702699]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:47 felhom sudo[702702]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives/adatlemez +Oct 29 12:59:47 felhom sudo[702702]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:47 felhom sudo[702702]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:47 felhom sudo[702705]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 12:59:47 felhom sudo[702705]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:47 felhom sudo[702705]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:47 felhom felhom-agent[697615]: time=2026-10-29T12:59:47.169+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/adatlemez guest_path=/mnt/felhom-drives/adatlemez durable_id=uuid:863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 29 12:59:47 felhom sudo[702662]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:47 felhom sudo[702711]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- ip route show default +Oct 29 12:59:47 felhom sudo[702711]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:47 felhom sudo[702723]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob ssh_port +Oct 29 12:59:47 felhom sudo[702723]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:47 felhom sudo[702723]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:47 felhom sudo[702680]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:47 felhom sudo[702728]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob operator_ips +Oct 29 12:59:47 felhom sudo[702728]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:47 felhom sudo[702728]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:47 felhom sudo[702729]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 12:59:47 felhom sudo[702729]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:48 felhom sudo[702711]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:48 felhom sudo[702740]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- cat /etc/network/interfaces +Oct 29 12:59:48 felhom sudo[702740]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:48 felhom sudo[702729]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:49 felhom sudo[702740]: pam_unix(sudo:session): session closed for user root +Oct 29 12:59:49 felhom sudo[702775]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- pgrep -x dhclient +Oct 29 12:59:49 felhom sudo[702775]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 12:59:50 felhom sudo[702775]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:02 felhom sudo[703094]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:00:02 felhom sudo[703094]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:02 felhom sudo[703094]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:02 felhom sudo[703097]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:00:02 felhom sudo[703097]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:02 felhom sudo[703097]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:03 felhom sudo[703145]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:00:03 felhom sudo[703145]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:03 felhom sudo[703145]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:03 felhom sudo[703148]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:00:03 felhom sudo[703148]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:04 felhom sudo[703148]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:04 felhom sudo[703151]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:00:04 felhom sudo[703151]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:04 felhom sudo[703151]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:04 felhom sudo[703154]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 13:00:04 felhom sudo[703154]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:04 felhom sudo[703154]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:04 felhom sudo[703157]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:00:04 felhom sudo[703157]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:04 felhom sudo[703157]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:04 felhom sudo[703169]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:00:04 felhom sudo[703169]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:04 felhom sudo[703169]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:04 felhom sudo[703172]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:00:04 felhom sudo[703172]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:04 felhom sudo[703172]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:04 felhom sudo[703175]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:00:04 felhom sudo[703175]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:04 felhom sudo[703175]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:04 felhom sudo[703178]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 13:00:04 felhom sudo[703178]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:04 felhom sudo[703178]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:04 felhom sudo[703181]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:00:04 felhom sudo[703181]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:04 felhom sudo[703181]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:06 felhom sudo[703204]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives +Oct 29 13:00:06 felhom sudo[703204]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:06 felhom sudo[703204]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:07 felhom sudo[703216]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:00:07 felhom sudo[703216]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:07 felhom sudo[703216]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:07 felhom sudo[703219]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:00:07 felhom sudo[703219]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:07 felhom sudo[703219]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:07 felhom sudo[703222]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/adatlemez/felhom-data +Oct 29 13:00:07 felhom sudo[703222]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:07 felhom sudo[703222]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:07 felhom sudo[703225]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/chown 100000:100000 /mnt/adatlemez/felhom-data +Oct 29 13:00:07 felhom sudo[703225]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:07 felhom sudo[703225]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:07 felhom sudo[703228]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives/adatlemez +Oct 29 13:00:07 felhom sudo[703228]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:07 felhom sudo[703228]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:07 felhom sudo[703231]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:00:07 felhom sudo[703231]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:07 felhom sudo[703231]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:07 felhom felhom-agent[697615]: time=2026-10-29T13:00:07.158+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/adatlemez guest_path=/mnt/felhom-drives/adatlemez durable_id=uuid:863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 29 13:00:16 felhom felhom-agent[697615]: time=2026-10-29T13:00:16.721+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:00:16 felhom sudo[703392]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f {{.State.Status}} felhom-controller +Oct 29 13:00:16 felhom sudo[703392]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:17 felhom sudo[703392]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:17 felhom sudo[703406]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:00:17 felhom sudo[703406]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:18 felhom sudo[703406]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:26 felhom sudo[703590]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives +Oct 29 13:00:26 felhom sudo[703590]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:26 felhom sudo[703590]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:27 felhom sudo[703602]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:00:27 felhom sudo[703602]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:27 felhom sudo[703602]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:27 felhom sudo[703605]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:00:27 felhom sudo[703605]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:27 felhom sudo[703605]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:27 felhom sudo[703608]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/adatlemez/felhom-data +Oct 29 13:00:27 felhom sudo[703608]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:27 felhom sudo[703608]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:27 felhom sudo[703611]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/chown 100000:100000 /mnt/adatlemez/felhom-data +Oct 29 13:00:27 felhom sudo[703611]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:27 felhom sudo[703611]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:27 felhom sudo[703614]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives/adatlemez +Oct 29 13:00:27 felhom sudo[703614]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:27 felhom sudo[703614]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:27 felhom sudo[703617]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:00:27 felhom sudo[703617]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:27 felhom sudo[703617]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:27 felhom felhom-agent[697615]: time=2026-10-29T13:00:27.178+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/adatlemez guest_path=/mnt/felhom-drives/adatlemez durable_id=uuid:863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 29 13:00:33 felhom sudo[703790]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:00:33 felhom sudo[703790]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:33 felhom sudo[703790]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:33 felhom sudo[703793]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:00:33 felhom sudo[703793]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:33 felhom sudo[703793]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:33 felhom sudo[703796]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:00:33 felhom sudo[703796]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:33 felhom sudo[703796]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:34 felhom sudo[703799]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 13:00:34 felhom sudo[703799]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:34 felhom sudo[703799]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:34 felhom sudo[703802]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:00:34 felhom sudo[703802]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:34 felhom sudo[703802]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:34 felhom sudo[703814]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:00:34 felhom sudo[703814]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:34 felhom sudo[703814]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:34 felhom sudo[703817]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:00:34 felhom sudo[703817]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:34 felhom sudo[703817]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:34 felhom sudo[703820]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:00:34 felhom sudo[703820]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:34 felhom sudo[703820]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:34 felhom sudo[703823]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 13:00:34 felhom sudo[703823]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:34 felhom sudo[703823]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:34 felhom sudo[703826]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:00:34 felhom sudo[703826]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:34 felhom sudo[703826]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:46 felhom sudo[704026]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives +Oct 29 13:00:46 felhom sudo[704026]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:46 felhom sudo[704026]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:46 felhom felhom-agent[697615]: time=2026-10-29T13:00:46.621+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:00:46 felhom sudo[704034]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- ip -4 -o addr show dev eth0 +Oct 29 13:00:46 felhom sudo[704034]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:46 felhom sudo[704045]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:00:46 felhom sudo[704045]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:46 felhom sudo[704045]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:46 felhom felhom-agent[697615]: time=2026-10-29T13:00:46.743+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:00:46 felhom felhom-agent[697615]: time=2026-10-29T13:00:46.752+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:00:46 felhom sudo[704051]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:00:46 felhom sudo[704051]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:46 felhom sudo[704052]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f {{.State.Status}} felhom-controller +Oct 29 13:00:46 felhom sudo[704052]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:46 felhom sudo[704051]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:47 felhom sudo[704062]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:00:47 felhom sudo[704062]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:47 felhom sudo[704062]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:47 felhom sudo[704065]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:00:47 felhom sudo[704065]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:47 felhom sudo[704065]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:47 felhom sudo[704068]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/adatlemez/felhom-data +Oct 29 13:00:47 felhom sudo[704068]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:47 felhom sudo[704068]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:47 felhom sudo[704071]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/chown 100000:100000 /mnt/adatlemez/felhom-data +Oct 29 13:00:47 felhom sudo[704071]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:47 felhom sudo[704071]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:47 felhom sudo[704074]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives/adatlemez +Oct 29 13:00:47 felhom sudo[704074]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:47 felhom sudo[704074]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:47 felhom sudo[704077]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:00:47 felhom sudo[704077]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:47 felhom sudo[704077]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:47 felhom felhom-agent[697615]: time=2026-10-29T13:00:47.184+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/adatlemez guest_path=/mnt/felhom-drives/adatlemez durable_id=uuid:863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 29 13:00:47 felhom sudo[704034]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:47 felhom sudo[704083]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- ip route show default +Oct 29 13:00:47 felhom sudo[704083]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:47 felhom sudo[704095]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob ssh_port +Oct 29 13:00:47 felhom sudo[704095]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:47 felhom sudo[704095]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:47 felhom sudo[704052]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:47 felhom sudo[704099]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob operator_ips +Oct 29 13:00:47 felhom sudo[704099]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:47 felhom sudo[704099]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:47 felhom sudo[704102]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:00:47 felhom sudo[704102]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:48 felhom sudo[704083]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:48 felhom sudo[704110]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- cat /etc/network/interfaces +Oct 29 13:00:48 felhom sudo[704110]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:48 felhom sudo[704102]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:49 felhom sudo[704110]: pam_unix(sudo:session): session closed for user root +Oct 29 13:00:49 felhom sudo[704150]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- pgrep -x dhclient +Oct 29 13:00:49 felhom sudo[704150]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:00:50 felhom sudo[704150]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:01 felhom felhom-agent[697615]: time=2026-10-29T13:01:01.293+01:00 level=INFO msg="poke received → triggering an immediate desired-state cycle" from=10.77.0.1:56840 +Oct 29 13:01:01 felhom felhom-agent[697615]: time=2026-10-29T13:01:01.293+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" +Oct 29 13:01:01 felhom sudo[704409]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:01 felhom sudo[704409]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:01 felhom sudo[704409]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:01 felhom sudo[704412]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:01 felhom sudo[704412]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:01 felhom sudo[704412]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:01 felhom sudo[704415]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +Oct 29 13:01:01 felhom sudo[704415]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:01 felhom sudo[704415]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:02 felhom sudo[704452]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Status}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' cloudflared +Oct 29 13:01:02 felhom sudo[704452]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:02 felhom sudo[704519]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:02 felhom sudo[704519]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:02 felhom sudo[704519]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:02 felhom sudo[704524]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:02 felhom sudo[704524]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:02 felhom sudo[704524]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:03 felhom sudo[704452]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:03 felhom sudo[704670]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:03 felhom sudo[704670]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:03 felhom sudo[704670]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:03 felhom sudo[704678]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:03 felhom sudo[704678]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704678]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:04 felhom sudo[704690]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:04 felhom sudo[704690]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704690]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:04 felhom sudo[704696]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 13:01:04 felhom sudo[704696]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704696]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:04 felhom sudo[704704]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:04 felhom sudo[704704]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704704]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:04 felhom sudo[704751]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:01:04 felhom sudo[704751]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704751]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:04 felhom sudo[704761]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config +Oct 29 13:01:04 felhom sudo[704761]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704761]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:04 felhom sudo[704764]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config +Oct 29 13:01:04 felhom sudo[704764]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704764]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:04 felhom sudo[704768]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:01:04 felhom sudo[704768]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704768]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:04 felhom sudo[704771]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob operator_ips +Oct 29 13:01:04 felhom sudo[704771]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704771]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:04 felhom sudo[704774]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:04 felhom sudo[704774]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704774]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:04 felhom sudo[704777]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:04 felhom sudo[704777]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom felhom-agent[697615]: time=2026-10-29T13:01:04.595+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:01:04 felhom sudo[704777]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:04 felhom felhom-agent[697615]: time=2026-10-29T13:01:04.605+01:00 level=WARN msg="controller-supervisor: RESTARTING the controller on the operator's press" vmid=9201 action_id=3 unit=felhom-controller-bootstrap.service +Oct 29 13:01:04 felhom sudo[704782]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- systemctl restart felhom-controller-bootstrap.service +Oct 29 13:01:04 felhom sudo[704782]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704783]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:04 felhom sudo[704783]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704783]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:04 felhom sudo[704788]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 13:01:04 felhom sudo[704788]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704788]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:04 felhom sudo[704791]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:04 felhom sudo[704791]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:04 felhom sudo[704791]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:06 felhom sudo[704782]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:06 felhom sudo[704930]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:01:06 felhom sudo[704930]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:06 felhom sudo[704942]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives +Oct 29 13:01:06 felhom sudo[704942]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:06 felhom sudo[704942]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:06 felhom sudo[704945]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:06 felhom sudo[704945]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:06 felhom sudo[704945]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:06 felhom sudo[704948]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:06 felhom sudo[704948]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:06 felhom sudo[704948]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:07 felhom sudo[705045]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:07 felhom sudo[705045]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:07 felhom sudo[705045]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:07 felhom sudo[705055]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:07 felhom sudo[705055]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:07 felhom sudo[705055]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:07 felhom sudo[705079]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/adatlemez/felhom-data +Oct 29 13:01:07 felhom sudo[705079]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:07 felhom sudo[705079]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:07 felhom sudo[705091]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/chown 100000:100000 /mnt/adatlemez/felhom-data +Oct 29 13:01:07 felhom sudo[705091]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:07 felhom sudo[705091]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:07 felhom sudo[705099]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives/adatlemez +Oct 29 13:01:07 felhom sudo[705099]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:07 felhom sudo[705099]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:07 felhom sudo[705105]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:07 felhom sudo[705105]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:07 felhom sudo[705105]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:07 felhom felhom-agent[697615]: time=2026-10-29T13:01:07.167+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/adatlemez guest_path=/mnt/felhom-drives/adatlemez durable_id=uuid:863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 29 13:01:07 felhom sudo[704930]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:07 felhom sudo[705341]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:07 felhom sudo[705341]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:07 felhom sudo[705341]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:07 felhom sudo[705344]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:07 felhom sudo[705344]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:07 felhom sudo[705344]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:07 felhom sudo[705347]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:07 felhom sudo[705347]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:07 felhom sudo[705347]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:07 felhom sudo[705350]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 13:01:07 felhom sudo[705350]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:07 felhom sudo[705350]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:07 felhom sudo[705353]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:07 felhom sudo[705353]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:07 felhom sudo[705353]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:08 felhom sudo[705365]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:08 felhom sudo[705365]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:08 felhom sudo[705365]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:08 felhom sudo[705368]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:08 felhom sudo[705368]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:08 felhom sudo[705368]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:08 felhom sudo[705371]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:08 felhom sudo[705371]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:08 felhom sudo[705371]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:08 felhom sudo[705374]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 13:01:08 felhom sudo[705374]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:08 felhom sudo[705374]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:08 felhom sudo[705377]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:08 felhom sudo[705377]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:08 felhom sudo[705377]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:09 felhom sudo[705382]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:01:09 felhom sudo[705382]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:10 felhom sudo[705382]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:12 felhom sudo[705514]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:01:12 felhom sudo[705514]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:12 felhom sudo[705524]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:12 felhom sudo[705524]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:12 felhom sudo[705524]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:12 felhom sudo[705527]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:12 felhom sudo[705527]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:12 felhom sudo[705527]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:12 felhom felhom-agent[697615]: time=2026-10-29T13:01:12.759+01:00 level=INFO msg="local-api: same-disk" vmid=9201 a=/mnt/sys_drive b=/mnt/felhom-drives/adatlemez different=true proven=false reason=vm +Oct 29 13:01:13 felhom sudo[705514]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:13 felhom felhom-agent[697615]: time=2026-10-29T13:01:13.475+01:00 level=WARN msg="audit: operator action" id=3 action=restart_controller arg=9201 by="the operator via the hub" outcome=done message="restarted; the controller answered after 9 s (Docker's health check: healthy)" +Oct 29 13:01:13 felhom felhom-agent[697615]: time=2026-10-29T13:01:13.475+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" +Oct 29 13:01:13 felhom sudo[705776]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:13 felhom sudo[705776]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:13 felhom sudo[705776]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:13 felhom sudo[705792]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:13 felhom sudo[705792]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:13 felhom sudo[705792]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:13 felhom sudo[705804]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +Oct 29 13:01:13 felhom sudo[705804]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:14 felhom sudo[705804]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:14 felhom sudo[705812]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Status}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' cloudflared +Oct 29 13:01:14 felhom sudo[705812]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:15 felhom sudo[705812]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:16 felhom sudo[705994]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:01:16 felhom sudo[705994]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:16 felhom sudo[705994]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:16 felhom sudo[705999]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config +Oct 29 13:01:16 felhom sudo[705999]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:16 felhom sudo[705999]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:16 felhom sudo[706002]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config +Oct 29 13:01:16 felhom sudo[706002]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:16 felhom sudo[706002]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:16 felhom sudo[706006]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:01:16 felhom sudo[706006]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:16 felhom sudo[706006]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:16 felhom sudo[706009]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob operator_ips +Oct 29 13:01:16 felhom sudo[706009]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:16 felhom sudo[706009]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:16 felhom felhom-agent[697615]: time=2026-10-29T13:01:16.717+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:01:16 felhom sudo[706014]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f {{.State.Status}} felhom-controller +Oct 29 13:01:16 felhom sudo[706014]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:17 felhom sudo[706014]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:17 felhom sudo[706057]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:01:17 felhom sudo[706057]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:18 felhom sudo[706057]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:26 felhom sudo[706273]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives +Oct 29 13:01:26 felhom sudo[706273]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:26 felhom sudo[706273]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:27 felhom sudo[706285]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:27 felhom sudo[706285]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:27 felhom sudo[706285]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:27 felhom sudo[706288]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:27 felhom sudo[706288]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:27 felhom sudo[706288]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:27 felhom sudo[706291]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/adatlemez/felhom-data +Oct 29 13:01:27 felhom sudo[706291]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:27 felhom sudo[706291]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:27 felhom sudo[706294]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/chown 100000:100000 /mnt/adatlemez/felhom-data +Oct 29 13:01:27 felhom sudo[706294]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:27 felhom sudo[706294]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:27 felhom sudo[706297]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives/adatlemez +Oct 29 13:01:27 felhom sudo[706297]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:27 felhom sudo[706297]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:27 felhom sudo[706300]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:27 felhom sudo[706300]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:27 felhom sudo[706300]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:27 felhom felhom-agent[697615]: time=2026-10-29T13:01:27.163+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/adatlemez guest_path=/mnt/felhom-drives/adatlemez durable_id=uuid:863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 29 13:01:32 felhom felhom-agent[697615]: time=2026-10-29T13:01:32.252+01:00 level=INFO msg="poke received → triggering an immediate desired-state cycle" from=10.77.0.1:45209 +Oct 29 13:01:32 felhom felhom-agent[697615]: time=2026-10-29T13:01:32.252+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" +Oct 29 13:01:32 felhom sudo[706498]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:32 felhom sudo[706498]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:32 felhom sudo[706498]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:32 felhom sudo[706501]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:32 felhom sudo[706501]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:32 felhom sudo[706501]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:32 felhom sudo[706504]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +Oct 29 13:01:32 felhom sudo[706504]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:32 felhom sudo[706504]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:32 felhom sudo[706514]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Status}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' cloudflared +Oct 29 13:01:32 felhom sudo[706514]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:34 felhom sudo[706514]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:34 felhom sudo[706728]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:01:34 felhom sudo[706728]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:34 felhom sudo[706728]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:34 felhom sudo[706733]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config +Oct 29 13:01:34 felhom sudo[706733]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:35 felhom sudo[706733]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:35 felhom sudo[706736]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config +Oct 29 13:01:35 felhom sudo[706736]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:35 felhom sudo[706736]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:35 felhom sudo[706740]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:01:35 felhom sudo[706740]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:35 felhom sudo[706740]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:35 felhom sudo[706743]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob operator_ips +Oct 29 13:01:35 felhom sudo[706743]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:35 felhom sudo[706743]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:35 felhom felhom-agent[697615]: time=2026-10-29T13:01:35.148+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:01:35 felhom felhom-agent[697615]: time=2026-10-29T13:01:35.155+01:00 level=WARN msg="controller-supervisor: RESTARTING the controller on the operator's press" vmid=9201 action_id=4 unit=felhom-controller-bootstrap.service +Oct 29 13:01:35 felhom sudo[706748]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- systemctl restart felhom-controller-bootstrap.service +Oct 29 13:01:35 felhom sudo[706748]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:36 felhom sudo[706748]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:36 felhom sudo[706868]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:01:36 felhom sudo[706868]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:36 felhom sudo[706880]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:36 felhom sudo[706880]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:37 felhom sudo[706880]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:37 felhom sudo[706883]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:37 felhom sudo[706883]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:37 felhom sudo[706883]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:37 felhom sudo[706868]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:38 felhom sudo[707258]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:38 felhom sudo[707258]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:38 felhom sudo[707258]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:38 felhom sudo[707261]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:38 felhom sudo[707261]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:38 felhom sudo[707261]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:38 felhom sudo[707264]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:38 felhom sudo[707264]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:38 felhom sudo[707264]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:38 felhom sudo[707267]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 13:01:38 felhom sudo[707267]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:38 felhom sudo[707267]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:38 felhom sudo[707270]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:38 felhom sudo[707270]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:38 felhom sudo[707270]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:38 felhom sudo[707282]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:38 felhom sudo[707282]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:38 felhom sudo[707282]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:38 felhom sudo[707285]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:38 felhom sudo[707285]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:38 felhom sudo[707285]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:38 felhom sudo[707288]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:38 felhom sudo[707288]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:38 felhom sudo[707288]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:38 felhom sudo[707291]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 13:01:38 felhom sudo[707291]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:38 felhom sudo[707291]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:38 felhom sudo[707294]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:38 felhom sudo[707294]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:38 felhom sudo[707294]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:39 felhom sudo[707299]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:01:39 felhom sudo[707299]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:40 felhom sudo[707299]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:42 felhom sudo[707421]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:01:42 felhom sudo[707421]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:43 felhom sudo[707443]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:43 felhom sudo[707443]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:43 felhom sudo[707443]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:43 felhom sudo[707446]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:43 felhom sudo[707446]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:43 felhom sudo[707446]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:43 felhom felhom-agent[697615]: time=2026-10-29T13:01:43.100+01:00 level=INFO msg="local-api: same-disk" vmid=9201 a=/mnt/sys_drive b=/mnt/felhom-drives/adatlemez different=true proven=false reason=vm +Oct 29 13:01:43 felhom sudo[707421]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:43 felhom felhom-agent[697615]: time=2026-10-29T13:01:43.657+01:00 level=WARN msg="audit: operator action" id=4 action=restart_controller arg=9201 by="the operator via the hub" outcome=done message="restarted; the controller answered after 9 s (Docker's health check: healthy)" +Oct 29 13:01:43 felhom felhom-agent[697615]: time=2026-10-29T13:01:43.657+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" +Oct 29 13:01:44 felhom sudo[707632]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:44 felhom sudo[707632]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:44 felhom sudo[707632]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:44 felhom sudo[707649]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:44 felhom sudo[707649]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:44 felhom sudo[707649]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:44 felhom sudo[707662]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +Oct 29 13:01:44 felhom sudo[707662]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:44 felhom sudo[707662]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:44 felhom sudo[707709]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Status}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' cloudflared +Oct 29 13:01:44 felhom sudo[707709]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:45 felhom sudo[707709]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:46 felhom sudo[707906]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:01:46 felhom sudo[707906]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:46 felhom sudo[707906]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:46 felhom sudo[707911]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config +Oct 29 13:01:46 felhom sudo[707911]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:46 felhom sudo[707911]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:46 felhom sudo[707914]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config +Oct 29 13:01:46 felhom sudo[707914]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:46 felhom sudo[707914]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:46 felhom sudo[707918]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:01:46 felhom sudo[707918]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:46 felhom sudo[707918]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:46 felhom sudo[707921]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob operator_ips +Oct 29 13:01:46 felhom sudo[707921]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:46 felhom sudo[707921]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:46 felhom sudo[707924]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives +Oct 29 13:01:46 felhom sudo[707924]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:46 felhom sudo[707924]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:46 felhom felhom-agent[697615]: time=2026-10-29T13:01:46.617+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:01:46 felhom sudo[707931]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- ip -4 -o addr show dev eth0 +Oct 29 13:01:46 felhom sudo[707931]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:46 felhom sudo[707940]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:01:46 felhom sudo[707940]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:46 felhom sudo[707940]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:46 felhom felhom-agent[697615]: time=2026-10-29T13:01:46.733+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:01:46 felhom felhom-agent[697615]: time=2026-10-29T13:01:46.740+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:01:46 felhom sudo[707946]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:01:46 felhom sudo[707946]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:46 felhom sudo[707945]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f {{.State.Status}} felhom-controller +Oct 29 13:01:46 felhom sudo[707945]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:46 felhom sudo[707946]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:46 felhom felhom-agent[697615]: time=2026-10-29T13:01:46.772+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:01:46 felhom sudo[707962]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:01:46 felhom sudo[707962]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:47 felhom sudo[707962]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:47 felhom sudo[707965]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:01:47 felhom sudo[707965]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:47 felhom sudo[707965]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:47 felhom sudo[707968]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/adatlemez/felhom-data +Oct 29 13:01:47 felhom sudo[707968]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:47 felhom sudo[707968]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:47 felhom sudo[707971]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/chown 100000:100000 /mnt/adatlemez/felhom-data +Oct 29 13:01:47 felhom sudo[707971]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:47 felhom sudo[707971]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:47 felhom sudo[707974]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives/adatlemez +Oct 29 13:01:47 felhom sudo[707974]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:47 felhom sudo[707974]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:47 felhom sudo[707977]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:01:47 felhom sudo[707977]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:47 felhom sudo[707977]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:47 felhom felhom-agent[697615]: time=2026-10-29T13:01:47.134+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/adatlemez guest_path=/mnt/felhom-drives/adatlemez durable_id=uuid:863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 29 13:01:47 felhom sudo[707931]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:47 felhom sudo[708008]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob ssh_port +Oct 29 13:01:47 felhom sudo[708008]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:47 felhom sudo[708008]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:47 felhom sudo[707945]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:47 felhom sudo[708014]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- ip route show default +Oct 29 13:01:47 felhom sudo[708014]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:47 felhom sudo[708016]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob operator_ips +Oct 29 13:01:47 felhom sudo[708016]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:47 felhom sudo[708015]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:01:47 felhom sudo[708015]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:47 felhom sudo[708016]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:48 felhom sudo[708014]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:48 felhom sudo[708027]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- cat /etc/network/interfaces +Oct 29 13:01:48 felhom sudo[708027]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:48 felhom sudo[708015]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:49 felhom sudo[708027]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:49 felhom sudo[708066]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- pgrep -x dhclient +Oct 29 13:01:49 felhom sudo[708066]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:49 felhom sudo[708069]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- ip -4 -o addr show dev eth0 +Oct 29 13:01:49 felhom sudo[708069]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:50 felhom sudo[708066]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:50 felhom sudo[708069]: pam_unix(sudo:session): session closed for user root +Oct 29 13:01:50 felhom sudo[708117]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker exec felhom-controller cat /opt/docker/felhom-controller/controller.yaml +Oct 29 13:01:50 felhom sudo[708117]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:01:51 felhom sudo[708117]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:00 felhom felhom-agent[697615]: time=2026-10-29T13:02:00.382+01:00 level=INFO msg="poke received → triggering an immediate desired-state cycle" from=10.77.0.1:35356 +Oct 29 13:02:00 felhom felhom-agent[697615]: time=2026-10-29T13:02:00.383+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" +Oct 29 13:02:00 felhom sudo[708363]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:02:00 felhom sudo[708363]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:00 felhom sudo[708363]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:00 felhom sudo[708366]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:02:00 felhom sudo[708366]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:00 felhom sudo[708366]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:00 felhom sudo[708369]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +Oct 29 13:02:00 felhom sudo[708369]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:00 felhom sudo[708369]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:01 felhom sudo[708377]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Status}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' cloudflared +Oct 29 13:02:01 felhom sudo[708377]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:02 felhom sudo[708377]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:02 felhom sudo[708583]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:02:02 felhom sudo[708583]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:02 felhom sudo[708583]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:03 felhom sudo[708588]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config +Oct 29 13:02:03 felhom sudo[708588]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:03 felhom sudo[708588]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:03 felhom sudo[708591]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config +Oct 29 13:02:03 felhom sudo[708591]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:03 felhom sudo[708591]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:03 felhom sudo[708595]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:02:03 felhom sudo[708595]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:03 felhom sudo[708595]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:03 felhom sudo[708598]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob operator_ips +Oct 29 13:02:03 felhom sudo[708598]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:03 felhom sudo[708598]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:03 felhom felhom-agent[697615]: time=2026-10-29T13:02:03.182+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:02:03 felhom felhom-agent[697615]: time=2026-10-29T13:02:03.189+01:00 level=WARN msg="controller-supervisor: RESTARTING the controller on the operator's press" vmid=9201 action_id=5 unit=felhom-controller-bootstrap.service +Oct 29 13:02:03 felhom sudo[708603]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- systemctl restart felhom-controller-bootstrap.service +Oct 29 13:02:03 felhom sudo[708603]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:04 felhom sudo[708603]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:04 felhom sudo[708759]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:02:04 felhom sudo[708759]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:05 felhom sudo[708771]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:02:05 felhom sudo[708771]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:05 felhom sudo[708771]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:05 felhom sudo[708774]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:02:05 felhom sudo[708774]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:05 felhom sudo[708774]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:05 felhom sudo[708759]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:06 felhom sudo[709163]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:02:06 felhom sudo[709163]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:06 felhom sudo[709163]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:06 felhom sudo[709166]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:02:06 felhom sudo[709166]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:06 felhom sudo[709166]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:06 felhom sudo[709169]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:02:06 felhom sudo[709169]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:06 felhom sudo[709169]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:06 felhom sudo[709172]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 13:02:06 felhom sudo[709172]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:06 felhom sudo[709172]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:06 felhom sudo[709175]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:02:06 felhom sudo[709175]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:06 felhom sudo[709175]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:06 felhom sudo[709182]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives +Oct 29 13:02:06 felhom sudo[709182]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:06 felhom sudo[709182]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:06 felhom sudo[709195]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:02:06 felhom sudo[709195]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:06 felhom sudo[709195]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:06 felhom sudo[709198]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:02:06 felhom sudo[709198]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:06 felhom sudo[709198]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:06 felhom sudo[709201]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:02:06 felhom sudo[709201]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:06 felhom sudo[709201]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:06 felhom sudo[709205]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb +Oct 29 13:02:06 felhom sudo[709205]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:06 felhom sudo[709205]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:06 felhom sudo[709208]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:02:06 felhom sudo[709208]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:06 felhom sudo[709208]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:07 felhom sudo[709214]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:02:07 felhom sudo[709214]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:07 felhom sudo[709214]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:07 felhom sudo[709217]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:02:07 felhom sudo[709217]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:07 felhom sudo[709217]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:07 felhom sudo[709220]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/adatlemez/felhom-data +Oct 29 13:02:07 felhom sudo[709220]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:07 felhom sudo[709220]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:07 felhom sudo[709223]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/chown 100000:100000 /mnt/adatlemez/felhom-data +Oct 29 13:02:07 felhom sudo[709223]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:07 felhom sudo[709223]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:07 felhom sudo[709226]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives/adatlemez +Oct 29 13:02:07 felhom sudo[709226]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:07 felhom sudo[709226]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:07 felhom sudo[709229]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:02:07 felhom sudo[709229]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:07 felhom sudo[709229]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:07 felhom felhom-agent[697615]: time=2026-10-29T13:02:07.152+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/adatlemez guest_path=/mnt/felhom-drives/adatlemez durable_id=uuid:863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 +Oct 29 13:02:07 felhom sudo[709244]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:02:07 felhom sudo[709244]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:08 felhom sudo[709244]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:10 felhom sudo[709377]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:02:10 felhom sudo[709377]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:11 felhom sudo[709389]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:02:11 felhom sudo[709389]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:11 felhom sudo[709389]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:11 felhom sudo[709392]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:02:11 felhom sudo[709392]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:11 felhom sudo[709392]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:11 felhom felhom-agent[697615]: time=2026-10-29T13:02:11.174+01:00 level=INFO msg="local-api: same-disk" vmid=9201 a=/mnt/sys_drive b=/mnt/felhom-drives/adatlemez different=true proven=false reason=vm +Oct 29 13:02:11 felhom sudo[709377]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:11 felhom felhom-agent[697615]: time=2026-10-29T13:02:11.831+01:00 level=WARN msg="audit: operator action" id=5 action=restart_controller arg=9201 by="the operator via the hub" outcome=done message="restarted; the controller answered after 9 s (Docker's health check: healthy)" +Oct 29 13:02:11 felhom felhom-agent[697615]: time=2026-10-29T13:02:11.831+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" +Oct 29 13:02:12 felhom sudo[709589]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:02:12 felhom sudo[709589]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:12 felhom sudo[709589]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:12 felhom sudo[709599]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:02:12 felhom sudo[709599]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:12 felhom sudo[709599]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:12 felhom sudo[709619]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +Oct 29 13:02:12 felhom sudo[709619]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:12 felhom sudo[709619]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:12 felhom sudo[709636]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Status}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' cloudflared +Oct 29 13:02:12 felhom sudo[709636]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:13 felhom sudo[709636]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:14 felhom sudo[709848]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:02:14 felhom sudo[709848]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:14 felhom sudo[709848]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:14 felhom sudo[709853]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config +Oct 29 13:02:14 felhom sudo[709853]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:14 felhom sudo[709853]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:14 felhom sudo[709856]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config +Oct 29 13:02:14 felhom sudo[709856]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:14 felhom sudo[709856]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:14 felhom sudo[709860]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:02:14 felhom sudo[709860]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:14 felhom sudo[709860]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:14 felhom sudo[709863]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob operator_ips +Oct 29 13:02:14 felhom sudo[709863]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:14 felhom sudo[709863]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:16 felhom felhom-agent[697615]: time=2026-10-29T13:02:16.674+01:00 level=INFO msg="poke received → triggering an immediate desired-state cycle" from=10.77.0.1:46324 +Oct 29 13:02:16 felhom felhom-agent[697615]: time=2026-10-29T13:02:16.674+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" +Oct 29 13:02:16 felhom felhom-agent[697615]: time=2026-10-29T13:02:16.718+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:02:16 felhom sudo[709910]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f {{.State.Status}} felhom-controller +Oct 29 13:02:16 felhom sudo[709910]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:17 felhom sudo[709922]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:02:17 felhom sudo[709922]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:17 felhom sudo[709922]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:17 felhom sudo[709925]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:02:17 felhom sudo[709925]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:17 felhom sudo[709925]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:17 felhom sudo[709928]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +Oct 29 13:02:17 felhom sudo[709928]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:17 felhom sudo[709928]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:17 felhom sudo[709936]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Status}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' cloudflared +Oct 29 13:02:17 felhom sudo[709936]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:17 felhom sudo[709910]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:17 felhom sudo[709949]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' felhom-controller +Oct 29 13:02:17 felhom sudo[709949]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:18 felhom sudo[709936]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:18 felhom sudo[709949]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:19 felhom sudo[710154]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:02:19 felhom sudo[710154]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:19 felhom sudo[710154]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:19 felhom sudo[710159]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config +Oct 29 13:02:19 felhom sudo[710159]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:19 felhom sudo[710159]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:19 felhom sudo[710162]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config +Oct 29 13:02:19 felhom sudo[710162]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:19 felhom sudo[710162]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:19 felhom sudo[710166]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:02:19 felhom sudo[710166]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:19 felhom sudo[710166]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:19 felhom sudo[710169]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob operator_ips +Oct 29 13:02:19 felhom sudo[710169]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:19 felhom sudo[710169]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:19 felhom felhom-agent[697615]: time=2026-10-29T13:02:19.423+01:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Oct 29 13:02:19 felhom felhom-agent[697615]: time=2026-10-29T13:02:19.430+01:00 level=WARN msg="audit: operator action" id=6 action=restart_controller arg=9201 by="the operator via the hub" outcome=refused message="refused: 3 restarts in 15 minutes — presses are refused until 12:32 UTC" +Oct 29 13:02:19 felhom felhom-agent[697615]: time=2026-10-29T13:02:19.430+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" +Oct 29 13:02:19 felhom sudo[710185]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:02:19 felhom sudo[710185]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:19 felhom sudo[710185]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:19 felhom sudo[710188]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:02:19 felhom sudo[710188]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:19 felhom sudo[710188]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:19 felhom sudo[710191]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage +Oct 29 13:02:19 felhom sudo[710191]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:19 felhom sudo[710191]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:20 felhom sudo[710199]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Status}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' cloudflared +Oct 29 13:02:20 felhom sudo[710199]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:21 felhom sudo[710199]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:22 felhom sudo[710407]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:02:22 felhom sudo[710407]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:22 felhom sudo[710407]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:22 felhom sudo[710412]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config +Oct 29 13:02:22 felhom sudo[710412]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:22 felhom sudo[710412]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:22 felhom sudo[710415]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config +Oct 29 13:02:22 felhom sudo[710415]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:22 felhom sudo[710415]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:22 felhom sudo[710419]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes +Oct 29 13:02:22 felhom sudo[710419]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:22 felhom sudo[710419]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:22 felhom sudo[710422]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nft list set inet felhom_oob operator_ips +Oct 29 13:02:22 felhom sudo[710422]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:22 felhom sudo[710422]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:26 felhom sudo[710564]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives +Oct 29 13:02:26 felhom sudo[710564]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:26 felhom sudo[710564]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:27 felhom sudo[710576]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda +Oct 29 13:02:27 felhom sudo[710576]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:27 felhom sudo[710576]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:27 felhom sudo[710579]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Oct 29 13:02:27 felhom sudo[710579]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:27 felhom sudo[710579]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:27 felhom sudo[710582]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/adatlemez/felhom-data +Oct 29 13:02:27 felhom sudo[710582]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:27 felhom sudo[710582]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:27 felhom sudo[710585]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/chown 100000:100000 /mnt/adatlemez/felhom-data +Oct 29 13:02:27 felhom sudo[710585]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:27 felhom sudo[710585]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:27 felhom sudo[710588]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/mkdir -p /mnt/felhom-drives/adatlemez +Oct 29 13:02:27 felhom sudo[710588]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:27 felhom sudo[710588]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:27 felhom sudo[710591]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/lxc-info -n 9201 -p -H +Oct 29 13:02:27 felhom sudo[710591]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +Oct 29 13:02:27 felhom sudo[710591]: pam_unix(sudo:session): session closed for user root +Oct 29 13:02:27 felhom felhom-agent[697615]: time=2026-10-29T13:02:27.158+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/adatlemez guest_path=/mnt/felhom-drives/adatlemez durable_id=uuid:863c67fd-3e21-4d9f-9a74-3c5fa406f2a2 diff --git a/documentation/audits/release-2026-10-29b/p1-restart/hub-presses.txt b/documentation/audits/release-2026-10-29b/p1-restart/hub-presses.txt new file mode 100644 index 00000000..81c14780 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p1-restart/hub-presses.txt @@ -0,0 +1,46 @@ +2026/10/29 12:55:12 [INFO] access token "cc-hosts": GET /hosts/tester-1-d70be4 → 200 +2026/10/29 12:55:12 [INFO] access token "cc-hosts": GET /hosts/tester-1-d70be4 → 200 +2026/10/29 12:56:41 [INFO] host-report from tester-1-d70be4 (1 guests, 3 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 20600 bytes) +2026/10/29 12:56:41 [INFO] DR-recipe host-half stored for customer tester-1 (host tester-1-d70be4, v1) +2026/10/29 12:56:43 [INFO] host tester-1-d70be4 cleared signed-op job 72750c05c087ad02 (executed or rejected) +2026/10/29 12:56:51 [INFO] host-report from tester-1-d70be4 (1 guests, 3 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 20411 bytes) +2026/10/29 12:56:51 [INFO] DR-recipe host-half stored for customer tester-1 (host tester-1-d70be4, v1) +2026/10/29 13:01:00 [INFO] host operator action #3 restart_controller 9201 requested for host tester-1-d70be4 by operator CLI (basic auth) from 10.42.0.1 — the agent acts on its next report (poked now) +2026/10/29 13:01:00 [INFO] access token "cc-hosts": POST /hosts/tester-1-d70be4/operator-action → 303 +2026/10/29 13:01:04 [INFO] host-report from tester-1-d70be4 (1 guests, 3 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 20602 bytes) +2026/10/29 13:01:04 [INFO] DR-recipe host-half stored for customer tester-1 (host tester-1-d70be4, v1) +2026/10/29 13:01:12 [INFO] Event from tester-1: controller_started (info) — Controller elindult (0.347.0) +2026/10/29 13:01:13 [INFO] DR-recipe app-half stored for customer tester-1 (v1) +2026/10/29 13:01:13 [INFO] Received report from tester-1 (5272 bytes) +2026/10/29 13:01:16 [INFO] host-report from tester-1-d70be4 (1 guests, 3 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 20839 bytes) +2026/10/29 13:01:16 [INFO] DR-recipe host-half stored for customer tester-1 (host tester-1-d70be4, v1) +2026/10/29 13:01:16 [INFO] host operator action #3 restart_controller 9201 on tester-1-d70be4 (pressed by operator CLI (basic auth) from 10.42.0.1): done — restarted; the controller answered after 9 s (Docker's health check: healthy) +2026/10/29 13:01:27 [INFO] Controller supervisor: controller_restarted_by_agent tester-1-d70be4/9201 (the operator via the hub (action #3)) +2026/10/29 13:01:31 [INFO] host operator action #4 restart_controller 9201 requested for host tester-1-d70be4 by operator CLI (basic auth) from 10.42.0.1 — the agent acts on its next report (poked now) +2026/10/29 13:01:31 [INFO] access token "cc-hosts": POST /hosts/tester-1-d70be4/operator-action → 303 +2026/10/29 13:01:35 [INFO] host-report from tester-1-d70be4 (1 guests, 3 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 20697 bytes) +2026/10/29 13:01:35 [INFO] DR-recipe host-half stored for customer tester-1 (host tester-1-d70be4, v1) +2026/10/29 13:01:42 [INFO] Event from tester-1: controller_started (info) — Controller elindult (0.347.0) +2026/10/29 13:01:43 [INFO] DR-recipe app-half stored for customer tester-1 (v1) +2026/10/29 13:01:43 [INFO] Received report from tester-1 (5312 bytes) +2026/10/29 13:01:46 [INFO] host-report from tester-1-d70be4 (1 guests, 3 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 20840 bytes) +2026/10/29 13:01:46 [INFO] DR-recipe host-half stored for customer tester-1 (host tester-1-d70be4, v1) +2026/10/29 13:01:46 [INFO] host operator action #4 restart_controller 9201 on tester-1-d70be4 (pressed by operator CLI (basic auth) from 10.42.0.1): done — restarted; the controller answered after 9 s (Docker's health check: healthy) +2026/10/29 13:01:59 [INFO] host operator action #5 restart_controller 9201 requested for host tester-1-d70be4 by operator CLI (basic auth) from 10.42.0.1 — the agent acts on its next report (poked now) +2026/10/29 13:01:59 [INFO] access token "cc-hosts": POST /hosts/tester-1-d70be4/operator-action → 303 +2026/10/29 13:02:03 [INFO] host-report from tester-1-d70be4 (1 guests, 3 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 20697 bytes) +2026/10/29 13:02:03 [INFO] DR-recipe host-half stored for customer tester-1 (host tester-1-d70be4, v1) +2026/10/29 13:02:10 [INFO] Event from tester-1: controller_started (info) — Controller elindult (0.347.0) +2026/10/29 13:02:11 [INFO] DR-recipe app-half stored for customer tester-1 (v1) +2026/10/29 13:02:11 [INFO] Received report from tester-1 (5312 bytes) +2026/10/29 13:02:14 [INFO] host-report from tester-1-d70be4 (1 guests, 3 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 20843 bytes) +2026/10/29 13:02:14 [INFO] DR-recipe host-half stored for customer tester-1 (host tester-1-d70be4, v1) +2026/10/29 13:02:14 [INFO] host operator action #5 restart_controller 9201 on tester-1-d70be4 (pressed by operator CLI (basic auth) from 10.42.0.1): done — restarted; the controller answered after 9 s (Docker's health check: healthy) +2026/10/29 13:02:16 [INFO] host operator action #6 restart_controller 9201 requested for host tester-1-d70be4 by operator CLI (basic auth) from 10.42.0.1 — the agent acts on its next report (poked now) +2026/10/29 13:02:16 [INFO] access token "cc-hosts": POST /hosts/tester-1-d70be4/operator-action → 303 +2026/10/29 13:02:19 [INFO] host-report from tester-1-d70be4 (1 guests, 3 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 20698 bytes) +2026/10/29 13:02:19 [INFO] DR-recipe host-half stored for customer tester-1 (host tester-1-d70be4, v1) +2026/10/29 13:02:22 [INFO] host-report from tester-1-d70be4 (1 guests, 3 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 20841 bytes) +2026/10/29 13:02:22 [INFO] DR-recipe host-half stored for customer tester-1 (host tester-1-d70be4, v1) +2026/10/29 13:02:22 [INFO] host operator action #6 restart_controller 9201 on tester-1-d70be4 (pressed by operator CLI (basic auth) from 10.42.0.1): refused — refused: 3 restarts in 15 minutes — presses are refused until 12:32 UTC +2026/10/29 13:02:28 [INFO] Controller supervisor: controller_restarted_by_agent tester-1-d70be4/9201 (the operator via the hub (action #5)) diff --git a/documentation/audits/release-2026-10-29b/p1-restart/press1-agent.txt b/documentation/audits/release-2026-10-29b/p1-restart/press1-agent.txt new file mode 100644 index 00000000..57fc9a5d --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p1-restart/press1-agent.txt @@ -0,0 +1,5 @@ +Oct 29 13:01:01 felhom felhom-agent[697615]: time=2026-10-29T13:01:01.293+01:00 level=INFO msg="poke received → triggering an immediate desired-state cycle" from=10.77.0.1:56840 +Oct 29 13:01:01 felhom felhom-agent[697615]: time=2026-10-29T13:01:01.293+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" +Oct 29 13:01:04 felhom felhom-agent[697615]: time=2026-10-29T13:01:04.605+01:00 level=WARN msg="controller-supervisor: RESTARTING the controller on the operator's press" vmid=9201 action_id=3 unit=felhom-controller-bootstrap.service +Oct 29 13:01:13 felhom felhom-agent[697615]: time=2026-10-29T13:01:13.475+01:00 level=WARN msg="audit: operator action" id=3 action=restart_controller arg=9201 by="the operator via the hub" outcome=done message="restarted; the controller answered after 9 s (Do +Oct 29 13:01:13 felhom felhom-agent[697615]: time=2026-10-29T13:01:13.475+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" diff --git a/documentation/audits/release-2026-10-29b/p1-restart/press2-agent.txt b/documentation/audits/release-2026-10-29b/p1-restart/press2-agent.txt new file mode 100644 index 00000000..bc4a32b6 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p1-restart/press2-agent.txt @@ -0,0 +1,11 @@ +running 2026-10-29T12:01:36.375483534Z healthy +Oct 29 13:01:01 felhom felhom-agent[697615]: time=2026-10-29T13:01:01.293+01:00 level=INFO msg="poke received → triggering an immediate desired-state cycle" from=10.77.0.1:56840 +Oct 29 13:01:01 felhom felhom-agent[697615]: time=2026-10-29T13:01:01.293+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" +Oct 29 13:01:04 felhom felhom-agent[697615]: time=2026-10-29T13:01:04.605+01:00 level=WARN msg="controller-supervisor: RESTARTING the controller on the operator's press" vmid=9201 action_id=3 unit=felhom-controller-bootstrap.service +Oct 29 13:01:13 felhom felhom-agent[697615]: time=2026-10-29T13:01:13.475+01:00 level=WARN msg="audit: operator action" id=3 action=restart_controller arg=9201 by="the operator via the hub" outcome=done message="restarted; the controller an +Oct 29 13:01:13 felhom felhom-agent[697615]: time=2026-10-29T13:01:13.475+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" +Oct 29 13:01:32 felhom felhom-agent[697615]: time=2026-10-29T13:01:32.252+01:00 level=INFO msg="poke received → triggering an immediate desired-state cycle" from=10.77.0.1:45209 +Oct 29 13:01:32 felhom felhom-agent[697615]: time=2026-10-29T13:01:32.252+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" +Oct 29 13:01:35 felhom felhom-agent[697615]: time=2026-10-29T13:01:35.155+01:00 level=WARN msg="controller-supervisor: RESTARTING the controller on the operator's press" vmid=9201 action_id=4 unit=felhom-controller-bootstrap.service +Oct 29 13:01:43 felhom felhom-agent[697615]: time=2026-10-29T13:01:43.657+01:00 level=WARN msg="audit: operator action" id=4 action=restart_controller arg=9201 by="the operator via the hub" outcome=done message="restarted; the controller an +Oct 29 13:01:43 felhom felhom-agent[697615]: time=2026-10-29T13:01:43.657+01:00 level=INFO msg="hub: out-of-band report triggered (watchdog/poke)" diff --git a/documentation/audits/release-2026-10-29b/p1-restart/press3-4-hub.txt b/documentation/audits/release-2026-10-29b/p1-restart/press3-4-hub.txt new file mode 100644 index 00000000..a54cdc2e --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p1-restart/press3-4-hub.txt @@ -0,0 +1,2 @@ +HUB #5: 2026/10/29 13:02:14 [INFO] host operator action #5 restart_controller 9201 on tester-1-d70be4 (pressed by operator CLI (basic auth) from 10.42.0.1): done — restarted; the controller answered after 9 s (Docker's health check: healthy) +HUB #6: 2026/10/29 13:02:22 [INFO] host operator action #6 restart_controller 9201 on tester-1-d70be4 (pressed by operator CLI (basic auth) from 10.42.0.1): refused — refused: 3 restarts in 15 minutes — presses are refused until 12:32 UTC diff --git a/documentation/audits/release-2026-10-29b/p2-health/health.csv b/documentation/audits/release-2026-10-29b/p2-health/health.csv new file mode 100644 index 00000000..65fed69c --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p2-health/health.csv @@ -0,0 +1,595 @@ +box,start,end,duration_ms,exit_code,status,failing_streak,container_started,read_at +tester-1,2026-10-29T11:12:07.201,2026-10-29T11:12:07.255,54,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:14:28Z +tester-1,2026-10-29T11:12:37.257,2026-10-29T11:12:37.310,53,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:14:28Z +tester-1,2026-10-29T11:13:07.310,2026-10-29T11:13:07.372,61,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:14:28Z +tester-1,2026-10-29T11:13:37.374,2026-10-29T11:13:37.430,56,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:14:28Z +tester-1,2026-10-29T11:14:07.431,2026-10-29T11:14:07.488,57,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:14:28Z +demo-hp,2026-10-29T11:12:23.861,2026-10-29T11:12:23.921,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:14:29Z +demo-hp,2026-10-29T11:12:53.923,2026-10-29T11:12:53.982,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:14:29Z +demo-hp,2026-10-29T11:13:23.983,2026-10-29T11:13:24.043,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:14:29Z +demo-hp,2026-10-29T11:13:54.045,2026-10-29T11:13:54.103,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:14:29Z +demo-hp,2026-10-29T11:14:24.104,2026-10-29T11:14:24.165,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:14:29Z +tester-1,2026-10-29T11:14:37.489,2026-10-29T11:14:37.542,52,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:15:00Z +demo-hp,2026-10-29T11:14:54.166,2026-10-29T11:14:54.228,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:15:02Z +tester-1,2026-10-29T11:15:07.543,2026-10-29T11:15:07.599,56,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:15:33Z +demo-hp,2026-10-29T11:15:24.229,2026-10-29T11:15:24.287,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:15:35Z +tester-1,2026-10-29T11:15:37.600,2026-10-29T11:15:37.653,53,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:16:06Z +tester-1,2026-10-29T11:16:07.654,2026-10-29T11:16:07.711,58,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:16:06Z +demo-hp,2026-10-29T11:15:54.288,2026-10-29T11:15:54.347,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:16:08Z +tester-1,2026-10-29T11:16:37.712,2026-10-29T11:16:37.766,54,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:16:39Z +demo-hp,2026-10-29T11:16:24.349,2026-10-29T11:16:24.407,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:16:41Z +tester-1,2026-10-29T11:17:07.768,2026-10-29T11:17:07.820,52,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:17:12Z +demo-hp,2026-10-29T11:16:54.408,2026-10-29T11:16:54.468,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:17:14Z +tester-1,2026-10-29T11:17:37.821,2026-10-29T11:17:37.873,53,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:17:45Z +demo-hp,2026-10-29T11:17:24.468,2026-10-29T11:17:24.526,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:17:47Z +tester-1,2026-10-29T11:18:07.875,2026-10-29T11:18:07.927,52,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:18:18Z +demo-hp,2026-10-29T11:17:54.528,2026-10-29T11:17:54.586,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:18:19Z +tester-1,2026-10-29T11:18:37.928,2026-10-29T11:18:37.988,60,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:18:51Z +demo-hp,2026-10-29T11:18:24.587,2026-10-29T11:18:24.648,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:18:52Z +tester-1,2026-10-29T11:19:07.989,2026-10-29T11:19:08.041,52,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:19:24Z +demo-hp,2026-10-29T11:18:54.650,2026-10-29T11:18:54.709,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:19:25Z +demo-hp,2026-10-29T11:19:24.710,2026-10-29T11:19:24.778,68,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:19:25Z +tester-1,2026-10-29T11:19:38.042,2026-10-29T11:19:38.111,69,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:19:57Z +demo-hp,2026-10-29T11:19:54.779,2026-10-29T11:19:54.839,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:19:58Z +tester-1,2026-10-29T11:20:08.113,2026-10-29T11:20:08.170,57,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:20:29Z +demo-hp,2026-10-29T11:20:24.840,2026-10-29T11:20:24.903,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:20:31Z +tester-1,2026-10-29T11:20:38.171,2026-10-29T11:20:38.228,57,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:21:03Z +demo-hp,2026-10-29T11:20:54.904,2026-10-29T11:20:54.964,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:21:04Z +tester-1,2026-10-29T11:21:08.229,2026-10-29T11:21:08.300,71,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:21:36Z +demo-hp,2026-10-29T11:21:24.965,2026-10-29T11:21:25.024,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:21:37Z +tester-1,2026-10-29T11:21:38.301,2026-10-29T11:21:38.378,77,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:22:09Z +tester-1,2026-10-29T11:22:08.379,2026-10-29T11:22:08.444,65,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:22:09Z +demo-hp,2026-10-29T11:21:55.025,2026-10-29T11:21:55.084,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:22:10Z +tester-1,2026-10-29T11:22:38.445,2026-10-29T11:22:38.498,53,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:22:41Z +demo-hp,2026-10-29T11:22:25.085,2026-10-29T11:22:25.146,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:22:43Z +tester-1,2026-10-29T11:23:08.499,2026-10-29T11:23:08.558,59,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:23:14Z +demo-hp,2026-10-29T11:22:55.148,2026-10-29T11:22:55.207,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:23:16Z +tester-1,2026-10-29T11:23:38.559,2026-10-29T11:23:38.633,74,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:23:47Z +demo-hp,2026-10-29T11:23:25.209,2026-10-29T11:23:25.267,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:23:49Z +tester-1,2026-10-29T11:24:08.634,2026-10-29T11:24:08.697,63,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:24:20Z +demo-hp,2026-10-29T11:23:55.268,2026-10-29T11:23:55.326,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:24:21Z +tester-1,2026-10-29T11:24:38.698,2026-10-29T11:24:38.760,61,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:24:53Z +demo-hp,2026-10-29T11:24:25.328,2026-10-29T11:24:25.387,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:24:54Z +demo-hp,2026-10-29T11:24:55.388,2026-10-29T11:24:55.459,71,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:24:54Z +tester-1,2026-10-29T11:25:08.761,2026-10-29T11:25:08.820,59,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:25:26Z +demo-hp,2026-10-29T11:25:25.461,2026-10-29T11:25:25.520,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:25:27Z +tester-1,2026-10-29T11:25:38.821,2026-10-29T11:25:38.893,72,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:25:59Z +demo-hp,2026-10-29T11:25:55.521,2026-10-29T11:25:55.581,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:26:00Z +tester-1,2026-10-29T11:26:08.894,2026-10-29T11:26:08.953,59,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:26:31Z +demo-hp,2026-10-29T11:26:25.582,2026-10-29T11:26:25.643,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:26:33Z +tester-1,2026-10-29T11:26:38.955,2026-10-29T11:26:39.012,57,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:27:05Z +demo-hp,2026-10-29T11:26:55.644,2026-10-29T11:26:55.704,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:27:06Z +tester-1,2026-10-29T11:27:09.014,2026-10-29T11:27:09.075,62,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:27:38Z +tester-1,2026-10-29T11:27:39.076,2026-10-29T11:27:39.139,63,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:27:38Z +demo-hp,2026-10-29T11:27:25.705,2026-10-29T11:27:25.778,73,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:27:39Z +tester-1,2026-10-29T11:28:09.142,2026-10-29T11:28:09.193,51,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:28:11Z +demo-hp,2026-10-29T11:27:55.778,2026-10-29T11:27:55.844,66,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:28:12Z +tester-1,2026-10-29T11:28:39.194,2026-10-29T11:28:39.255,61,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:28:44Z +demo-hp,2026-10-29T11:28:25.845,2026-10-29T11:28:25.905,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:28:45Z +tester-1,2026-10-29T11:29:09.256,2026-10-29T11:29:09.313,57,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:29:16Z +demo-hp,2026-10-29T11:28:55.906,2026-10-29T11:28:55.965,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:29:18Z +tester-1,2026-10-29T11:29:39.314,2026-10-29T11:29:39.376,61,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:29:49Z +demo-hp,2026-10-29T11:29:25.967,2026-10-29T11:29:26.026,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:29:51Z +tester-1,2026-10-29T11:30:09.377,2026-10-29T11:30:09.433,56,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:30:22Z +demo-hp,2026-10-29T11:29:56.028,2026-10-29T11:29:56.086,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:30:24Z +tester-1,2026-10-29T11:30:39.435,2026-10-29T11:30:39.491,56,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:30:55Z +demo-hp,2026-10-29T11:30:26.087,2026-10-29T11:30:26.145,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:30:56Z +demo-hp,2026-10-29T11:30:56.147,2026-10-29T11:30:56.217,70,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:30:56Z +tester-1,2026-10-29T11:31:09.493,2026-10-29T11:31:09.548,55,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:31:28Z +demo-hp,2026-10-29T11:31:26.219,2026-10-29T11:31:26.278,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:31:29Z +tester-1,2026-10-29T11:31:39.549,2026-10-29T11:31:39.603,54,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:32:01Z +demo-hp,2026-10-29T11:31:56.279,2026-10-29T11:31:56.337,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:32:02Z +tester-1,2026-10-29T11:32:09.605,2026-10-29T11:32:09.666,61,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:32:34Z +demo-hp,2026-10-29T11:32:26.338,2026-10-29T11:32:26.398,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:32:35Z +tester-1,2026-10-29T11:32:39.668,2026-10-29T11:32:39.736,68,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:33:07Z +demo-hp,2026-10-29T11:32:56.399,2026-10-29T11:32:56.460,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:33:08Z +tester-1,2026-10-29T11:33:09.737,2026-10-29T11:33:09.799,63,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:33:40Z +tester-1,2026-10-29T11:33:39.801,2026-10-29T11:33:39.865,64,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:33:40Z +demo-hp,2026-10-29T11:33:26.462,2026-10-29T11:33:26.520,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:33:41Z +tester-1,2026-10-29T11:34:09.866,2026-10-29T11:34:09.920,55,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:34:13Z +demo-hp,2026-10-29T11:33:56.522,2026-10-29T11:33:56.582,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:34:14Z +tester-1,2026-10-29T11:34:39.922,2026-10-29T11:34:39.979,57,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:34:45Z +demo-hp,2026-10-29T11:34:26.584,2026-10-29T11:34:26.641,57,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:34:47Z +tester-1,2026-10-29T11:35:09.980,2026-10-29T11:35:10.042,62,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:35:18Z +demo-hp,2026-10-29T11:34:56.643,2026-10-29T11:34:56.702,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:35:19Z +tester-1,2026-10-29T11:35:40.044,2026-10-29T11:35:40.103,59,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:35:51Z +demo-hp,2026-10-29T11:35:26.704,2026-10-29T11:35:26.763,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:35:52Z +tester-1,2026-10-29T11:36:10.105,2026-10-29T11:36:10.203,98,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:36:24Z +demo-hp,2026-10-29T11:35:56.765,2026-10-29T11:35:56.824,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:36:25Z +demo-hp,2026-10-29T11:36:26.825,2026-10-29T11:36:26.894,69,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:36:25Z +tester-1,2026-10-29T11:36:40.204,2026-10-29T11:36:40.263,59,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:36:57Z +demo-hp,2026-10-29T11:36:56.895,2026-10-29T11:36:56.958,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:36:58Z +tester-1,2026-10-29T11:37:10.265,2026-10-29T11:37:10.322,57,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:37:30Z +demo-hp,2026-10-29T11:37:26.959,2026-10-29T11:37:27.021,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:37:31Z +tester-1,2026-10-29T11:37:40.323,2026-10-29T11:37:40.384,61,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:38:03Z +demo-hp,2026-10-29T11:37:57.022,2026-10-29T11:37:57.082,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:38:04Z +tester-1,2026-10-29T11:38:10.386,2026-10-29T11:38:10.438,51,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:38:36Z +demo-hp,2026-10-29T11:38:27.083,2026-10-29T11:38:27.144,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:38:37Z +tester-1,2026-10-29T11:38:40.438,2026-10-29T11:38:40.500,61,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:39:09Z +tester-1,2026-10-29T11:39:10.501,2026-10-29T11:39:10.564,63,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:39:09Z +demo-hp,2026-10-29T11:38:57.145,2026-10-29T11:38:57.211,66,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:39:10Z +tester-1,2026-10-29T11:39:40.565,2026-10-29T11:39:40.623,58,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:39:42Z +demo-hp,2026-10-29T11:39:27.212,2026-10-29T11:39:27.275,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:39:43Z +tester-1,2026-10-29T11:40:10.624,2026-10-29T11:40:10.675,51,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:40:15Z +demo-hp,2026-10-29T11:39:57.276,2026-10-29T11:39:57.335,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:40:16Z +tester-1,2026-10-29T11:40:40.677,2026-10-29T11:40:40.748,71,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:40:47Z +demo-hp,2026-10-29T11:40:27.337,2026-10-29T11:40:27.397,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:40:49Z +tester-1,2026-10-29T11:41:10.750,2026-10-29T11:41:10.805,55,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:41:20Z +demo-hp,2026-10-29T11:40:57.398,2026-10-29T11:40:57.457,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:41:22Z +tester-1,2026-10-29T11:41:40.806,2026-10-29T11:41:40.870,63,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:41:53Z +demo-hp,2026-10-29T11:41:27.458,2026-10-29T11:41:27.519,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:41:55Z +tester-1,2026-10-29T11:42:10.871,2026-10-29T11:42:10.930,59,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:42:26Z +demo-hp,2026-10-29T11:41:57.521,2026-10-29T11:41:57.577,57,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:42:27Z +demo-hp,2026-10-29T11:42:27.579,2026-10-29T11:42:27.648,70,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:42:27Z +tester-1,2026-10-29T11:42:40.931,2026-10-29T11:42:40.995,64,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:42:59Z +demo-hp,2026-10-29T11:42:57.650,2026-10-29T11:42:57.708,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:43:00Z +tester-1,2026-10-29T11:43:10.997,2026-10-29T11:43:11.049,52,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:43:32Z +demo-hp,2026-10-29T11:43:27.710,2026-10-29T11:43:27.776,66,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:43:33Z +tester-1,2026-10-29T11:43:41.049,2026-10-29T11:43:41.111,61,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:44:05Z +demo-hp,2026-10-29T11:43:57.777,2026-10-29T11:43:57.841,64,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:44:06Z +tester-1,2026-10-29T11:44:11.111,2026-10-29T11:44:11.167,55,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:44:38Z +demo-hp,2026-10-29T11:44:27.843,2026-10-29T11:44:27.904,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:44:39Z +tester-1,2026-10-29T11:44:41.168,2026-10-29T11:44:41.233,65,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:45:11Z +tester-1,2026-10-29T11:45:11.235,2026-10-29T11:45:11.298,63,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:45:11Z +demo-hp,2026-10-29T11:44:57.905,2026-10-29T11:44:57.969,64,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:45:12Z +tester-1,2026-10-29T11:45:41.299,2026-10-29T11:45:41.363,64,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:45:44Z +demo-hp,2026-10-29T11:45:27.970,2026-10-29T11:45:28.030,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:45:45Z +tester-1,2026-10-29T11:46:11.364,2026-10-29T11:46:11.416,52,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:46:17Z +demo-hp,2026-10-29T11:45:58.031,2026-10-29T11:45:58.090,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:46:18Z +tester-1,2026-10-29T11:46:41.417,2026-10-29T11:46:41.481,64,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:46:49Z +demo-hp,2026-10-29T11:46:28.091,2026-10-29T11:46:28.152,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:46:51Z +tester-1,2026-10-29T11:47:11.483,2026-10-29T11:47:11.535,52,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:47:22Z +demo-hp,2026-10-29T11:46:58.154,2026-10-29T11:46:58.214,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:47:24Z +tester-1,2026-10-29T11:47:41.535,2026-10-29T11:47:41.597,61,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:47:55Z +demo-hp,2026-10-29T11:47:28.215,2026-10-29T11:47:28.275,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:47:57Z +demo-hp,2026-10-29T11:47:58.277,2026-10-29T11:47:58.350,73,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:47:57Z +tester-1,2026-10-29T11:48:11.599,2026-10-29T11:48:11.654,55,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:48:28Z +demo-hp,2026-10-29T11:48:28.351,2026-10-29T11:48:28.413,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:48:30Z +tester-1,2026-10-29T11:48:41.654,2026-10-29T11:48:41.717,63,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:49:01Z +demo-hp,2026-10-29T11:48:58.414,2026-10-29T11:48:58.475,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:49:02Z +tester-1,2026-10-29T11:49:11.719,2026-10-29T11:49:11.773,54,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:49:34Z +demo-hp,2026-10-29T11:49:28.477,2026-10-29T11:49:28.537,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:49:36Z +tester-1,2026-10-29T11:49:41.775,2026-10-29T11:49:41.837,63,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:50:07Z +demo-hp,2026-10-29T11:49:58.539,2026-10-29T11:49:58.601,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:50:09Z +tester-1,2026-10-29T11:50:11.838,2026-10-29T11:50:11.890,52,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:50:40Z +tester-1,2026-10-29T11:50:41.891,2026-10-29T11:50:41.957,66,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:50:40Z +demo-hp,2026-10-29T11:50:28.602,2026-10-29T11:50:28.664,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:50:41Z +tester-1,2026-10-29T11:51:11.959,2026-10-29T11:51:12.020,61,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:51:13Z +demo-hp,2026-10-29T11:50:58.665,2026-10-29T11:50:58.723,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:51:14Z +tester-1,2026-10-29T11:51:42.022,2026-10-29T11:51:42.076,54,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:51:46Z +demo-hp,2026-10-29T11:51:28.724,2026-10-29T11:51:28.783,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:51:47Z +tester-1,2026-10-29T11:52:12.077,2026-10-29T11:52:12.131,54,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:52:19Z +demo-hp,2026-10-29T11:51:58.783,2026-10-29T11:51:58.843,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:52:20Z +tester-1,2026-10-29T11:52:42.132,2026-10-29T11:52:42.194,62,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:52:52Z +demo-hp,2026-10-29T11:52:28.844,2026-10-29T11:52:28.901,57,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:52:53Z +tester-1,2026-10-29T11:53:12.196,2026-10-29T11:53:12.249,53,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:53:24Z +demo-hp,2026-10-29T11:52:58.902,2026-10-29T11:52:58.960,57,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:53:26Z +tester-1,2026-10-29T11:53:42.250,2026-10-29T11:53:42.321,72,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:53:57Z +demo-hp,2026-10-29T11:53:28.961,2026-10-29T11:53:29.019,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:53:59Z +demo-hp,2026-10-29T11:53:59.020,2026-10-29T11:53:59.089,69,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:53:59Z +tester-1,2026-10-29T11:54:12.323,2026-10-29T11:54:12.375,52,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:54:30Z +demo-hp,2026-10-29T11:54:29.090,2026-10-29T11:54:29.149,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:54:31Z +tester-1,2026-10-29T11:54:42.376,2026-10-29T11:54:42.428,52,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:55:02Z +demo-hp,2026-10-29T11:54:59.149,2026-10-29T11:54:59.208,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:55:03Z +tester-1,2026-10-29T11:55:12.429,2026-10-29T11:55:12.490,61,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:55:35Z +demo-hp,2026-10-29T11:55:29.209,2026-10-29T11:55:29.270,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:55:36Z +tester-1,2026-10-29T11:55:42.491,2026-10-29T11:55:42.550,59,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:56:07Z +demo-hp,2026-10-29T11:55:59.270,2026-10-29T11:55:59.336,66,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:56:08Z +tester-1,2026-10-29T11:56:12.551,2026-10-29T11:56:12.608,57,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:56:39Z +demo-hp,2026-10-29T11:56:29.338,2026-10-29T11:56:29.396,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:56:40Z +tester-1,2026-10-29T11:56:42.610,2026-10-29T11:56:42.675,66,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:57:11Z +tester-1,2026-10-29T11:57:12.677,2026-10-29T11:57:12.776,99,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:57:11Z +demo-hp,2026-10-29T11:56:59.397,2026-10-29T11:56:59.464,67,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:57:13Z +tester-1,2026-10-29T11:57:42.777,2026-10-29T11:57:42.829,52,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:57:44Z +demo-hp,2026-10-29T11:57:29.465,2026-10-29T11:57:29.525,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:57:45Z +tester-1,2026-10-29T11:58:12.830,2026-10-29T11:58:12.889,59,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:58:17Z +demo-hp,2026-10-29T11:57:59.526,2026-10-29T11:57:59.578,52,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:58:18Z +tester-1,2026-10-29T11:58:42.891,2026-10-29T11:58:42.949,57,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:58:49Z +demo-hp,2026-10-29T11:58:29.579,2026-10-29T11:58:29.651,72,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:58:51Z +tester-1,2026-10-29T11:59:12.950,2026-10-29T11:59:13.007,56,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:59:22Z +demo-hp,2026-10-29T11:58:59.652,2026-10-29T11:58:59.712,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:59:24Z +tester-1,2026-10-29T11:59:43.008,2026-10-29T11:59:43.068,60,0,healthy,0,2026-10-29T11:03:01,2026-10-29T11:59:55Z +demo-hp,2026-10-29T11:59:29.713,2026-10-29T11:59:29.771,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T11:59:57Z +tester-1,2026-10-29T12:00:13.070,2026-10-29T12:00:13.122,52,0,healthy,0,2026-10-29T11:03:01,2026-10-29T12:00:28Z +demo-hp,2026-10-29T11:59:59.773,2026-10-29T11:59:59.834,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:00:30Z +demo-hp,2026-10-29T12:00:29.835,2026-10-29T12:00:29.908,73,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:00:30Z +tester-1,2026-10-29T12:00:43.123,2026-10-29T12:00:43.195,72,0,healthy,0,2026-10-29T11:03:01,2026-10-29T12:01:01Z +demo-hp,2026-10-29T12:00:59.910,2026-10-29T12:00:59.977,67,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:01:02Z +tester-1,2026-10-29T12:01:11.210,2026-10-29T12:01:11.262,52,0,unhealthy,0,2026-10-29T12:01:06,2026-10-29T12:01:34Z +demo-hp,2026-10-29T12:01:29.978,2026-10-29T12:01:30.048,70,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:01:35Z +demo-hp,2026-10-29T12:02:00.049,2026-10-29T12:02:00.109,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:02:08Z +tester-1,2026-10-29T12:02:09.626,2026-10-29T12:02:09.684,58,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:02:40Z +tester-1,2026-10-29T12:02:39.685,2026-10-29T12:02:39.735,51,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:02:40Z +demo-hp,2026-10-29T12:02:30.110,2026-10-29T12:02:30.171,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:02:41Z +tester-1,2026-10-29T12:03:09.737,2026-10-29T12:03:09.796,59,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:03:12Z +demo-hp,2026-10-29T12:03:00.172,2026-10-29T12:03:00.232,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:03:14Z +tester-1,2026-10-29T12:03:39.797,2026-10-29T12:03:39.853,56,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:03:45Z +demo-hp,2026-10-29T12:03:30.234,2026-10-29T12:03:30.294,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:03:47Z +tester-1,2026-10-29T12:04:09.854,2026-10-29T12:04:09.910,56,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:04:18Z +demo-hp,2026-10-29T12:04:00.294,2026-10-29T12:04:00.352,57,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:04:20Z +tester-1,2026-10-29T12:04:39.911,2026-10-29T12:04:39.967,57,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:04:51Z +demo-hp,2026-10-29T12:04:30.352,2026-10-29T12:04:30.411,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:04:53Z +tester-1,2026-10-29T12:05:09.968,2026-10-29T12:05:10.027,59,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:05:24Z +demo-hp,2026-10-29T12:05:00.412,2026-10-29T12:05:00.468,56,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:05:25Z +tester-1,2026-10-29T12:05:40.028,2026-10-29T12:05:40.080,52,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:05:57Z +demo-hp,2026-10-29T12:05:30.470,2026-10-29T12:05:30.531,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:05:58Z +tester-1,2026-10-29T12:06:10.081,2026-10-29T12:06:10.133,52,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:06:30Z +demo-hp,2026-10-29T12:06:00.532,2026-10-29T12:06:00.591,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:06:31Z +demo-hp,2026-10-29T12:06:30.592,2026-10-29T12:06:30.660,68,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:06:31Z +tester-1,2026-10-29T12:06:40.134,2026-10-29T12:06:40.189,55,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:07:03Z +demo-hp,2026-10-29T12:07:00.662,2026-10-29T12:07:00.729,67,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:07:04Z +tester-1,2026-10-29T12:07:10.191,2026-10-29T12:07:10.244,53,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:07:36Z +demo-hp,2026-10-29T12:07:30.730,2026-10-29T12:07:30.795,65,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:07:37Z +tester-1,2026-10-29T12:07:40.245,2026-10-29T12:07:40.300,55,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:08:09Z +tester-1,2026-10-29T12:08:10.302,2026-10-29T12:08:10.366,64,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:08:09Z +demo-hp,2026-10-29T12:08:00.796,2026-10-29T12:08:00.859,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:08:10Z +tester-1,2026-10-29T12:08:40.367,2026-10-29T12:08:40.425,57,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:08:42Z +demo-hp,2026-10-29T12:08:30.861,2026-10-29T12:08:30.930,70,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:08:43Z +tester-1,2026-10-29T12:09:10.426,2026-10-29T12:09:10.477,51,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:09:15Z +demo-hp,2026-10-29T12:09:00.932,2026-10-29T12:09:00.994,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:09:16Z +tester-1,2026-10-29T12:09:40.479,2026-10-29T12:09:40.532,53,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:09:48Z +demo-hp,2026-10-29T12:09:30.996,2026-10-29T12:09:31.057,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:09:49Z +tester-1,2026-10-29T12:10:10.533,2026-10-29T12:10:10.588,55,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:10:21Z +demo-hp,2026-10-29T12:10:01.058,2026-10-29T12:10:01.118,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:10:22Z +tester-1,2026-10-29T12:10:40.589,2026-10-29T12:10:40.642,53,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:10:54Z +demo-hp,2026-10-29T12:10:31.120,2026-10-29T12:10:31.180,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:10:55Z +tester-1,2026-10-29T12:11:10.643,2026-10-29T12:11:10.694,51,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:11:26Z +demo-hp,2026-10-29T12:11:01.182,2026-10-29T12:11:01.243,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:11:28Z +tester-1,2026-10-29T12:11:40.695,2026-10-29T12:11:40.746,51,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:11:59Z +demo-hp,2026-10-29T12:11:31.244,2026-10-29T12:11:31.304,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:12:01Z +demo-hp,2026-10-29T12:12:01.306,2026-10-29T12:12:01.367,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:12:01Z +tester-1,2026-10-29T12:12:10.747,2026-10-29T12:12:10.801,55,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:12:32Z +demo-hp,2026-10-29T12:12:31.369,2026-10-29T12:12:31.429,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:12:34Z +tester-1,2026-10-29T12:12:40.802,2026-10-29T12:12:40.873,71,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:13:05Z +demo-hp,2026-10-29T12:13:01.430,2026-10-29T12:13:01.492,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:13:07Z +tester-1,2026-10-29T12:13:10.874,2026-10-29T12:13:10.934,61,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:13:38Z +demo-hp,2026-10-29T12:13:31.493,2026-10-29T12:13:31.553,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:13:40Z +tester-1,2026-10-29T12:13:40.936,2026-10-29T12:13:41.001,65,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:14:11Z +tester-1,2026-10-29T12:14:11.002,2026-10-29T12:14:11.056,54,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:14:11Z +demo-hp,2026-10-29T12:14:01.555,2026-10-29T12:14:01.615,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:14:13Z +tester-1,2026-10-29T12:14:41.057,2026-10-29T12:14:41.115,58,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:14:44Z +demo-hp,2026-10-29T12:14:31.617,2026-10-29T12:14:31.679,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:14:45Z +tester-1,2026-10-29T12:15:11.116,2026-10-29T12:15:11.170,54,0,healthy,0,2026-10-29T12:02:04,2026-10-29T12:15:17Z +demo-hp,2026-10-29T12:15:01.681,2026-10-29T12:15:01.739,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:15:18Z +tester-1,2026-10-29T12:15:47.931,2026-10-29T12:15:47.988,56,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:15:49Z +demo-hp,2026-10-29T12:15:31.740,2026-10-29T12:15:31.802,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:15:51Z +tester-1,2026-10-29T12:16:17.989,2026-10-29T12:16:18.050,61,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:16:22Z +demo-hp,2026-10-29T12:16:01.803,2026-10-29T12:16:01.869,67,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:16:23Z +tester-1,2026-10-29T12:16:48.052,2026-10-29T12:16:48.115,63,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:16:55Z +demo-hp,2026-10-29T12:16:31.870,2026-10-29T12:16:31.930,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:16:56Z +tester-1,2026-10-29T12:17:18.117,2026-10-29T12:17:18.176,60,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:17:28Z +demo-hp,2026-10-29T12:17:01.932,2026-10-29T12:17:01.992,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:17:29Z +tester-1,2026-10-29T12:17:48.177,2026-10-29T12:17:48.232,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:18:00Z +demo-hp,2026-10-29T12:17:31.994,2026-10-29T12:17:32.057,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:18:01Z +demo-hp,2026-10-29T12:18:02.058,2026-10-29T12:18:02.126,68,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:18:01Z +tester-1,2026-10-29T12:18:18.233,2026-10-29T12:18:18.320,87,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:18:32Z +demo-hp,2026-10-29T12:18:32.127,2026-10-29T12:18:32.187,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:18:34Z +tester-1,2026-10-29T12:18:48.321,2026-10-29T12:18:48.383,62,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:19:05Z +demo-hp,2026-10-29T12:19:02.188,2026-10-29T12:19:02.246,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:19:07Z +tester-1,2026-10-29T12:19:18.384,2026-10-29T12:19:18.439,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:19:38Z +demo-hp,2026-10-29T12:19:32.248,2026-10-29T12:19:32.308,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:19:39Z +tester-1,2026-10-29T12:19:48.440,2026-10-29T12:19:48.499,59,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:20:11Z +demo-hp,2026-10-29T12:20:02.309,2026-10-29T12:20:02.366,57,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:20:12Z +tester-1,2026-10-29T12:20:18.502,2026-10-29T12:20:18.567,65,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:20:44Z +demo-hp,2026-10-29T12:20:32.367,2026-10-29T12:20:32.439,72,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:20:45Z +tester-1,2026-10-29T12:20:48.569,2026-10-29T12:20:48.641,72,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:21:17Z +tester-1,2026-10-29T12:21:18.642,2026-10-29T12:21:18.696,54,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:21:17Z +demo-hp,2026-10-29T12:21:02.440,2026-10-29T12:21:02.511,71,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:21:18Z +tester-1,2026-10-29T12:21:48.697,2026-10-29T12:21:48.757,60,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:21:50Z +demo-hp,2026-10-29T12:21:32.513,2026-10-29T12:21:32.575,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:21:51Z +tester-1,2026-10-29T12:22:18.758,2026-10-29T12:22:18.818,59,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:22:23Z +demo-hp,2026-10-29T12:22:02.577,2026-10-29T12:22:02.638,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:22:24Z +tester-1,2026-10-29T12:22:48.820,2026-10-29T12:22:48.877,58,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:22:56Z +demo-hp,2026-10-29T12:22:32.639,2026-10-29T12:22:32.705,66,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:22:57Z +tester-1,2026-10-29T12:23:18.878,2026-10-29T12:23:18.940,62,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:23:29Z +demo-hp,2026-10-29T12:23:02.705,2026-10-29T12:23:02.768,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:23:30Z +tester-1,2026-10-29T12:23:48.941,2026-10-29T12:23:49.005,63,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:24:02Z +demo-hp,2026-10-29T12:23:32.769,2026-10-29T12:23:32.830,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:24:03Z +demo-hp,2026-10-29T12:24:02.831,2026-10-29T12:24:02.901,70,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:24:03Z +tester-1,2026-10-29T12:24:19.006,2026-10-29T12:24:19.060,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:24:35Z +demo-hp,2026-10-29T12:24:32.903,2026-10-29T12:24:32.963,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:24:36Z +tester-1,2026-10-29T12:24:49.062,2026-10-29T12:24:49.123,61,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:25:08Z +demo-hp,2026-10-29T12:25:02.964,2026-10-29T12:25:03.038,74,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:25:09Z +tester-1,2026-10-29T12:25:19.124,2026-10-29T12:25:19.179,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:25:40Z +demo-hp,2026-10-29T12:25:33.040,2026-10-29T12:25:33.097,57,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:25:42Z +tester-1,2026-10-29T12:25:49.181,2026-10-29T12:25:49.245,64,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:26:13Z +demo-hp,2026-10-29T12:26:03.098,2026-10-29T12:26:03.170,72,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:26:15Z +tester-1,2026-10-29T12:26:19.247,2026-10-29T12:26:19.301,54,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:26:46Z +demo-hp,2026-10-29T12:26:33.171,2026-10-29T12:26:33.233,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:26:48Z +tester-1,2026-10-29T12:26:49.302,2026-10-29T12:26:49.359,57,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:27:19Z +tester-1,2026-10-29T12:27:19.359,2026-10-29T12:27:19.414,54,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:27:19Z +demo-hp,2026-10-29T12:27:03.235,2026-10-29T12:27:03.287,52,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:27:21Z +tester-1,2026-10-29T12:27:49.415,2026-10-29T12:27:49.484,69,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:27:52Z +demo-hp,2026-10-29T12:27:33.287,2026-10-29T12:27:33.347,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:27:53Z +tester-1,2026-10-29T12:28:19.485,2026-10-29T12:28:19.545,60,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:28:25Z +demo-hp,2026-10-29T12:28:03.349,2026-10-29T12:28:03.419,70,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:28:26Z +tester-1,2026-10-29T12:28:49.546,2026-10-29T12:28:49.606,60,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:28:57Z +demo-hp,2026-10-29T12:28:33.420,2026-10-29T12:28:33.480,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:28:58Z +tester-1,2026-10-29T12:29:19.607,2026-10-29T12:29:19.659,52,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:29:30Z +demo-hp,2026-10-29T12:29:03.482,2026-10-29T12:29:03.541,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:29:31Z +tester-1,2026-10-29T12:29:49.660,2026-10-29T12:29:49.721,61,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:30:03Z +demo-hp,2026-10-29T12:29:33.542,2026-10-29T12:29:33.603,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:30:04Z +demo-hp,2026-10-29T12:30:03.604,2026-10-29T12:30:03.670,67,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:30:04Z +tester-1,2026-10-29T12:30:19.722,2026-10-29T12:30:19.773,51,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:30:36Z +demo-hp,2026-10-29T12:30:33.672,2026-10-29T12:30:33.731,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:30:37Z +tester-1,2026-10-29T12:30:49.774,2026-10-29T12:30:49.825,51,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:31:09Z +demo-hp,2026-10-29T12:31:03.732,2026-10-29T12:31:03.805,72,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:31:10Z +tester-1,2026-10-29T12:31:19.826,2026-10-29T12:31:19.881,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:31:41Z +demo-hp,2026-10-29T12:31:33.806,2026-10-29T12:31:33.866,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:31:43Z +tester-1,2026-10-29T12:31:49.882,2026-10-29T12:31:49.940,58,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:32:14Z +demo-hp,2026-10-29T12:32:03.867,2026-10-29T12:32:03.928,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:32:16Z +tester-1,2026-10-29T12:32:19.941,2026-10-29T12:32:19.996,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:32:47Z +demo-hp,2026-10-29T12:32:33.930,2026-10-29T12:32:33.989,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:32:49Z +tester-1,2026-10-29T12:32:49.997,2026-10-29T12:32:50.056,59,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:33:20Z +tester-1,2026-10-29T12:33:20.057,2026-10-29T12:33:20.118,61,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:33:20Z +demo-hp,2026-10-29T12:33:03.990,2026-10-29T12:33:04.061,71,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:33:22Z +tester-1,2026-10-29T12:33:50.120,2026-10-29T12:33:50.185,65,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:33:53Z +demo-hp,2026-10-29T12:33:34.062,2026-10-29T12:33:34.120,57,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:33:55Z +tester-1,2026-10-29T12:34:20.188,2026-10-29T12:34:20.242,54,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:34:26Z +demo-hp,2026-10-29T12:34:04.121,2026-10-29T12:34:04.184,64,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:34:28Z +tester-1,2026-10-29T12:34:50.243,2026-10-29T12:34:50.310,67,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:34:59Z +demo-hp,2026-10-29T12:34:34.185,2026-10-29T12:34:34.244,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:35:01Z +tester-1,2026-10-29T12:35:20.312,2026-10-29T12:35:20.364,53,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:35:32Z +demo-hp,2026-10-29T12:35:04.245,2026-10-29T12:35:04.307,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:35:33Z +demo-hp,2026-10-29T12:35:34.309,2026-10-29T12:35:34.376,67,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:35:33Z +tester-1,2026-10-29T12:35:50.365,2026-10-29T12:35:50.421,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:36:05Z +demo-hp,2026-10-29T12:36:04.377,2026-10-29T12:36:04.447,69,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:36:06Z +tester-1,2026-10-29T12:36:20.422,2026-10-29T12:36:20.482,60,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:36:38Z +demo-hp,2026-10-29T12:36:34.449,2026-10-29T12:36:34.508,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:36:39Z +tester-1,2026-10-29T12:36:50.484,2026-10-29T12:36:50.545,61,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:37:11Z +demo-hp,2026-10-29T12:37:04.509,2026-10-29T12:37:04.569,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:37:12Z +tester-1,2026-10-29T12:37:20.547,2026-10-29T12:37:20.603,57,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:37:44Z +demo-hp,2026-10-29T12:37:34.570,2026-10-29T12:37:34.631,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:37:45Z +tester-1,2026-10-29T12:37:50.605,2026-10-29T12:37:50.656,51,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:38:17Z +demo-hp,2026-10-29T12:38:04.632,2026-10-29T12:38:04.694,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:38:18Z +tester-1,2026-10-29T12:38:20.658,2026-10-29T12:38:20.709,51,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:38:49Z +tester-1,2026-10-29T12:38:50.710,2026-10-29T12:38:50.772,62,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:38:49Z +demo-hp,2026-10-29T12:38:34.695,2026-10-29T12:38:34.756,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:38:51Z +tester-1,2026-10-29T12:39:20.773,2026-10-29T12:39:20.824,51,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:39:22Z +demo-hp,2026-10-29T12:39:04.757,2026-10-29T12:39:04.818,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:39:24Z +tester-1,2026-10-29T12:39:50.826,2026-10-29T12:39:50.881,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:39:55Z +demo-hp,2026-10-29T12:39:34.820,2026-10-29T12:39:34.879,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:39:57Z +tester-1,2026-10-29T12:40:20.883,2026-10-29T12:40:20.938,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:40:28Z +demo-hp,2026-10-29T12:40:04.880,2026-10-29T12:40:04.938,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:40:30Z +tester-1,2026-10-29T12:40:50.939,2026-10-29T12:40:50.991,52,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:41:01Z +demo-hp,2026-10-29T12:40:34.940,2026-10-29T12:40:35.003,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:41:03Z +tester-1,2026-10-29T12:41:20.992,2026-10-29T12:41:21.050,58,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:41:34Z +demo-hp,2026-10-29T12:41:05.004,2026-10-29T12:41:05.077,73,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:41:36Z +demo-hp,2026-10-29T12:41:35.078,2026-10-29T12:41:35.147,70,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:41:36Z +tester-1,2026-10-29T12:41:51.051,2026-10-29T12:41:51.128,77,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:42:07Z +demo-hp,2026-10-29T12:42:05.148,2026-10-29T12:42:05.213,64,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:42:09Z +tester-1,2026-10-29T12:42:21.132,2026-10-29T12:42:21.187,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:42:40Z +demo-hp,2026-10-29T12:42:35.214,2026-10-29T12:42:35.275,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:42:41Z +tester-1,2026-10-29T12:42:51.187,2026-10-29T12:42:51.242,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:43:13Z +demo-hp,2026-10-29T12:43:05.276,2026-10-29T12:43:05.334,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:43:14Z +tester-1,2026-10-29T12:43:21.243,2026-10-29T12:43:21.297,54,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:43:46Z +demo-hp,2026-10-29T12:43:35.335,2026-10-29T12:43:35.397,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:43:47Z +tester-1,2026-10-29T12:43:51.298,2026-10-29T12:43:51.349,51,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:44:19Z +demo-hp,2026-10-29T12:44:05.398,2026-10-29T12:44:05.460,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:44:20Z +tester-1,2026-10-29T12:44:21.350,2026-10-29T12:44:21.409,59,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:44:52Z +tester-1,2026-10-29T12:44:51.411,2026-10-29T12:44:51.466,56,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:44:52Z +demo-hp,2026-10-29T12:44:35.462,2026-10-29T12:44:35.522,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:44:53Z +tester-1,2026-10-29T12:45:21.467,2026-10-29T12:45:21.518,51,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:45:25Z +demo-hp,2026-10-29T12:45:05.523,2026-10-29T12:45:05.584,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:45:26Z +tester-1,2026-10-29T12:45:51.520,2026-10-29T12:45:51.571,51,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:45:58Z +demo-hp,2026-10-29T12:45:35.585,2026-10-29T12:45:35.649,64,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:45:59Z +tester-1,2026-10-29T12:46:21.572,2026-10-29T12:46:21.626,54,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:46:31Z +demo-hp,2026-10-29T12:46:05.650,2026-10-29T12:46:05.709,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:46:32Z +tester-1,2026-10-29T12:46:51.627,2026-10-29T12:46:51.687,60,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:47:03Z +demo-hp,2026-10-29T12:46:35.710,2026-10-29T12:46:35.770,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:47:05Z +demo-hp,2026-10-29T12:47:05.772,2026-10-29T12:47:05.841,69,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:47:05Z +tester-1,2026-10-29T12:47:21.688,2026-10-29T12:47:21.745,57,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:47:36Z +demo-hp,2026-10-29T12:47:35.842,2026-10-29T12:47:35.902,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:47:38Z +tester-1,2026-10-29T12:47:51.746,2026-10-29T12:47:51.797,51,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:48:09Z +demo-hp,2026-10-29T12:48:05.903,2026-10-29T12:48:05.982,79,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:48:11Z +tester-1,2026-10-29T12:48:21.798,2026-10-29T12:48:21.849,51,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:48:42Z +demo-hp,2026-10-29T12:48:35.983,2026-10-29T12:48:36.046,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:48:43Z +tester-1,2026-10-29T12:48:51.850,2026-10-29T12:48:51.894,44,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:49:15Z +demo-hp,2026-10-29T12:49:06.047,2026-10-29T12:49:06.110,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:49:16Z +tester-1,2026-10-29T12:49:21.895,2026-10-29T12:49:21.976,81,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:49:48Z +demo-hp,2026-10-29T12:49:36.111,2026-10-29T12:49:36.175,64,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:49:49Z +tester-1,2026-10-29T12:49:51.977,2026-10-29T12:49:52.037,59,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:50:20Z +demo-hp,2026-10-29T12:50:06.176,2026-10-29T12:50:06.237,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:50:21Z +tester-1,2026-10-29T12:50:22.038,2026-10-29T12:50:22.094,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:50:53Z +tester-1,2026-10-29T12:50:52.095,2026-10-29T12:50:52.154,58,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:50:53Z +demo-hp,2026-10-29T12:50:36.238,2026-10-29T12:50:36.307,69,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:50:54Z +tester-1,2026-10-29T12:51:22.155,2026-10-29T12:51:22.206,51,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:51:25Z +demo-hp,2026-10-29T12:51:06.308,2026-10-29T12:51:06.371,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:51:26Z +tester-1,2026-10-29T12:51:52.207,2026-10-29T12:51:52.272,65,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:51:58Z +demo-hp,2026-10-29T12:51:36.372,2026-10-29T12:51:36.431,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:51:59Z +tester-1,2026-10-29T12:52:22.273,2026-10-29T12:52:22.332,59,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:52:30Z +demo-hp,2026-10-29T12:52:06.433,2026-10-29T12:52:06.493,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:52:31Z +tester-1,2026-10-29T12:52:52.333,2026-10-29T12:52:52.396,62,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:53:02Z +demo-hp,2026-10-29T12:52:36.495,2026-10-29T12:52:36.552,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:53:04Z +tester-1,2026-10-29T12:53:22.397,2026-10-29T12:53:22.448,51,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:53:35Z +demo-hp,2026-10-29T12:53:06.553,2026-10-29T12:53:06.629,76,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:53:36Z +demo-hp,2026-10-29T12:53:36.631,2026-10-29T12:53:36.699,68,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:53:36Z +tester-1,2026-10-29T12:53:52.449,2026-10-29T12:53:52.505,56,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:54:08Z +demo-hp,2026-10-29T12:54:06.700,2026-10-29T12:54:06.759,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:54:09Z +tester-1,2026-10-29T12:54:22.506,2026-10-29T12:54:22.562,56,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:54:40Z +demo-hp,2026-10-29T12:54:36.761,2026-10-29T12:54:36.829,69,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:54:41Z +tester-1,2026-10-29T12:54:52.563,2026-10-29T12:54:52.618,54,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:55:12Z +demo-hp,2026-10-29T12:55:06.831,2026-10-29T12:55:06.892,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:55:14Z +tester-1,2026-10-29T12:55:22.619,2026-10-29T12:55:22.671,52,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:55:45Z +demo-hp,2026-10-29T12:55:36.893,2026-10-29T12:55:36.954,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:55:46Z +tester-1,2026-10-29T12:55:52.672,2026-10-29T12:55:52.734,62,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:56:18Z +demo-hp,2026-10-29T12:56:06.956,2026-10-29T12:56:07.019,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:56:19Z +tester-1,2026-10-29T12:56:22.736,2026-10-29T12:56:22.788,52,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:56:50Z +demo-hp,2026-10-29T12:56:37.020,2026-10-29T12:56:37.079,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:56:52Z +tester-1,2026-10-29T12:56:52.789,2026-10-29T12:56:52.854,65,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:57:23Z +tester-1,2026-10-29T12:57:22.856,2026-10-29T12:57:22.918,62,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:57:23Z +demo-hp,2026-10-29T12:57:07.080,2026-10-29T12:57:07.143,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:57:24Z +tester-1,2026-10-29T12:57:52.918,2026-10-29T12:57:52.976,57,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:57:55Z +demo-hp,2026-10-29T12:57:37.144,2026-10-29T12:57:37.205,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:57:57Z +tester-1,2026-10-29T12:58:22.977,2026-10-29T12:58:23.033,56,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:58:28Z +demo-hp,2026-10-29T12:58:07.206,2026-10-29T12:58:07.265,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:58:29Z +tester-1,2026-10-29T12:58:53.035,2026-10-29T12:58:53.088,54,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:59:00Z +demo-hp,2026-10-29T12:58:37.267,2026-10-29T12:58:37.325,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:59:01Z +tester-1,2026-10-29T12:59:23.090,2026-10-29T12:59:23.144,54,0,healthy,0,2026-10-29T12:15:42,2026-10-29T12:59:33Z +demo-hp,2026-10-29T12:59:07.327,2026-10-29T12:59:07.383,56,0,healthy,0,2026-10-29T08:58:01,2026-10-29T12:59:34Z +tester-1,2026-10-29T12:59:53.145,2026-10-29T12:59:53.207,63,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:00:06Z +demo-hp,2026-10-29T12:59:37.385,2026-10-29T12:59:37.454,69,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:00:07Z +demo-hp,2026-10-29T13:00:07.455,2026-10-29T13:00:07.523,68,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:00:07Z +tester-1,2026-10-29T13:00:23.209,2026-10-29T13:00:23.262,53,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:00:38Z +demo-hp,2026-10-29T13:00:37.524,2026-10-29T13:00:37.582,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:00:39Z +tester-1,2026-10-29T13:00:53.263,2026-10-29T13:00:53.328,65,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:01:10Z +demo-hp,2026-10-29T13:01:07.584,2026-10-29T13:01:07.652,68,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:01:12Z +tester-1,2026-10-29T13:01:23.329,2026-10-29T13:01:23.381,52,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:01:43Z +demo-hp,2026-10-29T13:01:37.653,2026-10-29T13:01:37.718,65,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:01:45Z +tester-1,2026-10-29T13:01:53.382,2026-10-29T13:01:53.437,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:02:16Z +demo-hp,2026-10-29T13:02:07.720,2026-10-29T13:02:07.780,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:02:17Z +tester-1,2026-10-29T13:02:23.439,2026-10-29T13:02:23.490,51,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:02:49Z +demo-hp,2026-10-29T13:02:37.781,2026-10-29T13:02:37.849,68,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:02:50Z +tester-1,2026-10-29T13:02:53.492,2026-10-29T13:02:53.547,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:03:21Z +demo-hp,2026-10-29T13:03:07.850,2026-10-29T13:03:07.909,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:03:22Z +tester-1,2026-10-29T13:03:23.548,2026-10-29T13:03:23.609,61,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:03:54Z +tester-1,2026-10-29T13:03:53.610,2026-10-29T13:03:53.665,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:03:54Z +demo-hp,2026-10-29T13:03:37.911,2026-10-29T13:03:37.978,67,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:03:55Z +tester-1,2026-10-29T13:04:23.667,2026-10-29T13:04:23.720,53,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:04:26Z +demo-hp,2026-10-29T13:04:07.979,2026-10-29T13:04:08.039,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:04:28Z +tester-1,2026-10-29T13:04:53.721,2026-10-29T13:04:53.799,77,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:04:59Z +demo-hp,2026-10-29T13:04:38.040,2026-10-29T13:04:38.100,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:05:00Z +tester-1,2026-10-29T13:05:23.799,2026-10-29T13:05:23.851,52,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:05:32Z +demo-hp,2026-10-29T13:05:08.102,2026-10-29T13:05:08.168,66,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:05:33Z +tester-1,2026-10-29T13:05:53.852,2026-10-29T13:05:53.906,54,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:06:04Z +demo-hp,2026-10-29T13:05:38.169,2026-10-29T13:05:38.229,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:06:06Z +tester-1,2026-10-29T13:06:23.907,2026-10-29T13:06:23.960,52,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:06:37Z +demo-hp,2026-10-29T13:06:08.229,2026-10-29T13:06:08.290,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:06:38Z +demo-hp,2026-10-29T13:06:38.291,2026-10-29T13:06:38.360,69,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:06:38Z +tester-1,2026-10-29T13:06:53.961,2026-10-29T13:06:54.012,52,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:07:10Z +demo-hp,2026-10-29T13:07:08.361,2026-10-29T13:07:08.420,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:07:11Z +tester-1,2026-10-29T13:07:24.014,2026-10-29T13:07:24.070,56,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:07:43Z +demo-hp,2026-10-29T13:07:38.421,2026-10-29T13:07:38.481,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:07:44Z +tester-1,2026-10-29T13:07:54.071,2026-10-29T13:07:54.125,54,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:08:15Z +demo-hp,2026-10-29T13:08:08.482,2026-10-29T13:08:08.544,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:08:17Z +tester-1,2026-10-29T13:08:24.126,2026-10-29T13:08:24.182,57,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:08:48Z +demo-hp,2026-10-29T13:08:38.547,2026-10-29T13:08:38.606,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:08:50Z +tester-1,2026-10-29T13:08:54.185,2026-10-29T13:08:54.244,60,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:09:21Z +demo-hp,2026-10-29T13:09:08.607,2026-10-29T13:09:08.665,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:09:23Z +tester-1,2026-10-29T13:09:24.246,2026-10-29T13:09:24.312,66,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:09:54Z +tester-1,2026-10-29T13:09:54.314,2026-10-29T13:09:54.368,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:09:54Z +demo-hp,2026-10-29T13:09:38.666,2026-10-29T13:09:38.724,57,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:09:56Z +tester-1,2026-10-29T13:10:24.369,2026-10-29T13:10:24.429,60,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:10:27Z +demo-hp,2026-10-29T13:10:08.725,2026-10-29T13:10:08.795,70,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:10:28Z +tester-1,2026-10-29T13:10:54.431,2026-10-29T13:10:54.482,52,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:11:00Z +demo-hp,2026-10-29T13:10:38.796,2026-10-29T13:10:38.860,64,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:11:01Z +tester-1,2026-10-29T13:11:24.483,2026-10-29T13:11:24.536,52,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:11:33Z +demo-hp,2026-10-29T13:11:08.862,2026-10-29T13:11:08.922,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:11:34Z +tester-1,2026-10-29T13:11:54.536,2026-10-29T13:11:54.591,54,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:12:05Z +demo-hp,2026-10-29T13:11:38.923,2026-10-29T13:11:38.989,65,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:12:07Z +tester-1,2026-10-29T13:12:24.592,2026-10-29T13:12:24.648,55,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:12:38Z +demo-hp,2026-10-29T13:12:08.989,2026-10-29T13:12:09.047,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:12:40Z +demo-hp,2026-10-29T13:12:39.049,2026-10-29T13:12:39.118,69,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:12:40Z +tester-1,2026-10-29T13:12:54.648,2026-10-29T13:12:54.711,62,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:13:11Z +demo-hp,2026-10-29T13:13:09.120,2026-10-29T13:13:09.178,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:13:13Z +tester-1,2026-10-29T13:13:24.712,2026-10-29T13:13:24.773,61,0,healthy,0,2026-10-29T12:15:42,2026-10-29T13:13:44Z +demo-hp,2026-10-29T13:13:39.179,2026-10-29T13:13:39.238,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:13:45Z +demo-hp,2026-10-29T13:14:09.239,2026-10-29T13:14:09.311,71,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:14:18Z +tester-1,2026-10-29T13:14:20.055,2026-10-29T13:14:20.114,59,0,healthy,0,2026-10-29T13:14:14,2026-10-29T13:14:49Z +tester-1,2026-10-29T13:14:50.115,2026-10-29T13:14:50.199,84,0,healthy,0,2026-10-29T13:14:14,2026-10-29T13:14:49Z +demo-hp,2026-10-29T13:14:39.313,2026-10-29T13:14:39.376,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:14:50Z +tester-1,2026-10-29T13:15:20.203,2026-10-29T13:15:20.257,55,0,healthy,0,2026-10-29T13:14:14,2026-10-29T13:15:22Z +demo-hp,2026-10-29T13:15:09.377,2026-10-29T13:15:09.437,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:15:23Z +tester-1,2026-10-29T13:15:50.259,2026-10-29T13:15:50.340,81,0,healthy,0,2026-10-29T13:14:14,2026-10-29T13:15:55Z +demo-hp,2026-10-29T13:15:39.438,2026-10-29T13:15:39.497,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:15:56Z +tester-1,2026-10-29T13:16:20.342,2026-10-29T13:16:20.402,60,0,healthy,0,2026-10-29T13:14:14,2026-10-29T13:16:27Z +demo-hp,2026-10-29T13:16:09.497,2026-10-29T13:16:09.567,70,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:16:28Z +tester-1,2026-10-29T13:16:50.403,2026-10-29T13:16:50.567,163,0,healthy,0,2026-10-29T13:14:14,2026-10-29T13:16:59Z +demo-hp,2026-10-29T13:16:39.568,2026-10-29T13:16:39.642,74,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:17:01Z +tester-1,2026-10-29T13:17:20.568,2026-10-29T13:17:20.630,62,0,healthy,0,2026-10-29T13:14:14,2026-10-29T13:17:32Z +demo-hp,2026-10-29T13:17:09.643,2026-10-29T13:17:09.711,68,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:17:34Z +tester-1,2026-10-29T13:17:50.632,2026-10-29T13:17:50.685,53,0,healthy,0,2026-10-29T13:14:14,2026-10-29T13:18:05Z +demo-hp,2026-10-29T13:17:39.712,2026-10-29T13:17:39.773,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:18:07Z +tester-1,2026-10-29T13:18:20.686,2026-10-29T13:18:20.738,52,0,healthy,0,2026-10-29T13:14:14,2026-10-29T13:18:38Z +demo-hp,2026-10-29T13:18:09.774,2026-10-29T13:18:09.835,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:18:39Z +demo-hp,2026-10-29T13:18:39.837,2026-10-29T13:18:39.901,64,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:18:39Z +tester-1,2026-10-29T13:18:50.739,2026-10-29T13:18:50.799,60,0,healthy,0,2026-10-29T13:14:14,2026-10-29T13:19:10Z +demo-hp,2026-10-29T13:19:09.902,2026-10-29T13:19:09.959,57,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:19:12Z +tester-1,2026-10-29T13:19:20.800,2026-10-29T13:19:20.852,52,0,healthy,0,2026-10-29T13:14:14,2026-10-29T13:19:43Z +demo-hp,2026-10-29T13:19:39.961,2026-10-29T13:19:40.020,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:19:44Z +tester-1,2026-10-29T13:19:50.853,2026-10-29T13:19:50.912,59,0,healthy,0,2026-10-29T13:14:14,2026-10-29T13:20:15Z +demo-hp,2026-10-29T13:20:10.021,2026-10-29T13:20:10.084,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:20:17Z +demo-hp,2026-10-29T13:20:40.085,2026-10-29T13:20:40.154,69,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:20:50Z +tester-1,2026-10-29T13:20:50.381,2026-10-29T13:20:50.449,68,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:21:21Z +tester-1,2026-10-29T13:21:20.451,2026-10-29T13:21:20.504,53,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:21:21Z +demo-hp,2026-10-29T13:21:10.156,2026-10-29T13:21:10.215,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:21:22Z +tester-1,2026-10-29T13:21:50.506,2026-10-29T13:21:50.557,51,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:21:53Z +demo-hp,2026-10-29T13:21:40.216,2026-10-29T13:21:40.274,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:21:55Z +tester-1,2026-10-29T13:22:20.558,2026-10-29T13:22:20.612,54,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:22:27Z +demo-hp,2026-10-29T13:22:10.275,2026-10-29T13:22:10.332,57,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:22:28Z +tester-1,2026-10-29T13:22:50.613,2026-10-29T13:22:50.666,53,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:22:59Z +demo-hp,2026-10-29T13:22:40.334,2026-10-29T13:22:40.391,57,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:23:01Z +tester-1,2026-10-29T13:23:20.667,2026-10-29T13:23:20.720,53,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:23:32Z +demo-hp,2026-10-29T13:23:10.392,2026-10-29T13:23:10.449,57,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:23:34Z +tester-1,2026-10-29T13:23:50.721,2026-10-29T13:23:50.777,56,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:24:05Z +demo-hp,2026-10-29T13:23:40.450,2026-10-29T13:23:40.512,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:24:07Z +tester-1,2026-10-29T13:24:20.778,2026-10-29T13:24:20.830,52,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:24:38Z +demo-hp,2026-10-29T13:24:10.513,2026-10-29T13:24:10.578,64,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:24:40Z +demo-hp,2026-10-29T13:24:40.579,2026-10-29T13:24:40.649,70,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:24:40Z +tester-1,2026-10-29T13:24:50.831,2026-10-29T13:24:50.883,52,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:25:11Z +demo-hp,2026-10-29T13:25:10.650,2026-10-29T13:25:10.712,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:25:13Z +tester-1,2026-10-29T13:25:20.883,2026-10-29T13:25:20.935,52,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:25:44Z +demo-hp,2026-10-29T13:25:40.714,2026-10-29T13:25:40.774,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:25:45Z +tester-1,2026-10-29T13:25:50.936,2026-10-29T13:25:50.988,51,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:26:17Z +demo-hp,2026-10-29T13:26:10.775,2026-10-29T13:26:10.837,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:26:19Z +tester-1,2026-10-29T13:26:20.989,2026-10-29T13:26:21.044,56,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:26:50Z +tester-1,2026-10-29T13:26:51.045,2026-10-29T13:26:51.111,66,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:26:50Z +demo-hp,2026-10-29T13:26:40.838,2026-10-29T13:26:40.896,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:26:51Z +tester-1,2026-10-29T13:27:21.112,2026-10-29T13:27:21.165,53,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:27:23Z +demo-hp,2026-10-29T13:27:10.897,2026-10-29T13:27:10.959,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:27:24Z +tester-1,2026-10-29T13:27:51.166,2026-10-29T13:27:51.220,54,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:27:56Z +demo-hp,2026-10-29T13:27:40.961,2026-10-29T13:27:41.019,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:27:57Z +tester-1,2026-10-29T13:28:21.222,2026-10-29T13:28:21.275,53,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:28:29Z +demo-hp,2026-10-29T13:28:11.020,2026-10-29T13:28:11.086,67,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:28:30Z +tester-1,2026-10-29T13:28:51.276,2026-10-29T13:28:51.330,55,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:29:02Z +demo-hp,2026-10-29T13:28:41.088,2026-10-29T13:28:41.148,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:29:03Z +tester-1,2026-10-29T13:29:21.331,2026-10-29T13:29:21.384,53,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:29:35Z +demo-hp,2026-10-29T13:29:11.149,2026-10-29T13:29:11.210,61,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:29:36Z +tester-1,2026-10-29T13:29:51.385,2026-10-29T13:29:51.439,53,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:30:08Z +demo-hp,2026-10-29T13:29:41.211,2026-10-29T13:29:41.271,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:30:09Z +tester-1,2026-10-29T13:30:21.440,2026-10-29T13:30:21.499,59,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:30:41Z +demo-hp,2026-10-29T13:30:11.272,2026-10-29T13:30:11.330,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:30:42Z +demo-hp,2026-10-29T13:30:41.330,2026-10-29T13:30:41.399,68,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:30:42Z +tester-1,2026-10-29T13:30:51.501,2026-10-29T13:30:51.556,55,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:31:13Z +demo-hp,2026-10-29T13:31:11.400,2026-10-29T13:31:11.461,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:31:15Z +tester-1,2026-10-29T13:31:21.556,2026-10-29T13:31:21.611,55,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:31:46Z +demo-hp,2026-10-29T13:31:41.463,2026-10-29T13:31:41.531,68,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:31:48Z +tester-1,2026-10-29T13:31:51.613,2026-10-29T13:31:51.674,62,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:32:19Z +demo-hp,2026-10-29T13:32:11.533,2026-10-29T13:32:11.595,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:32:21Z +tester-1,2026-10-29T13:32:21.675,2026-10-29T13:32:21.736,61,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:32:52Z +tester-1,2026-10-29T13:32:51.737,2026-10-29T13:32:51.794,57,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:32:52Z +demo-hp,2026-10-29T13:32:41.596,2026-10-29T13:32:41.658,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:32:54Z +tester-1,2026-10-29T13:33:21.795,2026-10-29T13:33:21.853,58,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:33:25Z +demo-hp,2026-10-29T13:33:11.659,2026-10-29T13:33:11.718,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:33:27Z +tester-1,2026-10-29T13:33:51.854,2026-10-29T13:33:51.907,53,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:33:58Z +demo-hp,2026-10-29T13:33:41.719,2026-10-29T13:33:41.782,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:34:00Z +tester-1,2026-10-29T13:34:21.909,2026-10-29T13:34:21.975,67,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:34:31Z +demo-hp,2026-10-29T13:34:11.783,2026-10-29T13:34:11.845,62,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:34:32Z +tester-1,2026-10-29T13:34:51.977,2026-10-29T13:34:52.041,64,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:35:04Z +demo-hp,2026-10-29T13:34:41.846,2026-10-29T13:34:41.907,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:35:05Z +tester-1,2026-10-29T13:35:22.042,2026-10-29T13:35:22.096,53,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:35:37Z +demo-hp,2026-10-29T13:35:11.908,2026-10-29T13:35:11.967,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:35:38Z +tester-1,2026-10-29T13:35:52.097,2026-10-29T13:35:52.172,75,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:36:10Z +demo-hp,2026-10-29T13:35:41.968,2026-10-29T13:35:42.028,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:36:11Z +demo-hp,2026-10-29T13:36:12.030,2026-10-29T13:36:12.096,66,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:36:11Z +tester-1,2026-10-29T13:36:22.173,2026-10-29T13:36:22.239,65,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:36:43Z +demo-hp,2026-10-29T13:36:42.098,2026-10-29T13:36:42.157,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:36:44Z +tester-1,2026-10-29T13:36:52.240,2026-10-29T13:36:52.293,53,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:37:15Z +demo-hp,2026-10-29T13:37:12.159,2026-10-29T13:37:12.219,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:37:17Z +tester-1,2026-10-29T13:37:22.294,2026-10-29T13:37:22.353,59,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:37:48Z +demo-hp,2026-10-29T13:37:42.219,2026-10-29T13:37:42.278,58,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:37:50Z +tester-1,2026-10-29T13:37:52.354,2026-10-29T13:37:52.406,51,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:38:21Z +tester-1,2026-10-29T13:38:22.407,2026-10-29T13:38:22.481,74,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:38:21Z +demo-hp,2026-10-29T13:38:12.279,2026-10-29T13:38:12.340,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:38:22Z +tester-1,2026-10-29T13:38:52.482,2026-10-29T13:38:52.536,54,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:38:53Z +demo-hp,2026-10-29T13:38:42.341,2026-10-29T13:38:42.405,64,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:38:55Z +tester-1,2026-10-29T13:39:22.537,2026-10-29T13:39:22.597,60,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:39:26Z +demo-hp,2026-10-29T13:39:12.406,2026-10-29T13:39:12.469,63,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:39:28Z +tester-1,2026-10-29T13:39:52.598,2026-10-29T13:39:52.652,54,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:39:59Z +demo-hp,2026-10-29T13:39:42.470,2026-10-29T13:39:42.528,59,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:40:01Z +tester-1,2026-10-29T13:40:22.654,2026-10-29T13:40:22.709,55,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:40:32Z +demo-hp,2026-10-29T13:40:12.529,2026-10-29T13:40:12.589,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:40:34Z +tester-1,2026-10-29T13:40:52.709,2026-10-29T13:40:52.761,52,0,healthy,0,2026-10-29T13:20:45,2026-10-29T13:41:05Z +demo-hp,2026-10-29T13:40:42.590,2026-10-29T13:40:42.650,60,0,healthy,0,2026-10-29T08:58:01,2026-10-29T13:41:07Z +tester-1,2026-10-29T13:41:17.404,2026-10-29T13:41:17.467,63,0,healthy,0,2026-10-29T13:41:12,2026-10-29T13:41:38Z +demo-hp,2026-10-29T13:41:18.157,2026-10-29T13:41:18.225,68,0,healthy,0,2026-10-29T13:41:13,2026-10-29T13:41:39Z diff --git a/documentation/audits/release-2026-10-29b/p2-health/healthlog.py b/documentation/audits/release-2026-10-29b/p2-health/healthlog.py new file mode 100644 index 00000000..45738824 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p2-health/healthlog.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 +"""healthlog.py — R-1344 spike (Part 2, 2026-10-29): every 30 s, read each test box controller's Docker +health record (`docker inspect -f '{{json .State.Health}}' felhom-controller`, the probe Docker runs every 30 s: +`curl -f localhost:8080/api/health`, timeout 5 s) and append every NEW probe (keyed by its start time) to a CSV: +box, start, end, duration_ms, exit_code, status, failing_streak, container_started. Read-only; nothing on a box changes.""" +import csv, json, os, subprocess, sys, time +from datetime import datetime, timezone + +UNTIL = datetime.fromisoformat(sys.argv[1].replace("Z", "+00:00")) +OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "health.csv") +BOXES = {"tester-1": ["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=10", "root@192.168.0.154"], + "demo-hp": ["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=10", "hp"]} +seen = {b: set() for b in BOXES} +new = not os.path.exists(OUT) +f = open(OUT, "a", newline="") +w = csv.writer(f) +if new: + w.writerow(["box", "start", "end", "duration_ms", "exit_code", "status", "failing_streak", "container_started", "read_at"]) + + +def ts(s): + # Docker: 2026-10-29T11:11:37.145382734Z (nanoseconds) → microseconds for fromisoformat + s = s.rstrip("Z") + if "." in s: + a, b = s.split(".", 1) + s = a + "." + (b + "000000")[:6] + return datetime.fromisoformat(s).replace(tzinfo=timezone.utc) + + +while datetime.now(timezone.utc) < UNTIL: + for box, ssh in BOXES.items(): + now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + try: + r = subprocess.run(ssh + ["pct exec 9201 -- docker inspect -f '{{json .State.Health}}|{{.State.StartedAt}}' felhom-controller"], + capture_output=True, text=True, timeout=40) + line = [l for l in r.stdout.splitlines() if l.startswith("{")] + if not line: + w.writerow([box, "", "", "", "", "READ-FAILED", "", "", now]); f.flush(); continue + health, started = line[0].rsplit("|", 1) + h = json.loads(health) + for e in h.get("Log") or []: + if e["Start"] in seen[box]: + continue + seen[box].add(e["Start"]) + d = (ts(e["End"]) - ts(e["Start"])).total_seconds() * 1000 + w.writerow([box, e["Start"][:23], e["End"][:23], round(d), e["ExitCode"], h.get("Status"), h.get("FailingStreak"), started[:19], now]) + f.flush() + except Exception as ex: # a read that fails is a row, never a silent gap + w.writerow([box, "", "", "", "", "READ-ERROR " + type(ex).__name__, "", "", now]); f.flush() + time.sleep(30) diff --git a/documentation/audits/release-2026-10-29b/p3-clear-hold/p3-clear.log b/documentation/audits/release-2026-10-29b/p3-clear-hold/p3-clear.log new file mode 100644 index 00000000..00e2ab65 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p3-clear-hold/p3-clear.log @@ -0,0 +1,19 @@ +13:43:59 === phase clear at 2026-10-29T12:43:59Z +13:44:02 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1 Up 28 minutes (healthy) +13:44:06 [before] api: state=stopped hold='Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.' | file holds: OCI runtime exec failed: exec failed: unable to start container process: exec: "python3": executable file not found in $PATH +['gokapi'] +13:44:06 press: HTTP/1.1 303 See Other | Location: /hosts/tester-1-d70be4#operator-actions +13:49:20 box log: 2026/10/29 12:44:09 [INFO] [opaction] operator action #7 received: clear_hold +2026/10/29 12:44:09 [INFO] [settings] restore hold CLEARED for gokapi +2026/10/29 12:44:09 [WARN] [hold] gokapi: hold CLEARED by the operator via the hub, action #7 — it was unhealthy_stop (set 2026-10-29T12:28:14Z); the app is not started by this +2026/10/29 12:44:09 [INFO] [opaction] operator action #7 clear_hold: done — the hold on gokapi is cleared (it was: stopped by the box for a crash loop (2026-10-29T12:28:14Z) — the household's Start gives it one more try). The app is NOT started by this — the household's Start, or a restore, starts it +2026/10/29 12:44:44 [INFO] [holdreport] gokapi hold cleared — reported to the hub + +13:49:24 [after clear] api: state=stopped hold=None | file holds: OCI runtime exec failed: exec failed: unable to start container process: exec: "python3": executable file not found in $PATH +[] +14:00:55 next settings save: 2026/10/29 13:00:45 [INFO] [settings] Settings saved +14:00:58 [after the next save] api: state=stopped hold=None | file holds: OCI runtime exec failed: exec failed: unable to start container process: exec: "python3": executable file not found in $PATH +[] +14:02:09 hub page: No app is held on this box. | clear_hold gokapi | the hold on gokapi is cleared (it was: stopped by the box for a crash loop (2026-10-29T12:28:14Z) — the household's Start gives it one more try). The app is NOT started by this — the household's Start, or a restore, starts it +14:02:10 hub held-apps log: 0 'no longer held' line(s) +14:02:13 app events since the clear (F2 check): diff --git a/documentation/audits/release-2026-10-29b/p3-clear-hold/p3-hold.log b/documentation/audits/release-2026-10-29b/p3-clear-hold/p3-hold.log new file mode 100644 index 00000000..c2310808 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p3-clear-hold/p3-hold.log @@ -0,0 +1,15 @@ +13:27:53 === phase hold at 2026-10-29T12:27:53Z +13:27:56 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1 Up 12 minutes (healthy) +13:27:56 [1] gokapi already deployed — reusing +13:28:02 compose: 16: image: f0rc3/gokapi:v1.9.6 +22: entrypoint: +13:28:04 Restart → 200 {'ok': True, 'data': {'state': 'restarting'}, 'message': 'Stack gokapi restart requested — state now: restarting'} +13:28:18 [held] api: state=stopped hold='Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.' | file holds: OCI runtime exec failed: exec failed: unable to start container process: exec: "python3": executable file not found in $PATH +['gokapi'] +13:28:21 box log: 2026/10/29 12:28:14 [WARN] [deadapp] gokapi: crash_loop — 7 in 10m0s; STOPPING it (decision 28) +2026/10/29 12:28:14 [INFO] [stacks] Stack gokapi stopped successfully (took 0.1s) +2026/10/29 12:28:14 [WARN] [settings] restore hold SET for gokapi — the app stays stopped until it is cleared +2026/10/29 12:28:14 [WARN] [backup] gokapi is STOPPED by the box: crash_loop (trip 1 within 24h0m0s) — Start gives it one more try (decision 28) +2026/10/29 12:28:14 [INFO] Event pushed: app_stopped_unhealthy (warning) [hu-only] — A(z) gokapi alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod. + +13:28:51 hub page: gokapi | Clear hold | Clear the hold of gokapi (unhealthy_stop, since 2026-10-29 12:28)? The box's controller clears it at once; the household can then start the app. diff --git a/documentation/audits/release-2026-10-29b/p3-clear-hold/p3-teardown.log b/documentation/audits/release-2026-10-29b/p3-clear-hold/p3-teardown.log new file mode 100644 index 00000000..99dc4108 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p3-clear-hold/p3-teardown.log @@ -0,0 +1,9 @@ +14:02:37 === phase teardown at 2026-10-29T13:02:37Z +14:02:40 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1 Up 46 minutes (healthy) +14:02:46 [before remove] api: state=stopped hold=None | file holds: OCI runtime exec failed: exec failed: unable to start container process: exec: "python3": executable file not found in $PATH +[] +14:02:46 [X] stop -> 200 {'ok': True, 'message': 'Stack gokapi stop completed'} +14:03:18 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'gokapi', 'volumes_removed': ['gokapi_gokapi_config', 'gokapi_gokapi_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [] +14:03:26 [X] after remove: deployed=False leftovers='' +14:03:30 [after remove] api: state=not_deployed hold=None | file holds: OCI runtime exec failed: exec failed: unable to start container process: exec: "python3": executable file not found in $PATH +[] diff --git a/documentation/audits/release-2026-10-29b/p3-clear-hold/r1340-clear.log b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1340-clear.log new file mode 100644 index 00000000..cd08c743 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1340-clear.log @@ -0,0 +1,12 @@ +14:18:28 === phase clear at 2026-10-29T13:18:28Z +14:18:31 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1-r1340fault Up 4 minutes (healthy) +14:18:32 hub page: gokapi | Clear the hold of gokapi (restore_mixed, since 2026-10-29 13:17)? The box's controller clears it at once; the household can then start the app. | This hold protects data from a failed restore. If you clear it, the app may start on half-restored data. Check the restore copy first. | clear_hold gokapi | the hold on gokapi is cleared (it was: stopped by the box for a crash loop (2026-10-29T12:28:14Z) — the household's Start gives it one more try). The app is NOT started by this — the household's Start, or a restore, starts it | +14:18:32 press: HTTP/1.1 303 See Other | Location: /hosts/tester-1-d70be4#operator-actions +14:19:32 hub: 2026/10/29 14:18:32 [INFO] operator action #8 clear_hold gokapi requested for tester-1 (host tester-1-d70be4) by access token "cc-hosts" from 10.42.0.1 — the box acts on its next report +2026/10/29 14:18:51 [INFO] operator action #8 clear_hold gokapi for tester-1 (pressed by access token "cc-hosts" from 10.42.0.1): done — the hold on gokapi is cleared (it was: held after an off-site restore failed its database replay (2026-10-29T13:17:07Z) — clear the hold to start it). The app is NOT started by this — the household's Start, or a restore, starts it +14:19:36 box: 2026/10/29 13:18:35 [INFO] [settings] restore hold CLEARED for gokapi +2026/10/29 13:18:35 [WARN] [hold] gokapi: hold CLEARED by the operator via the hub, action #8 — it was restore_mixed (set 2026-10-29T13:17:07Z); the app is not started by this +2026/10/29 13:18:35 [INFO] [opaction] operator action #8 clear_hold: done — the hold on gokapi is cleared (it was: held after an off-site restore failed its database replay (2026-10-29T13:17:07Z) — clear the hold to start it). The app is NOT started by this — the household's Start, or a restore, starts it + +14:19:36 app after: state=stopped hold=None +14:19:39 copy on disk: 20261029T131705Z diff --git a/documentation/audits/release-2026-10-29b/p3-clear-hold/r1340-hold.log b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1340-hold.log new file mode 100644 index 00000000..8fb9e763 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1340-hold.log @@ -0,0 +1,24 @@ +14:16:48 === phase hold at 2026-10-29T13:16:48Z +14:16:51 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1-r1340fault Up 2 minutes (healthy) +14:16:54 volumes: ['gokapi_gokapi_config', 'gokapi_gokapi_data'] +14:16:59 blocker: 97d0f358ac5802285c6d +14:17:03 [R] restoring gokapi from snapshot 'helyi' (of 1 offered) +14:17:03 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +14:17:03 + 0.0s restore (True, None, None) +14:17:07 + 4.1s restore (False, None, None) +14:17:09 [R] after restore: state=stopped hold='a(z) gokapi visszaállítása 2026-10-29 13:17-kor félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás biztonsági okból leállítva marad. Ehhez segítség kell: vedd fel velünk a kapcsolatot' phase=None +14:17:12 restore: state=stopped hold='a(z) gokapi visszaállítása 2026-10-29 13:17-kor félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás biztonsági okból leállítva marad. Ehhez segítség kell: vedd fel velünk a kapcsolatot' +14:17:15 box log: 2026/10/29 13:17:06 [ERROR] [backup] Failed to remove volume gokapi_gokapi_config before the restore: Error response from daemon: remove gokapi_gokapi_config: volume is in use - [97d0f358ac5802285c6dfc2fd33b4f83b54043a85c31a840c4a5b5284b030fe1] — exit status 1 (not written into) +2026/10/29 13:17:07 [ERROR] [faultinject] stop-gokapi: FAULT INJECTED (fail left 0) +2026/10/29 13:17:07 [ERROR] [offbox] gokapi: stopping the app before the put-back FAILED: FAULT INJECTED at stop-gokapi (D63 test build) — nothing is put back; the copy stays at /mnt/sys_drive/felhom-data/backups/restore-undo/gokapi/20261029T131705Z; holding the app +2026/10/29 13:17:07 [WARN] [settings] restore hold SET for gokapi — the app stays stopped until it is cleared +2026/10/29 13:17:07 [ERROR] [offbox] gokapi: the restore stopped half-way after files/volumes/version had moved — HOLDING the app stopped for support (R-893); error was: failed to restore 1 volume(s): [gokapi_gokapi_config] +2026/10/29 13:17:07 [INFO] Event pushed: backup_run_failures (error) [hu-only] — App "gokapi" is HELD STOPPED for support: its restore from its own backup failed, and putting the live data back FAILED — stopping the app before the put-back failed, so nothing was put back: FAULT INJECTED at stop-gokapi (D63 test build). The app may be on a mix of the backup's and the live data. The copy of the live data is kept at /mnt/sys_drive/felhom-data/backups/restore-undo/gokapi/20261029T131705Z. The app will not start from any path until the hold is cleared (the hub's Clear hold button, or on the box: pct exec -- docker exec felhom-controller felhom-controller --clear-restore-hold ). Restore error: failed to restore 1 volume(s): [gokapi_gokapi_config] + +14:17:36 hub events for gokapi: 2026/10/29 14:10:14 [INFO] Event from tester-1: app_oom (warning) — Alkalmazás memóriája elfogyott: gokapi (gokapi) — egy folyamatát a memóriakorlát leállította +2026/10/29 14:13:28 [INFO] Event from tester-1: app_removed (info) — Alkalmazás eltávolítva: gokapi +2026/10/29 14:17:07 [INFO] Event from tester-1: backup_run_failures (error) — App "gokapi" is HELD STOPPED for support: its restore from its own backup failed, and putting the live data back FAILED — stopping the app before the put-back failed, so nothing was put back: FAULT INJECTED at stop-gokapi (D63 test build). The app may be on a mix of the backup's and the live data. The copy of the live data is kept at /mnt/sys_drive/felhom-data/backups/restore-undo/gokapi/20261029T131705Z. The app will not start from any path until the hold is cleared (the hub's Clear hold button, or on the box: pct exec -- docker exec felhom-controller felhom-controller --clear-restore-hold ). Restore error: failed to restore 1 volume(s): [gokapi_gokapi_config] +2026/10/29 14:17:16 [INFO] Event from tester-1: app_held (info) — App gokapi is held: held after an off-site restore failed its database replay (2026-10-29T13:17:07Z) — clear the hold to start it +2026/10/29 14:17:16 [DEBUG] [held-apps] tester-1/gokapi held (restore_mixed since 2026-10-29T13:17:07Z) +14:17:36 hub mail sent: 2026/10/29 14:10:14 [INFO] Operator email sent for tester-1/app_oom +2026/10/29 14:17:07 [INFO] Operator email sent for tester-1/backup_run_failures diff --git a/documentation/audits/release-2026-10-29b/p3-clear-hold/r1341-backup.log b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1341-backup.log new file mode 100644 index 00000000..04a325e0 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1341-backup.log @@ -0,0 +1,13 @@ +14:15:58 === phase backup at 2026-10-29T13:15:58Z +14:16:01 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1-r1340fault Up About a minute (healthy) +14:16:01 backup press → 200 {'ok': True, 'message': 'Mentés elindítva'} +14:16:46 log: 2026/10/29 13:16:15 [INFO] [backup] Volume dump: gokapi/gokapi_gokapi_data → 43.0 KB +2026/10/29 13:16:15 [INFO] [backup] Restarting gokapi after volume dump +2026/10/29 13:16:15 [INFO] [stacks] Starting stack: gokapi +2026/10/29 13:16:16 [INFO] [stacks] Stack gokapi started successfully (took 0.4s) +2026/10/29 13:16:19 [INFO] [stacks] Stack gokapi post-start status: +2026/10/29 13:16:19 [INFO] [stacks] gokapi f0rc3/gokapi:v1.9.6 running Up 3 seconds (health: starting) +2026/10/29 13:16:38 [INFO] [backup] App-data backup completed: 2 databases (545.9 KB total), 4 volume dump(s) (36.774s) +2026/10/29 13:16:38 [INFO] [backup] Recovery unit captured for gokapi → /mnt/sys_drive/felhom-data/backups/primary/gokapi (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) + +14:16:46 restore points: [{"time": "2026-10-29T13:16:15Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}] diff --git a/documentation/audits/release-2026-10-29b/p3-clear-hold/r1341-deploy.log b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1341-deploy.log new file mode 100644 index 00000000..58ce9163 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1341-deploy.log @@ -0,0 +1,20 @@ +14:09:32 === phase deploy at 2026-10-29T13:09:32Z +14:09:35 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1 Up 53 minutes (healthy) +14:09:35 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['GOKAPI_PASSWORD'] +14:09:35 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +14:09:56 [1] deployed, controller state=running, pinned={'gokapi': 'f0rc3/gokapi:v1.9.6'} +14:09:59 cap: 134217728 +/usr/bin/tail +ok +14:09:59 app: state=running hold=None deployed=True +14:10:03 container: f0rc3/gokapi:v1.9.6 Up 21 seconds (healthy) +14:15:26 === phase deploy at 2026-10-29T13:15:26Z +14:15:29 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1-r1340fault Up About a minute (healthy) +14:15:29 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['GOKAPI_PASSWORD'] +14:15:29 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +14:15:49 [1] deployed, controller state=running, pinned={'gokapi': 'f0rc3/gokapi:v1.9.6'} +14:15:53 cap: 134217728 +/usr/bin/tail +ok +14:15:53 app: state=running hold=None deployed=True +14:15:57 container: f0rc3/gokapi:v1.9.6 Up 21 seconds (healthy) diff --git a/documentation/audits/release-2026-10-29b/p3-clear-hold/r1341-oom.log b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1341-oom.log new file mode 100644 index 00000000..18a98737 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1341-oom.log @@ -0,0 +1,15 @@ +14:10:04 === phase oom at 2026-10-29T13:10:04Z +14:10:07 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1 Up 54 minutes (healthy) +14:10:10 host counter before: +14:10:14 hog: rc=137 +14:10:17 after: running=true oomkilled=true restarts=0 + +14:10:18 host counter after: +14:11:36 OOM lines (first 2 scans): 2026/10/29 13:10:14 [WARN] [deadapp] gokapi: container gokapi was OOM-killed (started 2026-10-29T13:09:40.403331677Z; kills 1, limit 128M, peak 128M; seen by docker,cgroup) +2026/10/29 13:10:14 [INFO] Event pushed: app_oom (warning) [hu-only] — Alkalmazás memóriája elfogyott: gokapi (gokapi) — egy folyamatát a memóriakorlát leállította + +14:12:44 app_oom events pushed: 1 + +14:12:48 OOM lines (after 4 scans): 2026/10/29 13:10:14 [WARN] [deadapp] gokapi: container gokapi was OOM-killed (started 2026-10-29T13:09:40.403331677Z; kills 1, limit 128M, peak 128M; seen by docker,cgroup) +2026/10/29 13:10:14 [INFO] Event pushed: app_oom (warning) [hu-only] — Alkalmazás memóriája elfogyott: gokapi (gokapi) — egy folyamatát a memóriakorlát leállította + diff --git a/documentation/audits/release-2026-10-29b/p3-clear-hold/r1341-teardown.log b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1341-teardown.log new file mode 100644 index 00000000..bd284e69 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1341-teardown.log @@ -0,0 +1,18 @@ +14:12:49 === phase teardown at 2026-10-29T13:12:49Z +14:12:52 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1 Up 57 minutes (healthy) +14:12:52 app: state=running hold=None deployed=True +14:12:55 container: f0rc3/gokapi:v1.9.6 Up 3 minutes (healthy) +14:12:56 [X] stop -> 200 {'ok': True, 'message': 'Stack gokapi stop completed'} +14:13:28 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'gokapi', 'volumes_removed': ['gokapi_gokapi_config', 'gokapi_gokapi_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [] +14:13:37 [X] after remove: deployed=False leftovers='' +14:13:37 app: state=not_deployed hold=None deployed=False +14:13:40 container: +14:19:53 === phase teardown at 2026-10-29T13:19:53Z +14:19:56 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1-r1340fault Up 5 minutes (healthy) +14:19:56 app: state=stopped hold=None deployed=True +14:19:59 container: +14:19:59 [X] stop -> 200 {'ok': True, 'message': 'Stack gokapi stop completed'} +14:20:31 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'gokapi', 'volumes_removed': ['gokapi_gokapi_config', 'gokapi_gokapi_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [] +14:20:40 [X] after remove: deployed=False leftovers='' +14:20:40 app: state=not_deployed hold=None deployed=False +14:20:43 container: diff --git a/documentation/audits/release-2026-10-29b/p3-clear-hold/r1342-backup.log b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1342-backup.log new file mode 100644 index 00000000..698d749a --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1342-backup.log @@ -0,0 +1,13 @@ +14:04:03 === phase backup at 2026-10-29T13:04:03Z +14:04:06 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1 Up 48 minutes (healthy) +14:04:06 backup press → 200 {'ok': True, 'message': 'Mentés elindítva'} +14:04:51 log: 2026/10/29 13:04:20 [INFO] [backup] Volume dump: gokapi/gokapi_gokapi_data → 43.0 KB +2026/10/29 13:04:20 [INFO] [backup] Restarting gokapi after volume dump +2026/10/29 13:04:20 [INFO] [stacks] Starting stack: gokapi +2026/10/29 13:04:20 [INFO] [stacks] Stack gokapi started successfully (took 0.4s) +2026/10/29 13:04:24 [INFO] [stacks] Stack gokapi post-start status: +2026/10/29 13:04:24 [INFO] [stacks] gokapi f0rc3/gokapi:v1.9.6 running Up 3 seconds (health: starting) +2026/10/29 13:04:44 [INFO] [backup] App-data backup completed: 2 databases (544.9 KB total), 4 volume dump(s) (37.715s) +2026/10/29 13:04:44 [INFO] [backup] Recovery unit captured for gokapi → /mnt/sys_drive/felhom-data/backups/primary/gokapi (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) + +14:04:51 restore points: [{"time": "2026-10-29T13:04:19Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}] diff --git a/documentation/audits/release-2026-10-29b/p3-clear-hold/r1342-deploy.log b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1342-deploy.log new file mode 100644 index 00000000..c2034c92 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1342-deploy.log @@ -0,0 +1,10 @@ +14:03:31 === phase deploy at 2026-10-29T13:03:31Z +14:03:34 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1 Up 47 minutes (healthy) +14:03:34 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['GOKAPI_PASSWORD'] +14:03:34 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +14:03:54 [1] deployed, controller state=running, pinned={'gokapi': 'f0rc3/gokapi:v1.9.6'} +14:03:58 cap: 134217728 +/usr/bin/tail +ok +14:03:58 app: state=running hold=None deployed=True +14:04:01 container: f0rc3/gokapi:v1.9.6 Up 21 seconds (healthy) diff --git a/documentation/audits/release-2026-10-29b/p3-clear-hold/r1342-r1342.log b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1342-r1342.log new file mode 100644 index 00000000..653de2e3 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p3-clear-hold/r1342-r1342.log @@ -0,0 +1,19 @@ +14:04:52 === phase r1342 at 2026-10-29T13:04:52Z +14:04:56 controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1 Up 49 minutes (healthy) +14:04:56 [R] restoring gokapi from snapshot 'helyi' (of 1 offered) +14:04:56 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +14:04:56 + 0.0s restore (True, None, None) +14:06:01 + 64.9s restore (False, None, None) +14:06:01 [R] after restore: state=running hold=None phase=None +14:06:05 restore: state=running hold=None +14:06:15 list text: Gokapi visszaállítás: 2026. 10. 29. 14:04 · 59,8 KB Törlés most Most nem törölhető: a doboz még figyeli, hogy az alkalmazás rendben elindult-e a visszaállítás után (legfeljebb 30 percig). +14:06:15 deletable rows: 0, off buttons: 1 +14:06:18 copy on disk: ['20261029T130458Z'] +14:06:18 delete mid-watch → {"error":"Most nem törölhető: a doboz még figyeli, hogy az alkalmazás rendben elindult-e a visszaállítás után (legfeljebb 30 percig).","ok":false,"redirect":""} + +409 +14:06:18 [X] stop -> 200 {'ok': True, 'message': 'Stack gokapi stop completed'} +14:06:50 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'gokapi', 'volumes_removed': ['gokapi_gokapi_config', 'gokapi_gokapi_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [] +14:06:59 [X] after remove: deployed=False leftovers='' +14:07:22 watch log after the remove: 2026/10/29 13:06:59 [INFO] [backup] gokapi: the app was removed during the stable-start check after its unit restore — no alarm; the restore copy, if the remove left it, is an unused one (/mnt/sys_drive/felhom-data/backups/restore-undo/gokapi/20261029T130458Z) (R-1342) + diff --git a/documentation/audits/release-2026-10-29b/p3-clear-hold/walk-9202-rc1.txt b/documentation/audits/release-2026-10-29b/p3-clear-hold/walk-9202-rc1.txt new file mode 100644 index 00000000..b647556a --- /dev/null +++ b/documentation/audits/release-2026-10-29b/p3-clear-hold/walk-9202-rc1.txt @@ -0,0 +1,7 @@ +controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1 Up About a minute (healthy) +containers: 6 total +not running/unhealthy: none +refusals since 2026-10-29T12:14:00Z: 0 +ERROR lines since: 0 +login page: 200 +file manager: 200 diff --git a/documentation/audits/release-2026-10-29b/perf/pages-cchosts.txt b/documentation/audits/release-2026-10-29b/perf/pages-cchosts.txt new file mode 100644 index 00000000..fb4fe109 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/perf/pages-cchosts.txt @@ -0,0 +1,20 @@ +/ 200 ttfb=0.690595s total=0.691202s size=10549 +/ 200 ttfb=0.063345s total=0.063863s size=10549 +/ 200 ttfb=0.009637s total=0.010230s size=10549 +/ 200 ttfb=0.015199s total=0.015702s size=10549 +/ 200 ttfb=0.009370s total=0.009942s size=10549 +/hosts 200 ttfb=0.302955s total=0.303342s size=10528 +/hosts 200 ttfb=0.023615s total=0.024208s size=10528 +/hosts 200 ttfb=0.044165s total=0.044735s size=10528 +/hosts 200 ttfb=0.023545s total=0.024152s size=10528 +/hosts 200 ttfb=0.051020s total=0.051508s size=10528 +/hosts/tester-1-d70be4 200 ttfb=0.388597s total=0.391578s size=55443 +/hosts/tester-1-d70be4 200 ttfb=0.111756s total=0.114312s size=55443 +/hosts/tester-1-d70be4 200 ttfb=0.287602s total=0.291129s size=55443 +/hosts/tester-1-d70be4 200 ttfb=0.160149s total=0.161979s size=55443 +/hosts/tester-1-d70be4 200 ttfb=0.306244s total=0.308237s size=55443 +/customers 403 ttfb=0.000917s total=0.000947s size=45 +/customers 403 ttfb=0.000697s total=0.000729s size=45 +/customers 403 ttfb=0.000729s total=0.000766s size=45 +/customers 403 ttfb=0.000591s total=0.000638s size=45 +/customers 403 ttfb=0.000610s total=0.000651s size=45 diff --git a/documentation/audits/release-2026-10-29b/perf/pages-hubtoken.txt b/documentation/audits/release-2026-10-29b/perf/pages-hubtoken.txt new file mode 100644 index 00000000..f95f5176 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/perf/pages-hubtoken.txt @@ -0,0 +1,5 @@ +/configuration 200 ttfb=0.260323s total=0.260896s size=21292 +/configuration 200 ttfb=0.001449s total=0.001924s size=21292 +/configuration 200 ttfb=0.001321s total=0.002110s size=21292 +/configuration 200 ttfb=0.001369s total=0.001926s size=21292 +/configuration 200 ttfb=0.001562s total=0.002072s size=21292 diff --git a/documentation/audits/release-2026-10-29b/release-controller-0.348.0/build-0.348.0-digests.txt b/documentation/audits/release-2026-10-29b/release-controller-0.348.0/build-0.348.0-digests.txt new file mode 100644 index 00000000..aad1af8d --- /dev/null +++ b/documentation/audits/release-2026-10-29b/release-controller-0.348.0/build-0.348.0-digests.txt @@ -0,0 +1,4 @@ +#22 naming to gitea.dooplex.hu/admin/felhom-controller:0.348.0 done +#22 naming to gitea.dooplex.hu/admin/felhom-controller:latest done +0.348.0: digest: sha256:672fc1c1deaaf1cbeecfc2c5b04da731b076a9194f4ce7ed9678339fcd4548bf size: 856 +latest: digest: sha256:672fc1c1deaaf1cbeecfc2c5b04da731b076a9194f4ce7ed9678339fcd4548bf size: 856 diff --git a/documentation/audits/release-2026-10-29b/release-controller-0.348.0/floors-per-box.txt b/documentation/audits/release-2026-10-29b/release-controller-0.348.0/floors-per-box.txt new file mode 100644 index 00000000..2d9319c7 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/release-controller-0.348.0/floors-per-box.txt @@ -0,0 +1,9 @@ +== tester-1 +HTTP/1.1 303 See Other +Location: /customers/tester-1?flash=floor_set +== demo-hp +HTTP/1.1 303 See Other +Location: /customers/demo-hp?flash=floor_set +== demo-felhom +HTTP/1.1 303 See Other +Location: /customers/demo-felhom?flash=floor_set diff --git a/documentation/audits/release-2026-10-29b/release-controller-0.348.0/floors-set-at.txt b/documentation/audits/release-2026-10-29b/release-controller-0.348.0/floors-set-at.txt new file mode 100644 index 00000000..e1d25f14 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/release-controller-0.348.0/floors-set-at.txt @@ -0,0 +1 @@ +2026-10-29T13:41:01Z diff --git a/documentation/audits/release-2026-10-29b/release-controller-0.348.0/rollout-wait.txt b/documentation/audits/release-2026-10-29b/release-controller-0.348.0/rollout-wait.txt new file mode 100644 index 00000000..acc2a590 --- /dev/null +++ b/documentation/audits/release-2026-10-29b/release-controller-0.348.0/rollout-wait.txt @@ -0,0 +1 @@ +13:41:24 tester-1=gitea.dooplex.hu/admin/felhom-controller:0.348.0|Up 9 seconds (healthy) demo-hp=gitea.dooplex.hu/admin/felhom-controller:0.348.0|Up 10 seconds (healthy) demo-felhom=gitea.dooplex.hu/admin/felhom-controller:0.348.0|Up 14 seconds (healthy) diff --git a/documentation/audits/release-2026-10-29b/release-controller-0.348.0/walk-3boxes.txt b/documentation/audits/release-2026-10-29b/release-controller-0.348.0/walk-3boxes.txt new file mode 100644 index 00000000..6f2a64ad --- /dev/null +++ b/documentation/audits/release-2026-10-29b/release-controller-0.348.0/walk-3boxes.txt @@ -0,0 +1,73 @@ +=== tester-1 +controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0 Up 25 seconds (healthy) +containers: 10 total +not running/unhealthy: none +refusals since 2026-10-29T13:41:00Z: 0 +ERROR lines since: 0 +login page: 200 +file manager: 200 +2026/10/29 13:41:13 [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30) +2026/10/29 13:41:13 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply +2026/10/29 13:41:23 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.348.0 (we are 0.348.0), no managed update running +ls: cannot access '/opt/docker/felhom-controller/data/ctl/': No such file or directory +=== demo-hp +controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0 Up 27 seconds (healthy) +containers: 26 total +not running/unhealthy: none +refusals since 2026-10-29T13:41:00Z: 0 +ERROR lines since: 0 +login page: 200 +file manager: 200 +2026/10/29 13:41:14 main.go:855: [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30) +2026/10/29 13:41:14 offsiteapply.go:148: [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply +2026/10/29 13:41:21 updater.go:104: [DEBUG] [selfupdate] SetFloor: floor "" → "0.348.0" +ls: cannot access '/opt/docker/felhom-controller/data/ctl/': No such file or directory +=== demo-felhom +controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0 Up 32 seconds (healthy) +containers: 9 total +not running/unhealthy: none +refusals since 2026-10-29T13:41:00Z: 0 +ERROR lines since: 0 +login page: 200 +file manager: 200 +2026/10/29 13:40:50 [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30) +2026/10/29 13:40:50 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply +2026/10/29 13:41:00 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.348.0 (we are 0.348.0), no managed update running +ls: cannot access '/opt/docker/felhom-controller/data/ctl/': No such file or directory +control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -) +== tester-1 (enkicsifelhom.hu): 6 public name(s) + OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1 + OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 + OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1 + OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1 + OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1 +== demo-hp (enkisfelhom.hu): 15 public name(s) + OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1 + OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1 + OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1 + OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1 + OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1 + OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1 + OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 + OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1 + OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1 + OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1 + OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1 + OK http://time.enkisfelhom.hu/r1033-check?q=1 -> 301 https://time.enkisfelhom.hu/r1033-check?q=1 + OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1 + OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1 +== demo-felhom (demo-felhom.eu): 8 public name(s) + OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1 + OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1 + OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1 + OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 + OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1 + OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1 + OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1 + OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1 +== every public name redirects http -> https +root@192.168.0.154: srw------- 1 root root 0 Oct 29 13:41 hold.sock +hp: srw------- 1 root root 0 Oct 29 13:41 hold.sock +felhom-pve: srw------- 1 root root 0 Oct 29 13:40 hold.sock diff --git a/documentation/audits/release-2026-10-29b/release-controller-0.348.0/walk-9202-rc2.txt b/documentation/audits/release-2026-10-29b/release-controller-0.348.0/walk-9202-rc2.txt new file mode 100644 index 00000000..7a20f14a --- /dev/null +++ b/documentation/audits/release-2026-10-29b/release-controller-0.348.0/walk-9202-rc2.txt @@ -0,0 +1,7 @@ +controller: gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.2 Up 2 minutes (healthy) +containers: 6 total +not running/unhealthy: none +refusals since 2026-10-29T13:37:30Z: 0 +ERROR lines since: 0 +login page: 200 +file manager: 200 diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 905637c7..75e40fa8 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -154,7 +154,7 @@ stopping line that lies. | **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC | -## App updates — 9 rows (P3 9) +## App updates — 10 rows (P3 10) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -167,8 +167,9 @@ stopping line that lies. | **R-462** | App updates | P3 | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s, measured). The catalog currency and the next-apps ranking: `audits/catalog-currency-2026-09-30.md` — 25 of 53 apps behind upstream inside a major, 19 across one (morning); after the session **31 apps can update themselves at night (+ nextcloud with a whole copy)**, 32 carry a proven step, 21 have no ladder.** **-- 2026-09-30 (evening, more-night-apps brief): ten more steps published on both venues** — first ladders for calibre-web, gitea, wger, crafty-controller, uptime-kuma, zipline (new front-door fixtures; gitea's installer form), within-major steps for emby, ghost, home-assistant, outline, rallly. **Night-updatable by the audit's method (a): 30 + 2 conditional at the session's start → 35 + 3 conditional** (calibre-web conditional: `files_may_change`). zipline got a two-step ladder 4.6.1 → 4.7.0 → 4.8.0 (the direct jump fails, R-742). Stopped: wanderer (the bench cannot run it, R-739). Fixture/harness defects fixed on the way: R-735, gitea's installer, the HTTPS backend on the bench, per-file names for `files_may_change`. `audits/more-night-apps-2026-09-30/` | **READY — rank P2-MEDIUM; owner: CC (scope already ruled, `09` §3 decision 6)** **Re-ranked 2026-10-03: P2→P3: 35+ apps already update by proven steps; an app without a ladder simply does not move, which is safe.** **NARROWED 2026-10-20 (night brief part 2, Part 5):** the fixture side is DONE for every app installed on a test box — 58 box fixtures run on both venues; of the 16 installed apps only bentopdf has none (it keeps no data). Upstream within a major (check-currency, registries read): privatebin 2.0.6→2.1.0, home-assistant 2026.9.4→2026.10.2, romm 5.3.1→5.4.0, plex 1.41.4→1.43.4, kimai 2.67.0→2.70.0 — the list left. **No edge was run:** the bench (guest 9401, stopped) and the drill catalog the box venue needs were outside the brief's machines and repos; the operator's word for both is on STATUS's morning list (`audits/night2-2026-10-20/P5/r462-list.txt`). | — | — | CC | | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC | | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). **2026-10-28 (R-1332): HTTP2Shell has NO fixed release yet** — v0.17.3's server carries Node.js 24.18.0 (`tsx index.ts`, behind the setup gate) and the newest v1.9.0 still Node.js 24.22.0 (fixed: 24.22.1); the web image is nginx, no Node. The big step waits for a v1.x with Node ≥ 24.22.1, or takes the runtime fix later as a lane step. | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:** **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC | +| **R-1347** | App updates | P3 | **After a successful restore lifts an update hold, the stack manager's „last update ended held" flag stays set until the next update or a controller restart — until then the crash-loop watcher does not watch the app and the nightly update run skips it.** Found by the D64 work 2026-10-29 (controller `5b3b541` clears the flag for `clear_hold` only, `ForgetUpdateHeld`). `clearUpdateHoldAfterRestore` lifts the hold in settings but not this cached flag. | OPEN | — | the restore's lift calls ForgetUpdateHeld too, red-first test (restore after an update hold → the watcher watches again) | CC | -## Backup & restore — 47 rows (P2 4, P3 25, P4 18) +## Backup & restore — 48 rows (P2 4, P3 26, P4 18) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -219,6 +220,7 @@ stopping line that lies. | **R-1340** | Backup & restore | P3 | **The operator's „HELD STOPPED" mail for a mixed restore says the DATABASE restore failed and was rolled back, and that the files and volumes are the snapshot's — whatever the cause.** Measured 2026-10-29 08:18 UTC on Tester 1 (D63's H5 live proof, test build): Radicale (no database) — the volume could not be removed (not written into), the put-back's stop failed, nothing moved; the mail read „its off-site database restore failed. The database was rolled back to the pre-restore copy and the live definition written back, but its files and named volumes are the snapshot's". The text (`controller/cmd/controller/main.go:1555`) predates the D20/D22 put-back, when the only mixed hold was R-893 option C. Support reads a wrong account of what is on the box. The household's sentence is conservative and stays. Fix: the operator text names the cause it was given (the replay error, the put-back step that failed, whether the copy is kept and where) instead of the option-C story; a test per cause. Evidence: `audits/d63-2026-10-29/run-h5.log`. | OPEN | — | the fix above, red first | CC | | **R-1342** | Backup & restore | P3 | **For 30 minutes after a restore the box watches whether the app runs stably, and its restore copy is the way back — but (1) D62's „Törlés most" could delete that copy during the watch, and (2) an app REMOVED during the watch made the box mail „did not start … the pre-restore copy is kept" when the watch ran out.** Found live on Tester 1, 2026-10-29: Gokapi restored 10:08 CET, its copy deleted by hand 10:10 (D62's own live proof), the app removed 10:11; at 10:39 CET the operator got a `backup_run_failures` error mail — both halves of its sentence false. (1) is a gap in D62 (released in controller 0.347.0): the operator's rule is „never while an operation runs on the app"; (2) is older (the R-1270 watch, since 2026-10-26). **Fixed on main** (controller `652a690`, not released — one release per session): the watch marks its copy (`Manager.watching`, under `m.mu`), the list reads it „checking" with one line and no delete date, the delete refuses it; the watch ends quietly when the provider says the app was removed (`StackRemovalReporter`, implemented by the production adapter — a compile-time assertion). Tests `TestR1342_*`, each seen failing by a mutation. Evidence: `audits/d63-2026-10-29/` (hub log of the mail, `run-p4b-d62.log`). | OPEN — fixed on main, not released; rides the next controller release | — | release it (next controller release) | CC | | **R-1343** | Backup & restore | P3 | **The operator's „clear a restore hold" route can be undone silently: the CLI clears `settings.json`, but the running controller keeps the hold in memory and writes it back at its next settings save (after each hub report, ~15 min) — and its hint names a restart without saying it must run INSIDE the guest and AT ONCE.** Measured 2026-10-29 on Tester 1: the operator ran `docker exec felhom-controller felhom-controller --clear-restore-hold radicale` at 11:42 CET („restore hold CLEARED … Settings saved"), then `systemctl restart felhom-bootstrap.service` on the Proxmox HOST (the hint's unit name does not exist there; the host's own bootstrap unit skipped itself — no harm); the guest controller was restarted at 11:53 and loaded `settings.json` WITH the hold again (its mtime: the old controller's save after a report). The CLI's own text knew half of this („the running controller keeps refusing until it reloads", measured on demo-hp 2026-08-22). Fix options: the CLI refuses while a controller runs and says „stop it first"; or the clear goes through the running controller (an operator-only local call) so memory and file agree; and the hint names „in the guest (pct exec -- systemctl restart felhom-controller-bootstrap.service), at once". | OPEN | — | pick the fix, red-first test that a save after the clear does not bring the hold back | CC | +| **R-1348** | Backup & restore | P3 | **The restore_mixed hold written at START-UP after an interrupted put-back fails sends no operator mail when it is set** — only the hold reporter's info `app_held` reaches the hub, and the hub's reminder after its grace period. Found by the R-1340 work 2026-10-29 (controller `5b3b541`): the mail sentences for unhealthy stop, failed update and waiting put-back exist and are tested, but the start-up path has no sender. | OPEN | — | send the operator mail from the start-up put-back failure (the R-1340 sentence), red-first test | CC | ## Storage & devices — 14 rows (P3 4, P4 10) @@ -323,7 +325,7 @@ stopping line that lies. | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | | **R-1344** | Monitoring & notifications | P3 | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** Operator brief 2026-10-29, Part 2: the agent may restart it only after a measurement shows a busy night (backups, updates) never looks „not answering". Built 2026-10-29 as ALARM ONLY (agent `637c92c`: 6 consecutive sweeps of Docker's health check „unhealthy" — on top of Docker's own 3 failed probes, ~4.5 min in all — set `unresponsive_since`; the hub mails `controller_unresponsive`, operator-only). The measurement: Docker's own probe record (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) logged every 30 s on Tester 1 and demo-hp from 2026-10-29 11:14 UTC to 2026-10-30 07:00 UTC — through tonight's real night chain (a day run cannot hold the whole-guest vzdump). | OPEN — measuring | — | read the measurement (worst probe times, any 5-s timeout, any „unhealthy"); if none fired: switch the alarm to a restart (same guards and brake, `controller_unresponsive_restart`) | CC | -## Hub & operator — 13 rows (P3 3, P4 10) +## Hub & operator — 14 rows (P3 4, P4 10) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -340,6 +342,7 @@ stopping line that lies. | **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator | | **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC | | **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC | +| **R-1346** | Hub & operator | P3 | **Hub pages load slowly for the operator, and the artifacts page's vouch has twice ended in a 504 from an nginx in front of the hub — while the save went through.** Operator, 2026-10-29 (the 0.169.0 vouch: 504 in the browser; the hub logged „Artifact manifest set: agent=0.169.0" at 14:06:09 CET). Measured the same afternoon INSIDE the cluster (hub ClusterIP, 5 tries each): the dashboard 0.01–0.69 s, Hosts 0.02–0.30 s, a host page 0.11–0.39 s, Configuration 0.002–0.26 s; the registry calls behind the artifacts page ~0.1 s each (11 agent + 7 golden versions, 8 in parallel, cached) — so neither the hub's own work nor the registry explains a minute. Suspect the road (browser → proxy/tunnel → DooPlex) or one handler not yet timed (the artifacts page and its POST are session-only, so no token can time them). The hub logs no request durations. Evidence: `audits/release-2026-10-29b/perf/`. | OPEN | — | (1) a slow-request line in the hub (method, path, status, duration ≥ 2 s); (2) time the public path from outside the LAN; (3) then fix where the time goes | CC | ## Business & legal — 14 rows (P2 4, P3 2, P4 8)