Files
felhom.eu/documentation/audits/night-2026-09-24/E/round-04.json
T
2026-09-24 14:21:14 +02:00

196 lines
12 KiB
JSON
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{
"round": 4,
"action": "failing_step",
"accident": {
"kind": "controller_killed",
"cmd": "kill -9 of the controller PID (1 of max 3)",
"fired_at": "14:09:22",
"result": "killed pid 2474 rc=0"
},
"app": "wishlist",
"started": "2026-09-24T12:08:51Z",
"before": {
"state": {
"deployed": true,
"state": "running",
"update_phase": null,
"hold_reason": null,
"hold_kind": null,
"update_error": null,
"ladder_steps_left": null
},
"badges": {
"hu": [
{
"title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.",
"text": "Naprakész"
}
],
"en": [
{
"title": "This app is running the newest version available.",
"text": "Up to date"
}
]
},
"page_hu": [
"Temperature high: 84°C (hwmon2) Rendszermonitor →",
"+ 1 további figyelmeztetés →",
"Az ügyfélszolgálat foglalkozik ezzel az alkalmazással, ezért az adatai megmaradnak: most csak magát az alkalmazást távolíthatod el."
],
"page_en": [
"Temperature high: 84°C (hwmon2) System monitor →",
"+ 1 more warning →",
"Support is working on this app, so its data stays: for now you can remove only the app itself."
]
},
"badge_catchup_s": 4.7,
"act": "Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}",
"armed_at": "14:09:13",
"armed_why": "at verifying",
"phases": [
[
6.1,
"backing-up"
],
[
8.2,
"safety-dump"
],
[
9.3,
"pulling"
],
[
10.3,
"copying"
],
[
21.4,
"starting"
],
[
21.9,
"verifying"
],
[
123.9,
"undoing"
],
[
141.8,
"undone"
]
],
"controller_back_s": 0.2,
"time_to_steady_s": 147.7,
"after": {
"state": {
"deployed": true,
"state": "running",
"update_phase": "undone",
"hold_reason": null,
"hold_kind": null,
"update_error": null,
"ladder_steps_left": null
},
"badges": {
"hu": [
{
"title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.",
"text": "Naprakész"
}
],
"en": [
{
"title": "This app is running the newest version available.",
"text": "Up to date"
}
]
},
"page_hu": [
"Temperature high: 78°C (hwmon2) Rendszermonitor →",
"+ 1 további figyelmeztetés →",
"A(z) wishlist frissítése 2026-09-24 14:10-kor nem sikerült. A doboz automatikusan visszaállította az előző változatot és az adatokat — semmi nem veszett el.",
"Az ügyfélszolgálat foglalkozik ezzel az alkalmazással, ezért az adatai megmaradnak: most csak magát az alkalmazást távolíthatod el."
],
"page_en": [
"Temperature high: 78°C (hwmon2) System monitor →",
"+ 1 more warning →",
"The update of wishlist at 2026-09-24 14:10 did not succeed. The box put back the previous version and its data automatically — nothing was lost.",
"Support is working on this app, so its data stays: for now you can remove only the app itself."
]
},
"readback": {
"A": true
},
"events": [],
"box_log": [
"2026-09-24T12:08:59Z INFO [stacks] Stack wishlist started successfully (took 0.4s)",
"2026-09-24T12:09:00Z INFO [backup] update pre-backup for wishlist: volume dump OK",
"2026-09-24T12:09:00Z INFO [backup] Recovery unit captured for wishlist → /mnt/sys_drive/felhom-data/backups/primary/wishlist (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)",
"2026-09-24T12:09:00Z INFO [backup] update pre-backup for wishlist: recovery unit captured (0 database dump(s))",
"2026-09-24T12:09:00Z INFO [backup] Tier 2 copied wishlist → /mnt/felhom-drives/scratch_hdd/backups/secondary/wishlist (323.7 KB, 0 leg(s), 0s)",
"2026-09-24T12:09:00Z INFO [backup] update pre-backup for wishlist: complete in 2.015s",
"2026-09-24T12:09:00Z INFO [stacks] update wishlist: precondition met after the backup — Tier 2 (second drive) copy from 2026-09-24T12:09:00Z",
"2026-09-24T12:09:00Z INFO [stacks] update wishlist: phase safety-dump",
"2026-09-24T12:09:00Z INFO [backup] Discovered 3 databases",
"2026-09-24T12:09:00Z INFO [backup] update safety dump for wishlist: the app has no database — nothing to copy (no-op)",
"2026-09-24T12:09:00Z INFO [stacks] update wishlist: safety dump done (0 file(s)) []",
"2026-09-24T12:09:01Z INFO [stacks] update wishlist: the undo copy will hold 2 named volume(s), 0.3 MiB",
"2026-09-24T12:09:01Z INFO [stacks] update wishlist: phase pinning",
"2026-09-24T12:09:01Z INFO [stacks] update wishlist: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/wishlist/docker-compose.yml (wishlist=ghcr.io/cmintey/wishlist:latest)",
"2026-09-24T12:09:01Z INFO [stacks] update wishlist: phase pulling",
"2026-09-24T12:09:02Z INFO [stacks] update wishlist: phase copying",
"2026-09-24T12:09:03Z INFO [stacks] Stack wishlist post-start status:",
"2026-09-24T12:09:03Z INFO [stacks] wishlist ghcr.io/cmintey/wishlist:v0.67.1@sha256:4b89bfb5b57ffb087c33978d21b09048e5c37f4506326e9ccd2705a0a1d86548 running Up 3 seconds (health: starting)",
"2026-09-24T12:09:24Z WARN [stacks] update recovery: wishlist was interrupted in verifying (started 2026-09-24T12:08:58Z) — the new version may have run; marking it Updating and RESUMING the health wait",
"2026-09-24T12:09:24Z WARN [update] 1 interrupted update(s) will resume after the backup side is wired: [wishlist]",
"2026-09-24T12:09:24Z INFO [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false)",
"2026-09-24T12:09:24Z INFO [scheduler] Registered periodic job: deadapp-check (every 30s)",
"2026-09-24T12:09:24Z INFO [scheduler] Registered periodic job: backup-cache (every 5m0s)",
"2026-09-24T12:09:24Z INFO [scheduler] Daily job tier2-backup scheduled for 2026-09-24 14:20 CEST",
"2026-09-24T12:09:24Z INFO [scheduler] Daily job offbox-backup scheduled for 2026-09-24 15:05 CEST",
"2026-09-24T12:09:24Z INFO [stacks] update wishlist: resuming after a controller restart — `up -d` then the health wait",
"2026-09-24T12:09:24Z INFO [backup] Found 10 DB dump files across drives",
"2026-09-24T12:09:24Z INFO [stacks] update wishlist: phase verifying",
"2026-09-24T12:09:25Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:09:25Z WARN [monitor] Temperature threshold breached: 78°C (limit: 75°C)",
"2026-09-24T12:09:25Z INFO [backup] Discovered 3 databases",
"2026-09-24T12:09:25Z INFO [backup] nextcloud: pruned old undo copy pre-restore-20260924T104416Z-nextcloud-mariadb.sql (keeping the newest 3)",
"2026-09-24T12:09:25Z INFO [backup] Recovery unit captured for nextcloud → /mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1)",
"2026-09-24T12:09:25Z INFO [backup] Backup status cache refreshed",
"2026-09-24T12:09:29Z WARN [api] Unauthorized request to /api/stacks/wishlist from 172.18.0.10:35772",
"2026-09-24T12:09:30Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:09:35Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:09:40Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:09:45Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:09:49Z INFO [bootrecon] \"gokapi\" is a boot orphan by intent but is HELD (held after a failed restore whose rollback also failed (2026-09-24T10:37:48Z) — clear the hold to start it) — NOT starting it; whatever is holding it owns its recovery",
"2026-09-24T12:09:49Z INFO [bootrecon] \"chaosoom\" is a boot orphan by intent but is HELD (held after a failed restore whose rollback also failed (2026-09-24T11:48:01Z) — clear the hold to start it) — NOT starting it; whatever is holding it owns its recovery",
"2026-09-24T12:09:50Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:09:55Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:10:00Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:10:05Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:10:10Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:10:15Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:10:20Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:10:25Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:10:30Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:10:35Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:10:40Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:10:45Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:10:50Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:10:55Z WARN Health probe wishlist: HTTP GET :8999/ → Get \"http://wishlist:8999/\": dial tcp 172.18.0.5:8999: connect: connection refused",
"2026-09-24T12:10:55Z ERROR [stacks] update wishlist FAILED after the new version was started: not healthy: not healthy within 1m30s (last: health check failing)",
"2026-09-24T12:10:55Z INFO [stacks] update wishlist: kept 3084 bytes of the app's own log at /opt/docker/stacks/wishlist/hold-logs/20260924T121055Z/compose-logs.txt before stopping it (R-621)",
"2026-09-24T12:10:55Z INFO [stacks] update wishlist: phase undoing",
"2026-09-24T12:10:55Z WARN [stacks] update wishlist: UNDO — putting back the previous version and its 2 volume copy(ies) (reason: not healthy: not healthy within 1m30s (last: health check failing))",
"2026-09-24T12:10:57Z INFO [stacks] pin wishlist: wishlist=ghcr.io/cmintey/wishlist:v0.67.1",
"2026-09-24T12:11:13Z INFO [stacks] update wishlist: event app_update_undone",
"2026-09-24T12:11:13Z WARN notifier disabled (no hub configured): DROPPED event app_update_undone (severity warning) — further app_update_undone events are logged at DEBUG only",
"2026-09-24T12:11:13Z INFO [stacks] update wishlist: UNDONE in 18s — the previous version is running on the data from before the update (the app's health check passed)",
"2026-09-24T12:11:15Z INFO [sync] Updated wishlist/.felhom.yml",
"2026-09-24T12:11:15Z INFO [api] Catalog sync completed: Sablonok frissítve — frissítve: wishlist"
],
"undo_copies_left": "0"
}