BIGNIGHT F2 power cut during backup: guard restarts the stopped app, true alarm; R-519 (torn point dated by its newest part); R-517 amended
gates / gates (push) Successful in 20s
gates / gates (push) Successful in 20s
This commit is contained in:
@@ -500,3 +500,14 @@ target=felhom-pbs … storage 'felhom-pbs' does not exist`), no operator mail fo
|
||||
| should have fired, did not | none — a 61 s outage is under the hub's 30-minute staleness threshold, by design |
|
||||
|
||||
Verdict **PASS**. Jellyfin's health probe answered 503 / refused for ≈ 45 s after its start (WARN lines), then healthy.
|
||||
|
||||
### F2 — power cut during the nightly backup (`phase5/F2-power-cut-during-backup.txt`, `phase5/F2/`)
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| how | `POST /api/backup/run` 19:40:03Z; the harness waited for the first „Stopping … for safe volume dump" (adventurelog, 19:40:07Z) and cut power at **19:40:08Z**; 33 s off; `qm start` 19:40:41Z |
|
||||
| what the customer saw | every app down ≈ 3½ min, as F1. Afterwards `/backups/apps`: „Utolsó adatbázis mentés **2026-09-14 21:40 (5 perce)**", six databases „21:40 · OK", every app „Utolsó: 5 perce"; `/backups` and `/dashboard`: **no word of an interruption**; dashboard tile „Utolsó mentés: 2026-09-14 **19:40**" (UTC beside local times — R-500); whole-system tile „– · Naprakész" with no backup listed (added to R-517) |
|
||||
| what the box did alone | controller 19:42:48Z: `[appstop] crash recovery: an app-data backup (volume dump) … was interrupted and left 1 app(s) stopped — restarting them: [adventurelog]` → restarted 19:42:52Z — **the app-stop guard restarted what it stopped** ✓. All 12 running on the same images at **+245 s**. AdventureLog's trip read back intact (3 places, visits). |
|
||||
| torn copy | on disk the adventurelog and bookstack units hold a **19:40:07 SQL dump beside 19:00 volume tars and a 19:02:34 manifest**; both restore points are dated 19:40:07Z → **R-519 (P2)** |
|
||||
| alarms fired | `backup_failed (error) — an app-data backup (volume dump) was interrupted by a controller restart — 1 app(s) were left stopped and have been restarted` + **operator e-mail** — **true**; `controller_started (info)` — true. No false dead-app alarm at +10 min |
|
||||
| should have fired, did not | a customer-side notice on the backups page (R-519) |
|
||||
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
F2 POST /api/backup/run 19:40:03
|
||||
{"ok":true,"message":"Mentés elindítva"}
|
||||
|
||||
stack stopped for dump: 2026/09/14 19:40:07 [INFO] [backup] Stopping adventurelog for safe volume dump
|
||||
F2 qm stop at 19:40:08 (+5 s into the backup)
|
||||
status: stopped
|
||||
F2 qm start at 19:40:41
|
||||
status: running
|
||||
F2 + 135s dashboard /api/health 200
|
||||
F2 + 137s privatebin running same images
|
||||
F2 + 137s gokapi running same images
|
||||
F2 + 138s vaultwarden running same images
|
||||
F2 + 138s jellyfin running same images
|
||||
F2 + 144s nextcloud running same images
|
||||
F2 + 149s bookstack running same images
|
||||
F2 + 155s uptime-kuma running same images
|
||||
F2 + 184s mealie running same images
|
||||
F2 + 189s docmost running same images
|
||||
F2 + 204s immich running same images
|
||||
F2 + 205s adventurelog running same images
|
||||
F2 + 245s paperless-ngx running same images
|
||||
F2 + 245s ALL 12 running
|
||||
@@ -0,0 +1 @@
|
||||
19:45:14 adventurelog after F2 200 [('Badacsony', 4.0, 1), ('Keszthely', 5.0, 1), ('Tihany', 5.0, 1)]
|
||||
@@ -0,0 +1,10 @@
|
||||
=== 19:45:09 /api/backup/status
|
||||
{"ok":true,"data":{"enabled":true,"running":false}}
|
||||
|
||||
=== /backups/apps
|
||||
Ütemezés | Adatbázis mentés | 02:30 | Következő: holnap 02:30 | Utolsó adatbázis mentés: | 2026-09-14 21:40 (5 perce) | Mentés most | Adatbázisok | Alkalmazás | Típus | Méret | Utolsó | Érvényesítés | Állapot | immich | PostgreSQL | 51.0 MB | 21:40 | 66 tábla | OK | nextcloud | MariaDB | 1.4 MB | 21:40 | 131 tábla | OK | paperless-ngx | PostgreSQL | 317.6 KB | 21:40 | 72 tábla | OK | adventurelog | PostgreSQL | 152.6 KB | 21:40 | 47 tábla | OK | bookstack | MariaDB | 66.4 KB | 21:40 | 41 tábla | OK | docmost | PostgreSQL | 139.9 KB | 21:40 | 42 tábla | OK | Alkalmazások mentési állapota | AdventureLog | Konfig + DB + Adatok | ▶ | 1. mentés | Auto | helyi | Utolsó: 5 perce | DB + Konfig + Adatok | 2. mentés | rsync | → hdd_1 | Naponta | Utolsó sikeres: 42 perce | Sikeres | 112.2 MB | DB + Konfig + Adatok | Fájlok visszaállítása | Teljes visszaállítás a másolatból | Beállítás | 3. mentés | Nincs beállítva | távoli (offsite) | Nincs távoli mentési cél beállítva | Beállítás | BookStack | Konfig + DB + Adatok | ▶ | 1. mentés | Auto | helyi | Utolsó: 5 perce | DB + Konfig + Adatok | 2. mentés | rsync | → hdd_1 | Naponta | Utolsó sikeres: 42 perce | Sikeres | 154.9 MB | DB + Konfig + Adatok | Fájlok visszaállítása | Teljes visszaállítás a másolatból | Beállítás | 3. mentés | Nincs beállítva | távoli (offsite) | Nincs távoli mentési cél beállítva | Beállítás | Docmost | Konfig + DB + A
|
||||
=== restore points adventurelog / bookstack
|
||||
{"ok":true,"data":[{"time":"2026-09-14T19:40:07Z","short_id":"helyi","tier":1,"drive_label":"Belső SSD (rendszer)"}]}
|
||||
|
||||
{"ok":true,"data":[{"time":"2026-09-14T19:40:07Z","short_id":"helyi","tier":1,"drive_label":"Belső SSD (rendszer)"}]}
|
||||
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
2026/09/14 19:42:48 [INFO] local-api: mount mp8 → /mnt/felhom-drives (storage=/mnt/felhom-drives, class=, backup=false)
|
||||
2026/09/14 19:42:48 [INFO] local-api: mount mp9 → /etc/felhom-bootstrap (storage=/var/lib/felhom-agent/guests/9201/bootstrap, class=, backup=false)
|
||||
2026/09/14 19:42:48 [INFO] local-api: mount mp0 → /var/lib/felhom (storage=local-lvm, class=, backup=true)
|
||||
2026/09/14 19:42:48 [WARN] [appstop] crash recovery: an app-data backup (volume dump) (op "volume-dump:adventurelog") was interrupted and left 1 app(s) stopped — restarting them: [adventurelog]
|
||||
2026/09/14 19:42:52 [INFO] [appstop] crash recovery: restarted adventurelog after the interrupted an app-data backup (volume dump)
|
||||
2026/09/14 19:42:52 [INFO] [scheduler] Daily job db-dump scheduled for 2026-09-15 02:30 CEST
|
||||
2026/09/14 19:42:52 [INFO] [scheduler] Registered periodic job: backup-cache (every 5m0s)
|
||||
2026/09/14 19:42:52 [INFO] [scheduler] Daily job tier2-backup scheduled for 2026-09-15 03:30 CEST
|
||||
2026/09/14 19:42:52 [INFO] [scheduler] Daily job offbox-backup scheduled for 2026-09-15 04:15 CEST
|
||||
2026/09/14 19:42:53 [INFO] Event pushed: backup_failed (error) — an app-data backup (volume dump) was interrupted by a controller restart — 1 app(s) were left stopped and have been restarted
|
||||
2026/09/14 19:42:57 [INFO] [backup] Found 6 DB dump files across drives
|
||||
2026/09/14 19:42:58 [INFO] [backup] Discovered 5 databases
|
||||
2026/09/14 19:42:58 [INFO] [backup] Discovered app data: 12 apps
|
||||
2026/09/14 19:42:59 [INFO] [backup] Backup status cache refreshed
|
||||
2026/09/14 19:43:48 [INFO] [bootrecon] boot window: budget 50s exhausted while the fleet was still changing — sweeping anyway
|
||||
2026/09/14 19:43:48 [INFO] [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start)
|
||||
@@ -0,0 +1,18 @@
|
||||
=== /backups 19:45:41
|
||||
fragment 'megszakad' 0
|
||||
fragment 'sikertelen' 0
|
||||
fragment 'hiba' 2
|
||||
fragment 'Hiba' 0
|
||||
fragment 'nem sikerült' 0
|
||||
fragment 'interrupt' 0
|
||||
fragment 'failed' 0
|
||||
↗ | Biztonsági mentés | enkicsifelhom.hu | A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer — így hibás fájlok ellen véd, lemezhiba ellen nem. Csatlakoztass egy második meghajtót a teljes védelemhez. | Ezt a meghajtót kijelölheted a rendszermentés helyéül — így egy lemezhiba után is vissza tudod állítani a rendszert. | Adatlemez | Kijelölöm | Tárhely áttekintés | Rendszer (/) | 23.0 GB / 68.7 GB (34%) | Adatlemez | 3.52 GB / 97.9 GB (4%) | DB mentések | 6 fájl | Rendszermentés (teljes mentés) | A teljes szerver — alkalmazások, beállítások és adatbázisok együtt — időszakos mentése, amelyből az egész készülék visszaállítható. Ezt a host-ügynök készíti és kezeli. | – | Utolsó teljes mentés | – | Méret / cél | Naprakész | Következő mentés | 0 órája — a mentési ablakon belül | – | Visszaállítás ellenőrizve | Még nem futott | Mentés most | Pillanatkép-mód: az alkalmazások csa
|
||||
=== /dashboard 19:45:42
|
||||
fragment 'megszakad' 0
|
||||
fragment 'sikertelen' 0
|
||||
fragment 'hiba' 0
|
||||
fragment 'Hiba' 0
|
||||
fragment 'nem sikerült' 0
|
||||
fragment 'interrupt' 0
|
||||
fragment 'failed' 0
|
||||
↗ | Vezérlőpult | enkicsifelhom.hu | 16 | Futó alkalmazás | 0 | Leállítva | 57 | Összes alkalmazás | Memória | 3.8 GB / 12 GB (33%) | CPU | 15% | Load: 2.45 / 2.69 / 1.18 | Rendszer (/) | 23.0 GB / 68.7 GB (34%) | Adatlemez | 3.52 GB / 97.9 GB (4%) | Lemezek állapota | QEMU QEMU HARDDISK | Nincs adat | 0 °C | QEMU QEMU HARDDISK | Nincs adat | 0 °C | Biztonsági mentés | Utolsó mentés: | 2026-09-14 19:40 | Adatbázisok: | 6 mentve | Telepített alkalmazások | AdventureLog | Utazási napló és kalandtervező | Fut | Megnyitás | Napló | BookStack | Egyszerű, könyv-szerű wiki és dokumentáció platform | Fut | Megnyitás | Napló | Cloudflare Tunnel | Biztonságos internetkapcsolat — a szerver portnyitás nélkül érhető el kívülről. | Fut | Védett | Docmost | Modern wiki és dokumentáció platform (Notion-szerű) | Fut | Megnyitás | Napló | FileBrowser | Fájlkezelő — a tárhely fájljainak böngészése a böngés
|
||||
@@ -0,0 +1,37 @@
|
||||
== adventurelog
|
||||
/mnt/sys_drive/felhom-data/backups/primary/adventurelog:
|
||||
drwxr-xr-x 2 root root 4096 19:02:34 compose
|
||||
drwxr-xr-x 2 root root 4096 19:40:07 db-dumps
|
||||
-rw-r--r-- 1 root root 1385 19:02:34 manifest.json
|
||||
drwxr-xr-x 2 root root 4096 19:00:35 volume-dumps
|
||||
|
||||
/mnt/sys_drive/felhom-data/backups/primary/adventurelog/db-dumps:
|
||||
-rw-r--r-- 1 root root 156311 19:40:07 adventurelog-postgres.sql
|
||||
|
||||
/mnt/sys_drive/felhom-data/backups/primary/adventurelog/volume-dumps:
|
||||
-rw-r--r-- 1 root root 2048 19:00:34 adventurelog_adventurelog_media.tar
|
||||
-rw-r--r-- 1 root root 117450752 19:00:35 adventurelog_adventurelog_postgres_data.tar
|
||||
"created_at": "2026-09-14T19:02:34Z"
|
||||
== bookstack
|
||||
/mnt/sys_drive/felhom-data/backups/primary/bookstack:
|
||||
drwxr-xr-x 2 root root 4096 19:02:34 compose
|
||||
drwxr-xr-x 2 root root 4096 19:40:07 db-dumps
|
||||
-rw-r--r-- 1 root root 1247 19:02:34 manifest.json
|
||||
drwxr-xr-x 2 root root 4096 19:00:48 volume-dumps
|
||||
|
||||
/mnt/sys_drive/felhom-data/backups/primary/bookstack/db-dumps:
|
||||
-rw-r--r-- 1 root root 67958 19:40:07 bookstack-mariadb.sql
|
||||
|
||||
/mnt/sys_drive/felhom-data/backups/primary/bookstack/volume-dumps:
|
||||
-rw-r--r-- 1 root root 1461248 19:00:47 bookstack_bookstack_config.tar
|
||||
-rw-r--r-- 1 root root 160836096 19:00:48 bookstack_bookstack_db_data.tar
|
||||
"created_at": "2026-09-14T19:02:34Z"
|
||||
== vaultwarden
|
||||
/mnt/sys_drive/felhom-data/backups/primary/vaultwarden:
|
||||
drwxr-xr-x 2 root root 4096 19:02:34 compose
|
||||
-rw-r--r-- 1 root root 1073 19:02:34 manifest.json
|
||||
drwxr-xr-x 2 root root 4096 19:02:33 volume-dumps
|
||||
|
||||
/mnt/sys_drive/felhom-data/backups/primary/vaultwarden/volume-dumps:
|
||||
-rw-r--r-- 1 root root 756736 19:02:33 vaultwarden_vaultwarden_data.tar
|
||||
"created_at": "2026-09-14T19:02:34Z"
|
||||
@@ -253,3 +253,48 @@
|
||||
19:39:47 bookstack:ok5/fail0 mealie:ok5/fail0 nextcloud:ok1/fail0
|
||||
19:39:52 bookstack:ok4/fail0 mealie:ok5/fail0 nextcloud:ok2/fail0
|
||||
19:39:57 bookstack:ok5/fail0 mealie:ok5/fail0 nextcloud:ok2/fail0
|
||||
19:40:02 bookstack:ok5/fail0 mealie:ok5/fail0 nextcloud:ok2/fail0
|
||||
19:40:07 bookstack:ok4/fail0 mealie:ok4/fail0 nextcloud:ok2/fail0
|
||||
19:40:12 bookstack:ok3/fail0 mealie:ok3/fail0 nextcloud:ok1/fail0
|
||||
19:40:17 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail0
|
||||
19:40:22 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:40:27 bookstack:ok0/fail0 mealie:ok0/fail0 nextcloud:ok0/fail0
|
||||
19:40:32 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:40:37 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail0
|
||||
19:40:42 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:40:47 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:40:52 bookstack:ok0/fail2 mealie:ok0/fail2 nextcloud:ok0/fail2
|
||||
19:40:57 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:41:02 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:41:07 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:41:12 bookstack:ok0/fail2 mealie:ok0/fail2 nextcloud:ok0/fail2
|
||||
19:41:17 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:41:22 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:41:27 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:41:32 bookstack:ok0/fail2 mealie:ok0/fail2 nextcloud:ok0/fail2
|
||||
19:41:37 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:41:42 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:41:47 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:41:52 bookstack:ok0/fail2 mealie:ok0/fail2 nextcloud:ok0/fail2
|
||||
19:41:57 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:42:02 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:42:07 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:42:12 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:42:17 bookstack:ok0/fail2 mealie:ok0/fail2 nextcloud:ok0/fail2
|
||||
19:42:22 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:42:27 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:42:32 bookstack:ok0/fail1 mealie:ok0/fail1 nextcloud:ok0/fail1
|
||||
19:42:37 bookstack:ok0/fail2 mealie:ok0/fail2 nextcloud:ok0/fail2
|
||||
19:42:42 bookstack:ok0/fail2 mealie:ok0/fail2 nextcloud:ok0/fail1
|
||||
19:42:47 bookstack:ok0/fail5 mealie:ok0/fail5 nextcloud:ok0/fail3
|
||||
19:42:52 bookstack:ok0/fail5 mealie:ok0/fail5 nextcloud:ok0/fail2
|
||||
19:42:57 bookstack:ok0/fail4 mealie:ok0/fail4 nextcloud:ok0/fail3
|
||||
19:43:02 bookstack:ok0/fail5 mealie:ok0/fail5 nextcloud:ok0/fail2
|
||||
19:43:07 bookstack:ok0/fail4 mealie:ok0/fail5 nextcloud:ok0/fail1
|
||||
19:43:12 bookstack:ok4/fail0 mealie:ok0/fail4 nextcloud:ok1/fail0
|
||||
19:43:17 bookstack:ok4/fail0 mealie:ok0/fail5 nextcloud:ok1/fail0
|
||||
19:43:22 bookstack:ok4/fail0 mealie:ok0/fail5 nextcloud:ok1/fail0
|
||||
19:43:27 bookstack:ok4/fail0 mealie:ok0/fail4 nextcloud:ok1/fail0
|
||||
19:43:32 bookstack:ok5/fail0 mealie:ok0/fail5 nextcloud:ok0/fail0
|
||||
19:43:37 bookstack:ok4/fail0 mealie:ok0/fail5 nextcloud:ok0/fail2
|
||||
19:43:42 bookstack:ok5/fail0 mealie:ok0/fail5 nextcloud:ok0/fail2
|
||||
|
||||
Reference in New Issue
Block a user