BIGNIGHT phase 4: R-517 (P1, backup page claims a failed PBS tier is current and present), R-518 (apps down ~8 min on 'a few seconds'); backup evidence
gates / gates (push) Successful in 20s

This commit is contained in:
2026-09-14 21:16:16 +02:00
parent e0366f1f05
commit 6aaa3a4a34
12 changed files with 247 additions and 0 deletions
@@ -405,3 +405,27 @@ for the Paperless OOM (R-514).
### Phase 3 interventions: **0** (every seed through the app's own front door or the product's own SMB share;
harness retries are recorded, none needed an act a customer could not make). Rows: R-512, R-513, R-514, R-515, R-516.
Evidence pulled off the box 19:00Z: `box-logs-phase3/`.
## Phase 4 — a month of routines (`phase4/`)
Backup pages read before (`phase4/backup-pages-before.txt`). The nightly's legs, each through its own endpoint, in
the nightly's order (window: db-dump W, Tier-2 W+60 m, off-site W+105 m, whole-system after):
| leg | endpoint | start → end | result |
|---|---|---|---|
| Tier 1 — DB dumps + recovery units („Mentés most" on Alkalmazások) | `POST /api/backup/run` | 19:00:26 → 19:02:38 | `db_dump count 6, 2m8s, success true` |
| Tier 2 — off-drive copies | `POST /api/backup/tier2` (no page calls it; the nightly does) | 19:02:5x → (below) | |
| Tier 3 — off-site | none: not configured on this record (restic off; PBS DR stuck, R-511) | — | not run |
| whole-system (Plane-2 local) | `POST /api/guest-backup/trigger` („Mentés most" on Áttekintés) | (below) | |
Catalog: **no app had a newer version** (12 × template == catalog, all `catalog_since 2026-07-18`). So a real one-step
bump: **privatebin `2.0.5 → 2.0.6`** (upstream Docker Hub 2026-08-08), catalog commit `d5d91e0` pushed **18:59:36Z**
with `catalog_since 2026-09-14`, gates OK. At 19:02:41Z the box still reads 2.0.5 (the 15-minute sync).
R-499 still present on `/stacks/bookstack/backup` („már szerepelnek a teljes rendszermentésben (PBS)").
Tier 2 result (controller log, `phase4/controller-log-t1-t2.txt`): `Tier 2 run complete: 12 app(s) processed` at
19:03:06Z, **17 s**. The eight system-disk apps copied to the data drive (`hdd_1/backups/secondary/…`, e.g. bookstack
154.9 MB); the four drive apps copied to the internal SSD marked **`[SSD: state-only]`** (immich 1.7 GB, nextcloud
1.3 GB, paperless 68.8 MB, jellyfin 0.6 MB) — definitions and databases, not the photos, as the Nextcloud page said
(„csak az adatbázis és a konfiguráció másolódik"). The status API's `running` flag covers only the DB leg, so it read
false during Tier 2 (the harness loop exited on it; the log is the observable).
@@ -0,0 +1,6 @@
POST /api/backup/run 19:00:26
{"ok":true,"message":"Mentés elindítva"}
19:00:26 running=True
19:02:38 running=False
{"ok":true,"data":{"db_dump":{"count":6,"duration":"2m8.275001343s","last_run":"2026-09-14T19:02:34.370521948Z","success":true},"enabled":true,"running":false}}
File diff suppressed because one or more lines are too long
@@ -0,0 +1,8 @@
=== GET /backups 18:59:06
↗ | Biztonsági mentés | enkicsifelhom.hu | A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer — így hibás fájlok ellen véd, lemezhiba ellen nem. Csatlakoztass egy második meghajtót a teljes védelemhez. | Ezt a meghajtót kijelölheted a rendszermentés helyéül — így egy lemezhiba után is vissza tudod állítani a rendszert. | Adatlemez | Kijelölöm | Tárhely áttekintés | Rendszer (/) | 19.5 GB / 68.7 GB (28%) | Adatlemez | 1.00 GB / 97.9 GB (1%) | DB mentések | – | Rendszermentés (teljes mentés) | A teljes szerver — alkalmazások, beállítások és adatbázisok együtt — időszakos mentése, amelyből az egész készülék visszaállítható. Ezt a host-ügynök készíti és kezeli. | ✓ | Utolsó teljes mentés | 2026-09-14 20:04 (55 perce) | 624.7 MB | Helyi tároló (local) | Naprakész | Következő mentés | 0 órája — a mentési ablakon belül | – | Visszaállítás ellenőrizve | Még nem futott | Mentés most | Pillanatkép-mód: az alkalmazások csak néhány másodpercre állnak le. | Mentési időablak | A mentések egymás után futnak: adatbázis-mentés, helyi másolat, távoli mentés, majd a teljes rendszermentés. | Mentési időablak kezdete | Mentés | Adatbázis-mentés: | 02:30 | Helyi másolat: | 03:30 | Távoli mentés: | 04:15 | Teljes rendszermentés: kb. | 04:30–08:30 | között | – | Adatmentés | – | Távoli rendszermentés | nincs beállítva | 0 | Adatbázis mentve
=== GET /backups/apps 18:59:06
↗ | Biztonsági mentés — Alkalmazások | enkicsifelhom.hu | Alkalmazás-mentések (adatbázis + konfiguráció) | Az egyes alkalmazások részletes, granulált mentése — adatbázis-kiírások, beállítások és alkalmazás-fájlok. A fenti teljes mentéstől függetlenül, alkalmazásonként visszaállítható. | Ütemezés | Adatbázis mentés | 02:30 | Következő: holnap 02:30 | Utolsó adatbázis mentés: | Még nem futott | Mentés most | Adatbázisok | Alkalmazás | Típus | Méret | Utolsó | Érvényesítés | Állapot | Alkalmazások mentési állapota | AdventureLog | Konfig + DB + Adatok | ▶ | 1. mentés | Auto | helyi | Utolsó: 10 perce | — | DB + Konfig + Adatok | 2. mentés | 1. mentés auto | Nincs 2. (off-drive) másolat | Beállítás | 3. mentés | Nincs beállítva | távoli (offsite) | Nincs távoli mentési cél beállítva | Beállítás | BookStack | Konfig + DB + Adatok | ▶ | 1. mentés | Auto | helyi | Utolsó: 40 perce | — | DB + Konfig + Adatok | 2. mentés | 1. mentés auto | Nincs 2. (off-drive) másolat | Beállítás | 3. mentés | Nincs beállítva | távoli (offsite) | Nincs távoli mentési cél beállítva | Beállítás | Docmost | Konfig + DB + Adatok | ▶ | 1. mentés | Auto | helyi | Utolsó: 40 perce | — | DB + Konfig + Adatok | 2. mentés | 1. mentés auto | Nincs 2. (off-drive) másolat | Beállítás | 3. mentés | Nincs beállítva | távoli (offsite) | Nincs távoli mentési cél beállítva | Beállítás | Gokapi | Konfig + Adatok | ▶ | 1. mentés | Auto | helyi | Utolsó: 40 perce | — | Konfig + Adatok | 2. mentés | 1. mentés auto | Nincs 2. (off-drive) másolat | Beállítás | 3. mentés | Nincs beállítva | távoli (offsite) | Nincs távoli mentési cél beállítva | Beállítás | Immich | Adatlemez | 531.8 MB | ▶ | 1. mentés | Auto | helyi | Utolsó: 35 perce | — | DB + Konfig + Adatok | 2. mentés | 1. mentés auto | Nincs 2. (off-drive) másolat | Beállítás | 3. mentés | Nincs beállítva | távoli (offsite) | Nincs távoli mentési cél beállítva | Beállítás | Jellyfin | Konfig + Adatok | ▶ | 1. mentés | Auto | helyi | Utolsó: 25 perce | — | Konfig + Adatok | 2. mentés | 1. mentés auto | Nincs 2. (off-drive) másolat | Beállítás | 3. mentés | Nincs beállítva | távoli (offsite) | Nincs távoli mentési cél beállítva | Beállítás | Mealie | Konfig + Adatok | ▶ | 1. mentés | Auto | helyi | Utolsó: 25 perce | — | Konfig + Adatok | 2. mentés | 1. mentés auto | Nincs 2. (off-drive) másolat | Beállítás | 3. mentés | Nincs beállítva | távoli (offsite) | Nincs távoli mentési cél beállítva | Beállítás | Nextcloud | Adatlemez | 389.6 MB | ▶ | 1. mentés | Auto | helyi | Utolsó: 35 perce | — | DB + Konfig + Adatok | 2. mentés | 1. mentés auto | Nincs 2. (off-drive) másolat | Beállítás | 3. mentés | Nincs beállítva | távoli (offsite) | Nincs távoli mentési cél beállítva | Beállítás | Paperless-ngx | Adatlemez | 24.0 KB | ▶ | 1. mentés | Auto | helyi | Utolsó: 30 perce | — | DB + Konfig + Adatok | 2. mentés | 1. mentés auto | Nincs 2. (off-drive) másolat | Beállítás | 3. mentés | Nincs beállítva | távoli (offsite) | Nincs távoli mentési cél beállítva | Beállítás | PrivateBin | Konfig + Adatok | ▶ | 1. mentés | Auto | helyi | Utolsó: 40 perce | — | Konfig + Adatok | 2. mentés | 1. mentés auto | Nincs 2. (off-drive) másolat | Beállítás | 3. mentés | Nincs beállítva | távoli (offsite) | Nincs távoli mentési cél beállítva | Beállítás | Uptime Kuma | Konfig + Adatok | ▶ | 1. mentés | Auto | helyi | Utolsó: 15 perce | — | Konfig + Adatok | 2. mentés | 1. mentés auto | Nincs 2. (off-drive) másolat | Beállítás | 3. mentés | Ninc
=== GET /backups/remote 18:59:06
↗ | Biztonsági mentés — Távoli mentés | enkicsifelhom.hu | Felhom offsite tárhely — igényelhető szolgáltatás. Titkosított, házon kívüli másolat a mentéseidről. Érdeklődj az üzemeltetőnél. | Távoli mentés (3. mentés) — titkosított, offsite | Az alkalmazás-mentések titkosított másolata egy távoli tárolóra — saját NAS vagy Felhom offsite tárhely — restic + SFTP kapcsolaton. A tároló csak titkosított adatot lát. Ez a 3-2-1 szabály „1 off-site" lába — független a helyi másodpéldánytól és a teljes rendszermentéstől. | Még nincs beállítva távoli mentési cél. | Távoli mentési cél beállítása | Cél címe (IP vagy hosztnév — NAS vagy SFTP-kiszolgáló) | SSH port | Felhasználó | Tároló útvonala a célgépen | Napi automatikus távoli mentés engedélyezve | SSH privát kulcs | A kulcsot 0600-as fájlba írjuk; sosem naplózzuk és nem tároljuk a beállításokban. | A célgép ismert-host sora (known_hosts) | A host-kulcs rögzítése (no blind TOFU). Lekérdezhető: | ssh-keyscan -p <port> <host> | Mentés
=== GET /backups/restore 18:59:06
↗ | Biztonsági mentés — Visszaállítás | enkicsifelhom.hu | Visszaállítás | Alkalmazás: | — Válasszon — | AdventureLog | BookStack | Docmost | Gokapi | Immich | Jellyfin | Mealie | Nextcloud | Paperless-ngx | PrivateBin | Uptime Kuma | Vaultwarden | Pillanatkép: | — Válasszon alkalmazást — | Még nincs mentés felhasználói adattal. | A visszaállítás felülírja az alkalmazás jelenlegi adatait a kiválasztott mentés állapotával. | Az alkalmazás a folyamat során automatikusan leáll és újraindul. | Megértettem, visszaállítás indítása. | Visszaállítás indítása | Importálás mentett csomagból (.fab) | Hordozható mentéscsomag (.fab) | Hordozható pillanatfelvétel — bárhol tárolhatod, és bármikor visszatöltheted egy meghajtóról. A folyamatos védelmet az 1–3. szintű mentés adja. | Jelszavas titkosítás (opcionális) | Üresen hagyva a csomag titkosítás nélkül készül. | AdventureLog | Letöltés (.fab) | BookStack | Letöltés (.fab) | Docmost | Letöltés (.fab) | Gokapi | Letöltés (.fab) | Immich | Letöltés (.fab) | Jellyfin | Letöltés (.fab) | Mealie | Letöltés (.fab) | Nextcloud | Letöltés (.fab) | Paperless-ngx | Letöltés (.fab) | PrivateBin | Letöltés (.fab) | Uptime Kuma | Letöltés (.fab) | Vaultwarden | Letöltés (.fab) | Összes letöltése (egyenként) | A csomagok egyenként készülnek és töltődnek le — pillanatfelvétel a mostani állapotról.
@@ -0,0 +1,80 @@
2026/09/14 19:01:56 [INFO] [stacks] Starting stack: nextcloud
2026/09/14 19:02:03 [INFO] [stacks] Stack nextcloud started successfully (took 6.4s)
2026/09/14 19:02:03 [INFO] [backup] Stopping paperless-ngx for safe volume dump
2026/09/14 19:02:03 [INFO] [stacks] Stopping stack: paperless-ngx
2026/09/14 19:02:06 [INFO] [stacks] Stack nextcloud post-start status:
2026/09/14 19:02:06 [INFO] [stacks] nextcloud nextcloud:34.0.1-apache running Up 3 seconds (health: starting)
2026/09/14 19:02:11 [INFO] [stacks] Stack paperless-ngx stopped successfully (took 7.9s)
2026/09/14 19:02:11 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 126.0 KB
2026/09/14 19:02:12 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 68.2 MB
2026/09/14 19:02:12 [INFO] [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_redis_data → 135.0 KB
2026/09/14 19:02:12 [INFO] [backup] Restarting paperless-ngx after volume dump
2026/09/14 19:02:12 [INFO] [stacks] Starting stack: paperless-ngx
2026/09/14 19:02:24 [INFO] [stacks] Stack paperless-ngx started successfully (took 11.2s)
2026/09/14 19:02:24 [INFO] [backup] Stopping privatebin for safe volume dump
2026/09/14 19:02:24 [INFO] [stacks] Stopping stack: privatebin
2026/09/14 19:02:24 [INFO] [stacks] Stack privatebin stopped successfully (took 0.6s)
2026/09/14 19:02:25 [INFO] [backup] Volume dump: privatebin/privatebin_privatebin_data → 18.5 KB
2026/09/14 19:02:25 [INFO] [backup] Restarting privatebin after volume dump
2026/09/14 19:02:25 [INFO] [stacks] Starting stack: privatebin
2026/09/14 19:02:26 [INFO] [stacks] Stack privatebin started successfully (took 0.5s)
2026/09/14 19:02:26 [INFO] [backup] Stopping uptime-kuma for safe volume dump
2026/09/14 19:02:26 [INFO] [stacks] Stopping stack: uptime-kuma
2026/09/14 19:02:27 [INFO] [stacks] Stack paperless-ngx post-start status:
2026/09/14 19:02:27 [INFO] [stacks] paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 running Up 3 seconds (health: starting)
2026/09/14 19:02:29 [INFO] [stacks] Stack privatebin post-start status:
2026/09/14 19:02:29 [INFO] [stacks] privatebin privatebin/pdo:2.0.5 running Up 3 seconds (health: starting)
2026/09/14 19:02:31 [INFO] [stacks] Stack uptime-kuma stopped successfully (took 4.5s)
2026/09/14 19:02:31 [INFO] [backup] Volume dump: uptime-kuma/uptime-kuma_uptime_kuma_data → 356.5 KB
2026/09/14 19:02:31 [INFO] [backup] Restarting uptime-kuma after volume dump
2026/09/14 19:02:31 [INFO] [stacks] Starting stack: uptime-kuma
2026/09/14 19:02:32 [INFO] [stacks] Stack uptime-kuma started successfully (took 0.4s)
2026/09/14 19:02:32 [INFO] [backup] Stopping vaultwarden for safe volume dump
2026/09/14 19:02:32 [INFO] [stacks] Stopping stack: vaultwarden
2026/09/14 19:02:32 [INFO] [stacks] Stack vaultwarden stopped successfully (took 0.4s)
2026/09/14 19:02:33 [INFO] [backup] Volume dump: vaultwarden/vaultwarden_vaultwarden_data → 739.0 KB
2026/09/14 19:02:33 [INFO] [backup] Restarting vaultwarden after volume dump
2026/09/14 19:02:33 [INFO] [stacks] Starting stack: vaultwarden
2026/09/14 19:02:33 [INFO] [stacks] Stack vaultwarden started successfully (took 0.5s)
2026/09/14 19:02:34 [INFO] [backup] App-data backup completed: 6 databases (52.7 MB total), 12 volume dump(s) (2m8.275s)
2026/09/14 19:02:34 [INFO] [backup] Recovery unit captured for adventurelog → /mnt/sys_drive/felhom-data/backups/primary/adventurelog (images=3, secrets-referenced=2, data_keys=1, portable-carried=2/2, withheld=0)
2026/09/14 19:02:34 [INFO] [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=2, data_keys=0, portable-carried=2/2, withheld=0)
2026/09/14 19:02:34 [INFO] [backup] Recovery unit captured for docmost → /mnt/sys_drive/felhom-data/backups/primary/docmost (images=3, secrets-referenced=2, data_keys=0, portable-carried=2/2, withheld=0)
2026/09/14 19:02:34 [INFO] [backup] Recovery unit captured for gokapi → /mnt/sys_drive/felhom-data/backups/primary/gokapi (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1)
2026/09/14 19:02:34 [INFO] [backup] Recovery unit captured for immich → /mnt/felhom-drives/hdd_1/backups/primary/immich (images=4, secrets-referenced=1, data_keys=0, portable-carried=1/1, withheld=0)
2026/09/14 19:02:34 [INFO] [backup] Recovery unit captured for jellyfin → /mnt/felhom-drives/hdd_1/backups/primary/jellyfin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/09/14 19:02:34 [INFO] [backup] Recovery unit captured for mealie → /mnt/sys_drive/felhom-data/backups/primary/mealie (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/09/14 19:02:34 [INFO] [backup] Recovery unit captured for nextcloud → /mnt/felhom-drives/hdd_1/backups/primary/nextcloud (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1)
2026/09/14 19:02:34 [INFO] [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1)
2026/09/14 19:02:34 [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/09/14 19:02:34 [INFO] [backup] Recovery unit captured for uptime-kuma → /mnt/sys_drive/felhom-data/backups/primary/uptime-kuma (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/09/14 19:02:34 [INFO] [backup] Recovery unit captured for vaultwarden → /mnt/sys_drive/felhom-data/backups/primary/vaultwarden (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1)
2026/09/14 19:02:35 [INFO] [stacks] Stack uptime-kuma post-start status:
2026/09/14 19:02:35 [INFO] [stacks] uptime-kuma louislam/uptime-kuma:2.4.0 running Up 3 seconds (health: starting)
2026/09/14 19:02:37 [INFO] [stacks] Stack vaultwarden post-start status:
2026/09/14 19:02:37 [INFO] [stacks] vaultwarden vaultwarden/server:1.36.0-alpine running Up 3 seconds (health: starting)
2026/09/14 19:02:49 [INFO] [api] Manual Tier 2 (off-drive) backup triggered
2026/09/14 19:02:50 [INFO] [backup] Tier 2 copied adventurelog → /mnt/felhom-drives/hdd_1/backups/secondary/adventurelog (112.2 MB, 0 leg(s), 1s)
2026/09/14 19:02:53 [INFO] [backup] Tier 2 copied bookstack → /mnt/felhom-drives/hdd_1/backups/secondary/bookstack (154.9 MB, 0 leg(s), 3s)
2026/09/14 19:02:53 [INFO] [backup] Tier 2 copied docmost → /mnt/felhom-drives/hdd_1/backups/secondary/docmost (50.3 MB, 0 leg(s), 0s)
2026/09/14 19:02:53 [INFO] [backup] Tier 2 copied gokapi → /mnt/felhom-drives/hdd_1/backups/secondary/gokapi (51.4 MB, 0 leg(s), 0s)
2026/09/14 19:03:01 [INFO] [backup] Tier 2 copied immich → /mnt/sys_drive/felhom-data/backups/secondary/immich (1.7 GB, 1 leg(s), 7s) [SSD: state-only]
2026/09/14 19:03:01 [INFO] [backup] Tier 2 copied jellyfin → /mnt/sys_drive/felhom-data/backups/secondary/jellyfin (614.7 KB, 0 leg(s), 0s) [SSD: state-only]
2026/09/14 19:03:01 [INFO] [backup] Tier 2 copied mealie → /mnt/felhom-drives/hdd_1/backups/secondary/mealie (1.4 MB, 0 leg(s), 0s)
2026/09/14 19:03:05 [INFO] [backup] Tier 2 copied nextcloud → /mnt/sys_drive/felhom-data/backups/secondary/nextcloud (1.3 GB, 1 leg(s), 4s) [SSD: state-only]
2026/09/14 19:03:05 [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sys_drive/felhom-data/backups/secondary/paperless-ngx (68.8 MB, 1 leg(s), 0s) [SSD: state-only]
2026/09/14 19:03:05 [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/hdd_1/backups/secondary/privatebin (34.9 KB, 0 leg(s), 0s)
2026/09/14 19:03:06 [INFO] [backup] Tier 2 copied uptime-kuma → /mnt/felhom-drives/hdd_1/backups/secondary/uptime-kuma (373.4 KB, 0 leg(s), 0s)
2026/09/14 19:03:06 [INFO] [backup] Tier 2 copied vaultwarden → /mnt/felhom-drives/hdd_1/backups/secondary/vaultwarden (759.3 KB, 0 leg(s), 0s)
2026/09/14 19:03:06 [INFO] [backup] Tier 2 run complete: 12 app(s) processed (incl. volume-only — F6)
2026/09/14 19:03:06 [WARN] [shares] tier-2: share beolvasas is not under a registered storage root — skipped
2026/09/14 19:03:06 [INFO] [shares] tier-2 copied 1 share(s) from /mnt/felhom-drives/hdd_1 → /mnt/sys_drive/felhom-data/backups/secondary/_shares (94.3 MB)
2026/09/14 19:03:06 [INFO] [shares] tier-2 run complete: 1 share leg(s), 94.3 MB, 0s
2026/09/14 19:03:23 [INFO] [web] manual whole-guest backup triggered (quiesce loop)
2026/09/14 19:03:23 [INFO] [quiesce] manual backup requested — quiescing now
2026/09/14 19:03:23 [INFO] [quiesce] backup due on 2 tier(s) — quiescing 12 stack(s): [adventurelog bookstack docmost gokapi immich jellyfin mealie nextcloud paperless-ngx privatebin uptime-kuma vaultwarden]
2026/09/14 19:03:23 [INFO] [stacks] Stopping stack: adventurelog
2026/09/14 19:03:27 [INFO] [stacks] Stack adventurelog stopped successfully (took 4.0s)
2026/09/14 19:03:27 [INFO] [stacks] Stopping stack: bookstack
2026/09/14 19:03:30 [INFO] [stacks] Stack bookstack stopped successfully (took 3.8s)
2026/09/14 19:03:30 [INFO] [stacks] Stopping stack: docmost
@@ -0,0 +1,79 @@
2026/09/14 19:03:23 [INFO] [web] manual whole-guest backup triggered (quiesce loop)
2026/09/14 19:03:23 [INFO] [quiesce] manual backup requested — quiescing now
2026/09/14 19:03:23 [INFO] [quiesce] backup due on 2 tier(s) — quiescing 12 stack(s): [adventurelog bookstack docmost gokapi immich jellyfin mealie nextcloud paperless-ngx privatebin uptime-kuma vaultwarden]
2026/09/14 19:03:23 [INFO] [stacks] Stopping stack: adventurelog
2026/09/14 19:03:27 [INFO] [stacks] Stack adventurelog stopped successfully (took 4.0s)
2026/09/14 19:03:27 [INFO] [stacks] Stopping stack: bookstack
2026/09/14 19:03:30 [INFO] [stacks] Stack bookstack stopped successfully (took 3.8s)
2026/09/14 19:03:30 [INFO] [stacks] Stopping stack: docmost
2026/09/14 19:03:32 [INFO] [stacks] Stack docmost stopped successfully (took 1.1s)
2026/09/14 19:03:32 [INFO] [stacks] Stopping stack: gokapi
2026/09/14 19:03:32 [INFO] [stacks] Stack gokapi stopped successfully (took 0.2s)
2026/09/14 19:03:32 [INFO] [stacks] Stopping stack: immich
2026/09/14 19:03:33 [INFO] [stacks] Stack immich stopped successfully (took 0.8s)
2026/09/14 19:03:33 [INFO] [stacks] Stopping stack: jellyfin
2026/09/14 19:03:33 [INFO] [stacks] Stack jellyfin stopped successfully (took 0.5s)
2026/09/14 19:03:33 [INFO] [stacks] Stopping stack: mealie
2026/09/14 19:03:35 [INFO] [stacks] Stack mealie stopped successfully (took 2.0s)
2026/09/14 19:03:35 [INFO] [stacks] Stopping stack: nextcloud
2026/09/14 19:03:37 [INFO] [stacks] Stack nextcloud stopped successfully (took 1.8s)
2026/09/14 19:03:37 [INFO] [stacks] Stopping stack: paperless-ngx
2026/09/14 19:03:44 [INFO] [stacks] Stack paperless-ngx stopped successfully (took 6.9s)
2026/09/14 19:03:44 [INFO] [stacks] Stopping stack: privatebin
2026/09/14 19:03:44 [INFO] [stacks] Stack privatebin stopped successfully (took 0.3s)
2026/09/14 19:03:44 [INFO] [stacks] Stopping stack: uptime-kuma
2026/09/14 19:03:49 [INFO] [stacks] Stack uptime-kuma stopped successfully (took 4.3s)
2026/09/14 19:03:49 [INFO] [stacks] Stopping stack: vaultwarden
2026/09/14 19:03:49 [INFO] [stacks] Stack vaultwarden stopped successfully (took 0.3s)
2026/09/14 19:03:49 [INFO] [quiesce] tier local: backup job backup-9201-1789412629454125782 started — polling
2026/09/14 19:04:00 [INFO] [quiesce] a backup cycle is already running — skipping this scheduled check
2026/09/14 19:04:01 [INFO] [scheduler] Running job: backup-cache
2026/09/14 19:04:01 [INFO] [backup] Found 6 DB dump files across drives
2026/09/14 19:04:01 [INFO] [backup] Discovered 0 databases
2026/09/14 19:04:01 [INFO] [backup] Discovered app data: 12 apps
2026/09/14 19:04:01 [INFO] [backup] Backup status cache refreshed
2026/09/14 19:04:01 [INFO] [scheduler] Job backup-cache completed (took 181ms)
2026/09/14 19:09:00 [INFO] [quiesce] a backup cycle is already running — skipping this scheduled check
2026/09/14 19:09:01 [INFO] [scheduler] Running job: backup-cache
2026/09/14 19:09:01 [INFO] [backup] Found 6 DB dump files across drives
2026/09/14 19:09:01 [INFO] [backup] Discovered 0 databases
2026/09/14 19:09:01 [INFO] [backup] Discovered app data: 12 apps
2026/09/14 19:09:01 [INFO] [backup] Backup status cache refreshed
2026/09/14 19:09:01 [INFO] [scheduler] Job backup-cache completed (took 522ms)
2026/09/14 19:09:59 [INFO] [quiesce] tier local: backup job backup-9201-1789412629454125782 done — next tier may start (app still quiesced)
2026/09/14 19:09:59 [INFO] [quiesce] tier felhom-pbs: backup job backup-9201-felhom-pbs-1789412999526947801 started — polling
2026/09/14 19:10:09 [INFO] [quiesce] unquiescing (backup failed): restarting 12 stack(s)
2026/09/14 19:10:09 [INFO] [stacks] Starting stack: adventurelog
2026/09/14 19:10:16 [INFO] [stacks] Stack adventurelog started successfully (took 6.6s)
2026/09/14 19:10:16 [INFO] [stacks] Starting stack: bookstack
2026/09/14 19:10:22 [INFO] [stacks] Stack bookstack started successfully (took 6.4s)
2026/09/14 19:10:22 [INFO] [stacks] Starting stack: docmost
2026/09/14 19:10:34 [INFO] [stacks] Stack docmost started successfully (took 11.7s)
2026/09/14 19:10:34 [INFO] [stacks] Starting stack: gokapi
2026/09/14 19:10:35 [INFO] [stacks] Stack gokapi started successfully (took 0.5s)
2026/09/14 19:10:35 [INFO] [stacks] Starting stack: immich
2026/09/14 19:10:46 [INFO] [stacks] Stack immich started successfully (took 11.3s)
2026/09/14 19:10:47 [INFO] [stacks] Starting stack: jellyfin
2026/09/14 19:10:47 [INFO] [stacks] Stack jellyfin started successfully (took 0.5s)
2026/09/14 19:10:47 [INFO] [stacks] Starting stack: mealie
2026/09/14 19:10:48 [INFO] [stacks] Stack mealie started successfully (took 0.6s)
2026/09/14 19:10:48 [INFO] [stacks] Starting stack: nextcloud
2026/09/14 19:10:55 [INFO] [stacks] Stack nextcloud started successfully (took 6.9s)
2026/09/14 19:10:56 [INFO] [stacks] Starting stack: paperless-ngx
2026/09/14 19:11:08 [INFO] [stacks] Stack paperless-ngx started successfully (took 12.2s)
2026/09/14 19:11:08 [INFO] [stacks] Starting stack: privatebin
2026/09/14 19:11:09 [INFO] [stacks] Stack privatebin started successfully (took 0.6s)
2026/09/14 19:11:09 [INFO] [stacks] Starting stack: uptime-kuma
2026/09/14 19:11:10 [INFO] [stacks] Stack uptime-kuma started successfully (took 1.0s)
2026/09/14 19:11:11 [INFO] [stacks] Starting stack: vaultwarden
2026/09/14 19:11:12 [INFO] [stacks] Stack vaultwarden started successfully (took 1.1s)
2026/09/14 19:11:12 [WARN] [quiesce] tier felhom-pbs: backup job backup-9201-felhom-pbs-1789412999526947801 failed
2026/09/14 19:11:12 [WARN] [quiesce] tier felhom-pbs has now failed 1 time(s) in a row — deferring its next quiesce by 15m0s (cap 4h0m0s) so the apps are not stopped again for a backup that cannot succeed
2026/09/14 19:11:12 [INFO] Event pushed: whole_guest_backup_failed (error) — Whole-guest backup FAILED on the felhom-pbs tier — retrying with backoff (next attempt in 15m0s)
2026/09/14 19:14:01 [INFO] [scheduler] Running job: backup-cache
2026/09/14 19:14:01 [INFO] [backup] Found 6 DB dump files across drives
2026/09/14 19:14:01 [INFO] [backup] Discovered 6 databases
2026/09/14 19:14:01 [INFO] [backup] Discovered app data: 12 apps
2026/09/14 19:14:01 [INFO] [backup] Recovery unit captured for privatebin → /mnt/sys_drive/felhom-data/backups/primary/privatebin (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
2026/09/14 19:14:01 [INFO] [backup] Backup status cache refreshed
2026/09/14 19:14:01 [INFO] [scheduler] Job backup-cache completed (took 492ms)
@@ -0,0 +1,9 @@
POST /api/guest-backup/trigger 19:03:23
{"data":{"started":true},"ok":true}
19:03:23 phase=done
{"data":{"backup":{"target_id":"local","vmid":9201,"archive":"local:backup/vzdump-lxc-9201-2026_09_14-20_04_01.tar.zst","mode":"snapshot","crash_consistent":true,"size_bytes":655026777,"success":true,"started_at":"2026-09-14T18:04:01Z","duration_seconds":26.176726096},"error":"","job_id":"backup-9201-1789409041243366069","phase":"done"},"ok":true}
(harness: the first poll read the previous 18:04 job's 'done' and stopped; polling for a NEW job id)
19:03:46 job=43366069 phase=done
19:03:51 job=54125782 phase=snapshotted
19:09:52 job=54125782 phase=done
{"data":{"backup":{"target_id":"local","vmid":9201,"archive":"local:backup/vzdump-lxc-9201-2026_09_14-21_03_49.tar.zst","mode":"snapshot","crash_consistent":true,"size_bytes":8877619753,"success":true,"started_at":"2026-09-14T19:03:49Z","duration_seconds":362.208662908},"error":"","job_id":"backup-9201-1789412629454125782","phase":"done"},"ok":true}
@@ -0,0 +1,19 @@
=== hub log tester-1 since 19:00Z
2026/09/14 21:02:50 [INFO] Event from tester-1: crossdrive_completed (info) — Másodlagos mentés elkészült: adventurelog
2026/09/14 21:02:53 [INFO] Event from tester-1: crossdrive_completed (info) — Másodlagos mentés elkészült: bookstack
2026/09/14 21:02:53 [INFO] Event from tester-1: crossdrive_completed (info) — Másodlagos mentés elkészült: docmost
2026/09/14 21:02:53 [INFO] Event from tester-1: crossdrive_completed (info) — Másodlagos mentés elkészült: gokapi
2026/09/14 21:03:01 [INFO] Event from tester-1: crossdrive_completed (info) — Másodlagos mentés elkészült: immich
2026/09/14 21:03:01 [INFO] Event from tester-1: crossdrive_completed (info) — Másodlagos mentés elkészült: jellyfin
2026/09/14 21:03:01 [INFO] Event from tester-1: crossdrive_completed (info) — Másodlagos mentés elkészült: mealie
2026/09/14 21:03:05 [INFO] Event from tester-1: crossdrive_completed (info) — Másodlagos mentés elkészült: nextcloud
2026/09/14 21:03:05 [INFO] Event from tester-1: crossdrive_completed (info) — Másodlagos mentés elkészült: paperless-ngx
2026/09/14 21:03:05 [INFO] Event from tester-1: crossdrive_completed (info) — Másodlagos mentés elkészült: privatebin
2026/09/14 21:03:06 [INFO] Event from tester-1: crossdrive_completed (info) — Másodlagos mentés elkészült: uptime-kuma
2026/09/14 21:03:06 [INFO] Event from tester-1: crossdrive_completed (info) — Másodlagos mentés elkészült: vaultwarden
2026/09/14 21:03:06 [INFO] Event from tester-1: crossdrive_completed (info) — Másodlagos mentés elkészült: Megosztások
2026/09/14 21:11:12 [INFO] Event from tester-1: whole_guest_backup_failed (error) — Whole-guest backup FAILED on the felhom-pbs tier — retrying with backoff (next attempt in 15m0s)
2026/09/14 21:11:13 [INFO] Operator email sent for tester-1/whole_guest_backup_failed
2026/09/14 21:12:43 [WARN] host tester-1-a61396 backup FAILED: target=felhom-pbs vmid=9201 err="proxmox: task UPID:felhom:00023B4C:0007D509:6AA84687:vzdump:9201:felhom-agent@pve!agent: failed: exitstatus \"could not activate storage 'felhom-pbs': storage 'felhom-pbs' does not exist could not get storage information for 'felhom-pbs': storage 'felhom-pbs' does not exist\""
=== hub customer page 19:15:31
Last report: 2 min ago · Controller 0.242.0 | Auto-refresh | (paused) | Tester 1 | ok | Controller | 0.242.0 | Last report 2 min ago | Containers 24/24 | Overview | Applications | Setup | Edit | Backup & DR | Events | 2 | Notifications | Host | Customer Info | Customer ID | tester-1 | Name | Tester 1 | Domain | enkicsifelhom.hu | Email | tester1@felhom.eu | Config | MANAGED | Config Created | 32d ago | Health | Status: ok | System | Hostname | tester-1 | OS | Debian GNU/Linux 12 (bookworm) | Kernel | 7.0.2-6-pve | CPU | AMD Ryzen Embedded V1756B with Radeon Vega Gfx (4 cores) | CPU | 7% | Memory | 31% | Storage | SSD | 33% | 23.0 / 68.7 GB | Adatlemez | 4% | 3.5 / 97.9 GB | Backup | App data (local) | Enabled | Last DB dump | 13 min ago | No off-site data reported — the controller has never sent an | offsite | object (no off-site tier, or a controller older than v0.109.0). | This is not the same as "no backups" | ; it means this page cannot say. | Containers (24/24) | Name | State | CPU | Memory | adventurelog | running | 0.1% | 236 MB | adventurelog-postgres | running | 0.0% | 46 MB | adventurelog-frontend | running | 0.0% | 127 MB | bookstack | running | 0.0% | 122 MB | bookstack-db | running | 0.0% | 141 MB | docmost | running | 0.6% | 242 MB | docmost-redis | running | 0.7% | 23 MB | docmost-postgres | running | 0.0% | 27 MB | gokapi | running | 0.0% | 33 MB | immich-server | running | 0.1% | 872 MB | immich-machine-learning | running | 0.2% | 594 MB | immich-postgres | running | 0.0% | 127 MB | immich-redis | running | 0.7% | 5 MB | jellyfin | running | 0.0% | 399 MB | mealie | running | 0.2% | 424 MB | nextcloud | running | 6.5% | 156 MB | nextcloud-redis | running | 0.5% | 4 MB | nextcloud-db | running | 0.1% | 151 MB | paperless-webserver | running | 0.1% | 628 MB | paperless-postgres | running | 3.7% | 26 MB | paperless-redis | running | 0.5% | 4 MB | privatebin | running | 0.0% | 55 MB | uptime-kuma | running | 0.5% | 236 MB | vaultwarden | running | 0.0% | 80 MB | App Telemetry | (last 7 days) | App | Memory (current) | Memory (avg 7d) | Memory (peak 7d) | Catalog Limit | Errors | Warnings | Logs | Immich | 1599 MB | 1640 MB | 2879 MB | 4096M | 167 | 48 | Request log tail | Paperless-ngx | 658 MB | 570 MB | 808 MB | 1152M | 444 | 46 | Request log tail | Docmost | 292 MB | 379 MB | 420 MB | 768M | 0 | 18 | Request log tail | AdventureLog | 408 MB | 342 MB | 457 MB | 384M | 50 | 3 | Request log tail | Nextcloud | 311 MB | 338 MB | 525 MB | 1024M | 20 | 45 | Request log tail | Mealie | 424 MB | 279 MB | 424 MB | 1000M | 14 | 8 | Request log tail
@@ -0,0 +1,5 @@
=== /backups/remote 19:04:01
↗ | Biztonsági mentés — Távoli mentés | enkicsifelhom.hu | Felhom offsite tárhely — igényelhető szolgáltatás. Titkosított, házon kívüli másolat a mentéseidről. Érdeklődj az üzemeltetőnél. | Távoli mentés (3. mentés) — titkosított, offsite | Az alkalmazás-mentések titkosított másolata egy távoli tárolóra — saját NAS vagy Felhom offsite tárhely — restic + SFTP kapcsolaton. A tároló csak titkosított adatot lát. Ez a 3-2-1 szabály „1 off-site" lába — független a helyi másodpéldánytól és a teljes rendszermentéstől. | Még nincs beállítva távoli mentési cél. | Távoli mentési cél beállítása | Cél címe (IP vagy hosztnév — NAS vagy SFTP-kiszolgáló) | SSH port | Felhasználó | Tároló útvonala a célgépen | Napi automatikus távoli mentés engedélyezve | SSH privát kulcs | A kulcsot 0600-as fájlba írjuk; sosem naplózzuk és nem tároljuk a beállításokban. | A célgép ismert-host sora (known_hosts) | A host-kulcs rögzítése (no blind TOFU). Lekérdezhető: | ssh-keyscan -p <port> <host> | Mentés
=== GET /backup/offbox/status
{"orphaned":false,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":0,"current_file":"","elapsed_sec":0,"phase":""},"snapshots":0,"status":""}
@@ -0,0 +1,6 @@
(seed file holds the three pastes of the last seeding run: 1month, never, 10min; the first two pastes' keys were not kept by the harness)
19:04:23 16b07513ec2946f2 1month not JSON — HTTP 404 (expired or gone)
19:04:23 eb1a716a315d7329 never not JSON — HTTP 404 (expired or gone)
19:04:23 32e4908873822f0e 10min not JSON — HTTP 404 (expired or gone)
19:04:23 pastes in seed file 3 read back equal 0
19:04:23 page version marker []
@@ -0,0 +1,5 @@
POST /api/backup/tier2 19:02:49
{"ok":true,"message":"2. mentés elindítva"}
19:02:49 running=False
{"ok":true,"data":{"db_dump":{"count":6,"duration":"2m8.275001343s","last_run":"2026-09-14T19:02:34.370521948Z","success":true},"enabled":true,"running":false}}
+2
View File
@@ -720,6 +720,8 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
| **R-514** | **[P2-MEDIUM] Paperless-ngx dies silently when a family uploads 20 documents at once: its worker is OOM-killed inside the catalog's 768 MB cap, 11 uploads fail, 8 wait forever, and the app still reads „Fut".** MEASURED 2026-09-14 (BIGNIGHT, VM 333, catalog paperless-ngx 2.20.15, `paperless-webserver` limit 805 306 368 B): 20 small 3-page PDFs posted through `/api/documents/post_document/` at 18:29:57Z. At 18:31:22Z the VM kernel logged `Memory cgroup out of memory: Killed process … (gs)` ×2, `([celeryd: celer)`, `([celery beat] -)` — constraint MEMCG of that container; `docker inspect` → `oomkilled=true restarts=0`. Paperless's own task list: **11 FAILURE (`WorkerLostError`), 1 STARTED, 8 PENDING**, unchanged 13 minutes later; **0 documents**. The controller shows the app running and healthy; no event, no alarm (`phase3/paperless-tasks.txt`, `paperless-oom-check.txt`). A household scanning a drawer of bills sees nothing arrive and no reason. **Fix shape:** raise the cap or set `PAPERLESS_TASK_WORKERS=1` / `PAPERLESS_THREADS_PER_WORKER=1` in the template, and let the dead-app/health check see an OOM-killed worker. | **READY — rank P2-MEDIUM; owner: CC (catalog)** |
| **R-515** | **[P3-LOW] The Paperless-ngx app page tells the customer to log in with `admin / admin`, and that login does not exist: the deploy form generates the admin password.** MEASURED 2026-09-14 (BIGNIGHT, VM 333): `/apps/paperless-ngx` „Első lépések — Jelentkezz be: admin / admin" and „Alapértelmezett belépés admin / admin"; the catalog template generates `PAPERLESS_ADMIN_PASSWORD` (`password:16`) and the token call with the generated value succeeded (`phase3/seed-paperless.txt`). A household following the page is refused at its first login. Same card also sends documents to „FileBrowser … import/paperless" — the FileBrowser login is R-513. **Fix:** the card points at „Beállítások → Automatikusan generált értékek" as gokapi's does. | **READY — rank P3-LOW; owner: CC (catalog)** |
| **R-516** | **[P3-LOW] English a customer meets on a fresh box and its apps' first screens — enumerated by the big night.** MEASURED 2026-09-14 (BIGNIGHT, VM 333, ISO 1.27.1, controller 0.242.0). Felhom-owned: (1) the dashboard menu item **„Debug"**; (2) the dashboard CPU tile **„Load: 0.29 / 0.39 / 0.37"**; (3) the launcher tile **„Filebrowser"** opens a login in English with no Felhom text (R-513); (4) the storage page mixes formal „Adjon hozzá / Csatlakoztasson" with the product's „te". App first screens a household meets before any Felhom text helps: (5) **Uptime Kuma 2.4 opens on „Which database would you like to use?"** (SQLite / Embedded MariaDB, „Next") — the app card's „Első lépések" does not mention it; (6) PrivateBin, Gokapi, AdventureLog and FileBrowser UIs are English (the apps' own). Already rows: the Proxmox installer screens (R-495, answered by the guide), `wiki.DOMAIN` (R-498). **Fix shape:** rename „Debug"/„Load" (controller); add the Uptime Kuma database step to its card, or pre-seed `db-config.json` for SQLite in the template (catalog). | **READY — rank P3-LOW; owner: CC (controller + catalog)** |
| **R-517** | **[P1-HIGH] After a failed off-site whole-system backup, „Biztonsági mentés" tells the customer the full backup is current and that a remote copy on separate hardware exists — neither is true.** MEASURED 2026-09-14 (BIGNIGHT, VM 333, controller 0.242.0, customer `tester-1` with the DR tier ticked but never provisioned — R-511): „Mentés most" at 19:03:23Z; the local tier succeeded (8 877 619 753 B, 362 s); the `felhom-pbs` tier then failed — agent: `could not activate storage 'felhom-pbs': storage 'felhom-pbs' does not exist`; `pvesm status` lists only `local` and `local-lvm`. At 19:15:36Z `/backups` read: „✗ · **Utolsó teljes mentés 2026-09-14 21:09 (5 perce) · 0 B · Biztonsági szerver – külön hardver (PBS) · Naprakész**" and „✓ **Távoli rendszermentés — külön hardveren (PBS)**" (`phase4/backup-pages-after.txt`). The successful 8.9 GB local backup is no longer shown; a 0-byte failed attempt is labelled up to date; the remote tier is ticked as present. The CLAUDE.md rule „presence is not success" in page form: an attempt's timestamp stands in for a result. The hub did raise a true `whole_guest_backup_failed (error)` to the operator; the customer's page says the opposite. **Fix shape:** the tile shows the newest SUCCESSFUL backup per tier, a failed tier as failed, and a tier whose storage does not exist as „nincs beállítva". | **READY — rank P1-HIGH; owner: CC (controller)** |
| **R-518** | **[P2-MEDIUM] „Mentés most" on the whole-system backup stops every app for about eight minutes while the page promises „csak néhány másodpercre".** MEASURED 2026-09-14 (BIGNIGHT, VM 333, 12 apps): the button's call quiesced all 12 stacks at 19:03:23Z (first stopped 19:03:27Z); the local vzdump ran 19:03:49 → 19:09:59Z; the controller then kept the apps stopped for the second (PBS) tier and restarted them at 19:10:09Z after it failed, the last started 19:11:12Z (`phase4/guest-backup-quiesce-log.txt`) — **≈ 7 m 45 s** with every app answering 404. The page under the button: „Pillanatkép-mód: az alkalmazások csak néhány másodpercre állnak le." A household pressing it at dinner loses every app for the length of the dump, and longer on a bigger box. **Fix shape:** state the real expected downtime (it scales with data), or quiesce per tier and not across a second tier's attempt; do not start a tier whose storage is absent (see R-517). | **READY — rank P2-MEDIUM; owner: CC (controller)** |
<!-- DUE-CHECKS-BEGIN — machine-readable. Parsed by scripts/due_checks_gate.py.
One row per dated check. The R-number must have a row above. Dates are UTC.