Rehearsal: R-1193 corrected and narrowed to P3 (the operator mailbox shows host_stale at 13:33 and app_oom; the household was told nothing); R-1192 second freeze + the app_oom hint; R-1194 (P3 VERIFY) fresh box's felhom-sshd config invalid; B6 #5 slow-line result

2026-10-20 19:10:42 +02:00
1 parent 78278502ac
commit 9db8aacdad
@@ -1 +1,15 @@
-16:32:09Z running pha
+16:32:09Z running phase=dump app= 0% / files 0/0 snaps=0 last_err=''
+16:34:13Z running phase= app=bookstack 100% 153.6 MB/153.6 MB files 8/8 snaps=0 last_err=''
+16:36:13Z running phase= app=immich 12.32862472832852% 150.6 MB/1.2 GB files 787/790 snaps=0 last_err=''
+16:38:14Z running phase= app=immich 12.32862472832852% 150.6 MB/1.2 GB files 787/790 snaps=0 last_err=''
+16:40:14Z running phase= app=immich 12.32862472832852% 150.6 MB/1.2 GB files 787/790 snaps=0 last_err=''
+16:42:14Z running phase= app=immich 12.32862472832852% 150.6 MB/1.2 GB files 787/790 snaps=0 last_err=''
+16:44:14Z running phase= app=immich 12.32862472832852% 150.6 MB/1.2 GB files 787/790 snaps=0 last_err=''
+16:46:14Z running phase= app=immich 12.32862472832852% 150.6 MB/1.2 GB files 787/790 snaps=0 last_err=''
+16:48:14Z running phase= app=mealie 0% / files 0/0 snaps=0 last_err=''
+16:50:14Z running phase= app=nextcloud 41.02783378851586% 423.9 MB/1.0 GB files 3107/3108 snaps=0 last_err=''
+16:52:14Z running phase= app=nextcloud 41.02783378851586% 423.9 MB/1.0 GB files 3107/3108 snaps=0 last_err=''
+16:54:14Z running phase= app=nextcloud 41.02783378851586% 423.9 MB/1.0 GB files 3107/3108 snaps=0 last_err=''
+16:56:15Z running phase= app=nextcloud 41.02783378851586% 423.9 MB/1.0 GB files 3107/3108 snaps=0 last_err=''
+16:58:15Z running phase= app=nextcloud 41.02783378851586% 423.9 MB/1.0 GB files 3107/3108 snaps=0 last_err=''
+17:00:15Z running phase= app=nextcloud 41.02783378851586% 423.9 MB/1.0 GB files 3107/3108 snaps=0 last_err=''
@@ -0,0 +1,5 @@
+qdisc netem 8001: root refcnt 2 limit 1000 loss 5% rate 1Mbit seed 8660933106302401915
+ Sent 87470204 bytes 76388 pkt (dropped 2885, overlimits 0 requeues 0) 
+ backlog 10598b 4p requeues 0
+17:01:02
+qdisc fq_codel 0: root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 
@@ -0,0 +1,2 @@
+17:01:12Z running app= nextcloud 41.02783378851586 423.9 MB / 1.0 GB snaps= 0 size=  dur=  err= 
+17:03:12Z error app=  0  /  snaps= 0 size=  dur= 30m10s err= offbox backup immich: exit status 1
@@ -0,0 +1,5 @@
+17:01:32 INFO [offbox] backed up nextcloud (/mnt/felhom-drives/hdd_1/backups/primary/nextcloud, 1 mandatory path(s))
+17:01:36 INFO [offbox] backed up paperless-ngx (/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx, 1 mandatory path(s))
+17:01:38 INFO [offbox] backed up radicale (/mnt/sys_drive/felhom-data/backups/primary/radicale, 0 mandatory path(s))
+17:01:49 ERROR [offbox] backup failed after 30m10s: offbox backup immich: exit status 1
+17:01:49 WARN [web] manual off-box backup failed: offbox backup immich: exit status 1
@@ -0,0 +1,7 @@
+200 2026-10-20T16:50:57Z
+17:01:32 INFO [offbox] backed up nextcloud (/mnt/felhom-drives/hdd_1/backups/primary/nextcloud, 1 mandatory path(s))
+17:01:36 INFO [offbox] backed up paperless-ngx (/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx, 1 mandatory path(s))
+17:01:38 INFO [offbox] backed up radicale (/mnt/sys_drive/felhom-data/backups/primary/radicale, 0 mandatory path(s))
+17:01:49 ERROR [offbox] backup failed after 30m10s: offbox backup immich: exit status 1
+17:01:49 INFO [offbox] manual run progress reporting ended after 30m10s
+17:01:49 WARN [web] manual off-box backup failed: offbox backup immich: exit status 1
@@ -0,0 +1,6 @@
+17:04:13Z press 302
+17:04:43Z running app=  0  /  snaps= 0 size=  dur= 30m10s err= 
+17:05:43Z running app=  0  /  snaps= 0 size=  dur= 30m10s err= 
+17:06:43Z running app= immich 25 309.8 MB / 1.2 GB snaps= 0 size=  dur= 30m10s err= 
+17:07:43Z running app= nextcloud 88 911.3 MB / 1.0 GB snaps= 0 size=  dur= 30m10s err= 
+17:08:43Z ok app=  0  /  snaps= 17 size= 1.1 GB dur= 3m38s err= 
@@ -0,0 +1,20 @@
+# B6 #5 — slow, lossy internet during the first big off-site run (2026-10-20)
+
+Expected (written before): the run slows but stays honest on the page; it either finishes or fails clearly, and the
+next run picks up (restic resumes from what is already uploaded); nothing is reported as safe that is not.
+
+| UTC | act | observed |
+|---|---|---|
+| 16:31:28 | `tc netem` on the VM's own tap (demo-hp): upload 1 Mbit/s + 5 % loss, both directions 5 % loss (`01-netem-on.txt`) | — |
+| 16:31:37 | „Távoli mentés most" | running, phase dump (`02-run-pressed.txt`) |
+| 16:34 | progress | BookStack 100 % 153.6 MB |
+| 16:36–16:46 | progress | **Immich frozen at „12.3 % · 150.6 MB / 1.2 GB · files 787/790" for 10+ minutes** — the number does not move while the line crawls |
+| ~16:47 | | the run moves on to Mealie, then Nextcloud (41 % frozen 16:50–17:01) — Immich had failed (its restic message fell out of the controller's log ring) |
+| 17:01:02 | shaping removed; only 87 MB had left the box in 30 min (`04-netem-off.txt`) | — |
+| 17:01:32–38 | | Nextcloud, Paperless, Radicale „backed up" |
+| 17:01:49 | | **run = error after 30 min 10 s: „offbox backup immich: exit status 1"; the page: „✗ Hiba … Utolsó hiba: offbox backup immich: exit status 1 … a pillanatképek száma még ismeretlen"** — English, technical, and silent about the apps that DID copy (`06*`, screens `b6-s5/`) |
+| 17:04–17:08 | second „Távoli mentés most" at full speed | **ok in 3 min 38 s; 17 snapshots; 1.1 GB** (`07-second-press.txt`) |
+
+Verdict: alarm TRUE (the run is marked failed), text poor (W25); time to heal = one more press at full speed. The
+per-app 2-hour limit (`offbox.go:39`) was not the cause — the failure was Immich's own upload breaking on the lossy
+line. Not measured: whether the NIGHT's run would have retried Immich by itself (the household pressed again first).