felhom.eu/REPORT-night-2026-10-21.md

REPORT — the night of 2026-10-21 → 22: make tomorrow's release safe

Unattended, 20:15 CEST → 08:00. Nothing released, deployed, published or vouched. Evidence: documentation/audits/night-2026-10-21/. Design read first: 07 §6.3/§8 (the restore ladder, R-893 option A), 03 §9 (guest sizing), 08 (notification groups), 10-localisation.md, the controller CHANGELOG v0.336.0.

Safest readings taken at the start (unattended): (1) Tester4 is the soak customer — a new customer would provision on ep0 and the Storage Box, which are read-only tonight; its old host was already deleted. (2) „One hub image" — so the node_down-after-delete finding is a row, not a second image. (3) The website is published by a push, so W2 is a patch file, not a commit. (4) The no-swap test ran after the box's last night job, not at 07:00 sharp (see Part 2).

1. Five lines

1. Soak (Part 1): NO FREEZE — 3 of the 4 pass lines met, 1 missed. 11 h on a fresh 8 GB box: no /api/health gap (largest interval 16 s), every night job ran, uploads complete except 1 photo of ~26 000 (R-1215); memory „full" pressure passed 20 % for ~3 min in the 03:30 second-drive job (peak 39 % host / 45 % guest over 10 s — R-1214).
2. No-swap test (Part 2): THE FREEZE CAME BACK. Host swap off, the same burst: the dashboard stopped answering within 20 s, dead up to 289 s at a time (24 of 125 probes failed), Immich lost 207 of 306 photos; it ended only when the Immich stream broke. Swap back on: normal at once. The agent warned once.
3. D22 built and proven: YES — the put-back covers the local, second-drive and kept restores and a restore cut by a power loss; proven on 9202 (a damaged backup and a killed controller both kept the newest row and the app running).
4. First-hour words: 22 fixed (controller W9–W14, W21, W22, W24, W25, W28, W29; hub W3–W5; catalog W16–W19; ISO source W1, W7); W6/W8 were already true; W2 is a ready patch (a publish).
5. R-1199: the hub mailed the household by a list saved two days before the box existed; the box never sent its own — fixed in the controller (decided by CC, may be reversed). R-1194: a timing false alarm — the first report runs before the emergency-door setup; fixed in the agent.

2. The morning list (copied from STATUS)

Morning list, 2026-10-22. CC decided one thing (you may reverse it): a box tells the hub the notification list its page shows, even if never saved (R-1199; the rehearsal household got 1 alarm of 9).

1. Release controller 0.338.0 (restore put-back incl. power cut, first-hour words, swap sentence)? On the scratch box a damaged backup and a killed controller both kept the newest data, app running (R-1177, R-1195, R-1178, R-1180).
2. Release and vouch agent 0.162.0 (swap = RAM, host-swap check, no false door alarm — R-1192, R-1194)? Soak, 11 h on a fresh 8 GB box: no freeze, every check answered, every night job ran; memory pressure passed the 20 % line for 3 min in the 03:30 backup (R-1214). With no host swap the freeze comes back; our installer gives 7.75 GB. I'd release.
3. Deploy hub 0.164.0 (host-swap alarm, bind-page words)? And delete the test host „Tester4-9aca27" (machine gone; my token cannot delete).
4. How was Tester 2's laptop installed — our installer, or its own Proxmox? (Without host swap the fix does not help.)
5. A publish: the website download button (patch ready), and with the next ISO the boot text and the cleared screen (R-1210). Secrets: keep or remove the REHEARSAL_2026_10_20_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_* names in sec.

3. Part 1 — the soak (numbers)

Box: VM 9322 on demo-hp (4 vCPU, 8 GiB, 256 GB + 200 GB), installed from the public ISO, customer Tester4; agent fixture 0.162.0-night.1 then night.2 by hand (provenance P1-soak/00-provenance.txt). Two channels: the box's own 10-s recorder (03-rec.jsonl, 3 632 samples) and a probe from DooPlex (02-outside.txt, 3 390 probes + demo-hp's view of the VM's disk reads).

	Load window 18:52–20:53 UTC	Night 20:53–04:05 UTC
health probes / failed	684 / 0	2 449 / 0
worst health gap (between answers)	10–11 s (slowest answer 1.27 s)	16 s at 01:30:36 (slowest answer 4.07 s)
worst memory „full" (10 s, host / guest)	31.2 % / 35.2 % (18:53:38, Immich model load; 1 sample)	39.4 % / 45.5 % (01:30:42, tier2 job; 12 samples > 20 %, ~3 min)
worst memory „full" (60 s)	—	20.1 % / 25.0 %
OOM kills / guest limit hits	0 / 0	0 / 0
guest swap used (max)	2 614 MB	2 720 MB

• Guest swap raised: 512 → 4096 MB by the fixture at 18:37:13 UTC (04-agent-fixture.txt).
• Uploads: Paperless 99 cycles × 53 PDFs, all accepted; Immich 87 cycles × 305 + the first upload — every user holds 303 photos (the set has 2 duplicates and 1 file Immich refuses by design), except cycle 36: 302 — one photo hung 10 min and was not stored (19:38 UTC, pressure 3.9 %, health 0.26 s; Immich's log lost at its 02:16 restart — R-1215); Nextcloud: both trees 3 031 of 3 033 files byte-identical to the manifest (sha256; the 2 are the uploader's by-design skips).
• Night jobs (box log 10-controller-log.txt): db-dump 00:30 (4 DBs, 4 m 14 s), tier2 01:30 (4 m 15 s), off-site 02:15 (8 apps, 72 snapshots, 5 m 53 s), abandon-sweep 03:10, proof 03:30 (actualbudget PASSED), integrity 04:00 (1 m 22 s) — all completed. The agent's whole-guest backup ran at 18:41 UTC (24 h cadence; next due after the night).
• Host swap the installer gives an 8 GB / 256 GB machine: 7.75 GB (LVM pve/swap, SwapTotal 8 126 460 kB; 02-host-swap-after-install.txt).
• Before (rehearsal, no fix): „full" 96 %, dashboard dead for about an hour. After: no gap over 16 s.

4. Part 3 — D22, each scenario

Scenario (9202, Rallly + Postgres, controller 0.338.0-rc.1)	Result
a normal local restore (the control)	unchanged: „1 adatkötet és az adatbázis visszaállítva", the expected rewind (A,B → A)
the unit's database volume archive damaged (cut to half)	put back: the live volume poured back, markers A,C (C = the newest live row), Rallly running, „…visszaállítása sikertelen. Nem változott semmi…" — on 0.331.0 (2026-10-20) the same case crash-looped Postgres and lost C
the controller killed (kill -9) right after the volume step, before the replay	on start: „interrupted restore is put back … 1 volume(s) poured back, definition=true", markers A,C,D (D newer than the backup), Rallly running, the record: „A visszaállítás félbeszakadt (áramszünet). Az alkalmazás adatai a visszaállítás előtti állapotban vannak."
off-site restore cut the same way	by test (TestR1195_AKilledOffsiteRestoreIsPutBackOnStart)
second drive: the unit refuses after the file half (R-1178)	by test: nothing moves, the app is never stopped
the night mirror during a restore (R-1180)	by test with the production rsync: skipped, mirrored the next night

Red-proofs seen failing, then passing: TestR1177_* (4 failed on the old restore_unit.go: the old „completed with data errors" text, no copy), TestR1195_* (the put-back at start switched off → „nothing poured back"; the off-site journal not written → no undo), TestR1195_TheRecordSays… (old „indítsd el újra"), TestR1178_* (1 restored + 1 replaced), TestR1180_* ([a.jpg b.jpg]). Three older tests that pinned the old outcome were changed on purpose (the app started on a half state is now held or put back). 9202 put back as found: controller 0.337.0, the same apps.

5. Part 2 — machines with no swap

The test (P2-noswap/): the soak box after its night, swapoff -a on the host at 04:06 UTC (48 s; it held 1.9 GB), the burst from 04:08:25 to 04:38:27 (Immich machine-learning restarted so its models load again, as on the first afternoon), then swapon -a. The first attempt (04:06–04:08) is VOID and kept: the load script restarted its names at 1, so Immich got duplicates and Nextcloud skipped existing folders (02-attempt1-VOID-duplicate-uploads.txt); the rerun used fresh names.

	No host swap (04:08–04:38)	Swap back on (04:38–04:44)
health probes / failed	125 / 24	34 / 0
gaps between answers	289 s, 172 s, 68 s	none
memory „full" (10 s, host / guest)	36 % / 47 % (94 of 180 samples > 20 %)	1.2 % / 1.3 %
VM disk reads (demo-hp's view)	~40 GB per 25 s at the worst	quiet
OOM kills	0	0
uploads	Immich 99 of 306, then its stream broke (a login answer that was not JSON); Paperless 52 of 53 in the first cycle, then 22 × 53; Nextcloud cut at the time limit	—

Said plainly: without host swap the freeze comes back — the same shape as the rehearsal (refaults, disk reads, no OOM kill), shorter only because the Immich stream died at 04:22 and took the load away. The agent logged its WARN once at the swapoff („host_swap_mb=938 guest_swap_mb=4096").

Built (commit only): agent 0da792e reports host.host_swap_mb (Proxmox node status = the kernel's SwapTotal — the same number as /proc/meminfo; used instead of reading the file because the agent already holds the node status) and host.guest_swap_mb, logs one WARN when short; GET /guest/memory carries both. Hub 0.164.0: operator-only host_swap_short / host_swap_ok, edge-triggered from the host's newest swap event. Controller install page, when memory is tight and the host's swap is below the guest's RAM: HU „A gépen nincs elég cserehely (swap), ezért nagy feltöltéskor a doboz lelassulhat, és percekig sem válaszolhat. Szólj a Felhomnak, mi rendbe tesszük." EN „The machine has too little swap space, so during large uploads the box can slow down and not answer for minutes. Tell Felhom; we will fix it." Creating host swap on a BYO host is a decision → R-1207 (D27).

6. Part 5 — the two checks

R-1199 (the household's mails went to the operator). Hub DB read with SELECT on named non-secret columns only (P5-checks/r1199-01-hub-db-read.txt): Tester4's stored list was saved 2026-10-18 18:18 — before the rehearsal box existed — and holds node_down, backup_failed, disk_critical, host_disk_critical, storage_fill_critical, offbox_repo_orphaned, app_update_undone, app_update_held, app_stopped_unhealthy: no disk_warning, no storage_disconnected, no crossdrive_failed. The rehearsal day's log: the one household mail was backup_failed. Mechanism: the box shows settings.DefaultEnabledEvents ticked when nothing was saved (controller/internal/settings/settings.go GetNotificationPrefs), but the start-up sync pushes only when an address or a clear was saved (settings.go SyncOnStartup, cmd/controller/main.go start-up goroutine), and the hub mails the household only by its stored list (hub/internal/notify/dispatcher.go processCustomer → isEventEnabled). Fix (controller ba22f9b): a box with no saved choice pushes the default list at start, with no address (the hub's no-clobber guard keeps the stored one). The box's page is the source — 08 („the household decides"). Decided by CC unattended — operator may reverse. TestR1199_* red-proved.

R-1194 (emergency SSH door „broken" on a new box). On the fresh soak box: the agent ran sshd -t -f /etc/felhom-sshd/sshd_config through sudo at 20:33:59, the config was applied at 20:34:00.434, and the hub mailed oob_degraded („config invalid") at 20:34 CEST; as the agent's user the check passes now (rc 0, port 8822 listening); without sudo it fails on the host key — which the agent never does. True candidate: the first report runs before the felhom-sshd loop's first pass (port 0, config never rendered) — a false alarm on every fresh box, operator key or not. Fix (agent 0da792e): no OOB stanza until the first pass has run (a failed pass counts). Red-proved.

7. Builds, commits, CI

Artifact	Version	From	Digest / sha256	State
controller	0.338.0-rc.1	d582feb	—	the D22 proof on 9202
controller	0.338.0-rc.2	f0abdd0	sha256:ae52c93ebb83af1c646986119a1b74b07f992ffb03017624d052ece6266ec748	smoke on 9202 (pages 200, 0 error lines); 9202 back on 0.337.0
hub	0.164.0	6b0d23c2	sha256:931594e333a4e8a9784eff2fb144d022ab46f89dd6a865a93e6100dc28dbd9dc	built, pushed, NOT deployed (manifest 0.163.0)
agent	0.162.0 candidate	0da792e, clean clone, release flags	5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928	not published; fixtures night.1 (286a7e3) and night.2 (0da792e) ran on the soak host

Commits. controller: d582feb, 37b79ca, 8c1e290, 28b05d6, 5776d54 (the helper's three: W9–W29), ba22f9b, f0abdd0, 1de8a55. agent: 0da792e, b310773. felhom.eu: 3c620f8d, f90e2229 (hub), 2fa5b8f3 (ISO source), 6b0d23c2, 2293b252, 9d3bf76d, 57c06ba9, 307d25b1, f576d7f7, c2c130ec, dee26da3 + this report. catalog: 13f9c9c (live through the catalog sync). CI: controller 2175, 2178, 2181, 2184, 2186, 2187, 2188, 2191 — all success; agent 2176, 2190 — success; catalog 2180 — success; felhom.eu 2177–2192 failed on one suite only (test_gate_decoys.py hit the 300 s limit on the CI runner; R-1209, fixed in c2c130ec) — after the fix: c2c130ec 2193, dee26da3 2194, 97f088ec 2195 — all success; the commit carrying this report is checked after its push (the final note)..

8. Teardown (three layers)

• Machine: VM 9322 night1021-soak and CT 9332 night1021-loader destroyed on demo-hp (names checked first) at ~04:47 UTC; their disk folders on nvme-scratch are gone (used 301 → 208 GB). The thin pool local-lvm read the same before and after (26 159 563 KiB used) — never written. The public ISO on local was there before and stays. demo-hp's own clock and swap untouched. The night's evidence was copied off first (logs, recorder, Immich counts).
• Host (hub): the host Tester4-9aca27 stays on the hub (agent 0.162.0-night.2); CC's token cannot delete a host — the operator deletes it (morning list 3). Until then it reads down. The customer Tester4 stays (the kept one).
• Off-site: ep0 (read only) holds for Tester4 one copy, ct/9201/2026-10-20T12:17:26Z, from the rehearsal box — nothing from tonight; nothing removed. The Storage Box holds Tester4's off-site repository with tonight's 72 snapshots — left (a new Tester4 box sets it aside by design).
• 9202: back on controller 0.337.0 with the apps it had; the D22 test app removed through the product.

9. Rows

Before 176; after 184; opened 9 (R-1207 host swap on a no-swap host — D27; R-1208 node_down after an operator delete; R-1209 the CI time limit — closed the same night; R-1210 first-hour words waiting for a publish; R-1211 the household backup_failed mail; R-1212 two texts ignoring the box language; R-1213 operator detail left on two pages; R-1214 memory pressure in the night's second-drive job; R-1215 one Immich upload hung); closed 1 (R-1209). Moved to WAITING-ON-OPERATOR (fixed on main, ship with the morning releases): R-1177, R-1178, R-1180, R-1195 (controller), R-1194 (agent). R-1188 (a) fixed. unproven.py --summary: unchanged, 35 of 55 not walked.

10. Observations

• The decoy suite timed out on CI on every felhom.eu push of the night — FILED: R-1209 (fixed, closed).

• node_down mailed for a customer whose host the operator had deleted — FILED: R-1208.

• A household backup_failed mail with no app and no step — FILED: R-1211 (the helper's).

• Two backup texts ignore the box language — FILED: R-1212. Operator detail on two pages — FILED: R-1213.

• TestR603_* was red on controller main since 0.337.0 (CI runs gates, not go test) — NOT-A-FINDING: fixed in the session (two English apostrophes), small.

• The first resend press on a spent link before the new box registered sent nothing — NOT-A-FINDING: a live link existed (minted at the host delete, mailed 16:49); the page says „if it was a real link" by design.

• The helper chose different words than the brief for W25 („a következő éjjel", the retry is the same night when pressed by day) and W29 („Leszerelve. Nem költöztettünk át adatot róla; ami rajta volt, a meghajtón maradt." — „adat nem volt rajta" would be false) — NOT-A-FINDING: the brief's texts were examples; the reasons are in the CHANGELOG.

• On the installed app pages Mealie already showed the right address (R-498); W19 was wrong only on the install page — NOT-A-FINDING beyond the catalog fix.

• An empty backups/restore-undo/ folder stays after a put-back (only the per-app folder is removed) — NOT-A-FINDING: harmless, no data.

• The soak load script's Nextcloud stream ran past the 2 hours (each stream finishes its current pass) — NOT-A-FINDING: harness.

• Memory „full" above 20 % in the night's second-drive job (~3 min) — FILED: R-1214. One Immich upload hung and was not stored — FILED: R-1215.

• Without host swap the freeze comes back — recorded on R-1207 (the operator's D27), not a new row.

• One execCommand failed: exit status 1 at 00:30:49 — NOT-A-FINDING: the memory check's docker inspect of all containers ran 1 s after actualbudget was stopped for its dump; the code keeps the other answers by design.

• One Radicale health probe timed out at 01:30:24 (the second-drive job starting) — NOT-A-FINDING: the next probe passed, no alarm, the household's door answered throughout.

• The guest hit its memory limit 101 times at 18:39 UTC, during the app installs, before the load — NOT-A-FINDING: page cache reclaim (0 OOM kills, swap 40 KB used), never again in 11 h.

• A 637 MB vzdump-lxc-9100-… on the fresh box's local — NOT-A-FINDING: the installer's golden archive (9100 = the golden build guest, felhom-host-install.sh); the restore test skips it with a reason.

• The byte-check tool tools/nc_compare.py turns TLS checking off — NOT-A-FINDING: a measurement script against a throwaway box with a self-signed certificate and a test domain that does not resolve; the box is destroyed.

• The first no-swap attempt reused the soak's names (Immich duplicates, Nextcloud skips) — NOT-A-FINDING: harness; stopped after 2 min, kept as VOID, rerun with fresh names (tools/load2.py now takes a name tag).

• CC ran an old one-off register script by mistake at 04:07 UTC; it added 16 duplicate rows to the register file — NOT-A-FINDING: caught at once by the row count, the file restored from git before anything was committed; the script renamed so it cannot run again. Fixed without a row: the R-603 test, the restore-record log line after a put-back, the CHANGELOG blank line. Secrets: names only — NIGHT_2026_10_21_ROOTPW, NIGHT_2026_10_21_PAIRING_CODE, NIGHT_2026_10_21_BIND_TOKEN, NIGHT_2026_10_21_SETUP_CODE (the last three spent). The bind link and the setup code were read from the mailbox through the mail connector, so both one-time values entered this session's record (spent within minutes). A scan of the evidence for every secret used tonight: 0 hits — all 34 secret names of the night, the rehearsal and the fixes run, across the 75 evidence files, this report, STATUS, CONTEXT and the register (positive control: a known word in the evidence was found). Poster: no fact in felhom-system-poster.facts.md changed (nothing released). Instruction files: none edited.