felhom.eu/REPORT-night-2026-10-21.md

# REPORT — the night of 2026-10-21 → 22: make tomorrow's release safe

Unattended, 20:15 CEST → 08:00. Nothing released, deployed, published or vouched. Evidence:
`documentation/audits/night-2026-10-21/`. Design read first: `07` §6.3/§8 (the restore ladder, R-893 option A),
`03` §9 (guest sizing), `08` (notification groups), `10-localisation.md`, the controller CHANGELOG v0.336.0.

**Safest readings taken at the start (unattended):** (1) Tester4 is the soak customer — a new customer would provision on
ep0 and the Storage Box, which are read-only tonight; its old host was already deleted. (2) „One hub image" — so the
`node_down`-after-delete finding is a row, not a second image. (3) The website is published by a push, so W2 is a patch
file, not a commit. (4) The no-swap test ran after the box's last night job, not at 07:00 sharp (see Part 2).

## 1. Five lines

1. **Soak (Part 1): NO FREEZE — 3 of the 4 pass lines met, 1 missed.** 11 h on a fresh 8 GB box: no `/api/health` gap (largest interval 16 s), every night job ran, uploads complete except 1 photo of ~26 000 (R-1215); memory „full" pressure passed 20 % for ~3 min in the 03:30 second-drive job (peak 39 % host / 45 % guest over 10 s — R-1214).
2. **No-swap test (Part 2): THE FREEZE CAME BACK.** Host swap off, the same burst: the dashboard stopped answering within 20 s, dead up to 289 s at a time (24 of 125 probes failed), Immich lost 207 of 306 photos; it ended only when the Immich stream broke. Swap back on: normal at once. The agent warned once.
3. **D22 built and proven: YES** — the put-back covers the local, second-drive and kept restores and a restore cut by a
   power loss; proven on 9202 (a damaged backup and a killed controller both kept the newest row and the app running).
4. **First-hour words: 22 fixed** (controller W9–W14, W21, W22, W24, W25, W28, W29; hub W3–W5; catalog W16–W19; ISO
   source W1, W7); W6/W8 were already true; W2 is a ready patch (a publish).
5. **R-1199:** the hub mailed the household by a list saved two days before the box existed; the box never sent its own
   — fixed in the controller (decided by CC, may be reversed). **R-1194:** a timing false alarm — the first report runs
   before the emergency-door setup; fixed in the agent.

## 2. The morning list (copied from STATUS)

**Morning list, 2026-10-22.** CC decided one thing (you may reverse it): a box tells the hub the notification list its page shows, even if never saved (R-1199; the rehearsal household got 1 alarm of 9).
1. Release controller 0.338.0 (restore put-back incl. power cut, first-hour words, swap sentence)? On the scratch box a damaged backup and a killed controller both kept the newest data, app running (R-1177, R-1195, R-1178, R-1180).
2. Release and vouch agent 0.162.0 (swap = RAM, host-swap check, no false door alarm — R-1192, R-1194)? Soak, 11 h on a fresh 8 GB box: no freeze, every check answered, every night job ran; memory pressure passed the 20 % line for 3 min in the 03:30 backup (R-1214). With no host swap the freeze comes back; our installer gives 7.75 GB. I'd release.
3. Deploy hub 0.164.0 (host-swap alarm, bind-page words)? And delete the test host „Tester4-9aca27" (machine gone; my token cannot delete).
4. How was Tester 2's laptop installed — our installer, or its own Proxmox? (Without host swap the fix does not help.)
5. A publish: the website download button (patch ready), and with the next ISO the boot text and the cleared screen (R-1210). Secrets: keep or remove the `REHEARSAL_2026_10_20_*`, `FIXES_2026_10_21_*`, `NIGHT_2026_10_21_*` names in sec.

## 3. Part 1 — the soak (numbers)

Box: VM 9322 on demo-hp (4 vCPU, 8 GiB, 256 GB + 200 GB), installed from the public ISO, customer Tester4; agent fixture
`0.162.0-night.1` then `night.2` by hand (provenance `P1-soak/00-provenance.txt`). Two channels: the box's own 10-s
recorder (`03-rec.jsonl`, 3 632 samples) and a probe from DooPlex (`02-outside.txt`, 3 390 probes + demo-hp's view of the VM's
disk reads).

| | Load window 18:52–20:53 UTC | Night 20:53–04:05 UTC |
|---|---|---|
| health probes / failed | 684 / 0 | 2 449 / 0 |
| worst health gap (between answers) | 10–11 s (slowest answer 1.27 s) | **16 s** at 01:30:36 (slowest answer 4.07 s) |
| worst memory „full" (10 s, host / guest) | 31.2 % / 35.2 % (18:53:38, Immich model load; 1 sample) | **39.4 % / 45.5 %** (01:30:42, tier2 job; 12 samples > 20 %, ~3 min) |
| worst memory „full" (60 s) | — | 20.1 % / 25.0 % |
| OOM kills / guest limit hits | 0 / 0 | 0 / 0 |
| guest swap used (max) | 2 614 MB | 2 720 MB |

- **Guest swap raised:** 512 → 4096 MB by the fixture at 18:37:13 UTC (`04-agent-fixture.txt`).
- **Uploads:** Paperless 99 cycles × 53 PDFs, all accepted; Immich 87 cycles × 305 + the first upload — every user holds
  303 photos (the set has 2 duplicates and 1 file Immich refuses by design), except cycle 36: 302 — **one photo hung 10 min
  and was not stored** (19:38 UTC, pressure 3.9 %, health 0.26 s; Immich's log lost at its 02:16 restart — R-1215);
  Nextcloud: both trees 3 031 of 3 033 files byte-identical to the manifest (sha256; the 2 are the uploader's by-design skips).
- **Night jobs (box log `10-controller-log.txt`):** db-dump 00:30 (4 DBs, 4 m 14 s), tier2 01:30 (4 m 15 s), off-site
  02:15 (8 apps, 72 snapshots, 5 m 53 s), abandon-sweep 03:10, proof 03:30 (actualbudget PASSED), integrity 04:00
  (1 m 22 s) — all completed. The agent's whole-guest backup ran at 18:41 UTC (24 h cadence; next due after the night).
- **Host swap the installer gives an 8 GB / 256 GB machine: 7.75 GB** (LVM `pve/swap`, SwapTotal 8 126 460 kB;
  `02-host-swap-after-install.txt`).
- **Before (rehearsal, no fix):** „full" 96 %, dashboard dead for about an hour. **After:** no gap over 16 s.

## 4. Part 3 — D22, each scenario

| Scenario (9202, Rallly + Postgres, controller 0.338.0-rc.1) | Result |
|---|---|
| a normal local restore (the control) | unchanged: „1 adatkötet és az adatbázis visszaállítva", the expected rewind (A,B → A) |
| the unit's database volume archive damaged (cut to half) | put back: the live volume poured back, markers **A,C** (C = the newest live row), Rallly running, „…visszaállítása sikertelen. Nem változott semmi…" — on 0.331.0 (2026-10-20) the same case crash-looped Postgres and lost C |
| the controller killed (kill -9) right after the volume step, before the replay | on start: „interrupted restore is put back … 1 volume(s) poured back, definition=true", markers **A,C,D** (D newer than the backup), Rallly running, the record: „A visszaállítás félbeszakadt (áramszünet). Az alkalmazás adatai a visszaállítás előtti állapotban vannak." |
| off-site restore cut the same way | by test (`TestR1195_AKilledOffsiteRestoreIsPutBackOnStart`) |
| second drive: the unit refuses after the file half (R-1178) | by test: nothing moves, the app is never stopped |
| the night mirror during a restore (R-1180) | by test with the production rsync: skipped, mirrored the next night |

**Red-proofs seen failing, then passing:** `TestR1177_*` (4 failed on the old `restore_unit.go`: the old „completed with
data errors" text, no copy), `TestR1195_*` (the put-back at start switched off → „nothing poured back"; the off-site
journal not written → no undo), `TestR1195_TheRecordSays…` (old „indítsd el újra"), `TestR1178_*` (1 restored +
1 replaced), `TestR1180_*` ([a.jpg b.jpg]). Three older tests that pinned the old outcome were changed on purpose
(the app started on a half state is now held or put back). 9202 put back as found: controller 0.337.0, the same apps.

## 5. Part 2 — machines with no swap

**The test** (`P2-noswap/`): the soak box after its night, `swapoff -a` on the host at 04:06 UTC (48 s; it held 1.9 GB),
the burst from 04:08:25 to 04:38:27 (Immich machine-learning restarted so its models load again, as on the first afternoon),
then `swapon -a`. The first attempt (04:06–04:08) is VOID and kept: the load script restarted its names at 1, so Immich got
duplicates and Nextcloud skipped existing folders (`02-attempt1-VOID-duplicate-uploads.txt`); the rerun used fresh names.

| | No host swap (04:08–04:38) | Swap back on (04:38–04:44) |
|---|---|---|
| health probes / failed | 125 / **24** | 34 / 0 |
| gaps between answers | **289 s**, 172 s, 68 s | none |
| memory „full" (10 s, host / guest) | **36 % / 47 %** (94 of 180 samples > 20 %) | 1.2 % / 1.3 % |
| VM disk reads (demo-hp's view) | ~40 GB per 25 s at the worst | quiet |
| OOM kills | 0 | 0 |
| uploads | Immich 99 of 306, then its stream broke (a login answer that was not JSON); Paperless 52 of 53 in the first cycle, then 22 × 53; Nextcloud cut at the time limit | — |

**Said plainly:** without host swap the freeze comes back — the same shape as the rehearsal (refaults, disk reads, no OOM
kill), shorter only because the Immich stream died at 04:22 and took the load away. The agent logged its WARN once at the
swapoff („host_swap_mb=938 guest_swap_mb=4096").

**Built (commit only):** agent `0da792e` reports `host.host_swap_mb` (Proxmox node status = the kernel's SwapTotal — the
same number as `/proc/meminfo`; used instead of reading the file because the agent already holds the node status) and
`host.guest_swap_mb`, logs one WARN when short; `GET /guest/memory` carries both. Hub 0.164.0: operator-only
`host_swap_short` / `host_swap_ok`, edge-triggered from the host's newest swap event. Controller install page, when memory
is tight and the host's swap is below the guest's RAM: **HU** „A gépen nincs elég cserehely (swap), ezért nagy
feltöltéskor a doboz lelassulhat, és percekig sem válaszolhat. Szólj a Felhomnak, mi rendbe tesszük." **EN** „The machine
has too little swap space, so during large uploads the box can slow down and not answer for minutes. Tell Felhom; we will
fix it." Creating host swap on a BYO host is a decision → R-1207 (D27).

## 6. Part 5 — the two checks

**R-1199 (the household's mails went to the operator).** Hub DB read with `SELECT` on named non-secret columns only
(`P5-checks/r1199-01-hub-db-read.txt`): Tester4's stored list was saved **2026-10-18 18:18** — before the rehearsal box
existed — and holds `node_down, backup_failed, disk_critical, host_disk_critical, storage_fill_critical,
offbox_repo_orphaned, app_update_undone, app_update_held, app_stopped_unhealthy`: no `disk_warning`, no
`storage_disconnected`, no `crossdrive_failed`. The rehearsal day's log: the one household mail was `backup_failed`.
**Mechanism:** the box shows `settings.DefaultEnabledEvents` ticked when nothing was saved
(`controller/internal/settings/settings.go` `GetNotificationPrefs`), but the start-up sync pushes only when an address or
a clear was saved (`settings.go` `SyncOnStartup`, `cmd/controller/main.go` start-up goroutine), and the hub mails the
household only by its stored list (`hub/internal/notify/dispatcher.go` `processCustomer` → `isEventEnabled`). **Fix
(controller `ba22f9b`):** a box with no saved choice pushes the default list at start, with no address (the hub's
no-clobber guard keeps the stored one). The box's page is the source — `08` („the household decides"). *Decided by CC
unattended — operator may reverse.* `TestR1199_*` red-proved.

**R-1194 (emergency SSH door „broken" on a new box).** On the fresh soak box: the agent ran `sshd -t -f
/etc/felhom-sshd/sshd_config` **through sudo** at 20:33:59, the config was applied at 20:34:00.434, and the hub mailed
`oob_degraded` („config invalid") at 20:34 CEST; as the agent's user the check passes now (rc 0, port 8822 listening);
without sudo it fails on the host key — which the agent never does. **True candidate: the first report runs before the
felhom-sshd loop's first pass** (port 0, config never rendered) — a false alarm on every fresh box, operator key or not.
**Fix (agent `0da792e`):** no OOB stanza until the first pass has run (a failed pass counts). Red-proved.

## 7. Builds, commits, CI

| Artifact | Version | From | Digest / sha256 | State |
|---|---|---|---|---|
| controller | `0.338.0-rc.1` | `d582feb` | — | the D22 proof on 9202 |
| controller | `0.338.0-rc.2` | `f0abdd0` | `sha256:ae52c93ebb83af1c646986119a1b74b07f992ffb03017624d052ece6266ec748` | smoke on 9202 (pages 200, 0 error lines); 9202 back on 0.337.0 |
| hub | `0.164.0` | `6b0d23c2` | `sha256:931594e333a4e8a9784eff2fb144d022ab46f89dd6a865a93e6100dc28dbd9dc` | built, pushed, NOT deployed (manifest 0.163.0) |
| agent | `0.162.0` candidate | `0da792e`, clean clone, release flags | `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` | not published; fixtures `night.1` (`286a7e3`) and `night.2` (`0da792e`) ran on the soak host |

**Commits.** controller: `d582feb`, `37b79ca`, `8c1e290`, `28b05d6`, `5776d54` (the helper's three: W9–W29), `ba22f9b`,
`f0abdd0`, `1de8a55`. agent: `0da792e`, `b310773`. felhom.eu: `3c620f8d`, `f90e2229` (hub), `2fa5b8f3` (ISO source),
`6b0d23c2`, `2293b252`, `9d3bf76d`, `57c06ba9`, `307d25b1`, `f576d7f7`, `c2c130ec`, `dee26da3` + this report.
catalog: `13f9c9c` (live through the catalog sync). **CI:** controller 2175, 2178, 2181, 2184, 2186, 2187, 2188, 2191 —
all success; agent 2176, 2190 — success; catalog 2180 — success; felhom.eu 2177–2192 **failed on one suite only**
(`test_gate_decoys.py` hit the 300 s limit on the CI runner; R-1209, fixed in `c2c130ec`) — after the fix: `c2c130ec` 2193, `dee26da3` 2194, `97f088ec` 2195 — all success; the commit carrying this report is checked after its push (the final note)..

## 8. Teardown (three layers)

- **Machine:** VM 9322 `night1021-soak` and CT 9332 `night1021-loader` destroyed on demo-hp (names checked first) at
  ~04:47 UTC; their disk folders on nvme-scratch are gone (used 301 → 208 GB). The thin pool `local-lvm` read the same before
  and after (26 159 563 KiB used) — never written. The public ISO on `local` was there before and stays. demo-hp's own clock
  and swap untouched. The night's evidence was copied off first (logs, recorder, Immich counts).
- **Host (hub):** the host `Tester4-9aca27` stays on the hub (agent `0.162.0-night.2`); CC's token cannot delete a host —
  **the operator deletes it** (morning list 3). Until then it reads down. The customer Tester4 stays (the kept one).
- **Off-site:** ep0 (read only) holds for Tester4 one copy, `ct/9201/2026-10-20T12:17:26Z`, from the rehearsal box —
  nothing from tonight; nothing removed. The Storage Box holds Tester4's off-site repository with tonight's 72 snapshots —
  left (a new Tester4 box sets it aside by design).
- **9202:** back on controller 0.337.0 with the apps it had; the D22 test app removed through the product.

## 9. Rows

Before **176**; after **184**; opened **9** (R-1207 host swap on a no-swap host — D27; R-1208 `node_down` after an
operator delete; R-1209 the CI time limit — closed the same night; R-1210 first-hour words waiting for a publish; R-1211
the household `backup_failed` mail; R-1212 two texts ignoring the box language; R-1213 operator detail left on two pages; R-1214 memory pressure in the night's second-drive job; R-1215 one Immich upload hung);
closed **1** (R-1209). Moved to WAITING-ON-OPERATOR (fixed on main, ship with the morning releases): R-1177, R-1178,
R-1180, R-1195 (controller), R-1194 (agent). R-1188 (a) fixed. `unproven.py --summary`: unchanged, 35 of 55 not walked.

## 10. Observations

- The decoy suite timed out on CI on every felhom.eu push of the night — **FILED: R-1209** (fixed, closed).
- `node_down` mailed for a customer whose host the operator had deleted — **FILED: R-1208**.
- A household `backup_failed` mail with no app and no step — **FILED: R-1211** (the helper's).
- Two backup texts ignore the box language — **FILED: R-1212**. Operator detail on two pages — **FILED: R-1213**.
- `TestR603_*` was red on controller main since 0.337.0 (CI runs gates, not `go test`) — **NOT-A-FINDING:** fixed in
  the session (two English apostrophes), small.
- The first `resend` press on a spent link before the new box registered sent nothing — **NOT-A-FINDING:** a live link
  existed (minted at the host delete, mailed 16:49); the page says „if it was a real link" by design.
- The helper chose different words than the brief for W25 („a következő éjjel", the retry is the same night when pressed
  by day) and W29 („Leszerelve. Nem költöztettünk át adatot róla; ami rajta volt, a meghajtón maradt." — „adat nem volt
  rajta" would be false) — **NOT-A-FINDING:** the brief's texts were examples; the reasons are in the CHANGELOG.
- On the installed app pages Mealie already showed the right address (R-498); W19 was wrong only on the install page —
  **NOT-A-FINDING** beyond the catalog fix.
- An empty `backups/restore-undo/` folder stays after a put-back (only the per-app folder is removed) —
  **NOT-A-FINDING:** harmless, no data.
- The soak load script's Nextcloud stream ran past the 2 hours (each stream finishes its current pass) —
  **NOT-A-FINDING:** harness.

- Memory „full" above 20 % in the night's second-drive job (~3 min) — **FILED: R-1214**. One Immich upload hung and was
  not stored — **FILED: R-1215**.
- Without host swap the freeze comes back — **recorded on R-1207** (the operator's D27), not a new row.
- One `execCommand failed: exit status 1` at 00:30:49 — **NOT-A-FINDING:** the memory check's `docker inspect` of all
  containers ran 1 s after actualbudget was stopped for its dump; the code keeps the other answers by design.
- One Radicale health probe timed out at 01:30:24 (the second-drive job starting) — **NOT-A-FINDING:** the next probe passed,
  no alarm, the household's door answered throughout.
- The guest hit its memory limit 101 times at 18:39 UTC, during the app installs, before the load — **NOT-A-FINDING:** page
  cache reclaim (0 OOM kills, swap 40 KB used), never again in 11 h.
- A 637 MB `vzdump-lxc-9100-…` on the fresh box's `local` — **NOT-A-FINDING:** the installer's golden archive (9100 = the
  golden build guest, `felhom-host-install.sh`); the restore test skips it with a reason.
- The byte-check tool `tools/nc_compare.py` turns TLS checking off — **NOT-A-FINDING:** a measurement script against a
  throwaway box with a self-signed certificate and a test domain that does not resolve; the box is destroyed.
- The first no-swap attempt reused the soak's names (Immich duplicates, Nextcloud skips) — **NOT-A-FINDING:** harness;
  stopped after 2 min, kept as VOID, rerun with fresh names (`tools/load2.py` now takes a name tag).
- CC ran an old one-off register script by mistake at 04:07 UTC; it added 16 duplicate rows to the register file —
  **NOT-A-FINDING:** caught at once by the row count, the file restored from git before anything was committed; the
  script renamed so it cannot run again.
**Fixed without a row:** the R-603 test, the restore-record log line after a put-back, the CHANGELOG blank line.
**Secrets:** names only — `NIGHT_2026_10_21_ROOTPW`, `NIGHT_2026_10_21_PAIRING_CODE`, `NIGHT_2026_10_21_BIND_TOKEN`,
`NIGHT_2026_10_21_SETUP_CODE` (the last three spent). The bind link and the setup code were read from the mailbox through
the mail connector, so both one-time values entered this session's record (spent within minutes). A scan of the evidence
for every secret used tonight: 0 hits — all 34 secret names of the night, the rehearsal and the fixes run, across the 75 evidence files, this report, STATUS, CONTEXT and the register (positive control: a known word in the evidence was found). **Poster:** no fact in `felhom-system-poster.facts.md` changed (nothing
released). **Instruction files:** none edited.