night 2026-10-21: soak result (no freeze in 11 h; memory pressure above 20 % for 3 min in the tier2 job, R-1214; 1 Immich upload lost, R-1215), no-swap test (the freeze comes back without host swap, R-1207), teardown, the night report

Rows 182 -> 184. STATUS morning list item 2 carries the soak result; CONTEXT the two results.

documentation/backlog/OPEN-ITEMS.md

@@ -124,7 +124,7 @@ stopping line that lies.
 | **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC |
 | **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator |
 
-## Apps & catalog — 14 rows (P3 6, P4 8)
+## Apps & catalog — 15 rows (P3 6, P4 9)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -142,6 +142,7 @@ stopping line that lies.
 | **R-1067** | Apps & catalog | P4 | **The catalog still carries `pi_compatible:` on every template, and nothing reads it any more.** controller v0.315.0 removed the „Pi kompatibilis" badge and the „csak x86" tag from every page (Raspberry Pi is no longer supported, so one was noise and the other was a warning on almost every app). The `ResourceHints.PiCompatible` struct field and the catalog key were deliberately LEFT: a box still on 0.314.0 reads the key, and removing it would make that box show „csak x86" on every app. **Gated on the floor:** once the fleet floor is at or above 0.315.0, drop `pi_compatible:` from the templates and the field from `stacks.ResourceHints` in the same pair of commits. Harmless until then — an unread key costs nothing. | **BLOCKED** — filed 2026-10-15. Checked 2026-10-16 (apps burndown): the floor is 0.321.0 and demo-hp, Tester 1 and the N100 run 0.321.0, but **Tester-2 still reports 0.292.0** (its newest report 2026-10-04; hub DB with WAL) — so nothing removed. | every box on ≥ 0.315.0 — today Tester-2 | Drop the catalog key and the struct field; grep the hub and the website for Pi wording at the same time (none found 2026-10-15) | CC |
 | **R-1043** | Apps & catalog | P4 | **Sonarr, Radarr and Prowlarr build their login redirect as an absolute `http://` URL.** They trust forwarded headers only from their „Trusted networks” (the household set `192.168.0.0/24`; traefik is `172.18.x`), so they ignore `X-Forwarded-Proto: https` and answer `Location: http://<name>/login` to an https visitor. Since controller v0.312.0 that hop is sent back to https by traefik (a GET, no password), so it is cosmetic; a template could seed traefik's network as trusted. (`audits/https-only-2026-10-14/``1A-3-4.txt`) | **OPEN** — filed 2026-10-14 | — | Seed the docker network as a trusted proxy in the *arr templates (with the R-1030 seed), measure the redirect | CC |
 | **R-1099** | Apps & catalog | P4 | **The N100 may hold RomM games in the old upload place, now shown under a fake „roms” platform.** Until 2026-10-16 RomM's own upload wrote to the drive's `roms/roms/<platform>/`; after the mapping fix (catalog `fb18471`, R-1089) those files are at RomM's `library/roms/roms/…`. Measured on 9201: such a game showed as missing in its platform and again under a platform „roms” (`audits/romm-ghost-2026-10-16/B/b-after.txt`). Files put in with the file manager are not affected — they are now found. The N100 runs RomM (hub `/apps/romm`); CC does not log in to the lived-in box. | OPEN — filed 2026-10-16 | Operator: look at `roms/roms/` on the N100's drive in the file manager | If it holds platform folders: move them up one level into `roms/`, then run „Scan” in RomM; if it is empty, close | Operator |
+| **R-1215** | Apps & catalog | P4 | **One Immich photo upload hung 10 minutes with no answer and was not stored, while the box was healthy.** Night soak 2026-10-21 (`audits/night-2026-10-21/P1-soak/08-load/proof.log`): cycle 36, `IMG_20120021.jpg` sent 19:38:30 UTC, the uploader gave up at its 600 s read limit; Immich's database holds 302 photos for that user, the other cycles 303 (`P1-soak/09-immich-cycle36.txt`). At that moment memory „full" pressure was 3.9 % and `/api/health` answered in 0.26 s. 1 of about 26 000 uploads that night. Not diagnosed: Immich restarted at 02:16 for the off-site dump, so its log of 19:38 is gone. The Immich phone app retries a failed upload. | OPEN | — | Watch the next long upload run with the Immich server log saved; file upstream only with a log | CC |
 
 
 ## App updates — 4 rows (P3 4)
@@ -240,17 +241,18 @@ stopping line that lies.
 | **R-1054** | Security & access | P3 | **The hub's operator/global API key is still one bearer that can do everything on `/api/v1`, including reading every box's break-glass root password (`GET /admin/hosts/{id}/recovery-credential`).** Hub v0.152.0 gave the web surface scoped tokens (`05` §17) but tokens are refused on `/api/v1` by design. Who uses the global key today is not listed in one place. Fix: list its consumers; move each operator use to a scoped token on the web side (or a token class on the API), and keep the global key only where a box path needs it — or retire it. | OPEN — filed 2026-10-15 by the token build | — | List the global key's consumers (pod env, scripts, runbooks) | CC |
 | **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f <conf>` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WAITING-ON-OPERATOR — fixed on main, unreleased; ships with the agent release (D26) | the agent release (morning list 2) | Release; the next fresh install must raise no oob_degraded; then close | operator |
 
-## Box system & updates — 13 rows (P1 1, P2 2, P3 6, P4 4)
+## Box system & updates — 14 rows (P1 1, P2 2, P3 7, P4 4)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
-| **R-1192** | Box system & updates | P1 | **On the minimum machine (8 GB), the household's first afternoon froze the box for about an hour: the dashboard and every app stopped answering, and only pulling the plug brought it back.** Rehearsal 2026-10-20 on a fresh box (controller 0.336.0, agent 0.161.0): the install page let the household book 8 apps to 3638 of 3712 MB (98 %) — the guest has 4096 MB in all — and told it on every page „Az alkalmazások csúcsterhelése meghaladhatja a rendelkezésre álló memóriát. **Normál használat mellett ez nem okoz problémát.**" Then the first uploads (53 PDFs to Paperless, photos to Immich, a file tree to Nextcloud — sequential per app) → from ~13:05 UTC `/api/health` gave no answer in 15 s; the VM read ~4 GB/min from its disk (158 → 476 GB) with the network still; the box's own history shows load 139–152 and no samples for 36 min; it came back at 14:00 for minutes and froze again; the plug pulled at 14:11 → dashboard in 2 min 22 s, stable after. Lost on the way: 8 Paperless documents failed (5 ordinary invoices among them), the Immich upload cut at photo 150, Nextcloud at 251 of 3 393 files. Cause NOT diagnosed (no shell in the box; memory read ~2.3 GB while load was 150 — not plain „memory full"); the one hint: the hub's `app_oom` at 13:34 — immich-machine-learning killed by its memory limit. Reproduced a SECOND time at 15:27:55 with only one Immich upload + one chunked Nextcloud upload running (~35 min, then froze again; plug pulled 16:13); per-app memory now logged every 30 s (`B4-use/memwatch.txt`). Evidence `audits/rehearsal-2026-10-20/journal.md` (12:56–14:23), `B4-use/box-metrics-after-plug.txt`, `B4-use/vm-counters-before-plug.txt`, `B4-use/after-plug-watch.txt`. | WAITING-ON-OPERATOR **DIAGNOSED 2026-10-21 with numbers, fix committed (agent `286a7e3`), proven by applying the same setting by hand** (`audits/fixes-2026-10-21/P4-freeze/`). Reproduced on demo-hp: a nested 8 GB VM, Tester4 from the public ISO, the same 8 apps, the same uploads (generator seed 20261020) → the dashboard dead 13:28:18–13:34:48 UTC. **Mechanism:** the guest's swap is capped at Proxmox's LXC default of 512 MB (the agent never set `swap`); Immich's analysis loaded its models (+1.15 GB, `immich-machine-learning` 70 → 1 224 MB) and the server grew (+0.4 GB) on top of 8 apps' 1.75 GB, so the 4096 MB guest filled (anon 3 585 MB, swap 511/512 MB) and could evict only file pages: ~360 MB left for ~25 containers' code → 150 000–2 700 000 refaults and up to 12 GB of reads per 10 s, memory „full" pressure 96 %, IO 97 %, load 162, 0 OOM kills; the host had 2.2 GB RAM and 7.4 GB swap free. **Experiment:** `pct set 9201 --swap 4096` live at 13:34:06 → „full" pressure 96 % → 1.7 % in 2 min, dashboard back 42 s later, uploads running on. **Proof (2 h, 13:43–15:43, swap = RAM by hand):** the same uploads repeated as fresh work (Paperless 110 cycles × 53 PDFs, Immich 111 new users × 306 photos, two Nextcloud trees): 720 samples, max „full" pressure 12 %, load 33, 0 limit hits, every dashboard probe answered (slowest 1.26 s). Controller 0.337.0: the false „Normál használat mellett ez nem okoz problémát." replaced. Nextcloud's small-file WebDAV upload ran ~17 files/min under this load — slow, not lost. | the operator's agent release + vouch | Release agent with the swap fix (swap = RAM at bring-up, at RAM resize, once at start); after the update read one box's `pct config` (swap = memory) and its `guest-swap: raised` log line; then close | operator |
+| **R-1192** | Box system & updates | P1 | **On the minimum machine (8 GB), the household's first afternoon froze the box for about an hour: the dashboard and every app stopped answering, and only pulling the plug brought it back.** Rehearsal 2026-10-20 on a fresh box (controller 0.336.0, agent 0.161.0): the install page let the household book 8 apps to 3638 of 3712 MB (98 %) — the guest has 4096 MB in all — and told it on every page „Az alkalmazások csúcsterhelése meghaladhatja a rendelkezésre álló memóriát. **Normál használat mellett ez nem okoz problémát.**" Then the first uploads (53 PDFs to Paperless, photos to Immich, a file tree to Nextcloud — sequential per app) → from ~13:05 UTC `/api/health` gave no answer in 15 s; the VM read ~4 GB/min from its disk (158 → 476 GB) with the network still; the box's own history shows load 139–152 and no samples for 36 min; it came back at 14:00 for minutes and froze again; the plug pulled at 14:11 → dashboard in 2 min 22 s, stable after. Lost on the way: 8 Paperless documents failed (5 ordinary invoices among them), the Immich upload cut at photo 150, Nextcloud at 251 of 3 393 files. Cause NOT diagnosed (no shell in the box; memory read ~2.3 GB while load was 150 — not plain „memory full"); the one hint: the hub's `app_oom` at 13:34 — immich-machine-learning killed by its memory limit. Reproduced a SECOND time at 15:27:55 with only one Immich upload + one chunked Nextcloud upload running (~35 min, then froze again; plug pulled 16:13); per-app memory now logged every 30 s (`B4-use/memwatch.txt`). Evidence `audits/rehearsal-2026-10-20/journal.md` (12:56–14:23), `B4-use/box-metrics-after-plug.txt`, `B4-use/vm-counters-before-plug.txt`, `B4-use/after-plug-watch.txt`. | WAITING-ON-OPERATOR **DIAGNOSED 2026-10-21 with numbers, fix committed (agent `286a7e3`), proven by applying the same setting by hand** (`audits/fixes-2026-10-21/P4-freeze/`). Reproduced on demo-hp: a nested 8 GB VM, Tester4 from the public ISO, the same 8 apps, the same uploads (generator seed 20261020) → the dashboard dead 13:28:18–13:34:48 UTC. **Mechanism:** the guest's swap is capped at Proxmox's LXC default of 512 MB (the agent never set `swap`); Immich's analysis loaded its models (+1.15 GB, `immich-machine-learning` 70 → 1 224 MB) and the server grew (+0.4 GB) on top of 8 apps' 1.75 GB, so the 4096 MB guest filled (anon 3 585 MB, swap 511/512 MB) and could evict only file pages: ~360 MB left for ~25 containers' code → 150 000–2 700 000 refaults and up to 12 GB of reads per 10 s, memory „full" pressure 96 %, IO 97 %, load 162, 0 OOM kills; the host had 2.2 GB RAM and 7.4 GB swap free. **Experiment:** `pct set 9201 --swap 4096` live at 13:34:06 → „full" pressure 96 % → 1.7 % in 2 min, dashboard back 42 s later, uploads running on. **Proof (2 h, 13:43–15:43, swap = RAM by hand):** the same uploads repeated as fresh work (Paperless 110 cycles × 53 PDFs, Immich 111 new users × 306 photos, two Nextcloud trees): 720 samples, max „full" pressure 12 %, load 33, 0 limit hits, every dashboard probe answered (slowest 1.26 s). Controller 0.337.0: the false „Normál használat mellett ez nem okoz problémát." replaced. Nextcloud's small-file WebDAV upload ran ~17 files/min under this load — slow, not lost. **NIGHT SOAK 2026-10-21→22 (fresh box from the public ISO, the agent fix as a by-hand fixture, `audits/night-2026-10-21/P1-soak/`):** 2 h of the rehearsal uploads, then the night chain untouched — 3 389 probes, none failed, largest interval between answers 16 s, slowest 4.1 s; 0 OOM kills; every night job ran (dump, second drive, off-site 8 apps, sweep, proof, integrity). Memory „full" pressure went above 20 % twice: one 10-s sample at the Immich model load (31/35 %) and ~3 min in the 03:30 second-drive job (39/45 %) — R-1214. 1 photo of ~26 000 not stored (a 10-min hang at low pressure) — R-1215. The freeze did not come back. | the operator's agent release + vouch | Release agent with the swap fix (swap = RAM at bring-up, at RAM resize, once at start); after the update read one box's `pct config` (swap = memory) and its `guest-swap: raised` log line; then close | operator |
 | **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
 | **R-982** | Box system & updates | P3 | **The Proxmox host accepts root password login over SSH from the LAN (measured on demo-hp), so anyone on the household network can try the root password.** Design-shaped: the break-glass password exists for the console. Fix: key-only SSH for root; the password stays for the console. Evidence: `audits/security-review-2026-10-12/evidence/demo-hp-host-reads.txt`. | OPEN — design-proposal | — | Decide; then set it in the installer and the config bundle | CC |
 | **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC |
 | **R-974** | Box system & updates | P3 | **A compromised host agent can likely become root on the host through the root wrapper that applies PBS settings: the wrapper accepts a folder path inside the agent's own writable tree and then acts on it as root.** Static (the symlink behaviour was reasoned, not run). Same class as R-861. Fix: a fixed root-owned folder, and refuse symlinks before any root action. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix the wrapper with a test on a throwaway box | CC |
 | **R-972** | Box system & updates | P2 | **The shipped controller and hub programs are built with Go 1.24.13 and golang.org/x/crypto v0.31.0, and the agent with Go 1.26.0, so dozens of fixed Go security bugs (including html/template escaping fixes) are still in what runs on boxes and the hub.** Go 1.24 has had no security fixes since Go 1.26 was released (Go supports the two newest major releases: https://go.dev/doc/devel/release#policy); the scanners name fixes in Go 1.26.9 and 1.27.2. Also: the controller image's Debian 12 base has 12 fixable critical packages, and the Samba image has 48 fixable high ones. Fix: build all three on a supported, patched Go; update x/crypto; rebuild the base images. Scanner output: `audits/security-review-2026-10-12/scanners/`. | **NARROWED** 2026-10-16 (burndown 2): **controller v0.322.0 released** on Go 1.27.2 + x/crypto v0.58.0, runtime debian:12.13-slim + `apt-get upgrade` (Debian 13 deliberately not: restic 0.18, R-1105) — scanner: critical 16 → 4, fixable criticals and highs in the OS 0; govulncheck 62 → 1 (module-level, `openpgp` not linked). **Hub v0.156.0 deployed** 2026-10-16 19:26 on the operator's yes (Go 1.27.2, alpine 3.24.2: 0 critical, 0 high); Synced/Healthy, reports from all three boxes after the start, the floor reads 0.322.0. **Agent: Go 1.27.2 committed (`37b99d7`), tested, NOT released** — the next vouched agent release carries it. Open: the Samba image (48 fixable high, not rebuilt); docker-compose plugin's own Go (4 high, upstream). Evidence `audits/burndown2-2026-10-16/P4/` **2026-10-16 (evening): the Samba part is DONE in controller v0.323.0** — felhom-samba 1.2.0 on Alpine 3.24.2 (0 critical / 0 high, was 48 fixable high), proven on demo-hp with the same household password (smbclient, nmbd, mDNS, WS-Discovery); on the three test boxes now, on every box with the global floor (R-1105's night check). The controller's Debian 13 OS part: 1 critical / 70 high, none fixable. Left: docker-compose plugin's own Go (4 high, upstream); the agent release; restic's binary is R-1107. **2026-10-17 (evening): agent v0.159.1 (Go 1.27.2) RELEASED and running on demo-hp, Tester 1 and demo-felhom** (signed jobs on the operator's yes); waiting for the operator's vouch — then only the upstream compose plugin is left (close then, recorded). | the next vouched agent release | Close when the global floor carries v0.323.0 and the next vouched agent release carries Go 1.27.2 (compose plugin: upstream, recorded) | CC |
-| **R-1207** | Box system & updates | P2 | **A host with no (or too little) swap gets nothing from the guest swap fix (R-1192), and the agent never adds host swap.** The fix lets a guest swap as much as its RAM; that swap lives on the HOST. Measured 2026-10-21 night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap (LVM `pve/swap`), so the installer's boxes are covered; a BYO host (or one with swap turned off) may have none. Now reported and said (agent `0da792e`: `host_swap_mb` / `guest_swap_mb`, one WARN; hub 0.164.0 `host_swap_short` to the operator; the install page sentence) — but nothing adds swap. The no-swap test on the soak VM (`audits/night-2026-10-21/P2-noswap/`) says what happens without it. | WAITING-ON-OPERATOR | decision: should the agent (or the installer for BYO) create host swap when it is short, or should an 8 GB BYO host without swap be refused at install? | Operator decides; CC builds the pick | operator |
+| **R-1207** | Box system & updates | P2 | **A host with no (or too little) swap gets nothing from the guest swap fix (R-1192), and the agent never adds host swap.** The fix lets a guest swap as much as its RAM; that swap lives on the HOST. Measured 2026-10-21 night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap (LVM `pve/swap`), so the installer's boxes are covered; a BYO host (or one with swap turned off) may have none. Now reported and said (agent `0da792e`: `host_swap_mb` / `guest_swap_mb`, one WARN; hub 0.164.0 `host_swap_short` to the operator; the install page sentence) — but nothing adds swap. The no-swap test on the soak VM (`audits/night-2026-10-21/P2-noswap/`) says what happens without it. | WAITING-ON-OPERATOR **NO-SWAP TEST 2026-10-22 04:08–04:38 UTC (`audits/night-2026-10-21/P2-noswap/`):** the soak box, `swapoff -a` on the host, the same burst → the freeze CAME BACK within 20 s: `/api/health` dead for 289 s, then 172 s and 68 s (24 of 125 probes failed), memory „full" 36 % host / 47 % guest, the VM reading ~40 GB per 25 s from disk; Immich lost 207 of 306 photos and its stream broke; the box recovered only when the Immich stream stopped (04:23). After `swapon -a`: 0 failed probes, „full" ≤ 1.3 %. The agent logged its one WARN at the swapoff (host 938 MB < guests 4096 MB). | decision: should the agent (or the installer for BYO) create host swap when it is short, or should an 8 GB BYO host without swap be refused at install? | Operator decides; CC builds the pick | operator |
+| **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | OPEN | — | Measure the tier2 job alone on the 8 GB box (per-step pressure); decide whether 20 % over 10 s is the right line, or limit the copy (ionice / cgroup IO weight) | CC |
 | **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator |
 | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
 | **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement | — | — | CC |