REPORT — the day brief of 2026-10-21: fix the four serious problems from the rehearsal, and the Sunday clock change

Separate from the shared REPORT.md on purpose: another session works in this clone (repo rule: a second session writes REPORT-<topic>.md). Evidence: documentation/audits/fixes-2026-10-21/. Design read first: 03 §8a, 07 §7.0/§6/§8.2, 05 (scheduler, escrow store), 04 (self-scoped reads), 09 §3 decisions D19, 78, R-241.

1. Five lines

R-1176 (a rebuilt box gets its off-site backups back) — FIXED AND LIVE. Hub 0.163.0 (operator yes, 11:46 UTC); proven end to end on demo-hp: no fresh key, recovery offered, wrong code refused, right code opened the history, Nextcloud 3 031 of 3 033 files byte-identical, the next off-site runs appended (63 → 64).
R-1175 (shares on two drives) — FIXED AND LIVE. Controller 0.337.0 (global floor 15:54 UTC), D25: a gone source keeps its copy until the household removes it.
R-1191 (an imported app keeps its family lock) — FIXED AND LIVE. Controller 0.337.0; proven on 9202: a stranger gets 401 after export → remove → import (it was 200).
R-1192 (the 8 GB freeze) — CAUSE MEASURED, FIX COMMITTED, NOT RELEASED. The guest's swap was capped at Proxmox's 512 MB default; the agent fix (swap = RAM) is committed (286a7e3); its release and vouch are the operator's (D26). Proven by applying the same setting by hand for 2 hours of the same uploads.
Ready for Tester 2: NO — one step left: the agent release with the swap fix. Everything else on the P1 list is live.

2. R-1192 — the mechanism, the numbers, the fix

One sentence: the guest's swap is capped at 512 MB (Proxmox's LXC default — the agent never set swap), so when Immich loaded its analysis models (+1.15 GB) on top of 8 apps, the 4096 MB guest could evict only program code, and ~25 containers re-read their code from disk without end.

	the freeze (13:26–13:34 UTC, cap 512 MB)	the proof (13:43–15:43 UTC, cap = RAM by hand)
guest private memory (anon), max	3 585 MB	2 782 MB
guest swap used, max	511 MB of 512 (full)	2 555 MB
refaults per 10 s, max	2 741 774	187 462
disk reads per 10 s, max	12 GB	0.9 GB
memory „full" pressure (avg10), max	96 %	12 %
IO pressure, max	97 %	53 %
load, max	162	33
guest „limit hit" events per 10 s, max	3 889	0
OOM kills	0	0
dashboard (DooPlex probe every 10 s)	dead 13:28:18–13:34:48	every probe answered, slowest 1.26 s

The experiment between them: pct set 9201 --swap 4096 at 13:34:06, uploads still running → memory „full" 96 % → 1.7 % within 2 minutes, the dashboard back 42 s after the change. The host had 2.2 GB RAM and 7.4 GB swap free throughout. Load for the proof: the same uploads as fresh work, repeated — Paperless 110 cycles × 53 PDFs (emptied each time), Immich 111 new users × 306 photos (305 accepted each time; one photo is refused every cycle), two Nextcloud trees (WebDAV small files, ~20 files/min under this load — slow, nothing lost). Method: a recorder on the box (tools/rec.py, 10 s, copied to DooPlex every minute) and the second channel on DooPlex (tools/outside.sh: the dashboard's /api/health and QEMU's disk counters from demo-hp).

Fix: agent 286a7e3 (commit only): swap = memory at bring-up, at a RAM resize, and once at agent start for every pool guest whose swap is below its RAM (live, no restart, never lowers). Tests red-proved. The controller half (0.337.0): „Normál használat mellett ez nem okoz problémát." is gone; the install page now says „Az alkalmazások csúcsterhelése meghaladhatja a rendelkezésre álló memóriát. Ilyenkor a doboz nagy feltöltések alatt lelassulhat, és akár percekig sem válaszol. Ha ez gyakran előfordul, távolíts el egy alkalmazást." No catalog change and no admission-ceiling change: the measured cause is the swap cap, and the order of preference (catalog → controller → agent) gave way to the proof. R-1193 (an earlier, plainer alarm): a design note in its row; build it only if a freeze is seen after the agent release.

3. R-1176 — the shape, the order check, the live proof

Shape (operator ruling): store.ReattachRetainedEscrow moves the newest retained row of the customer's newest escrow-ACKED host deletion into host_escrow for its one live host, in one transaction (no other live host, no live package, the deleted host not live again, an identity blob present). Called on the agent's host report and before the controller ACK's escrow status. Never opened, re-sealed or logged; audit escrow_carried (host ids + hash); two live hosts → escrow_carry_refused mailed once. Self-scope unchanged: the new host reads only rows attached to it (its retained GET for the old host is 403).

Order check (measured, not assumed): the hub carried at the agent's FIRST report (12:04:21 UTC); the controller's settle gate went GO at 12:06:25 after its first ACK; at 12:06:28 it logged „NOT minting a repository password: the hub holds a sealed recovery package". So no controller change was needed; 0.337.0 still records the escrow before SetFloor (a pinned belt for a hub that carries later).

Live proof (demo-hp VM 9321, Tester4, public ISO 1.29.0, no operator press before the install): bind (one OCR misread of the pairing code cost one of five tries — harness), claim, the dashboard offered „A korábbi, házon kívüli mentéseid visszaszerezhetők"; a wrong code: „A megadott helyreállítási kódot nem fogadtuk el … Semmi nem változott"; the right code unlocked 9 groups; Nextcloud reinstalled („előbb telepítsd újra" is the page's own order), „Teljes visszaállítás" with the placement confirm → „3113 fájl és 2 adatkötet és az adatbázis visszaállítva"; sha256 through Nextcloud's own WebDAV against the rehearsal's A5 manifest: 3 031 of 3 033 byte-identical — the 2 missing never reached the old box (the 1.5 GB file Nextcloud refused, a symlink rclone skips); 4 extra files are the rehearsal's later uploads. Off-site: two runs, 63 → 64 snapshots in the same 1.4 GB store, orphaned: false; the hub's full log has no set-aside and no key change.

4. Versions, commits, tests

Component	Version	State	Commits
hub	0.163.0	deployed 11:46 UTC (operator yes), Synced/Healthy	`9688e76e`, manifest `cd3d0598`
controller	0.337.0 (`sha256:264af313…`)	released: rc.1 on 9202; per-box floors 13:59 → three test boxes on 0.337.0 by 14:06; box walk green; global floor 0.337.0 / MinAgent 0.158.1 at 15:54:29 UTC, read back; per-box floors cleared	`8f5bea0`, `5929b8e`, `7848d4a`
agent	(0.162.0 when released)	committed, not released	`286a7e3`
catalog, installer	—	unchanged	—

Tests (all pass; each red-proved — seen failing first): hub TestR1183_* (3), TestR1176_* (store 7, api 3), TestR1182_* (store 2, api 1), TestR1203_*; controller TestR1175_* (3), TestR1181_* (2), TestD25_* (3), TestR1179_*, TestR1188c_*, TestR1191_* (2), TestR1176_EscrowIsRecordedBeforeTheFloorIsKnown, TestR1182_* (2); agent TestR1192_* (2) + the resize test's swap assertion. CI: green by job id for every push (see §5).

5. Morning list, teardown, rows, observations

Morning list (operator): D26 — release + vouch the agent with the swap fix; delete host Tester4-f6b67c (asked today); R-1205 (the whole-box tier waits for „Re-issue PBS credentials" after a failed second install); the REHEARSAL_2026_10_20_* / FIXES_2026_10_21_* names in sec.

Operator acts today: the hub-deploy yes; the root credential reveal for the freeze test (~2 min); the host delete (done 16:49 UTC — the delete appears 6 minutes after a box goes off, R-30's rule).

Teardown (three layers):

Machine (demo-hp, Tier 0): VM 9321 rh1021-box2 and helper CT 9331 rh1021-loader destroyed with --purge at 16:27:40 UTC (names checked first); no image left on nvme-scratch; 341, 9201, 9202, 9401 untouched. 9202 moved from 0.337.0-rc.1 to 0.337.0 by hand (it does not take the hub's floor). The public ISO 1.29.0 was only read.
Hub: the operator deleted host Tester4-f6b67c (16:49:37 UTC, escrow acknowledged): no Tester4 host, no current package, the household's package retained under Tester4-f6b67c with the same hash, repo_tag 'live' — a future rebuild carries it again; the WireGuard peer gone (P6-release/05-teardown-hub.txt).
ep0 / Storage Box (read only, nothing removed): Tester4's off-site store gained 2 snapshots (63 → 64, normal traffic); the hub's registrar pinned the rebuilt box's key and removed the dead box's line (normal enrollment). No PBS descriptor was made (R-1205), so nothing new on ep0's PBS.

Rows: before 182 (181 after a parallel session closed R-1201); after 176; opened 2 (R-1205, R-1206); closed 7 (R-1176, R-1203, R-1175, R-1179, R-1181, R-1182, R-1191). R-1183 → VERIFY (due 2026-10-26); R-1192 → WAITING-ON-OPERATOR; R-1188 (c) fixed. Capability map A12, A13 → PROVEN-LIVE. unproven.py --summary: unchanged, 35 of 55 not walked.

Observations:

PBS auto re-issue refused for the rebuilt Tester4 (latest deletion un-acked) — FILED: R-1205.
Future-dated snapshots from the clock test show as „Legutóbbi mentés: 2026. 10. 28." — FILED: R-1206.
The pairing-code OCR misread one character — NOT-A-FINDING: harness (a human reads the screen); corrected against the hub's record.
Immich's first admin needs the household's dashboard session (the setup gate) — NOT-A-FINDING: designed.
The full restore asks to confirm a new place (backup on hdd_1, now the internal disk) — NOT-A-FINDING: designed (R-351).
Nextcloud's small-file WebDAV upload ~20 files/min under heavy concurrent load — NOT-A-FINDING: slow, not lost; the new install-page sentence says big uploads can be slow.
kubectl logs -l reads only 10 lines — NOT-A-FINDING (a tooling trap; saved to memory, two checks redone over the full log).
Tester 2 last reported 2026-10-04 on 0.292.0; the floor reaches it only when it reports — NOT-A-FINDING (R-1139 covers the box).

Secrets: names only — FIXES_2026_10_21_PAIRING_CODE, FIXES_2026_10_21_BIND_TOKEN, FIXES_2026_10_21_SETUP_CODE (one-time, spent), FIXES_2026_10_21_ROOTPW (the operator's reveal; the box is torn down). The bind link and the setup code were read from the mailbox by the mail connector, so both one-time values entered this session's record (spent within minutes, as in the rehearsal). A scan of all 45 evidence files for every secret used today: 0 hits. Nothing secret in any commit.

11 KiB Raw Blame History Unescape Escape