REPORT — night 2026-10-22 → 23: the stray-copy fix, D30/D31 built, a fresh review, one full test night, the DooPlex survey

Night brief 20:15 → 08:00, unattended. Nothing was released, deployed, published or vouched. No change on ep0 or to DooPlex's configuration. Design read: 07-backup-architecture.md §6.3, §8 (now §8.2b); 03-host-agent.md and 06-offsite-connectivity.md (the WireGuard peer, the key, the full-list push to ep0); 05-hub-architecture.md (re-issue after a host delete, §15.4b); 01-topology-and-trust.md, runbooks/total-loss-of-dooplex.md, runbooks/RUNBOOK-hub-db-offsite-backup.md, runbooks/gitea-restore.md, runbooks/secrets.md.

Safest readings taken (unattended): (1) D29/D30/D31 are the operator's accepted picks (brief §0); built as picked. (2) The test box binds to Tester4 with the bind link the hub minted at 2026-10-22 07:37 (unconsumed, valid to 10-29) — no new link was minted. (3) The test box's root password is read from the hub vault (the break-glass path) because the day-0 install rotates it. (4) No hub image is built tonight: the morning builds and deploys hub 0.167.0 in one step (a hub image build in an unattended session is not CC's, 09 §3 decision 162). (5) Review fixes to code that copies or deletes customer data were made only with a red-proof AND a 9202 proof (rc.8).

0. Five lines (plain words)

Stray-copy fix proven: the night copy's container can no longer outlive the controller — removed by name on an error or timeout, every leftover removed at start, a restore refused while a copy runs; proven twice on the scratch box (rc.5, and rc.8 after two more review rounds).
D30 and D31 built: D30 proven live — a box's tunnel that ep0 refused healed itself with a new key in 20 minutes (and a short clock jump heals by itself, no new key needed); D31 built and red-proved (hub, not deployed).
The review: four fresh reviews and the commits' own security review — 41 points; every real one is fixed (code, red-proved; copy code also on the scratch box) or a row; one question for you (D31's reach, morning item 1).
The test night: PASS — a fresh 8 GB box with the release set, a household and 3.5 h of uploads: at night 0 of 1 833 health checks failed (worst gap 12 s), every night job ran and passed, nothing uploaded was lost, the 03:30 copy peaked at 14 % memory stall (39 % on 2026-10-21). One gap: no whole-box copy to ep0 (R-1232, likely an old copy counted).
DooPlex's three biggest gaps: no printed break-glass sheet (without it no off-site copy opens), the image registry and agent packages exist only on DooPlex (no update, reinstall or install after a loss), and the website, installer and hub all hang on the home line.

4. Part 4 — one full test night (Tester4-16fdef, nested VM 9344)

Installed tonight from ISO 1.30.0; the release set by hand (controller rc.6 → rc.8, agent 0.163.0-night.1 → 0.162.99 = agent main, the D4 drop-in by the bundle's own steps); the rehearsal's 8 apps; household seed 20261023 (3 414 files, 3.5 GB); uploads 20:04–23:38 UTC; the night chain untouched. Evidence: audits/night-2026-10-23/P4-testnight/ (00-provenance.txt first — it records the extra load below).

	Load window 19:56–23:38 UTC	Night 23:38–05:00 UTC
health probes / failed	1 175 / 2 (both explained, below)	1 833 / 0
worst gap between answers	23 s (22:03, my planned controller swap)	12 s (slowest answer 0.27 s)
worst memory „full" (10 s, host / guest)	22.7 % / 24.3 % (1 sample)	28.2 % / 30.6 % (00:32, the dump leg; 1 sample)
samples > 20 %	2, never two in a row	4, never two in a row
the 03:30 second-drive copy	—	13.6 % / 17.4 % at most (2026-10-21: 39.4 % / 45.5 %, ~3 min)
OOM kills	0	0

Night jobs, all on time, all passed: dumps 00:30 (3 m 55 s; Immich's model cache „excluded by the template — not dumped" = D21); second drive 01:30 (8 apps, 4 m 47 s, each copy in its named capped container, none left after); off-site 02:15 (8 apps, 80 snapshots, 5 m 27 s); proof 03:30 PASSED; integrity 04:00 PASSED (100 % re-read).
Uploads: Immich — 90 users, each exactly 303 photos (the set's 306 minus 2 duplicates and 1 empty file refused by design): nothing lost (R-1215 did not recur); Paperless — every upload accepted; Nextcloud — 3 031 of 3 033 files byte-identical to the manifest (sha256; the 2 are the uploader's by-design skips: the 1.5 GB file over its 1 GB limit and the link to it — the same as 2026-10-21) (18-nextcloud-sha256.txt).
D4: root password over SSH refused, root key works, the web page's password login works (console not re-run tonight; proven 2026-10-22). D30: proven on this box before the uploads (Part 2).
The two failed probes: 20:16:34 — the hub judged the box „stuck" while it only waited for the household's recovery code, re-issued the off-site login, and the box's controller restarted (R-1230, fixed in hub source tonight); 22:03:40 — my planned switch to rc.8.
Extra load, said plainly: the first (void) load attempt's loop was not stopped (my pkill -f killed its own shell first) and ran beside the planned load 19:56–22:14 — about twice the planned upload work for two hours. The box carried it without a gap.
Not seen tonight: the whole-box copy to ep0 did not run; most likely the old Tester4 box's ep0 copy (same guest number) counts as this box's weekly copy — R-1232 (P2, likely, not proven). The local whole-box tier ran at install (19:20) and is due again after 24 h, correctly.
Other: one [ERROR] execCommand failed line naming nothing (also on 2026-10-21) — R-1233.

3. Part 3 — the fresh review: findings → verdicts

Three read-only reviewers (controller v0.339.0..main, agent v0.162.0..main + the D4 patches, hub 0.165.0..main + the website patch), then a fourth on tonight's own new code, then the commits' automatic security review. Each finding checked at file:line by the main session.

#	Finding	Verdict
C1	the capped copy's container outlives the controller	REAL → fixed (Part 1)
C2	two more `.fab` import paths merge old and new data	REAL, pre-existing → R-1187 (recovery code) — not a hold
C3	the import refusal also catches an unhealthy app, wrong advice	REAL → text fixed; the path → R-1187
C4	`-v` creates a missing path on the daemon side	REAL → `--mount`
C5	the catch-up holds the whole-box backup during its slow checks	REAL → fixed, red-proved
C6, C8	comments / README overstate	REAL → fixed
C7	image tag, no `--pull never`	REAL → fixed; „no memory cgroup": NOT-A-FINDING (every guest is an LXC on a cgroup-v2 Proxmox kernel)
A1	`systemctl reload ssh` fails on a stopped / socket-activated sshd → bundle route frozen	REAL → D4 patch v2 (`try-reload-or-restart`)
A2	nothing checks the drop-in is in force	REAL → v2 reads `sshd -T` back, reports `sshd_dropin_not_in_force`
A3	an undone bundle leaves sshd reloaded with the drop-in	REAL → v2 reloads in undo
A4	downgrade / rollout order	NOT-A-FINDING: bundles go forward by signed jobs only; the morning list ships both halves in one step
A5	the installer swallows a failed reload	REAL → installer patch v2 warns
A6, A7	a wrong citation; „may only ever" wording	REAL → v2
A8 / H1	after a reboot the future stamp is gone, nothing alarms	REAL → R-1223
H2, H7	„ok" said to be mailed; WARN for ok	REAL → comment and log level fixed
H3	the operator cooldown is per customer, not per host	REAL → R-1224
H4	the alarm repeats after the 90-day event prune	NOT-A-FINDING: a reminder every 90 days for a tunnel still dead is wanted; host_swap does the same
H5	a wire-gate entry is false now	REAL → fixed
H6	two reports at once could raise two events	NOT-A-FINDING: the agent reports from one loop, one at a time
W1	the website patch applies, no over-promise	—
W2	8 GB minimum cannot run Immich	REAL → R-1225 (the operator's wording)
W3	FAQ: data leaves home only encrypted (Cloudflare reads tunnel traffic)	REAL, pre-existing → R-1226 (the operator's wording)
N1	D31's „never made a key" = „held no escrow row" — a deleted install that held a PBS secret is skipped too	REAL as a mechanism → hold or go (morning item 1); CC: go
N2	one failed `docker rm` refuses every restore until a restart	REAL → fixed (per-name strays)
N3	the start-up retry removes this process's own copies	REAL → fixed
N4	a second copy of the same destination kills the first	REAL → fixed (one per destination)
N5	`RunTier2` from the „Tier 2 on" save ignores a running restore	REAL, pre-existing → fixed
N6	a power cut during the key rotation can leave no key	REAL → fixed (fsync, link, rename over)
N7	rotation can repeat hourly; the first old key overwritten	REAL → fixed (first key kept, bound saved)
N8	„within about 15 minutes"	REAL → text fixed
N9	the import refusal advises removing an unhealthy app	REAL → text fixed
S1	(security review) the start-up removal could hit a copy starting between list and remove	REAL → fixed (the destination's lock)
S2	(security review) a restore beginning between the copy's check and its record	REAL → fixed (one step under the lock; every leg asks again)
S3	(security review) a wait stamp dated after now blocks the re-key	REAL → fixed

1. Part 1 — the stray-copy fix

The capped second-drive copy (R-1214) ran rsync -a --delete in its own container; killing the docker CLI (the 60-minute limit, a restart, a self-update) left the container deleting outside every guard. Controller 66a69d4, 912e592, then after the reviews 943c517 (C5), 9d9e404 (N2–N5, N9), 1babfe9 (two races): a fixed name + label; removed by name on any error; every leftover removed at start before the scheduler and the web server; a restore refused while a copy runs or may run; one copy per destination at a time; RunTier2 refuses during a restore for every caller, in one step with its record; --mount, the image ID, --pull never, --no-healthcheck. Proof on 9202: rc.5 and again rc.8 (audits/night-2026-10-23/P1-straycopy/): a restart in the middle of a 16 GB copy → the old container removed at start, by name, logged; a restore probe during the copy refused; the redone copy equal to the source; a 15-second test build → the timed-out container removed, the partial copy stopped growing; two runs into the same destination → the second waited, no kill, no failure. 9202 back on 0.339.0.

2. Part 2 — D30, D31, D29

D30 (agent ae69ad4, hardened 39fb9cb, b9b831f; hub 84751cb0, c6649dd4): live on the test box (P2-D30/). Run 1: a 2-hour clock jump healed by itself in 3 minutes — ep0 started a handshake (it had traffic for the box) — and the agent correctly did not re-key. Run 2 (ep0's handshake starts dropped on the box, the rehearsal's condition): dead tunnel → 15 min → RE-KEYED → the hub took the new key in place (existed=false generation=3 sync=ok, same 10.77.0.5) → the box switched at its next desired-state fetch → a handshake with the new key, tunnel ping ok. The live hub (0.165.0) stored the alarm's input (handshake_future_s=6851); the alarm and its clearing are pinned by hub tests (0.167.0, not deployed). Nothing new on ep0: the hub's existing full-list push.
D31 (hub 84751cb0): the F-14 re-issue reads the newest deletion of a host that held a key; red-proved. The review's N1 is the morning's hold-or-go question.
D29: recorded (CONTEXT, R-925).

5. Part 5 — the DooPlex survey (read only)

documentation/audits/dooplex-survival-2026-10-22/PLAN.md: every thing on DooPlex with where it lives, what copies it, where the copy is, who can restore it, how long — measured from unit states, journals, sizes, pods, DNS. Both nightly off-site jobs and both weekly restore tests are green. total-loss-of-dooplex.md marked step by step (works today / works once / never tried; one stale line fixed: the mail hold shipped in hub 0.145.0). Ten gaps, each with two options and CC's pick; four go to the operator as D32–D35. The three biggest: the paper sheet (without S1 no off-site copy opens), the registry and agent packages (only on DooPlex — no update, reinstall or install after a loss), and the home line (website, installer, hub — new installs stop).

4. Records

Commits.

felhom-controller: 66a69d4, 912e592, 943c517, 9d9e404, 1babfe9, 23f77f9 (docs).
felhom-agent: ae69ad4, 39fb9cb, b9b831f, b45c47b (REPORT).
felhom.eu: 84751cb0 (hub D30/D31 + Part 1 evidence), 75854432 (D30 proof, D4 v2, survey, 03/05/07, CONTEXT), c6649dd4 (hub text), 70a600d0 (register), 1805f8bb (STATUS), + the report commit.
CI by commit (Gitea jobs, all success, attempt 1): controller 66a69d4 2227, 912e592 2228, 943c517 2231, 9d9e404 2233, 1babfe9 2234, 23f77f9 2240; agent ae69ad4 2229, 39fb9cb 2235, b9b831f 2237, b45c47b 2241; felhom.eu 84751cb0 2230, 75854432 2232, c6649dd4 2236, 1805f8bb 2238 (with 70a600d0), e94d0256 2239 (with 14388e21), 5dc17cab 2242 (with f56f3328), 7802da72 2243 (with 1cb3e571) — a commit pushed with a later one has no job of its own; the last report commit's job is checked after this file.

Images. Controller 0.340.0-rc.5 (= 66a69d4), rc.6 (= 943c517), rc.7 (= 9d9e404, built, not used: the security review came first), rc.8 (= 1babfe9). No hub image (the morning builds 0.167.0 with its deploy). A test-only controller image (rc.5 + a 15 s limit) lived only on DooPlex and 9202 and was removed from both.

Dated checks due today, done: R-32 (Tester4's listing 72 → 80, none vanished), R-1173 (the other outcome).

Rows: before 180, after 191. Opened 11: R-1223 (the tunnel reboot gap), R-1224 (the per-customer cooldown), R-1225 and R-1226 (two website sentences, D36), R-1227 (the registry, D33), R-1228 (the home line, D35), R-1229 (ArgoCD settings), R-1230 (a held box reported stuck — fixed in hub source), R-1231 (the recovery page says „older"), R-1232 (the first ep0 copy of a reinstalled box), R-1233 (an error line naming nothing). Closed 0 — the night's fixes close on release (R-1214, R-1204, R-1205, R-1230 are READY). Updated: R-1214, R-1204, R-1205, R-925, R-1187, R-982, R-232, R-923, R-924.

Teardown, three layers.

Machine: 9202 back on 0.339.0, healthy, test files and the test image removed, no copy container (rc.5 / rc.8 stay in its own image store, as earlier rc images). The test box: VM 9344 destroyed with its disks, after every log was copied off (P4-testnight/13–21).
Host (demo-hp): VM 9344 and loader CT 9345 destroyed (own IDs; disks were on nvme-scratch, never the thin pool); /tmp clean. Standing items untouched: VM 341, CT 9401, the ISOs in local (see R-548).
Hub: host Tester4-16fdef stays registered (it will read „down") — the operator deletes it on the Hosts page, with appliance 49 (morning item 6). Tester4's off-site store holds tonight's 8 new snapshots (80 in all; left, as on 2026-10-21); its PBS token was re-issued automatically at the install (F-14, product behaviour); its WireGuard peer holds the D30 key until the host is deleted.
DooPlex: nothing changed — the survey read only; builds and the scratchpad only; the local test image removed.

System poster: no listed fact changed (no machine, time, tier or custody moved; the WireGuard key rotation is not a poster fact) — none of the three actions needed. Instruction files: none edited.

Observations.

demo-hp (a standing box): its local whole-box tier was refused 4 times tonight by the agent's space check (8.5 GB free, 9.2 GB needed), and the controller stopped 11 apps for about a minute before each try — FILED: R-548 (updated; the superseded installer ISOs on local are the easy half).
The capped copy inherited the controller's health check („starting") — NOT-A-FINDING: small, fixed in the session (--no-healthcheck, 912e592).
One outside health check answered 502 at 20:16:34Z: the hub's false „stuck" re-issue restarted the controller — FILED: R-1230 (fixed in hub source tonight).
The Nextcloud tree upload takes ~3 h under the load (same as 2026-10-21: 18:53 → 21:56). — NOT-A-FINDING: the same load gives the same time; not a regression.
The soak tools assumed the setup gate takes the dashboard cookie; the gate now hands out its own cookie (felhom_gate) through /__gate/start. — NOT-A-FINDING: the product works as designed; the tool was updated.
The first load attempt (19:55Z) got 404/401 everywhere and was stopped after 2 minutes, logs kept as void-1955/ in the loader. — NOT-A-FINDING: the tool's gate cookie (above); nothing on the box changed.
The VOID 9202 run's restore probe started a restore of a nonexistent app, failed at once. — NOT-A-FINDING: no data; explained in P1-straycopy/README.md.
A test request's error printed one character of a test password into the session (a Python traceback). — NOT-A-FINDING for the product; the probe now encodes the password and prints nothing of it.
The off-site unlock was refused once: the agent fixture's pre-release version string („0.163.0-night.1") is not a version the controller can compare, so its capability read „unknown" — NOT-A-FINDING for the product (no pre-release agent is ever released; the fixture was rebuilt as 0.162.99); the page's wording for „unknown" — FILED: R-1231.
The household's off-site snapshots carry the rehearsal's future date (2026-10-28) — NOT-A-FINDING: known, R-1206; retention was not due tonight, so its case did not come up.
The night copy sat at the guest's memory limit while it ran (77 845 limit events, 0 OOM) with almost no stall — NOT-A- FINDING: the capped copy's cache is reclaimed inside the guest by design; the consequence (pressure, health) is the measure, and it improved.
CC's own mistakes: (1) a commit (70a600d0) was made while its fast gate failed — the command chained the commit after printing the gate's code; the push hook refused it, STATUS was fixed, and both commits went up together with green gates. (2) Two gate runs at once (mine and the push hook's) made the decoy test fail; it passes alone. (3) The void load loop ran on for 2 h 18 min because my pkill -f load2.py matched its own shell (a known trap); found at 22:14, stopped by pid, recorded in the provenance; the same trap hit once more at the teardown (a pkill -f killed my own command; the teardown was rerun). (4) The first agent fixture used a version string the controller cannot compare, which cost one refused unlock.

19 KiB Raw Blame History Unescape Escape