felhom.eu/REPORT-night-fixes-2026-10-05.md

REPORT — the night's fixes, the power cut by day, a box that is off at night (2026-10-05)

Brief: "fix what the 2026-10-04 night found …" (operator, 2026-10-05). Evidence: documentation/audits/night-fixes-2026-10-05/. Architecture read before the claims: 07-backup-architecture.md (§6.1, the Tier-3 row), 11-os-updates.md (§5.4.1, §8), 09 §3 (decisions 53, 68–74, 81, 86), audits/offsite-append-only-2026-10-03/DESIGN.md.

1. The Part table

Part	State	Note
§1 rulings recorded first	done	09 decisions 100–103; R-870 (tokens, like R-831); Tester 2's page in runbooks/target-selection.md
A — clean-up guard (R-867)	done	controller v0.294.0; tests run restic 0.14.0's policy (proven identical to the binary); 5 red-proofs; live windows on both demo boxes with counts; R-867 and R-95 closed (DUE-CHECKS entry removed)
B — image clean-up race (R-863, R-864)	done	one lock for every pulling compose verb; the night's shape reproduced in a test and live on 9202 (fired at minute 3, skipped, install OK first time)
C — R8 real download (R-865)	changed	fix + tests + red-proof done. Live: the INSTALLED wrapper's measure read 12 802 456 B on demo-hp. No R8 refusal line was produced: it needs < 500 MB free on demo-hp's guest = 28.5 GB written into a thin pool with 18.7 GB free — it would have stopped every guest
D — R-868, R-869, R-866	done, with a second agent release	R-868's first fix (v0.144.0) was measured NOT to work live (broken stderr pipe); v0.144.1 fixed it and the A5 shape then delivered one applied report. R-866 live with the hub blackholed. R-869 test + red-proof
E — power cut mid-update (A1 by day)	done	operator's go; crash 06:13:55 UTC, back by itself in 37 s; the next pass failed → R-876 (P2); by-hand repair; demo-hp left with every package current
F — a box off at night (spike)	done	read only; partF/FINDINGS.md; no architecture covers it → R-871; R-872..R-874; STATUS decision A/B
G — releases, golden, records	done	controller v0.294.0; agent v0.144.0 + v0.144.1 (decision 108); golden 0.294.0 baked + vouched (gate OK); 07, 11, 00, 09 updated

2. Claims in the brief that turned out wrong (or only partly true)

1. "The policy's constants can drive the guard's line" — true, and done — but with a cost the brief did not name: a line derived from keep-daily (calendar days) can no longer catch a past-dated gap-fill that steers a 7–8-day-old keep, which the 8-day line caught while refusing every honest window. Past-dated fakes can never make the policy drop a snapshot inside the last keep-daily calendar days, so the line now catches only the skew-window shape. The rest is R-822's residual, bounded by the cap and the hub's count check (decision 104).
2. "The same race exists in the update and restore paths" — the update path: NO, it was already guarded (no pass while any app is updating, since v0.284). The restore and undo paths: exposed in principle (they pull inside compose up), not measured; they now hold the same lock.
3. "Dropping -s downloads nothing" — TRUE, measured on 9202 (archive cache 10 → 10 files, versions unchanged).
4. "A box off at W runs nothing when it comes back" — mostly true: the database dumps, second copy, off-site copy and app updates never catch up; but the whole-guest backup DOES catch up (48 h safety valve, about every 2 days for an evening-only box), and the OS leg follows it by day under the night label.
5. "The night A5 showed the wrapper finished" (R-868's premise, and v0.144.0's design) — the packages were installed, but the wrapper process itself died on its first log line after the agent's death. Found only by running it live.
6. Part E: "the next pass must repair dpkg first and finish" — it does NOT (R-876).

3. What was proven, with numbers

• Part A. Tests: the measured shape (7 d + 5 s → allowed), 60 nights × 3 apps with two same-day manual runs across a month boundary (never refused, ≥ 45 removed), weekly windows, the skew-window fake (refused), the lab's 13 future fakes (refused). Red-proofs: 8-day line back → 3 fail; each refusal dropped → its test fails. Lab: restic 0.14.0 binary vs the test simulation over 92 snapshots: identical 72 removals. Live: demo-felhom 15 snapshots, predicted removals 92e49f62, 343d57a5 → window 5 removed exactly those (16 → 14 incl. the run's new one); demo-hp 136 snapshots, predicted 18 → window 6 removed exactly those (145 → 127). Hub rows pruned, no event, no mail, key audit 0 findings.
• Part B. Live on 9202: controller start 05:28:35, install 05:31:15, clean-up 05:31:36 "skipped — … pulling images now", BookStack deployed 05:31:54 (35.5 s), retry 05:33:36 ran (0 deleted). Teardown through the product verified.
• Part C. download_bytes = 12802456 B from the installed wrapper on demo-hp (was 0).
• Part D. R-866: block=SAVED(2026-10-05T05:23:26Z; hub unreachable: … connect: invalid argument). R-868 (v0.144.1): killed 06:04:00, wrapper DONE upgraded=13, kept copy, hub report id 63 applied (13) at 06:09:05, once.
• Part E. See 11 §8.4: back in 37 s; apps healthy within 4 min; household saw one timeline line; one operator mail os_update_failed (true); next pass FAILED (R-876); after dpkg --configure -a the pass installed 12; the guest's package list equals the pre-test one (279 lines). The first crash attempt did not fire (my watcher's pattern missed apt's dpkg call; the pass installed normally) — rolled back again and repeated.
• Part F. audits/night-fixes-2026-10-05/partF/FINDINGS.md; two claims re-checked in source by me.

4. Rows

Register before 341, after 340. Closed (8): R-95, R-863, R-864, R-865, R-866, R-867, R-868, R-869. Opened (7): R-870 (tokens, operator), R-871 (a box off at night — the operator's decision), R-872 (P2, no missed-backup alarm for a box down at 05:00), R-873 (nightly "cannot be reached" mail), R-874 (restore-test never runs on short sessions), R-875 (P4, kept-report reason text), R-876 (P2, the repair misses dpkg's update journal). STATUS updated. unproven.py --summary: unchanged — 55 claims, 35 not walked (no number moved).

5. Teardown, three layers

• Machines: 9202: BookStack removed through the product (no container, stack dir, volume); controller 0.294.0 (set by hand, allowed there). demo-hp 9201: every package current, list identical to before; my scripts and logs in /root removed after copying. The bake VM: CT 9100 destroyed, token shredded, reverted to virgin, qemu gone.
• Hosts: the blackhole routes on felhom-pve removed (in the same script). Nothing provisioned.
• Hub: two one-shot clean-up grants (consumed); floors for demo-hp, demo-felhom, tester-1 → 0.294.0; artifacts vouched (agent 0.144.1, golden 0.294.0, min_agent 0.131.0); 12 signed jobs (3× agent 0.144.0, 3× bundle 0.144.0, 3× agent 0.144.1, 3× bundle 0.144.1). Tester 2: read only, nothing sent (offline all session).
• Scratch secrets (hub password, hub key, controller password, deploy secrets) shredded at the end.

6. Notes

• The hub pod log prints a customer's e-mail address in "Customer email sent to …" lines (seen by the Part F helper; not recorded anywhere).
• demo-hp's System page read "no running customer guest" / "unknown" for several minutes after the crash although the guest ran — R-853 (facts late after a boot), not a new defect.