felhom.eu/REPORT-night-fixes-2026-10-05.md

# REPORT — the night's fixes, the power cut by day, a box that is off at night (2026-10-05)

Brief: "fix what the 2026-10-04 night found …" (operator, 2026-10-05). Evidence: `documentation/audits/night-fixes-2026-10-05/`.
Architecture read before the claims: `07-backup-architecture.md` (§6.1, the Tier-3 row), `11-os-updates.md` (§5.4.1, §8),
`09` §3 (decisions 53, 68–74, 81, 86), `audits/offsite-append-only-2026-10-03/DESIGN.md`.

## 1. The Part table

| Part | State | Note |
|---|---|---|
| §1 rulings recorded first | **done** | `09` decisions 100–103; R-870 (tokens, like R-831); Tester 2's page in `runbooks/target-selection.md` |
| A — clean-up guard (R-867) | **done** | controller v0.294.0; tests run restic 0.14.0's policy (proven identical to the binary); 5 red-proofs; live windows on both demo boxes with counts; R-867 and R-95 closed (DUE-CHECKS entry removed) |
| B — image clean-up race (R-863, R-864) | **done** | one lock for every pulling compose verb; the night's shape reproduced in a test and live on 9202 (fired at minute 3, skipped, install OK first time) |
| C — R8 real download (R-865) | **changed** | fix + tests + red-proof done. Live: the INSTALLED wrapper's measure read 12 802 456 B on demo-hp. **No R8 refusal line was produced**: it needs < 500 MB free on demo-hp's guest = 28.5 GB written into a thin pool with 18.7 GB free — it would have stopped every guest |
| D — R-868, R-869, R-866 | **done, with a second agent release** | R-868's first fix (v0.144.0) was measured NOT to work live (broken stderr pipe); v0.144.1 fixed it and the A5 shape then delivered one `applied` report. R-866 live with the hub blackholed. R-869 test + red-proof |
| E — power cut mid-update (A1 by day) | **done** | operator's go; crash 06:13:55 UTC, back by itself in 37 s; **the next pass failed** → R-876 (P2); by-hand repair; demo-hp left with every package current |
| F — a box off at night (spike) | **done** | read only; `partF/FINDINGS.md`; no architecture covers it → R-871; R-872..R-874; STATUS decision A/B |
| G — releases, golden, records | **done** | controller v0.294.0; agent v0.144.0 + v0.144.1 (decision 108); golden 0.294.0 baked + vouched (gate OK); `07`, `11`, `00`, `09` updated |

## 2. Claims in the brief that turned out wrong (or only partly true)

1. **"The policy's constants can drive the guard's line"** — true, and done — but with a cost the brief did not name: a
   line derived from keep-daily (calendar days) can no longer catch a past-dated gap-fill that steers a 7–8-day-old
   keep, which the 8-day line caught while refusing every honest window. Past-dated fakes can never make the policy drop
   a snapshot inside the last keep-daily calendar days, so the line now catches only the skew-window shape. The rest is
   R-822's residual, bounded by the cap and the hub's count check (decision 104).
2. **"The same race exists in the update and restore paths"** — the update path: NO, it was already guarded (no pass
   while any app is updating, since v0.284). The restore and undo paths: exposed in principle (they pull inside
   `compose up`), not measured; they now hold the same lock.
3. **"Dropping `-s` downloads nothing"** — TRUE, measured on 9202 (archive cache 10 → 10 files, versions unchanged).
4. **"A box off at W runs nothing when it comes back"** — mostly true: the database dumps, second copy, off-site copy and
   app updates never catch up; but the whole-guest backup DOES catch up (48 h safety valve, about every 2 days for an
   evening-only box), and the OS leg follows it by day under the `night` label.
5. **"The night A5 showed the wrapper finished"** (R-868's premise, and v0.144.0's design) — the packages were installed,
   but the wrapper process itself died on its first log line after the agent's death. Found only by running it live.
6. Part E: "the next pass must repair dpkg first and finish" — it does NOT (R-876).

## 3. What was proven, with numbers

- **Part A.** Tests: the measured shape (7 d + 5 s → allowed), 60 nights × 3 apps with two same-day manual runs across a
  month boundary (never refused, ≥ 45 removed), weekly windows, the skew-window fake (refused), the lab's 13 future fakes
  (refused). Red-proofs: 8-day line back → 3 fail; each refusal dropped → its test fails. Lab: restic 0.14.0 binary vs
  the test simulation over 92 snapshots: identical 72 removals. **Live:** demo-felhom 15 snapshots, predicted removals
  `92e49f62`, `343d57a5` → window 5 removed exactly those (16 → 14 incl. the run's new one); demo-hp 136 snapshots,
  predicted 18 → window 6 removed exactly those (145 → 127). Hub rows `pruned`, no event, no mail, key audit 0 findings.
- **Part B.** Live on 9202: controller start 05:28:35, install 05:31:15, clean-up 05:31:36 "skipped — … pulling images
  now", BookStack deployed 05:31:54 (35.5 s), retry 05:33:36 ran (0 deleted). Teardown through the product verified.
- **Part C.** `download_bytes = 12802456 B` from the installed wrapper on demo-hp (was 0).
- **Part D.** R-866: `block=SAVED(2026-10-05T05:23:26Z; hub unreachable: … connect: invalid argument)`. R-868 (v0.144.1):
  killed 06:04:00, wrapper `DONE upgraded=13`, kept copy, hub report id 63 `applied` (13) at 06:09:05, once.
- **Part E.** See `11` §8.4: back in 37 s; apps healthy within 4 min; household saw one timeline line; one operator mail
  `os_update_failed` (true); next pass FAILED (R-876); after `dpkg --configure -a` the pass installed 12; the guest's
  package list equals the pre-test one (279 lines). The first crash attempt did not fire (my watcher's pattern missed
  apt's dpkg call; the pass installed normally) — rolled back again and repeated.
- **Part F.** `audits/night-fixes-2026-10-05/partF/FINDINGS.md`; two claims re-checked in source by me.

## 4. Rows

Register before **341**, after **340**. Closed (8): R-95, R-863, R-864, R-865, R-866, R-867, R-868, R-869. Opened (7):
R-870 (tokens, operator), R-871 (a box off at night — the operator's decision), R-872 (P2, no missed-backup alarm for a
box down at 05:00), R-873 (nightly "cannot be reached" mail), R-874 (restore-test never runs on short sessions),
R-875 (P4, kept-report reason text), **R-876 (P2, the repair misses dpkg's update journal)**. STATUS updated.
`unproven.py --summary`: unchanged — 55 claims, 35 not walked (no number moved).

## 5. Teardown, three layers

- **Machines:** 9202: BookStack removed through the product (no container, stack dir, volume); controller 0.294.0 (set by
  hand, allowed there). demo-hp 9201: every package current, list identical to before; my scripts and logs in `/root`
  removed after copying. The bake VM: CT 9100 destroyed, token shredded, reverted to `virgin`, qemu gone.
- **Hosts:** the blackhole routes on felhom-pve removed (in the same script). Nothing provisioned.
- **Hub:** two one-shot clean-up grants (consumed); floors for demo-hp, demo-felhom, tester-1 → 0.294.0; artifacts
  vouched (agent 0.144.1, golden 0.294.0, min_agent 0.131.0); 12 signed jobs (3× agent 0.144.0, 3× bundle 0.144.0,
  3× agent 0.144.1, 3× bundle 0.144.1). Tester 2: read only, nothing sent (offline all session).
- Scratch secrets (hub password, hub key, controller password, deploy secrets) shredded at the end.

## 6. Notes

- The hub pod log prints a customer's e-mail address in "Customer email sent to …" lines (seen by the Part F helper; not
  recorded anywhere).
- demo-hp's System page read "no running customer guest" / "unknown" for several minutes after the crash although the
  guest ran — R-853 (facts late after a boot), not a new defect.