felhom.eu/documentation/audits/night-burndown-2026-10-05/NIGHT-LOG.md

# Night log — the burn-down night, 2026-10-05 → 06

Brief: `drills/NIGHT-burndown-2026-10-05.md` (workspace root). One line per row touched:
row · result (fixed / closed-stale / failed / moved to A or D / needs-live) · minutes · commit.

**Baselines read from live source at 21:03 CEST:** felhom.eu `30650cad6e` · controller `6f1ba1fe43` (v0.297.0) · agent
`208fac8027` (v0.147.0) · catalog `4828dc754d` · hub v0.137.0 (CHANGELOG head) · register **199** (19 P2, 95 P3, 85 P4).

**How the night is worked:** six helper sessions, one per repo lane, each in its own git worktree and branch cut from
`origin/main` (controller ×2, hub, installer, agent, catalog). Helpers commit locally, never push. The lead reviews,
cherry-picks onto `main`, writes CHANGELOG, closes rows, pushes and watches CI.

## Rows

| Row | Result | Min | Commit |
|---|---|---|---|
| R-888 | closed — ruling (decision 130) | 5 | (this commit) |
| R-337 | closed — ruling (decision 130) | 2 | (this commit) |
| R-375 | closed — ruling (decision 130) | 2 | (this commit) |
| R-126 | ruling recorded (decision 128); build → helper ctrl-a | 2 | (this commit) |
| R-856 | ruling recorded (decision 129); build → helper ctrl-a | 2 | (this commit) |
| R-349 | agent half fixed (`64f704d`); hub half → helper hub; row open | 15 | felhom-agent `64f704d` |
| R-25 | agent half fixed (`769c4c3`); controller half → helper ctrl-b; row open | 25 | felhom-agent `769c4c3` |
| R-531 | closed-stale (all three facts measured, ruled, built) | 10 | this batch |
| R-444 | moved to A — needs a new root grant + a scheduled action on customer disks | 10 | — |
| R-99 | moved to A — deleting on a customer datastore is the row's own separate ruling; ep0 fenced | 5 | — |
| R-270 | agent: no agent half (controller lane) | 5 | — |
| R-275, R-276, R-881 | fixed on main (installer 1.32.0); ships with tag `installer-v1.32.0` | 35 | `85de3f9b` |
| R-306 | fixed on main (installer 1.32.0) | 15 | `17af3b65` |
| R-130 | fixed on main — decision 131 (renamed, not enforced) | 15 | `efe76093` |
| R-180 | fixed on main | 20 | `7f3944ef` |
| R-179 | fixed on main (NAS units; data-drive units left, GL6-F2) | 25 | `c11d4fbf` |
| R-310 | fixed on main + runbook line | 15 | `b43705ac` |
| R-274 | fixed on main (disclosure + pinning tests; check shipped with R-297) | 15 | `f790734e` |
| R-879 | fixed (hub v0.138.0) + roll-back command; decision 132; close after deploy | 75 | `5f060e3d`, `91e4ace9` |
| R-136 | fixed (hub v0.138.0); close after deploy | 20 | `e221476f` |
| R-283 | fixed (hub v0.138.0): box's own claim state shown; un-claim NOT taken (operator's call) | 30 | `daeed175` |
| R-540 | moved to D (multi-box design + money) | 10 | — |
| R-435 | moved to D (per-app counts, two repos, unmeasured threshold) | 10 | — |
| R-315 | fixed and closed (gate tooling is live on push) | 25 | `900e6d86` |
| R-325 | felhom.eu half fixed; controller half → helper ctrl-c | 15 | `7ee00d59` |
| R-422 | fixed and closed | 25 | `bf179847` |
| R-426 | partial (20 → 16 exemptions); row stays open | 15 | `68df84bc` |
| R-502 | moved to A (a Docker-running gate on DooPlex needs an operator word) | 10 | — |
| R-349 | hub half fixed (hub v0.138.0) | 20 | `a5b4d29a` |
| R-276 | hub half fixed (hub v0.138.0) | 15 | `0a460bcd` |
| R-729 | fixed on controller main | 20 | controller `27b373b` |
| R-545 | fixed on controller main | 20 | controller `27b373b` |
| R-618 | moved to A — The row itself calls that 'a design question, not a defect, raised here rather than decided'; trusting docker' | 5 | — |
| R-645 | moved to A — The row lists three unchosen shapes (the CLI refuses to lift an UPDATE hold; the lift also puts the pin back;  | 5 | — |
| R-314 | moved to D — The box has no operator door into a RUNNING controller: its HTTP surface is the household's session (the debug | 10 | — |
| R-35 | moved to D — Both directions the row names are designs: hot-apply needs a per-field ruling on which controller.yaml changes | 10 | — |
| R-844 | moved to D — Blocked on a box-side household timeline (a new surface — a design); nothing to wire it into today. | 5 | — |
| R-878 | needs a live measurement — The row says 'measure a large volume first' and the fix is 'if it matters'; the next attempt needs a timed cat | 5 | — |
| R-279 | moved to D — The box has no operator door into a RUNNING controller: its HTTP surface is the household's session (the debug | 5 | — |
| R-177 | moved to D — The box has no operator door into a RUNNING controller: its HTTP surface is the household's session (the debug | 5 | — |
| R-621 | fixed on controller main | 15 | controller `0b93e1a` |
| R-839 | fixed on controller main | 25 | controller `4867ec1` |
| R-271 | fixed on controller main | 20 | controller `f885100` |
| R-521 | fixed on controller main | 25 | controller `800b32c` |
| R-607 | fixed on controller main | 30 | controller `0b349e2` |
| R-615 | fixed on controller main | 20 | controller `0b349e2` |
| R-25 | fixed on controller main | 20 | controller `1355391` |
| (hub, no row) | fixed: Docker set first-seen stamped on the tick; flaky page test on a fixed clock | 25 | felhom.eu `6f78bd9f` |
| R-578 | fixed and closed (test only) | 40 | controller `9271f33` |
| R-569 | fixed on controller main | 35 | controller `d15ad10` |
| R-492 | fixed on controller main | 35 | controller `7b8216a` |
| R-576 | fixed and closed (gate) | 35 | controller `074ddda` |
| R-574 | fixed on controller main | 30 | controller `391967b` |
| R-488 | fixed and closed (tests 444 s → 2 s; a real ssh from a test removed) | 60 | controller `8ce496f` |
| R-325 | fixed and closed (both halves; drift check kept — one line why in the row) | 25 | controller `daeed27` |
| R-126 | fixed on controller main | 15 | controller `33c3973` |
| R-856 | moved to A — Building 'the same grace' changes nothing. If the operator wants the crash-boot app mails gone, the  | 20 | — |
| R-138 | moved to D — Needs design: where 'this zone is shared' is recorded (hub customer/zone model) and the policy (HTTP | 8 | — |
| R-270 | fixed on controller main | 25 | controller `6fac245` |
| R-616 | fixed on controller main | 25 | controller `28a5203` |
| R-498 | fixed on controller main | 20 | controller `05a447e` |
| R-682 | fixed on controller main | 35 | controller `4347983` |
| R-757 | fixed on controller main | 20 | controller `1991c74` |
| R-522 | fixed on controller main | 25 | controller `b36a85a` |
| R-575 | fixed on controller main | 20 | controller `b05be0a` |
| R-585 | fixed on controller main | 25 | controller `ca89e70` |
| R-79 | moved to D — Needs a design: keep the wire English and give each issue a KIND + key (like WarningKindAt for one w | 5 | — |
| R-516 | fixed on controller main | 20 | controller `d7efa7b` |
| R-718 | fixed on controller main | 15 | controller `c4f4019` |
| R-298 | moved to D — Needs a design ruling: is 'user-data AND backup target on one drive' supported (then the agent must  | 10 | — |
| R-756 | needs a live reading — A fix that walks up to a mounted ancestor would also loosen the boot reconciler's start gate (DriveL | 10 | — |
| (live, 9202) | the live-proof helper was stopped at 23:30 after 90 min with no result; 9202's catalog pointer put back byte-identical; no proof run — R-776, R-613 stay held, R-763/R-764 ship as a hidden app | — | `live-9202-teardown.txt` |
| R-518, R-638, R-528 | group D: one-page design proposals written (no code), in this folder | 40 | this commit |

## Release window 1 — 00:30 to 00:58 CEST (evidence: `delivery/`)

| Act | Result |
|---|---|
| catalog | pushed `c265b37` 00:30 (R-758, R-127a, R-798, R-763, R-764, R-786, R-731; R-776/R-613 held) |
| controller v0.298.0 | image built + pushed 00:31 (MinAgent 0.131.0) |
| golden 0.298.0 | **first bake unpinned** (no `GOLDEN_DOCKER_PKGS` — runbook gap, now fixed) → never vouched, re-baked pinned, sha `a2e730e9…`, round trip equal, token leak 0 with control, VM back on `virgin` |
| hub v0.138.0 | manifest `edde9e13`, synced 22:48:42Z, healthz 200 at 22:49:13Z (31 s), System page 200; 16 values sealed; raw DB copy 0 plaintext; 3 box reports after deploy, 0 auth failures; cookie `__Host-hub_session` |
| agent v0.148.0 | released (tag, sha `3e68a087…`, bundle `a6fa4f58…`), vouched, signed `agent_update` ×3 → all 0.148.0 by 22:50Z; signed `agent_config_update` ×3 → root files 0.148.0 by 22:58Z; host pages „matches vouched” |
| vouch + floors | agent 0.148.0 / golden 0.298.0 / min_agent 0.131.0; floors 0.298.0 for demo-hp, demo-felhom, tester-1 only; **global floor not touched; Tester 2 not touched** (DOWN, nothing sent) |
| delivery | demo-hp + demo-felhom `felhom-controller:0.298.0 (healthy)` at 22:51:29Z; Tester 1 „Controller elindult (0.298.0)” |
| rows | 25 closed as delivered (164 open after) |
| fixed without a row | `test_golden_currency_gate.py` used the local date, the gate UTC — red between 00:00 and 02:00 CEST (red-proved) |
| R-426 | felhom.eu side done: 16 → 12 exemptions; 3 foolable gates fixed (hostinstall R-185 count, age install, one-register leading verdict); decision 135 | 25 | felhom.eu night/gates ×4 |
| R-426 | agent + catalog decoys (8 exemptions removed; image-pins: 3 holes closed); 4 left (controller); agent go test red 21:25–01:55 from the installer change — fixed (test reads ISO first-boot files the installer only names) | 35 | agent `b2b82ae`, catalog `75f35842` |
| R-585 | the rest fixed on controller main (all producers) | 40 | controller `ff1758a` |
| R-621 | hold panel fixed on controller main (decision 136) | 25 | controller `0f2eab7` |
| R-127 | leg (b) moved to D | 10 | — |
| R-516 | items 7–10, 12 fixed on controller main; item 11 → new row R-889 | 60 | controller `e4774e6`, `2e5ef7e` |
| R-426 | CLOSED — exemption list 20 → 0 (controller decoys `7044b11`) | 30 | this batch |
| R-889 | OPENED — disk percent is used/total, not df's; moving it moves every fill alarm (a decision) | 10 | this batch |
| (agent, no row) | `go test ./internal/osupdate` red on DooPlex 21:25–01:55 (bundle test read the installer's new KEPT names as written files) — fixed | 10 | agent `b2b82ae` |
| (register) | section headings recounted (they still carried the 199-row counts) | 5 | this batch |
| R-516 | the bundle's last 49 formal forms → te-form; gate widened (61 → 0); narrowed to Go/template literals | 25 | controller `4c3c203` |

## The two night watches (§5)

| Watch | Result |
|---|---|
| R-872 at 05:00 | `Deadline check: 4 customers, 0 backup missed … 1 skipped (down)`. From a hub.db copy (+ -wal): Tester 2 first reported 2026-10-04 16:13:44Z → 34.8 h old at 05:00, under the 48 h line → not judged, by design, but SILENTLY. No alarm owed, none fired. Fixed without a row: the early returns now log why (hub, unreleased; red-proved). Dated check moved to 2026-10-07. `r872-watch.txt` |
| R-887 lost CI jobs | **2** of ~30 runs: felhom.eu 1384 (21:23Z) and felhom-controller 1401 (00:45Z) — no log, failed after 10–13.5 min; each re-run once → success. `r887-lost-jobs.txt` |

Seen, not filed (a documented design choice, F2 in `hub/internal/monitor/host_staleness.go`): a box already down when the
hub restarts is announced again once — Tester 2 got a `host_down` operator mail at each of yesterday's six hub restarts.