Files
felhom.eu/documentation/audits/night-burndown-2026-10-05/NIGHT-LOG.md
T

11 KiB
Raw Blame History

Night log — the burn-down night, 2026-10-05 → 06

Brief: drills/NIGHT-burndown-2026-10-05.md (workspace root). One line per row touched: row · result (fixed / closed-stale / failed / moved to A or D / needs-live) · minutes · commit.

Baselines read from live source at 21:03 CEST: felhom.eu 30650cad6e · controller 6f1ba1fe43 (v0.297.0) · agent 208fac8027 (v0.147.0) · catalog 4828dc754d · hub v0.137.0 (CHANGELOG head) · register 199 (19 P2, 95 P3, 85 P4).

How the night is worked: six helper sessions, one per repo lane, each in its own git worktree and branch cut from origin/main (controller ×2, hub, installer, agent, catalog). Helpers commit locally, never push. The lead reviews, cherry-picks onto main, writes CHANGELOG, closes rows, pushes and watches CI.

Rows

Row Result Min Commit
R-888 closed — ruling (decision 130) 5 (this commit)
R-337 closed — ruling (decision 130) 2 (this commit)
R-375 closed — ruling (decision 130) 2 (this commit)
R-126 ruling recorded (decision 128); build → helper ctrl-a 2 (this commit)
R-856 ruling recorded (decision 129); build → helper ctrl-a 2 (this commit)
R-349 agent half fixed (64f704d); hub half → helper hub; row open 15 felhom-agent 64f704d
R-25 agent half fixed (769c4c3); controller half → helper ctrl-b; row open 25 felhom-agent 769c4c3
R-531 closed-stale (all three facts measured, ruled, built) 10 this batch
R-444 moved to A — needs a new root grant + a scheduled action on customer disks 10 —
R-99 moved to A — deleting on a customer datastore is the row's own separate ruling; ep0 fenced 5 —
R-270 agent: no agent half (controller lane) 5 —
R-275, R-276, R-881 fixed on main (installer 1.32.0); ships with tag installer-v1.32.0 35 85de3f9b
R-306 fixed on main (installer 1.32.0) 15 17af3b65
R-130 fixed on main — decision 131 (renamed, not enforced) 15 efe76093
R-180 fixed on main 20 7f3944ef
R-179 fixed on main (NAS units; data-drive units left, GL6-F2) 25 c11d4fbf
R-310 fixed on main + runbook line 15 b43705ac
R-274 fixed on main (disclosure + pinning tests; check shipped with R-297) 15 f790734e
R-879 fixed (hub v0.138.0) + roll-back command; decision 132; close after deploy 75 5f060e3d, 91e4ace9
R-136 fixed (hub v0.138.0); close after deploy 20 e221476f
R-283 fixed (hub v0.138.0): box's own claim state shown; un-claim NOT taken (operator's call) 30 daeed175
R-540 moved to D (multi-box design + money) 10 —
R-435 moved to D (per-app counts, two repos, unmeasured threshold) 10 —
R-315 fixed and closed (gate tooling is live on push) 25 900e6d86
R-325 felhom.eu half fixed; controller half → helper ctrl-c 15 7ee00d59
R-422 fixed and closed 25 bf179847
R-426 partial (20 → 16 exemptions); row stays open 15 68df84bc
R-502 moved to A (a Docker-running gate on DooPlex needs an operator word) 10 —
R-349 hub half fixed (hub v0.138.0) 20 a5b4d29a
R-276 hub half fixed (hub v0.138.0) 15 0a460bcd
R-729 fixed on controller main 20 controller 27b373b
R-545 fixed on controller main 20 controller 27b373b
R-618 moved to A — The row itself calls that 'a design question, not a defect, raised here rather than decided'; trusting docker' 5 —
R-645 moved to A — The row lists three unchosen shapes (the CLI refuses to lift an UPDATE hold; the lift also puts the pin back; 5 —
R-314 moved to D — The box has no operator door into a RUNNING controller: its HTTP surface is the household's session (the debug 10 —
R-35 moved to D — Both directions the row names are designs: hot-apply needs a per-field ruling on which controller.yaml changes 10 —
R-844 moved to D — Blocked on a box-side household timeline (a new surface — a design); nothing to wire it into today. 5 —
R-878 needs a live measurement — The row says 'measure a large volume first' and the fix is 'if it matters'; the next attempt needs a timed cat 5 —
R-279 moved to D — The box has no operator door into a RUNNING controller: its HTTP surface is the household's session (the debug 5 —
R-177 moved to D — The box has no operator door into a RUNNING controller: its HTTP surface is the household's session (the debug 5 —
R-621 fixed on controller main 15 controller 0b93e1a
R-839 fixed on controller main 25 controller 4867ec1
R-271 fixed on controller main 20 controller f885100
R-521 fixed on controller main 25 controller 800b32c
R-607 fixed on controller main 30 controller 0b349e2
R-615 fixed on controller main 20 controller 0b349e2
R-25 fixed on controller main 20 controller 1355391
(hub, no row) fixed: Docker set first-seen stamped on the tick; flaky page test on a fixed clock 25 felhom.eu 6f78bd9f
R-578 fixed and closed (test only) 40 controller 9271f33
R-569 fixed on controller main 35 controller d15ad10
R-492 fixed on controller main 35 controller 7b8216a
R-576 fixed and closed (gate) 35 controller 074ddda
R-574 fixed on controller main 30 controller 391967b
R-488 fixed and closed (tests 444 s → 2 s; a real ssh from a test removed) 60 controller 8ce496f
R-325 fixed and closed (both halves; drift check kept — one line why in the row) 25 controller daeed27
R-126 fixed on controller main 15 controller 33c3973
R-856 moved to A — Building 'the same grace' changes nothing. If the operator wants the crash-boot app mails gone, the 20 —
R-138 moved to D — Needs design: where 'this zone is shared' is recorded (hub customer/zone model) and the policy (HTTP 8 —
R-270 fixed on controller main 25 controller 6fac245
R-616 fixed on controller main 25 controller 28a5203
R-498 fixed on controller main 20 controller 05a447e
R-682 fixed on controller main 35 controller 4347983
R-757 fixed on controller main 20 controller 1991c74
R-522 fixed on controller main 25 controller b36a85a
R-575 fixed on controller main 20 controller b05be0a
R-585 fixed on controller main 25 controller ca89e70
R-79 moved to D — Needs a design: keep the wire English and give each issue a KIND + key (like WarningKindAt for one w 5 —
R-516 fixed on controller main 20 controller d7efa7b
R-718 fixed on controller main 15 controller c4f4019
R-298 moved to D — Needs a design ruling: is 'user-data AND backup target on one drive' supported (then the agent must 10 —
R-756 needs a live reading — A fix that walks up to a mounted ancestor would also loosen the boot reconciler's start gate (DriveL 10 —
(live, 9202) the live-proof helper was stopped at 23:30 after 90 min with no result; 9202's catalog pointer put back byte-identical; no proof run — R-776, R-613 stay held, R-763/R-764 ship as a hidden app — live-9202-teardown.txt
R-518, R-638, R-528 group D: one-page design proposals written (no code), in this folder 40 this commit

Release window 1 — 00:30 to 00:58 CEST (evidence: delivery/)

Act Result
catalog pushed c265b37 00:30 (R-758, R-127a, R-798, R-763, R-764, R-786, R-731; R-776/R-613 held)
controller v0.298.0 image built + pushed 00:31 (MinAgent 0.131.0)
golden 0.298.0 first bake unpinned (no GOLDEN_DOCKER_PKGS — runbook gap, now fixed) → never vouched, re-baked pinned, sha a2e730e9…, round trip equal, token leak 0 with control, VM back on virgin
hub v0.138.0 manifest edde9e13, synced 22:48:42Z, healthz 200 at 22:49:13Z (31 s), System page 200; 16 values sealed; raw DB copy 0 plaintext; 3 box reports after deploy, 0 auth failures; cookie __Host-hub_session
agent v0.148.0 released (tag, sha 3e68a087…, bundle a6fa4f58…), vouched, signed agent_update ×3 → all 0.148.0 by 22:50Z; signed agent_config_update ×3 → root files 0.148.0 by 22:58Z; host pages „matches vouched”
vouch + floors agent 0.148.0 / golden 0.298.0 / min_agent 0.131.0; floors 0.298.0 for demo-hp, demo-felhom, tester-1 only; global floor not touched; Tester 2 not touched (DOWN, nothing sent)
delivery demo-hp + demo-felhom felhom-controller:0.298.0 (healthy) at 22:51:29Z; Tester 1 „Controller elindult (0.298.0)”
rows 25 closed as delivered (164 open after)
fixed without a row test_golden_currency_gate.py used the local date, the gate UTC — red between 00:00 and 02:00 CEST (red-proved)
R-426 felhom.eu side done: 16 → 12 exemptions; 3 foolable gates fixed (hostinstall R-185 count, age install, one-register leading verdict); decision 135
R-426 agent + catalog decoys (8 exemptions removed; image-pins: 3 holes closed); 4 left (controller); agent go test red 21:25–01:55 from the installer change — fixed (test reads ISO first-boot files the installer only names)
R-585 the rest fixed on controller main (all producers)
R-621 hold panel fixed on controller main (decision 136)
R-127 leg (b) moved to D
R-516 items 7–10, 12 fixed on controller main; item 11 → new row R-889
R-426 CLOSED — exemption list 20 → 0 (controller decoys 7044b11)
R-889 OPENED — disk percent is used/total, not df's; moving it moves every fill alarm (a decision)
(agent, no row) go test ./internal/osupdate red on DooPlex 21:25–01:55 (bundle test read the installer's new KEPT names as written files) — fixed
(register) section headings recounted (they still carried the 199-row counts)
R-516 the bundle's last 49 formal forms → te-form; gate widened (61 → 0); narrowed to Go/template literals

The two night watches (§5)

Watch Result
R-872 at 05:00 Deadline check: 4 customers, 0 backup missed … 1 skipped (down). From a hub.db copy (+ -wal): Tester 2 first reported 2026-10-04 16:13:44Z → 34.8 h old at 05:00, under the 48 h line → not judged, by design, but SILENTLY. No alarm owed, none fired. Fixed without a row: the early returns now log why (hub, unreleased; red-proved). Dated check moved to 2026-10-07. r872-watch.txt
R-887 lost CI jobs 2 of ~30 runs: felhom.eu 1384 (21:23Z) and felhom-controller 1401 (00:45Z) — no log, failed after 10–13.5 min; each re-run once → success. r887-lost-jobs.txt

Seen, not filed (a documented design choice, F2 in hub/internal/monitor/host_staleness.go): a box already down when the hub restarts is announced again once — Tester 2 got a host_down operator mail at each of yesterday's six hub restarts.