felhom.eu/REPORT-chaos-fixes-2026-09-17.md

REPORT — the chaos-night fixes: the quiet alarm, the restore record, the recovery-code timing, the slow crash loop

2026-09-17. R-549, R-550, R-546, R-539. Per-repo detail: felhom-controller/REPORT.md (v0.246.0), felhom-agent/REPORT.md (v0.132.0). This file covers felhom.eu (hub v0.117.0, the setting, the documents) and the whole task. Written as REPORT-<topic>.md because REPORT.md holds the earlier session's work.

Claims in the brief that turned out wrong — named first

1. „Readiness = escrow.pbs_storage_id set." The agent's preflight ok covers five blocking items; pbs_storage_id is the one R-546's box showed. The controller reads the combined ok.
2. „The backup tiers persist their records atomically." Checked at felhom-controller/controller/internal/settings/settings.go:727-752 — TRUE (tmp + rename, .bak recovery).
3. „Ruling A is a config change plus a document line, not code." WRONG. hub/internal/web/rollup.go controllerStatus hardcoded 30 m / 1 h while both checkers and hostStatus read the config — the dashboard would have turned amber 15 minutes before the alarm could fire. Hub v0.117.0 fixes it.
4. „Verify from the log line that the checker runs with 45 m." No log line printed the threshold. Hub v0.117.0 adds it to both „checker initialized" lines.
5. „The failure shows a raw error." Not in a browser — the page hid its start form behind the checklist. The raw -storage stderr came from the chaos-night harness calling the API directly.
6. „Emit the event from the agent" / „make the interval configurable for the test." The agent has no event channel (the hub mints from heartbeat timestamps); and no test interval was needed — five real kills 8 minutes apart reached the production threshold in the session.

1. Confirmed baselines (re-verified at start)

felhom-controller 714d5bce0920 v0.245.0 (MinAgent 0.131.0) · felhom-agent e98b857684f4 v0.131.0 · felhom.eu ea25c6c5b1f1 hub v0.116.0 · highest row R-550 · golden waiver to 2026-09-27.

2. Files (felhom.eu)

hub/internal/web/{rollup.go,configs.go,server.go,rollup_test.go}, hub/internal/monitor/{controller_supervisor.go,controller_supervisor_test.go,staleness.go,host_staleness.go}, hub/internal/notify/{dispatcher.go,templates.go}, hub/internal/api/{handler.go,chaosnight_events_test.go}, hub/CHANGELOG.md, manifests/hub.yaml, .claude/rules/hub.md, CONTEXT.md, STATUS.md, documentation/architecture/{03-host-agent.md,05-hub-architecture.md,07-backup-architecture.md,08-alarm-ladder.md}, documentation/runbooks/VOLUNTEER-first-hour.md, documentation/backlog/{OPEN-ITEMS.md,CLOSED-ITEMS.md}, documentation/audits/evidence-chaos-fixes-2026-09-17/ (6 files), this report.

3. Commits

felhom.eu: 469bfa5 (chaos-night leftover evidence), 37ae31f (hub v0.117.0), 06334e1 (manifest: 0.117.0 + 45m), 3c1882a (docs, register, evidence), and the commit carrying this report. felhom-agent: 18d03bd (v0.132.0), release-record CHANGELOG commit, 77cd70f (REPORT). Tag v0.132.0. felhom-controller: 0fe315b (v0.246.0), 29e2acb (REPORT).

4–5. Tests

Hub: go build/vet/test ./... green, 18 packages. Agent: 30 packages. Controller: 28 packages. Red-proofs, each seen failing then passing — hub: status follows threshold; slow-crashloop movement; operator-only registration; Hungarian subject. Agent: no counter; once-per-24h guard; persistence. Controller: restore record across restart; startup helper; main() wiring; restore page card; bar held back; waiting card; start refusal. Two of my own test mistakes corrected and recorded (a body-based fallback check; a -storage needle matching a menu id).

6. Deployed versions

Hub 0.117.0 (ArgoCD Synced/Healthy, revision == manifest HEAD, pod ready). Agent 0.132.0 on demo-hp and the N100 (signed jobs, committed). Controller 0.246.0 on demo-hp guest 9201. Proof lines in the evidence directory.

7. NOT live-validated

• R-546's readiness branches (bar held back, waiting card, start refusal) — no Tier-0 box is paused AND agent-connected. R-551.
• The escrow ceremony passing once ready — deliberately not run: the hub keeps one escrow per host, and a ceremony on demo-hp would supersede the standing box's escrow.
• Controller 0.246.0 is on 9201 only; the fleet floor was not raised (below).

8. Evidence copied off before each teardown

Yes. B.4(a) log written by the run itself to DooPlex before teardown; teardown logged separately; Part C logs off the box as they ran.

9. Teardown — three layers

• Machine: throwaway homebox on 9201 removed with data and backups (0 containers, 0 volumes); 10 of 10 standing apps up; password and scripts shredded in the guest.
• Host: nothing provisioned; demo-hp agent config untouched (the pbs_storage_id removal was considered and not done).
• Hub: no host or customer record created or deleted; the test events stay as history. Stated effect: 9201's slow crash-loop stays raised until 2026-09-18 09:22Z and cannot mail again before then.

An error of mine, caught by a gate before it was pushed

Closing R-539, I wrote open(CLOSED-ITEMS.md,'w').write(open(CLOSED-ITEMS.md).read() + row). Python opens — and empties — the file for writing before it reads it, so the closed register fell from 215 rows to 1. The pre-push instructions gate refused the push: citations of closed rows (R-549, and R-320 in unprompted-work.md) suddenly pointed at nothing. Nothing damaged was pushed. The file was restored from pushed commit 3c1882a and R-539 appended with read-then-write; the diff against 3c1882a is exactly one added line, and the open register exactly one removed line. The earlier closures (R-546/R-549/R-550) used read-then-write and were intact. Lesson kept: never open a file for writing in the same expression that reads it.

A recommendation not followed, with its reason

The brief's rules say „floor raised to deliver it". The controller floor was not raised to 0.246.0. The release was validated on one guest; raising a floor above golden 0.245.0 would roll it to Peti's box as well, and R-552 (my own gap) was found during validation. One line of evidence is not a fleet decision made unattended — the operator's to take, and cheap to take.

Observations

1. An interrupted-restore notice for a removed app never clears. FILED: R-552
2. No Tier-0 box can exercise the paused + agent-connected escrow state. FILED: R-551
3. Controller handler.go and agent controllersupervisor_test.go were not gofmt-clean before this task. NOT-A-FINDING: pre-existing formatting only; reformatting unrelated lines was left out to keep the diffs reviewable.

Addendum — the operator's two „yes" answers, carried out (2026-09-17, afternoon)

1. Controller floor raised to 0.246.0 (declared MinAgent 0.131.0). Hub: Global controller-version floor set to "0.246.0" and managed floor SERVED for demo-felhom … from declared; the N100 ran 0.246.0 within seconds (at/above floor 0.246.0 (we are 0.246.0)). demo-hp runs 0.246.0 (it has a per-customer override at 0.243.0). Peti's box is DOWN on the hub (last controller 0.115.0) and receives it when it reports. Evidence: audits/evidence-chaos-fixes-2026-09-17/decision1-floor-0246.txt. The „recommendation not followed" above is therefore superseded by the operator's decision.

2. Agent 0.132.0 vouched — which required a golden. The first vouch was refused by the hub's R-120 gate (golden 0.245.0 is older than the newest controller the fleet reports (0.246.0)); nothing was stored. Asked, the operator chose to bake. Golden 0.246.0 baked by RUNBOOK-manual-build §4.1, sha 05b7559d…, amd64, all markers, token leak 0 (control 1), registry 200 before teardown; then vouched together with agent 0.132.0: Artifact manifest set: agent=0.132.0 golden=0.246.0 min_agent="0.131.0" wrapper_sha=true, read back exactly. golden_currency_gate.py moved from WAIVED to OK. Evidence: tests/golden-0.246.0-2026-09-17/.

Mistakes of mine on the way, none reaching the registry: the first bake attempt used the arm64 template (my version sort), aborted before anything was built; stopping it, a self-matching pkill killed my own shell; the second attempt's watcher was killed for low memory and its post-bake steps were done by hand.

Observation. 4. golden_currency_gate.py reported the newest bake as 0.242.0 although 0.243.0–0.245.0 were baked and published, because those bakes filed their logs under audits/ rather than documentation/tests/golden-<ver>-<date>/. NOT-A-FINDING: a recording slip in earlier bakes (including 0.245.0, mine), not a gate defect — the gate reads the place the runbook names; 0.246.0 is filed there and the gate reads it.