chaos-night fixes: rulings recorded, guide reordered, three rows closed, two filed
gates / gates (push) Successful in 21s

Architecture: 08 records ruling A (45 m, the round-9 arithmetic, the cost) and
the two new event types with their audiences; 03 records the slow counter as
built; 07 records the restore-record persistence as a REVERSED design for the
restore record only; CONTEXT.md carries the day's rulings.

Guide: the recovery code moves after the first apps and waits for the yellow bar.

Register: R-549 and R-550 closed PROVEN-LIVE; R-546 closed on red-proofed tests
with its live walk owed by R-551 (no Tier-0 box is paused AND agent-connected).
R-552 filed: an interrupted-restore notice for a removed app never clears -
found in my own v0.246.0 after the release was built.

Evidence: Part A (hub prints 45m/1h30m), Part C delivery on HP and N100, B.4(a)
live proof and its teardown.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
2026-09-17 11:00:29 +02:00
parent 06334e164c
commit 3c1882a0a4
11 changed files with 163 additions and 13 deletions
@@ -121,6 +121,18 @@ a long window (N restarts in 24 h) raising a `warning`. Not built in the 2026-09
told to measure the budget rather than change it. Evidence:
`audits/evidence-drill-0243-2026-09-16/phase2-f9dprime.txt`.
**[BUILT 2026-09-17 — agent v0.132.0, hub v0.117.0] The slow counter.** Beside the unchanged 3-in-15
brake: restarts the supervisor performed in the last **24 hours**; at the **fifth**, the heartbeat stanza
sets `slow_crashloop_since` (moving at most once per 24 h) and the hub raises
`controller_slow_crashloop` (**warning, operator-only**). It never stops restarting — the fast brake stays
the only brake. **Persisted** per guest at `/var/lib/felhom-agent/guests/<vmid>/controller-restarts-24h.json`
(tmp + rename, 0600) so an agent restart or reboot does not reset it; the fast record stays in memory,
and the reason it does (a persisted „give up" could outlive the fix) does not apply to a counter that
only warns. **Deliberate kills count** — the supervisor cannot tell an operator's `docker kill` from a
crash (measured 2026-09-15). Delivered to demo-hp and the N100 by one operator-signed `agent_update` each
(ruling 1 of 2026-09-16); both logged `slow_crashloop_max=5 slow_crashloop_window=24h0m0s` at start.
Evidence: `audits/evidence-chaos-fixes-2026-09-17/partC-agent-update.txt`.
Signed payloads carry a **nonce + expiry** (anti-replay: a captured "restore" job cannot be
re-injected later) and a target binding (host + guest id) so a signature can't be retargeted.
Notification-on-destructive-op is an **audit signal, never the guard** — a compromised hub
@@ -122,6 +122,17 @@ distinct action (`audits/CAMPAIGN-8-backup-restore-2026-07-27.md:520`).
not by a customer. `00-capability-map.md:75` still carries "A customer (not the operator) performs a
restore via UI alone" as **MISSING as evidence** — by absence of the run, not by a product gap.
**[RULING 2026-09-17, operator — a design choice REVERSED] The restore record survives a restart
(R-550, controller v0.246.0).** The async restore status (`internal/backup/opstatus.go`) was in memory
by choice — „same precedent as notification cooldowns". Chaos night round 10 measured the cost: a
restore accepted, the box hard-reset four seconds later, and the status answering the Go zero value, so
the household could never learn whether it finished. **For the restore record only**, it is now written
atomically to `restore-status.json` in the controller's state directory at both ends of an op; at
startup a record still marked running becomes a failed, interrupted result („A visszaállítás megszakadt
(a doboz újraindult) — indítsd el újra."), shown per app on the restore page until that app's next
restore, and raised once as `restore_interrupted`. **Notification cooldowns stay in memory** — the
precedent is kept for what it was written for.
### Lane 2 — the operator: guest and host recovery
Rebuilding an LXC guest, rebuilding a host, and re-establishing a box's identity are **operator
@@ -198,6 +198,26 @@ same box — the agent's dead-man's-switch rather than the controller's — and
would have kept exactly the F9 silence on the host plane. Everything else keeps the hour (pinned by
`TestOperatorCooldown_NodeLivenessBypassesQuietHour`, which also asserts an unrelated type still waits).
**Operator ruling A, 2026-09-17 (R-549) — „the box went quiet" waits THREE report cycles, not two.**
The staleness threshold (`alerting.stale_threshold`, configuration in `manifests/hub.yaml`) moves from
**30 m to 45 m**; `node_down` and `host_down` follow at 2× = **90 m**. The reason is chaos night round 9
(`audits/DRILL-chaos-night-2026-09-17.md`): report cadence 15 m; a failed push is retried for ~100 s and
then given up (correctly — a report is a snapshot); the measured gap to the next good report was
**29 m 59 s** against a 30-minute threshold. One missed push spent the entire budget, so ordinary jitter
would page the operator about a box that was healthy and had already repaired itself. **The cost,
stated:** a truly dead box now pages 15 minutes later (45 m instead of 30), and „the box is down" 30
minutes later (90 m instead of 60). **One value, everywhere:** both staleness checkers, the host status and
— since hub v0.117.0 — the customer status on the dashboard read the same threshold (`controllerStatus`
hardcoded 30 m / 1 h until then; pinned by `TestControllerStatus_FollowsConfiguredThreshold`). A running
hub prints it at startup (`node_stale after 45m0s, node_down after 1h30m0s`).
**Two event types added 2026-09-17, with who receives them:**
| event | severity | who | minted by | why that audience |
|---|---|---|---|---|
| `controller_slow_crashloop` | warning | **operator only** | hub, from the agent's `slow_crashloop_since` moving (agent v0.132.0, hub v0.117.0) | host ids and vmids; the household's side of it is the dashboard coming back each time |
| `restore_interrupted` | warning | **the household** (and the operator) | controller v0.246.0 at startup, once per interruption | they pressed restore, were told it started, and can run it again — Hungarian `customerMessages` entry |
| Family | Grain | Key carries | Why |
|---|---|---|---|
| app down (`app_start_failed`) | **per APP** | `…:<stack_name>` | no digest exists for it — see below |