docs: 11 §5.7 System page, §5.8 Docker slow lane BUILT, §5.9 crash restart; 00/03/07/08; decisions 90-94 (CC unattended); runbooks docker-undo + crash-guard; register R-852 R-835 R-848 R-849 R-851 R-854 closed, R-853 R-855 R-856 opened, R-812 R-840 narrowed (334 -> 332); live evidence
gates / gates (push) Successful in 33s
gates / gates (push) Successful in 33s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -332,11 +332,31 @@ must never overlap a backup, a restore-test or a self-update.~~
|
||||
- **Operator:** a hub event for each update and each failure; a fleet view showing each box's OS release,
|
||||
how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest
|
||||
approved release raises an alarm (`08`).
|
||||
- **[FACT, hub v0.132.0] The System page** (`/system`, R-852, decision 89): one row per box — ring and switch with
|
||||
buttons, the tunnel, host Proxmox / running and next-boot kernel / Debian / release / pending / not covered / held /
|
||||
reboot needed / `kernel.panic` / oops / crash restarts / the guard, guest Debian / release / pending / restart needed,
|
||||
Docker engine / containerd / live-restore / release, the last leg — and above it the releases, what ring 0 runs, "Approve
|
||||
now" and "Approve Docker set". Colours are the alarm thresholds (decision 94). Hosts shows Proxmox / kernel too.
|
||||
- **Household:** one line on the timeline in both languages, informal voice: what was updated and
|
||||
whether the box restarted. Telling households in advance that the box may restart at night is a
|
||||
**promise to users**. That is the operator's decision when the slow lane is built.
|
||||
|
||||
### 5.8 The Docker engine slow lane — DESIGN (2026-10-04, nothing built) `[PROPOSAL]`
|
||||
### 5.8 The Docker engine slow lane — BUILT 2026-10-04 (agent v0.142.0, hub v0.132.0) `[FACT]`
|
||||
|
||||
**As built** (evidence `audits/os-docker-crash-2026-10-04/partB/`; decisions 87, 93):
|
||||
- **live-restore ON**: the golden bakes it (`build-golden.sh` 3.1.0, fail-closed assertion); an installed box gets it once
|
||||
by the wrapper's `live-restore-on` (merge into daemon.json + `systemctl reload docker`). Measured: the same container ids
|
||||
after (9202 6/6 by hand — R10 refuses a scratch guest by design —, demo-hp 24/24, demo-felhom 5/5).
|
||||
- **The step** runs in the night leg after a healthy guest and host step, **ring 0 only**, `select pending-docker`,
|
||||
allowed by the wrapper only with the root-owned `ring0_slow_lane` mark. **Ring 1 and every undo** only through a signed
|
||||
`os_docker_step` the wrapper re-verifies itself (decision 93). Measured: 29.7.x → 29.8.2 on both demo boxes, every id
|
||||
kept; a signed undo to 29.7.2 on demo-hp (41 s, ids kept) and back; demo-felhom as ring 1 by a signed job; a replayed
|
||||
job refused by the agent (nonce).
|
||||
- **Approval**: the hub never approves a Docker set automatically; the System page's button works after every ring-0 box
|
||||
ran the set in 2 healthy night Docker steps (`OS_DOCKER_APPROVE_NIGHTS` TEST override, logged). An approval nudges no
|
||||
box. Undo: `runbooks/os-updates-docker-undo.md`.
|
||||
|
||||
**The design as written before the build:**
|
||||
|
||||
Built from C5 (measured) and R-835. **The one decision it needs is in STATUS: `live-restore` on, fleet-wide.**
|
||||
|
||||
@@ -360,6 +380,25 @@ Built from C5 (measured) and R-835. **The one decision it needs is in STATUS: `l
|
||||
- **Not covered here:** the golden's own engine (baked weekly; a new golden carries the approved set), and BYO hosts
|
||||
(the guest is ours on both, so the lane applies there too).
|
||||
|
||||
### 5.9 A crashed host restarts, with a limit — BUILT 2026-10-04 (agent v0.142.0, installer 1.30.0) `[FACT]`
|
||||
|
||||
Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-docker-crash-2026-10-04/partC/`.
|
||||
- **Measured first (demo-hp, the operator's word before each crash):** `kernel.panic = 10` + `echo c >
|
||||
/proc/sysrq-trigger` → the box restarted by itself in 54 s, on the same kernel (the saved default), the agent up 18 s
|
||||
after the boot. `kernel.panic` set by `sysctl -w` is gone after the restart (back to 0) — it must be set at every boot.
|
||||
- **The crash signal** (decision 90): `efi_pstore` is on, yet a real panic saved NOTHING; the journal and `last` show
|
||||
only "no shutdown". The guard uses a **clean-stop marker** (the unit's ExecStop at every orderly shutdown); a boot
|
||||
without it followed a crash, a power cut or a hard reset — counted alike.
|
||||
- **The guard** (`felhom-crash-guard`, early boot unit + hourly re-arm timer): armed → `kernel.panic = 10`; the 2nd
|
||||
unclean boot within 60 minutes TRIPS it (`kernel.panic = 0`), so **the 3rd crash within the hour leaves the box off**
|
||||
(decision 92 — the operator's words); re-arms after 24 h of normal running or `felhom-crash-guard rearm`. A crash before
|
||||
the unit runs (very early boot) leaves the box off: the safe side. `panic_on_oops` stays 0; an oops is reported
|
||||
(decision 91).
|
||||
- **Telling people:** each unclean boot = an operator mail (`host_crash_restart`) + the household's line; the trip = an
|
||||
alarm (`host_crash_guard_tripped`); re-arm and oops announced once (`08` §6.3). The System page shows the guard.
|
||||
- **Measured live:** crash 1 → back in 54 s; crash 2 → back in 53 s, guard tripped; crash 3 → **stayed off** until the
|
||||
operator switched it on; the hub mailed the trip (15 min after the boot — R-853); re-armed by hand.
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks and edge cases
|
||||
@@ -432,7 +471,7 @@ Each step returns to the operator for go or no-go.
|
||||
3. **Host Debian, fast lane** (no kernel, no Proxmox packages). **BUILT 2026-10-04** — agent v0.141.1, hub v0.131.1;
|
||||
§8.2.
|
||||
4. **Fleet view and alarms** (§5.7). **BUILT 2026-10-04** — hub v0.131.0/v0.131.1; §8.3.
|
||||
5. **Slow lane: Docker engine.**
|
||||
5. **Slow lane: Docker engine.** **BUILT 2026-10-04** — agent v0.142.0, hub v0.132.0; §5.8.
|
||||
6. **Slow lane: host kernel and Proxmox packages, with the reboot.**
|
||||
7. **Later:** the Proxmox major upgrade (PVE 9 → 10), drilled on ring 0 first.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user