R-840/R-859/R-860 records: 11 §5.3.1 + §5.4.2, 03, 04, 00; runbooks config-bundle + os-updates-test-waits; register 333→334 (R-840/859/860 closed, R-861/862 opened); STATUS; report; evidence
gates / gates (push) Successful in 31s
gates / gates (push) Successful in 31s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -0,0 +1,69 @@
|
||||
# RUNBOOK — the config bundle: a box's root-owned files by a signed job (R-840)
|
||||
|
||||
Design: `architecture/11-os-updates.md` §5.4.2. Decision 96 (`09` §3). Built 2026-10-04: agent v0.143.0, hub v0.133.0,
|
||||
installer 1.31.0. Evidence: `audits/r840-config-bundle-2026-10-04/`.
|
||||
|
||||
## What it is
|
||||
|
||||
Every root-owned file the installer writes for the agent (both sudoers files, the five wrappers, the crash guard and
|
||||
its units, the agent and rollback units, the start-limit drop-in, the mgmt watchdog, the OOB belt's files) travels as
|
||||
ONE file, `felhom-config-bundle.json`, published beside the agent binary (`felhom-agent/<version>/`). A new box gets it
|
||||
from the installer; an installed box gets it by a signed `agent_config_update`. The box's own root-owned
|
||||
`felhom-os-apply` checks the signature, the sha, every path and every file before it writes anything, keeps the
|
||||
previous copies, self-checks, and puts everything back on a failure.
|
||||
|
||||
**The trust root is never in a bundle.** `/etc/felhom/operator-signers` and `/etc/felhom/os-trust.json` decide who may
|
||||
sign; no bundle may add, remove or change them. A box that has no signers file gets exactly the installer's pinned key
|
||||
(`felhom-op-1`), and only through a job that key signed. Signer rotation is a separate act (`04` §3).
|
||||
|
||||
## Send a bundle to a box (CC or the operator, on DooPlex)
|
||||
|
||||
1. The bundle's sha: the release output (`release-agent.sh` prints `bundle :`), or the hub's Configuration page after
|
||||
vouching (the hub resolves it from the registry by exact name).
|
||||
2. Sign and queue (key `felhom-op-1`; the hub key is Secret `felhom-system/report-api`, read into a file, never printed):
|
||||
```bash
|
||||
felhom-opsign -op agent_config_update -host <host_id> -key-id felhom-op-1 -key <operational key> \
|
||||
-agent-version <X.Y.Z> -bundle-sha256 <sha> -ttl 45m -upload http://<hub ClusterIP>:8080 -hub-key "$(cat <file>)"
|
||||
```
|
||||
3. The agent takes it on its next job poll (measured 4–15 min). Positive observables in the box's journal:
|
||||
`os-apply: BUNDLE DONE agent=<X> written=<n> … self-check=ok`, then `capability probe after the config bundle ok=71
|
||||
total=71`. The hub's System page "Root files" column shows the version.
|
||||
4. **Undo** = send the previous release's bundle the same way. The previous copies also stay on the box in
|
||||
`/var/lib/felhom-os-apply/bundle-prev/<time>-before-<version>/` (the last 3).
|
||||
|
||||
## A box from before agent v0.143.0 — the ONE by-hand step (bootstrap)
|
||||
|
||||
Such a box's `felhom-os-apply` has no bundle mode, and no signed job can write a root file there (that gap IS
|
||||
R-840). So once per box, as root, install the bundle-aware `felhom-os-apply` — and nothing else:
|
||||
|
||||
```bash
|
||||
curl -fsSL -o /root/felhom-bundle-bootstrap.sh https://felhom.eu/scripts/felhom-bundle-bootstrap.sh
|
||||
sha256sum /root/felhom-bundle-bootstrap.sh # f4a455fad9b1acac15c0a73a349d6f5d4353f734a3c426d8377d2e632762c548 (installer 1.31.0)
|
||||
bash /root/felhom-bundle-bootstrap.sh 0.143.0 8d7273cf5313ef62b867cb6f831c631923a436452d6f90b8ff7f0771170396ba
|
||||
```
|
||||
|
||||
It must end with `DONE. This box can now take signed config bundles. Nothing else was changed.` It stops with
|
||||
`STOP:` and changes nothing on a wrong sha. Then send the bundle by the signed job (above). Done on demo-hp and
|
||||
demo-felhom on 2026-10-04 (`audits/r840-config-bundle-2026-10-04/partB/b2`, `b3`).
|
||||
|
||||
### Tester 2 (`Tester-2-be8404`) — the operator's steps (decision 97)
|
||||
|
||||
CC has no route to Tester 2: its door (`felhom-sshd` on 8822) admits only the operator's WireGuard peer, and the
|
||||
`felhom-op` user cannot become root by itself.
|
||||
|
||||
1. Connect your WireGuard operator tunnel (`operations/nodes.md`, "OOB belt").
|
||||
2. `ssh -p 8822 felhom-op@10.77.0.5`
|
||||
3. Hub → Hosts → `Tester-2-be8404` → Console access → **Reveal** the root password (the hub writes one line on
|
||||
Tester 2's own timeline: "the operator requested your console password" — that is by design).
|
||||
4. `su -` with that password, then the three commands above.
|
||||
5. Tell CC "bootstrap done". CC sends the bundle by the signed job and reads it back on the System page.
|
||||
|
||||
## Release side
|
||||
|
||||
`scripts/release-agent.sh` builds and publishes the bundle with every release (reproducible: same source → same sha)
|
||||
and verifies it by download. Vouching an agent version stores its bundle sha; the install manifest serves it to the
|
||||
installer. A box behind the vouched bundle for 7 days raises `os_config_bundle_behind` (operator mail;
|
||||
`OS_ALARM_BUNDLE_BEHIND_AFTER`).
|
||||
|
||||
**Adding a root-owned file:** add it to `BUNDLE_FILES` in `felhom-agent/configs/felhom-os-apply` — never a new
|
||||
installer fetch. `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks.
|
||||
@@ -0,0 +1,35 @@
|
||||
# RUNBOOK — test waits for OS-update approvals, and how they end
|
||||
|
||||
Design: `architecture/11-os-updates.md` §5.3, §5.3.1, §5.8. Row R-859. Hub v0.133.0.
|
||||
|
||||
## The ruled waits
|
||||
|
||||
- Guest and host fast lane: every ring-0 box runs the set healthy for **24 h** and through **1** night run → the hub
|
||||
approves it automatically.
|
||||
- Docker engine set: every ring-0 box ran it in **2** healthy night Docker steps → the operator's "Approve Docker set"
|
||||
button works.
|
||||
|
||||
## A test wait (only for a test, never to ship)
|
||||
|
||||
Set one of these env vars on the hub Deployment (via `manifests/hub.yaml`, synced), restart:
|
||||
`OS_APPROVE_AFTER=<duration>`, `OS_APPROVE_NIGHTS=<n>`, `OS_DOCKER_APPROVE_NIGHTS=<n>`. The hub logs
|
||||
`TEST CONFIGURATION` at start.
|
||||
|
||||
**The rule (R-859): test approvals end with the test.** Every approval made while ANY of the three is set is stored
|
||||
with a `test` mark (amber "TEST approval" on the System page). At the next start WITHOUT the overrides, every test
|
||||
approval that no real approval has superseded is **cancelled**: no ring-1 box installs it from then on (ring-1 boxes are
|
||||
bumped), and the operator gets an `os_release_cancelled` mail. What boxes already installed stays. The same set is
|
||||
approved again by the ruled wait, as a real release.
|
||||
|
||||
So: **remove the override and restart the hub as the last step of every test.** Leaving it set leaves the test
|
||||
approvals in force for real boxes.
|
||||
|
||||
## The 2026-10-04 fact
|
||||
|
||||
On 2026-10-04 no release could pass the ruled 24 h + 1 night, so the guest and host sets were approved under the test
|
||||
wait (12:39 / 12:41 UTC, 1.5 h after first seen). **Tester 2** (bound 16:06 UTC) installed both on its first night run
|
||||
(16:24 / 16:25 UTC): 49 guest and 106 host packages. Why the risk was small: ring 0 (both demo boxes) had run the same
|
||||
sets healthy since 11:07 / 12:24 UTC and kept running them; the packages are Debian (Security) fixes only; Tester 2
|
||||
reported both runs `applied, healthy`. Hub v0.133.0's one-time backfill marked those two automatic approvals as test
|
||||
approvals and cancelled them at its start (2026-10-04 18:20 UTC, with two older ones of the same day). The operator's own
|
||||
Docker button approval of that day stays in force (decided by CC unattended — operator may reverse).
|
||||
Reference in New Issue
Block a user