felhom.eu/documentation/runbooks/config-bundle.md

RUNBOOK — the config bundle: a box's root-owned files by a signed job (R-840)

Design: architecture/11-os-updates.md §5.4.2. Decision 96 (09 §3). Built 2026-10-04: agent v0.143.0, hub v0.133.0, installer 1.31.0. Evidence: audits/r840-config-bundle-2026-10-04/.

What it is

Every root-owned file the installer writes for the agent (both sudoers files, the five wrappers, the crash guard and its units, the agent and rollback units, the start-limit drop-in, the mgmt watchdog, the OOB belt's files) travels as ONE file, felhom-config-bundle.json, published beside the agent binary (felhom-agent/<version>/). A new box gets it from the installer; an installed box gets it by a signed agent_config_update. The box's own root-owned felhom-os-apply checks the signature, the sha, every path and every file before it writes anything, keeps the previous copies, self-checks, and puts everything back on a failure.

The trust root is never in a bundle. /etc/felhom/operator-signers and /etc/felhom/os-trust.json decide who may sign; no bundle may add, remove or change them. A box that has no signers file gets exactly the installer's pinned key (felhom-op-1), and only through a job that key signed. Signer rotation is a separate act (04 §3).

Send a bundle to a box (CC or the operator, on DooPlex)

1. The bundle's sha: the release output (release-agent.sh prints bundle :), or the hub's Configuration page after vouching (the hub resolves it from the registry by exact name).
2. Sign and queue (key felhom-op-1; the hub key is Secret felhom-system/report-api, read into a file, never printed):

   felhom-opsign -op agent_config_update -host <host_id> -key-id felhom-op-1 -key <operational key> \
     -agent-version <X.Y.Z> -bundle-sha256 <sha> -ttl 45m -upload http://<hub ClusterIP>:8080 -hub-key "$(cat <file>)"
   

3. The agent takes it on its next job poll (measured 4–15 min). Positive observables in the box's journal: os-apply: BUNDLE DONE agent=<X> written=<n> … self-check=ok, then capability probe after the config bundle ok=71 total=71. The hub's System page "Root files" column shows the version.
4. Undo = send the previous release's bundle the same way. The previous copies also stay on the box in /var/lib/felhom-os-apply/bundle-prev/<time>-before-<version>/ (the last 3).

A box from before agent v0.143.0 — the ONE by-hand step (bootstrap)

Such a box's felhom-os-apply has no bundle mode, and no signed job can write a root file there (that gap IS R-840). So once per box, as root, install the bundle-aware felhom-os-apply — and nothing else:

curl -fsSL -o /root/felhom-bundle-bootstrap.sh https://felhom.eu/scripts/felhom-bundle-bootstrap.sh
sha256sum /root/felhom-bundle-bootstrap.sh     # f4a455fad9b1acac15c0a73a349d6f5d4353f734a3c426d8377d2e632762c548 (installer 1.31.0)
bash /root/felhom-bundle-bootstrap.sh 0.143.0 8d7273cf5313ef62b867cb6f831c631923a436452d6f90b8ff7f0771170396ba

It must end with DONE. This box can now take signed config bundles. Nothing else was changed. It stops with STOP: and changes nothing on a wrong sha. Then send the bundle by the signed job (above). Done on demo-hp and demo-felhom on 2026-10-04 (audits/r840-config-bundle-2026-10-04/partB/b2, b3).

Tester 2 (Tester-2-be8404) — the operator's steps (decision 97)

CC has no route to Tester 2: its door (felhom-sshd on 8822) admits only the operator's WireGuard peer, and the felhom-op user cannot become root by itself.

1. Connect your WireGuard operator tunnel (operations/nodes.md, "OOB belt").
2. ssh -p 8822 felhom-op@10.77.0.5
3. Hub → Hosts → Tester-2-be8404 → Console access → Reveal the root password (the hub writes one line on Tester 2's own timeline: "the operator requested your console password" — that is by design).
4. su - with that password, then the three commands above.
5. Tell CC "bootstrap done". CC sends the bundle by the signed job and reads it back on the System page.

Release side

scripts/release-agent.sh builds and publishes the bundle with every release (reproducible: same source → same sha) and verifies it by download. Vouching an agent version stores its bundle sha; the install manifest serves it to the installer. A box behind the vouched bundle for 7 days raises os_config_bundle_behind (operator mail; OS_ALARM_BUNDLE_BEHIND_AFTER).

Adding a root-owned file: add it to BUNDLE_FILES in felhom-agent/configs/felhom-os-apply — never a new installer fetch. test_every_root_file_the_installer_writes_is_in_the_bundle fails on a path the bundle lacks.