felhom.eu/documentation/runbooks/config-bundle.md

# RUNBOOK — the config bundle: a box's root-owned files by a signed job (R-840)

Design: `architecture/11-os-updates.md` §5.4.2. Decision 96 (`09` §3). Built 2026-10-04: agent v0.143.0, hub v0.133.0,
installer 1.31.0. Evidence: `audits/r840-config-bundle-2026-10-04/`.

## What it is

Every root-owned file the installer writes for the agent (both sudoers files, the five wrappers, the crash guard and
its units, the agent and rollback units, the start-limit drop-in, the mgmt watchdog, the OOB belt's files) travels as
ONE file, `felhom-config-bundle.json`, published beside the agent binary (`felhom-agent/<version>/`). A new box gets it
from the installer; an installed box gets it by a signed `agent_config_update`. The box's own root-owned
`felhom-os-apply` checks the signature, the sha, every path and every file before it writes anything, keeps the
previous copies, self-checks, and puts everything back on a failure.

**The trust root is never in a bundle.** `/etc/felhom/operator-signers` and `/etc/felhom/os-trust.json` decide who may
sign; no bundle may add, remove or change them. A box that has no signers file gets exactly the installer's pinned key
(`felhom-op-1`), and only through a job that key signed. Signer rotation is a separate act (`04` §3).

## Send a bundle to a box (CC or the operator, on DooPlex)

1. The bundle's sha: the release output (`release-agent.sh` prints `bundle :`), or the hub's Configuration page after
   vouching (the hub resolves it from the registry by exact name).
2. Sign and queue (key `felhom-op-1`; the hub key is Secret `felhom-system/report-api`, read into a file, never printed):
   ```bash
   felhom-opsign -op agent_config_update -host <host_id> -key-id felhom-op-1 -key <operational key> \
     -agent-version <X.Y.Z> -bundle-sha256 <sha> -ttl 45m -upload http://<hub ClusterIP>:8080 -hub-key "$(cat <file>)"
   ```
3. The agent takes it on its next job poll (measured 4–15 min). Positive observables in the box's journal:
   `os-apply: BUNDLE DONE agent=<X> written=<n> … self-check=ok`, then `capability probe after the config bundle ok=71
   total=71`. The hub's System page "Root files" column shows the version.
4. **Undo** = send the previous release's bundle the same way. The previous copies also stay on the box in
   `/var/lib/felhom-os-apply/bundle-prev/<time>-before-<version>/` (the last 3).

## A release whose bundle ADDS a path — the step bundle (R-880, decision 124)

The box's INSTALLED `felhom-os-apply` checks every path of an incoming bundle against its OWN table (R16). So when a
release adds a path (agent v0.146.1 added four), every box on an older bundle refuses it. Send a step first:

```bash
# the bundle the boxes run now — check its sha against the hub's Root files / config-bundle record
curl -fsS -o base.json https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/<old>/felhom-config-bundle.json
python3 felhom-agent/scripts/build-step-bundle.py base.json <new>-step1 step.json      # prints the step sha
curl -u admin:<token from a file> -X PUT --upload-file step.json \
  https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/<new>-step1/felhom-config-bundle.json   # 201
# per box: agent_update <new>  →  agent_config_update <new>-step1 (step sha)  →  agent_config_update <new> (release sha)
```

The step is the old bundle with ONLY `felhom-os-apply` replaced, so the old wrapper accepts it (`written=1 same=20`);
the new wrapper then accepts the release's bundle. Done this way on demo-hp, demo-felhom and Tester 1 on 2026-10-05
(`audits/hub-safety-2026-10-05/partH/`). Keep the step package while any box may still be on the old bundle.

## A box from before agent v0.143.0 — the ONE by-hand step (bootstrap)

Such a box's `felhom-os-apply` has no bundle mode, and no signed job can write a root file there (that gap IS
R-840). So once per box, as root, install the bundle-aware `felhom-os-apply` — and nothing else:

```bash
curl -fsSL -o /root/felhom-bundle-bootstrap.sh https://felhom.eu/scripts/felhom-bundle-bootstrap.sh
sha256sum /root/felhom-bundle-bootstrap.sh     # f4a455fad9b1acac15c0a73a349d6f5d4353f734a3c426d8377d2e632762c548 (installer 1.31.0)
bash /root/felhom-bundle-bootstrap.sh 0.143.0 8d7273cf5313ef62b867cb6f831c631923a436452d6f90b8ff7f0771170396ba
```

It must end with `DONE. This box can now take signed config bundles. Nothing else was changed.` It stops with
`STOP:` and changes nothing on a wrong sha. Then send the bundle by the signed job (above). Done on demo-hp and
demo-felhom on 2026-10-04 (`audits/r840-config-bundle-2026-10-04/partB/b2`, `b3`).

### Tester 2 (`Tester-2-be8404`) — the operator's steps (decision 97)

CC has no route to Tester 2: its door (`felhom-sshd` on 8822) admits only the operator's WireGuard peer, and the
`felhom-op` user cannot become root by itself.

1. Connect your WireGuard operator tunnel (`operations/nodes.md`, "OOB belt").
2. `ssh -p 8822 felhom-op@10.77.0.5`
3. Hub → Hosts → `Tester-2-be8404` → Console access → **Reveal** the root password (the hub writes one line on
   Tester 2's own timeline: "the operator requested your console password" — that is by design).
4. `su -` with that password, then the three commands above.
5. Tell CC "bootstrap done". CC sends the bundle by the signed job and reads it back on the System page.

## Release side

`scripts/release-agent.sh` builds and publishes the bundle with every release (reproducible: same source → same sha)
and verifies it by download. Vouching an agent version stores its bundle sha; the install manifest serves it to the
installer. A box behind the vouched bundle for 7 days raises `os_config_bundle_behind` (operator mail;
`OS_ALARM_BUNDLE_BEHIND_AFTER`).

**Adding a root-owned file:** add it to `BUNDLE_FILES` in `felhom-agent/configs/felhom-os-apply` — never a new
installer fetch. `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks.