felhom.eu/REPORT-r840-config-bundle-2026-10-04.md

# REPORT — the config bundle (R-840), test approvals (R-859), the Docker-socket self-heal (R-860), Tester 2, drill-r50

2026-10-04, evening–night. Brief: "a signed route that brings an installed box's root-owned parts up to date (R-840)…".
Architecture read first: `11-os-updates.md` (§5.3, §5.4, §5.8, §5.9, §8), `03-host-agent.md` (§3, §4, §11),
`04-control-plane-authorization.md` (§3), `07-backup-architecture.md` (§6.4). Evidence: `documentation/audits/r840-config-bundle-2026-10-04/`.

## The Part table

| Part | Result | Why / what changed |
|---|---|---|
| A — what Tester 2 has | **done (read through the hub; no route to the box)** | The brief's timestamp reasoning was wrong (UTC vs local). Tester 2 has installer 1.30.0, agent 0.142.0, the crash guard armed, `live-restore` on, Docker 29.8.2, the re-made golden. It lacks only the R-858 wrapper fix. |
| A — the `felhom-pbs` warning | **done: not a fault** | By design (R-723): a new box's first-hour tier skip is recorded, never mailed; the tier came 7 min later and backed up at 16:31 UTC. No row. |
| B — the bundle route | **done, live on both demo boxes** | Agent 0.143.0 (bundle mode in `felhom-os-apply`, signed `agent_config_update`), hub 0.133.0, installer 1.31.0. Changed: the installer is NOT the self-update wrapper (it cannot do it); a one-time bootstrap is needed on older boxes. |
| C — Tester 2 | **act 1 sent, NOT delivered (box offline since 18:06 UTC); act 2 waits for the operator; act 3 not needed** | Act 1: queued 18:35 UTC — see "Part C" below. Act 2 needs the one by-hand bootstrap (R-862, the operator's tunnel). Act 3: `live-restore` is already on. |
| D — test approvals end | **done, live** | Hub 0.133.0 cancelled the 4 test approvals of 2026-10-04 at start (18:20 UTC). Changed: the operator's own Docker button approval was not backfilled (CC decision). |
| E — self-heal after any Docker restart | **done, live (9202 + demo-hp)** | Controller 0.293.0. Changed: only a `docker.socket` restart breaks it; a dockerd crash or `systemctl restart docker` does not. |
| F — new installs start with the approved fixes | **built; tonight's golden carries none** | `build-golden.sh` 3.2.0 `GOLDEN_GUEST_PKGS`. No guest release is in force tonight (the test one was cancelled), so the golden keeps the template (49 Debian updates pending for the next real approval). |
| G — drill-r50 | **done** | Deleted through the product (operator's yes: it touched ep0). On ep0: its PBS namespace did not exist; its WireGuard peer left (5 → 4 peers pushed). |
| H — release, golden, records | **done** | Agent 0.143.0, controller 0.293.0, hub 0.133.0, installer 1.31.0, golden 0.293.0 baked + vouched. Floor 0.293.0 for the two demo customers only (Tester 2 untouched). |

## Claims in the brief that turned out wrong

1. **The timestamp reasoning about Tester 2 (§1).** Tester 2 was bound at **16:06 UTC = 18:06 local**; the releases were
   compared in local time. At 18:06 local, installer 1.30.0 (17:36), agent 0.142.0 (vouched ~17:49) and the re-made golden
   (17:48) were all current. Measured (hub records): agent **0.142.0**, crash guard **armed** (`kernel.panic` 10), `live-restore`
   **on**, Docker **29.8.2**; the wrapper reports a facts mode (so it is not "old"). Only agent 0.142.1's R-858 wrapper fix is
   missing — and the root files of 0.142.0 equal 0.143.0's in every file but `felhom-os-apply`.
2. **"The felhom-pbs warning is a fault."** It is the designed first-hour behaviour (R-723): recorded, not mailed.
3. **"The self-update wrapper can install a bundle safely."** It cannot: it is a fixed sh script that swaps one binary. The
   bundle is installed by `felhom-os-apply` (already reachable through the agent's sudoers line). And **no** existing root
   helper can install the first bundle on an older box — one by-hand bootstrap per older box (done on both demo boxes).
4. **"A restart of an existing container picks up the new socket."** True — measured twice (controller exit → restart policy;
   `docker restart traefik`): both came back on the current inode, same container ids.
5. **"dockerd restarts on its own … the box shows DOWN until a person acts."** Only when the socket FILE is re-created
   (`docker.socket` restart, i.e. a docker-ce upgrade or by hand). A dockerd crash or `systemctl restart docker` keeps it (measured).
6. **Act 3 for Tester 2 (`live-restore` reload)** is not needed: it is already on (from the golden).

## Part A — Tester 2, read back (hub records; CC has no route to the box)

| Item | Brief expected | Measured | Source |
|---|---|---|---|
| bound / enrolled | 16:06 (read as local) | 16:06:27 UTC bind, 16:07:04 UTC host | `appliance_registrations`, `hosts` |
| installer | 1.29.0 | **1.30.0** (the crash guard is installed — only 1.30.0 does that) | host report `system.facts.host.crash_guard` |
| agent | 0.141.1 | **0.142.0** | `/hosts` |
| PBS wrapper sha | — | `104db0a4…` = every release's | host report `wrapper_sha256` |
| `felhom-os-apply` | old (no facts, no Docker) | 0.142.0's (it answers the facts mode) | facts present |
| sudoers | — | not readable from the hub; 0.142.0's file equals 0.143.0's (tag diff); capability probe all ok | `/hosts/Tester-2-be8404` |
| crash guard / `kernel.panic` | absent / 0 | **armed / 10** | facts |
| operator-signers | absent | not readable from the hub; installer 1.30.0 writes it | inference |
| `live-restore` / Docker | off / unpinned | **on / 29.8.2** | facts |
| golden | first 0.292.0 bake | the re-bake (live-restore + 29.8.2) | facts |
| OS releases installed | TEST-approved | guest `os-guest-20261004-123933` (49 pkgs, 16:24) and host `os-host-20261004-124133` (106, 16:25), both approved "auto" 1.5 h after first seen = the TEST wait | `os_reports`, `os_releases` |

## Part C — Tester 2, what happened

- **Act 1** (signed `agent_update` to 0.143.0): queued 18:35 UTC, **not delivered** — Tester 2 stopped reporting at
  18:06 UTC (host 18:05:48, controller 18:06:24). It was also silent 17:13–18:05 UTC, and its controller started at
  18:06:20 UTC, so the box restarted or was switched on in between. CC sent Tester 2 nothing before 18:35 UTC; the cause
  is outside this work (power, network, or the tester). The job expires 19:20 UTC; if Tester 2 returns later, it is
  refused as expired (harmless) and must be re-signed.
- **Act 2** (bundle): waits for the operator's one-time bootstrap (R-862).
- **Act 3** (`live-restore`): not needed — already on.
- No Docker step, no reboot, no app change was sent. Read-back of the System page line: still agent 0.142.0, Root files
  `unknown`, guard armed, `live-restore` on (last report 18:05 UTC).

## Part B — the bundle

Shape, checks, trust rule, bootstrap: `11` §5.4.2; runbook `runbooks/config-bundle.md`. Red-proofs: 22 of 22 wrapper rules
(`partB/redproof.txt`), hub 6 of 6 (`partB/hub-bundle-redproof.txt`), Go executor tests. Live:

| Step | Box | Result |
|---|---|---|
| read the box's files vs the bundle | both | 21 of 22 identical; only `felhom-os-apply` differed (`b1`) |
| bootstrap, wrong sha | demo-hp | `STOP`, nothing changed (`b2`) |
| bootstrap | both | the new `felhom-os-apply`, self-check ok (`b2`, `b3`) |
| job A, wrong sha | demo-hp | refused by the agent, nothing changed (`b4`) |
| job B, the 0.143.0 bundle | both | written 0, same 21, kept 1, self-check ok, probe 71/71 (`b4`) |
| a bundle with one deliberate change | demo-hp | written 1 (that file), self-check ok (`b6`) |
| its undo (the 0.143.0 bundle again) | demo-hp | written 1, the release file back (sha `b71d8698…`), probe 71/71 (`b6`) |
| replay of job B | demo-hp | `REJECTED … replay (nonce already seen)` (`b6`) |
| the installer's new path | demo-felhom | written 0, record `installer`, services active (`b5`) |

## Part D — test approvals

Marked, cancelled at start, amber, operator event, ring-1 bump; red-proofs 6 of 6 (`partD/d-redproof.txt`). Live at the
18:20 UTC start: `os-guest-20261004-123933`, `os-20261004-091417`, `os-host-20261004-124133`, `os-host-20261004-124034`
cancelled; one mail (three held by the cooldown); the System page lists them; the Docker release stays (`partD/d2`). The
2026-10-04 fact and why the risk was small: `runbooks/os-updates-test-waits.md`.

## Part E — the measured heal

| Box | Break | Controller back | traefik back | Container ids |
|---|---|---|---|---|
| 9202 (controller 0.291.0, before the fix) | `systemctl restart docker.socket` | never by itself (blind 2+ min, health "healthy") | never | same |
| 9202 (0.293.0) | same | +73 s | +104 s | same (`e7`) |
| demo-hp 9201 (0.293.0) | same | +88 s | +120 s | 21 of 21 same (`e8`) |

`systemctl restart docker` and `kill -9 dockerd`: socket inode unchanged, nothing to heal (`e1`, `e2`). Red-proof 9 of 9 (`e6`).

## Part F — the first-night count

Golden 0.293.0 (`documentation/tests/golden-0.293.0-2026-10-04/`): no guest release in force → template versions kept;
**49** Debian updates pending in the baked guest, which the next REAL approval (tomorrow, after 24 h + a night) will bring.
A ring-1 box from this golden installs **0** on its first night until then. The host: the agent's first leg already runs
right after the first whole-guest backup (Tester 2: 17 min after enrolment, 106 host packages in 44 s) — an installer pass
would cost ~45 s and run before any backup exists; not built.

## Part G — drill-r50

What the product delete touched (read first, `partG/g1`): hub rows (host, guest, reports 185, telemetry 185, the recovery
credential, the DR recipe, the claim, an appliance registration, the WireGuard peer 10.77.0.4), ep0's PBS tenancy
(`tenantsync.Deprovision`, namespace `drill-r50`), ep0's WireGuard peer list. No Storage Box (no off-site tier), no mail
(no address). Asked the operator (ep0); yes. Result: `deprovision ok (existed=false)` — nothing destroyed on ep0; the peer
left ep0 at the next push (5 → 4); `/hosts` without drill-r50; the delete preview 404s (`partG/`).

## Rows

Before **333**, after **334**. Closed: **R-840** (built), **R-859** (opened and closed), **R-860** (opened and closed).
Opened: **R-861** (P2 — the agent's sudoers is root-equivalent; read, not exploited), **R-862** (P3 — Tester 2's bootstrap,
operator). R-857 not fixed: avoided by baking under a new controller version.

## Decisions taken by CC (operator may reverse)

1. The one-time backfill marks only the AUTOMATIC test approvals; your Docker button approval of 2026-10-04 stays in force.
2. The bundle alarm fires after 7 days behind (`OS_ALARM_BUNDLE_BEHIND_AFTER`).
3. The bundle lives inside `felhom-os-apply` (no new sudoers line), not in a new tool.
4. The controller fixes Part E (it reaches every box by the floor), not the agent (that would need a sudoers change).
5. The demo customers' floor is 0.293.0; the global floor stays 0.292.0 so Tester 2's controller does not move this session.

## Teardown

- Machines: 9202 left on controller 0.293.0, working (all apps up). demo-hp and demo-felhom: agent 0.143.0, bundle 0.143.0,
  controller 0.293.0; the test line removed again on demo-hp. Drill VM off and at `virgin`.
- Hosts: the bootstrap script, the installer harness and the test script removed from `/root` on both demo hosts. The
  bundles' previous copies stay in `/var/lib/felhom-os-apply/bundle-prev/` by design.
- Hub: the registry's test version `0.143.0-r840test` deleted (204, then 404); drill-r50 removed; the scratchpad copy of
  the hub DB deleted at the end.