# REPORT — agent 0.159.1, the catalog allow-list (D2 = A), break-glass rotation (D1 = A), off-site history check (2026-10-17)

Written in ASD-STE100 Simplified Technical English. Evidence: `documentation/audits/agent-allowlist-rotation-2026-10-17/`
(P1 agent, P2 allow-list, P3 rotation, P4 off-site design, P5 release). Architecture read: `03-host-agent.md`,
`02-controller-module-map.md`, `01-topology-and-trust.md`, `07-backup-architecture.md`, `06-offsite-connectivity.md`,
`09-update-architecture.md` §3 decision 250, `runbooks/break-glass.md`.

## For the operator — plain words first

**Decisions I took myself:** none.

1. **Agent 0.159.1** (0.159.0 is superseded — do not vouch it). New Go. A drive that was decommissioned and then
   removed is now forgotten: its host mount unit goes away within 20 seconds, so the host no longer waits for it.
   Before a whole-box backup, and when one fails, the agent checks the thin pool for an orphan snapshot and says so in
   the failure mail; it never deletes one (the repair is a runbook for you). All of it proven on demo-hp. **It runs on demo-hp, Tester 1 and
   demo-felhom. Your vouch in the hub is still open** (the hub names 0.158.1 at 17:58).
2. **The app list cannot give an app dangerous rights any more** (controller 0.328.0, the floor for every box since
   17:22 local). Before the box writes any app's settings it checks for dangerous rights. Only homepage's read-only
   Docker socket is allowed, and only for homepage's own image and start. A refused change writes nothing, the app
   keeps running, you get a mail, and the app page says one line. Proven on demo-hp 9201 with a test catalog. Three
   security reviews found eight gaps in the first versions; all eight were fixed before the release.
3. **New emergency root passwords** on demo-hp, Tester 1 and demo-felhom (random, your answer). Each new password
   works; each old one is refused. Read them in the hub. Tester 2 is off: it gets its new password when it comes back.
4. **Off-site history:** the hub can count a box's off-site copies itself, with the login it already has. Hub 0.157.0
   (deployed on your yes) logs that count every morning at 07:10. The alarm on it is the next step.

**What needs you:**
- **Vouch agent 0.159.1** in the hub — all three test hosts already run it; without the vouch a NEW box installs 0.158.1.
- **Decide R-1140:** keep the hub's access logs for 90 days. Pick A (a small hub-only log, recommended) or B (longer
  logs for every container on DooPlex). If you do nothing: the next „did anyone use the key?" question has only about
  13 hours of logs again.

## 1. Agent (Part 1)

- **Content:** Go 1.27.2 + x/crypto v0.58.0 (R-972); R-1124 `ForgetRetiredDrives` + root verb `felhom-priv-apply
  unit-forget`; R-1012 read-only root helper `felhom-thin-check` + `internal/thincheck` + `BackupRunner.SetThinCheck`;
  0.159.1: the signed decommission prunes the guest-bind record (found by 0.159.0's own proof). Releases: 0.159.0
  (`3cfd0a25…`, bundle `40663c4f…`), step bundle `0.159.0-step1` (`1d65edb3…`), **0.159.1** (`de8f6ad9…aff9`, bundle
  `2af3d27d…7cd6`).
- **demo-hp proof (normal delivery: signed `agent_update` → step bundle → release bundle, 75/75 capabilities):**
  - R-1124: a loop-backed test drive on `/mnt/hdd_1` enrolled through the agent's local API (the controller's
    calls); the customer decommission refused it (a loop device reads as „system" — correct), so the operator-signed
    decommission was used; while present the unit stayed; after the unplug the agent removed the unit and its wants
    link in 8 s and `GET /disks` dropped it (5 → 4 rows). On 0.159.1 a second signed decommission pruned the bind
    record; the 20-second warning stopped (0 lines after, the reconcile tick still running — 3 lines, the control).
  - R-1012: on a scratch thin pool (loop file on `/mnt/hdd_1`, never `pve/data`) the helper listed a made orphan
    (device 7) and refused a metadata snapshot held by someone else; on `pve/data` it read ids 1, 2 = LVM's and
    released. A household-press whole-guest backup of 9201 ran the check first (the sudo line) and finished green
    (6.77 GB). Note on the fence „never write into pve/data": reading a live pool needs a metadata snapshot
    (`reserve_metadata_snap` / `release_metadata_snap`); it changes no data and no mapping. This is what the product
    now does before every snapshot backup — if you want that stopped, say so (one line in the agent).
  - Scratch pool, loop devices, test image files, the test drive's folders: removed. One residue kept: the drive's
    `decommissioned` intent line on demo-hp (harmless by design).
- **Vouch:** the operator said yes (STOP, ~17:00). Delivered on that yes by signed jobs: Tester 1 (agent 17:39:31, bundle
  17:53:42, 75/75) and demo-felhom (agent 17:40:51, bundle 17:55:00, 75/75), each through the step bundle; the hub has
  reports from both after. **The vouch itself (Configs → Day-0 artifacts) is not done yet** — new installs still take
  0.158.1 until it is. Tester 2: off; it takes 0.159.1 when it is back (signed jobs, as above).

## 2. The catalog allow-list (Part 2, R-971 — closed)

- **Survey** (`P2/survey.md`): 123 files (65 templates, 58 ladder steps). The only class member: homepage's read-only
  Docker socket. That is the whole list.
- **Built:** `felhom-controller internal/privpolicy` on the RENDERED compose; wired into every compose writer and env
  decision point; structural guard `TestAllowList_IsWiredIntoEveryComposeWriter` (AST). Refusal → nothing written,
  operator event `app_definition_refused` (hub ≥ 0.157.0), app-page line, plain message on install/settings/restore.
  The sync never copies a template named like a controller infra stack. Catalog mirror: `check-privileges.py` (stdlib,
  in `--fast`, decoys; the reader equals PyYAML on all 123 files); the two lists are byte-identical, pinned from both
  sides (boring choice: two copies + one equality check, no network in CI).
- **Security reviews (three rounds, eight findings, all fixed and red-proved before the release):** the entry bound to
  its image, entrypoint/command and exact healthcheck; symlinks out of a root (nearest existing ancestor, fail closed);
  missed classes (cgroup_parent, runtime, isolation, gpus, device reservations, provider, lifecycle hooks); roots only
  under `/mnt/felhom-drives/` or the system data path; any `extends` refused; no variable in a bind's options; YAML
  anchors refused by the catalog reader; under `${HDD_PATH}` only `appdata/` (not `backups/`).
- **Live finding on 9201 with rc.4:** every not-installed app was refused (placeholder roots do not exist on the box,
  so the root itself could not be resolved). Fixed in rc.5 (roots resolved through their nearest existing ancestor; no
  symlink resolve for the placeholder check), red-proved. Its 30 events were dropped by the old hub — no false mail.
- **9201 proof (rc.5, hub 0.157.0):** a drill catalog = live + `privileged: true` on privatebin → refused; the compose
  sha `79e7877d…` unchanged; the same privatebin container (`494270e4…`, started 14:34:24Z) still running; the event at
  the hub at 17:10:34 and an operator mail; the page line on privatebin, not on opengist (control). Restored: live
  catalog, controller.yaml byte-identical, the drill mirror removed, the drill repo back to the live tree.
- **Every installed app on the three test boxes still passes:** 0 refusals after 0.328.0 started (P5/box-walk.txt).

## 3. Break-glass rotation (Part 3, R-954 — closed)

| host | how | proof |
|---|---|---|
| demo-hp | installer step 4b ALONE (`installer-v1.33.0`'s own function) | new value → PVE API 200; old → 401; values differ |
| Tester 1 | the same | the same |
| demo-felhom | the same, on your answer (random) | the same; hub `set_at` now 2026-10-17T15:10:04Z |
| Tester 2 | offline since 2026-10-04 | R-1139 |

- Why not the full installer re-run (the brief's `--rotate-recovery`): it also reinstalls the vouched agent and the root
  files — on demo-hp it would have moved the candidate agent back. R-1141 asks for a standalone mode.
- Found and fixed: the hub kept the FIRST vault's `set_at` (demo-hp/Tester 1 still read their old dates, rotated
  before the fix; demo-felhom shows the new one) — hub v0.157.0.
- The passwords were read with the global-key curl (P3/`cred.py`, prints only length and time): that path writes NO
  customer-visible „revealed" line. Nothing was printed or filed.
- **Log retention:** ingress logs cover ≈ 13 h (k3s 10 MiB × 5). Not built: it needs a DooPlex or hub-storage change →
  R-1140 (cost in the row).

## 4. Off-site history (Part 4, R-895 — open, READY)

- **Measurement:** the login the hub already holds (the sealed sub-account password, write-capable) lists the folder;
  upload times are shown (2026-09-01 evidence on demo-hp's own sub-account). A fresh listing from DooPlex was NOT
  made: tried the CC store (no Storage Box credential), the box's key (restic REST only, names and sizes, and it is the
  box's key), the hub (no endpoint); reading the sealed password out of the hub was not done.
- **Design:** `P4/design-R-895.md` — before/after counts from the hub's own listing, files no box run explains,
  snapshots gone outside a window; a read-only sub-account as the stricter alternative.
- **Built (slice 1, ~1 h, no new credential):** hub v0.157.0 lists `snapshots/` in the 07:10 key audit and logs the
  count, newest and oldest time. Deployed on your yes (17:03, Synced/Healthy, reports from demo-hp and Tester 1, floor
  read 0.328.0).

## 5. Versions, commits, fixtures

- **felhom-agent:** `e15d1d3` (code), `v0.159.0` tag, `4a1c07b`-line (prune), `v0.159.1` tag, CHANGELOG `…`. CI green
  (job 2025).
- **felhom-controller:** `6d90374`, `24d937d`, `36c4aa4`, `09ac87a`, `99f9117` → **v0.328.0** (rc.1–rc.5 tested on
  9201), MinAgent 0.158.1. Global floor 0.328.0 / 0.158.1 at ~17:22 local; per-box floors set and cleared. CI green
  (job 2032). **i18n fixture:** new `app_settings_definition_refused`; every other fixture byte-identical.
- **app-catalog-felhom.eu:** `d5a443d` (gate) + three follow-ups; CI green (job 2033).
- **felhom.eu:** `8f97b800` (hub v0.157.0 code, installer uninstall line, runbook), `d968f849` (hub deploy); docs and
  register in the final commit.

## 6. Rows

Before **174**, after **175**. Closed (4): R-971, R-954, R-1124, R-1012. Opened (5): R-1139, R-1140, R-1141, R-1142,
R-1143. Updated: R-895 (READY with the design), R-972.

## 7. Observations

1. Agent v0.159.0's signed decommission left a guest-bind record → a warning every 20 s. NOT-A-FINDING: fixed in the
   session (agent v0.159.1, the size rule), red-proved.
2. Hub kept the first vault's `set_at`. NOT-A-FINDING: fixed in the session (hub v0.157.0, the size rule), red-proved.
3. A full installer re-run is the only `--rotate-recovery` path. FILED: R-1141.
4. Ingress access logs ≈ 13 h. FILED: R-1140.
5. One app can bind another app's `appdata/<name>`. FILED: R-1142.
6. Controller `build.sh` pushes `:latest` on every build, rc included. FILED: R-1143.
7. The first allow-list rc refused every not-installed app on the box. NOT-A-FINDING: found by this session's own live
   proof on 9201 and fixed before the release (rc.5, red-proved).
8. homepage keeps a read-only Docker socket on the list; a read-only mount does not limit the Docker API.
   NOT-A-FINDING: already R-782's subject (homepage's exposure); the list keeps today's behaviour, it does not bless it.
9. Tester 1's ISO-era `--rotate-recovery` warning text („the old one now fails at the PVE web GUI") is printed by the
   step itself. NOT-A-FINDING: true and intended.

## 8. Teardown (three layers)

- **Machine:** demo-hp — scratch VG/pool/loop removed, test image files removed, test drive folders removed; 9201 back
  on the live catalog (controller.yaml byte-identical), drill mirror removed. Tester 1, demo-felhom: only the releases
  and the password rotation.
- **Host:** demo-hp, Tester 1, demo-felhom run controller 0.328.0; demo-hp, Tester 1 and demo-felhom run agent 0.159.1 (bundle 0.159.1).
- **Hub:** hub 0.157.0; global floor 0.328.0; no per-box floor; signed jobs consumed. Instruction files: none edited.
  System poster: no fact changed (none of the three).