felhom.eu/REPORT-agent-allowlist-rotation-2026-10-17.md

REPORT — agent 0.159.1, the catalog allow-list (D2 = A), break-glass rotation (D1 = A), off-site history check (2026-10-17)

Written in ASD-STE100 Simplified Technical English. Evidence: documentation/audits/agent-allowlist-rotation-2026-10-17/ (P1 agent, P2 allow-list, P3 rotation, P4 off-site design, P5 release). Architecture read: 03-host-agent.md, 02-controller-module-map.md, 01-topology-and-trust.md, 07-backup-architecture.md, 06-offsite-connectivity.md, 09-update-architecture.md §3 decision 250, runbooks/break-glass.md.

For the operator — plain words first

Decisions I took myself: none.

1. Agent 0.159.1 (0.159.0 is superseded — do not vouch it). New Go. A drive that was decommissioned and then removed is now forgotten: its host mount unit goes away within 20 seconds, so the host no longer waits for it. Before a whole-box backup, and when one fails, the agent checks the thin pool for an orphan snapshot and says so in the failure mail; it never deletes one (the repair is a runbook for you). All of it proven on demo-hp. It runs on demo-hp, Tester 1 and demo-felhom. Your vouch in the hub is still open (the hub names 0.158.1 at 17:58).
2. The app list cannot give an app dangerous rights any more (controller 0.328.0, the floor for every box since 17:22 local). Before the box writes any app's settings it checks for dangerous rights. Only homepage's read-only Docker socket is allowed, and only for homepage's own image and start. A refused change writes nothing, the app keeps running, you get a mail, and the app page says one line. Proven on demo-hp 9201 with a test catalog. Three security reviews found eight gaps in the first versions; all eight were fixed before the release.
3. New emergency root passwords on demo-hp, Tester 1 and demo-felhom (random, your answer). Each new password works; each old one is refused. Read them in the hub. Tester 2 is off: it gets its new password when it comes back.
4. Off-site history: the hub can count a box's off-site copies itself, with the login it already has. Hub 0.157.0 (deployed on your yes) logs that count every morning at 07:10. The alarm on it is the next step.

What needs you:

• Vouch agent 0.159.1 in the hub — all three test hosts already run it; without the vouch a NEW box installs 0.158.1.
• Decide R-1140: keep the hub's access logs for 90 days. Pick A (a small hub-only log, recommended) or B (longer logs for every container on DooPlex). If you do nothing: the next „did anyone use the key?" question has only about 13 hours of logs again.

1. Agent (Part 1)

• Content: Go 1.27.2 + x/crypto v0.58.0 (R-972); R-1124 ForgetRetiredDrives + root verb felhom-priv-apply unit-forget; R-1012 read-only root helper felhom-thin-check + internal/thincheck + BackupRunner.SetThinCheck; 0.159.1: the signed decommission prunes the guest-bind record (found by 0.159.0's own proof). Releases: 0.159.0 (3cfd0a25…, bundle 40663c4f…), step bundle 0.159.0-step1 (1d65edb3…), 0.159.1 (de8f6ad9…aff9, bundle 2af3d27d…7cd6).
• demo-hp proof (normal delivery: signed agent_update → step bundle → release bundle, 75/75 capabilities):
  • R-1124: a loop-backed test drive on /mnt/hdd_1 enrolled through the agent's local API (the controller's calls); the customer decommission refused it (a loop device reads as „system" — correct), so the operator-signed decommission was used; while present the unit stayed; after the unplug the agent removed the unit and its wants link in 8 s and GET /disks dropped it (5 → 4 rows). On 0.159.1 a second signed decommission pruned the bind record; the 20-second warning stopped (0 lines after, the reconcile tick still running — 3 lines, the control).
  • R-1012: on a scratch thin pool (loop file on /mnt/hdd_1, never pve/data) the helper listed a made orphan (device 7) and refused a metadata snapshot held by someone else; on pve/data it read ids 1, 2 = LVM's and released. A household-press whole-guest backup of 9201 ran the check first (the sudo line) and finished green (6.77 GB). Note on the fence „never write into pve/data": reading a live pool needs a metadata snapshot (reserve_metadata_snap / release_metadata_snap); it changes no data and no mapping. This is what the product now does before every snapshot backup — if you want that stopped, say so (one line in the agent).
  • Scratch pool, loop devices, test image files, the test drive's folders: removed. One residue kept: the drive's decommissioned intent line on demo-hp (harmless by design).
• Vouch: the operator said yes (STOP, ~17:00). Delivered on that yes by signed jobs: Tester 1 (agent 17:39:31, bundle 17:53:42, 75/75) and demo-felhom (agent 17:40:51, bundle 17:55:00, 75/75), each through the step bundle; the hub has reports from both after. The vouch itself (Configs → Day-0 artifacts) is not done yet — new installs still take 0.158.1 until it is. Tester 2: off; it takes 0.159.1 when it is back (signed jobs, as above).

2. The catalog allow-list (Part 2, R-971 — closed)

• Survey (P2/survey.md): 123 files (65 templates, 58 ladder steps). The only class member: homepage's read-only Docker socket. That is the whole list.
• Built: felhom-controller internal/privpolicy on the RENDERED compose; wired into every compose writer and env decision point; structural guard TestAllowList_IsWiredIntoEveryComposeWriter (AST). Refusal → nothing written, operator event app_definition_refused (hub ≥ 0.157.0), app-page line, plain message on install/settings/restore. The sync never copies a template named like a controller infra stack. Catalog mirror: check-privileges.py (stdlib, in --fast, decoys; the reader equals PyYAML on all 123 files); the two lists are byte-identical, pinned from both sides (boring choice: two copies + one equality check, no network in CI).
• Security reviews (three rounds, eight findings, all fixed and red-proved before the release): the entry bound to its image, entrypoint/command and exact healthcheck; symlinks out of a root (nearest existing ancestor, fail closed); missed classes (cgroup_parent, runtime, isolation, gpus, device reservations, provider, lifecycle hooks); roots only under /mnt/felhom-drives/ or the system data path; any extends refused; no variable in a bind's options; YAML anchors refused by the catalog reader; under ${HDD_PATH} only appdata/ (not backups/).
• Live finding on 9201 with rc.4: every not-installed app was refused (placeholder roots do not exist on the box, so the root itself could not be resolved). Fixed in rc.5 (roots resolved through their nearest existing ancestor; no symlink resolve for the placeholder check), red-proved. Its 30 events were dropped by the old hub — no false mail.
• 9201 proof (rc.5, hub 0.157.0): a drill catalog = live + privileged: true on privatebin → refused; the compose sha 79e7877d… unchanged; the same privatebin container (494270e4…, started 14:34:24Z) still running; the event at the hub at 17:10:34 and an operator mail; the page line on privatebin, not on opengist (control). Restored: live catalog, controller.yaml byte-identical, the drill mirror removed, the drill repo back to the live tree.
• Every installed app on the three test boxes still passes: 0 refusals after 0.328.0 started (P5/box-walk.txt).

3. Break-glass rotation (Part 3, R-954 — closed)

host	how	proof
demo-hp	installer step 4b ALONE (installer-v1.33.0's own function)	new value → PVE API 200; old → 401; values differ
Tester 1	the same	the same
demo-felhom	the same, on your answer (random)	the same; hub set_at now 2026-10-17T15:10:04Z
Tester 2	offline since 2026-10-04	R-1139

• Why not the full installer re-run (the brief's --rotate-recovery): it also reinstalls the vouched agent and the root files — on demo-hp it would have moved the candidate agent back. R-1141 asks for a standalone mode.
• Found and fixed: the hub kept the FIRST vault's set_at (demo-hp/Tester 1 still read their old dates, rotated before the fix; demo-felhom shows the new one) — hub v0.157.0.
• The passwords were read with the global-key curl (P3/cred.py, prints only length and time): that path writes NO customer-visible „revealed" line. Nothing was printed or filed.
• Log retention: ingress logs cover ≈ 13 h (k3s 10 MiB × 5). Not built: it needs a DooPlex or hub-storage change → R-1140 (cost in the row).

4. Off-site history (Part 4, R-895 — open, READY)

• Measurement: the login the hub already holds (the sealed sub-account password, write-capable) lists the folder; upload times are shown (2026-09-01 evidence on demo-hp's own sub-account). A fresh listing from DooPlex was NOT made: tried the CC store (no Storage Box credential), the box's key (restic REST only, names and sizes, and it is the box's key), the hub (no endpoint); reading the sealed password out of the hub was not done.
• Design: P4/design-R-895.md — before/after counts from the hub's own listing, files no box run explains, snapshots gone outside a window; a read-only sub-account as the stricter alternative.
• Built (slice 1, ~1 h, no new credential): hub v0.157.0 lists snapshots/ in the 07:10 key audit and logs the count, newest and oldest time. Deployed on your yes (17:03, Synced/Healthy, reports from demo-hp and Tester 1, floor read 0.328.0).

5. Versions, commits, fixtures

• felhom-agent: e15d1d3 (code), v0.159.0 tag, 4a1c07b-line (prune), v0.159.1 tag, CHANGELOG …. CI green (job 2025).
• felhom-controller: 6d90374, 24d937d, 36c4aa4, 09ac87a, 99f9117 → v0.328.0 (rc.1–rc.5 tested on 9201), MinAgent 0.158.1. Global floor 0.328.0 / 0.158.1 at ~17:22 local; per-box floors set and cleared. CI green (job 2032). i18n fixture: new app_settings_definition_refused; every other fixture byte-identical.
• app-catalog-felhom.eu: d5a443d (gate) + three follow-ups; CI green (job 2033).
• felhom.eu: 8f97b800 (hub v0.157.0 code, installer uninstall line, runbook), d968f849 (hub deploy); docs and register in the final commit.

6. Rows

Before 174, after 175. Closed (4): R-971, R-954, R-1124, R-1012. Opened (5): R-1139, R-1140, R-1141, R-1142, R-1143. Updated: R-895 (READY with the design), R-972.

7. Observations

1. Agent v0.159.0's signed decommission left a guest-bind record → a warning every 20 s. NOT-A-FINDING: fixed in the session (agent v0.159.1, the size rule), red-proved.
2. Hub kept the first vault's set_at. NOT-A-FINDING: fixed in the session (hub v0.157.0, the size rule), red-proved.
3. A full installer re-run is the only --rotate-recovery path. FILED: R-1141.
4. Ingress access logs ≈ 13 h. FILED: R-1140.
5. One app can bind another app's appdata/<name>. FILED: R-1142.
6. Controller build.sh pushes :latest on every build, rc included. FILED: R-1143.
7. The first allow-list rc refused every not-installed app on the box. NOT-A-FINDING: found by this session's own live proof on 9201 and fixed before the release (rc.5, red-proved).
8. homepage keeps a read-only Docker socket on the list; a read-only mount does not limit the Docker API. NOT-A-FINDING: already R-782's subject (homepage's exposure); the list keeps today's behaviour, it does not bless it.
9. Tester 1's ISO-era --rotate-recovery warning text („the old one now fails at the PVE web GUI") is printed by the step itself. NOT-A-FINDING: true and intended.

8. Teardown (three layers)

• Machine: demo-hp — scratch VG/pool/loop removed, test image files removed, test drive folders removed; 9201 back on the live catalog (controller.yaml byte-identical), drill mirror removed. Tester 1, demo-felhom: only the releases and the password rotation.
• Host: demo-hp, Tester 1, demo-felhom run controller 0.328.0; demo-hp, Tester 1 and demo-felhom run agent 0.159.1 (bundle 0.159.1).
• Hub: hub 0.157.0; global floor 0.328.0; no per-box floor; signed jobs consumed. Instruction files: none edited. System poster: no fact changed (none of the three).