agent v0.38.0: DR recipe — emit secret-free storage/guest/PBS half in host-report

DR recipe slice (agent half), grounded in SPIKE-dr-recipe-2026-06-16. Additive
`dr_recipe` host-report section = the non-secret reconstruction scaffolding the
operator must rebuild before PBS bytes can land. Built by pure
BuildDRRecipeHostHalf from facts the report already collects (no new reads):
guests[] sizing, drives[] (user-data by durable_id/role/mount/intent),
pve_storage[] (storage.cfg), pbs coordinates.

BOUNDARY (Phase-1 lesson): every field is an identifier/intent/size/coordinate —
never a key/password/token/hash/ENC:. PBS key stays in escrow; restic password
stays in escrow; the recipe names only the coordinates the restore targets.

Tests: BuildDRRecipeHostHalf selection, NoPBS, NoSecrets (boundary mirror),
dr_recipe key-set in the cross-repo golden contract test. recipe_version=1,
ignore-unknown on read.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-16 18:32:15 +02:00
parent c8d7ec6bb5
commit 8abc1b8852
11 changed files with 396 additions and 82 deletions
+40 -79
View File
@@ -1,95 +1,56 @@
# REPORT — felhom-agent v0.37.0 (host-reboot remount by durable-id/UUID)
# REPORT — felhom-agent v0.38.0 (DR recipe: agent storage/guest/PBS half)
**TASK — Close out the reboot story (agent half: Task A + Task A2).**
**TASK — DR recipe slice (agent half).** Emit the secret-free reconstruction-recipe scaffolding the
agent already owns, as an additive host-report section. Grounded in `SPIKE-dr-recipe-2026-06-16.md`.
## Task A — host-reboot remount resolves by filesystem UUID, never a cached node
## What
### Diagnose-first (root cause, pinned LIVE on felhom-pve)
The recipe is the non-secret **re-provision plan** that complements escrow (keys) and PBS/restic
(bytes): guest sizing, drive inventory (durable-id → role → mount → intent), PVE storage defs, and PBS
coordinates — the host/guest/storage scaffolding the operator must rebuild on new hardware *before* the
PBS bytes can land. The agent owns every fact (`StorageTarget`, `GuestSpec`, `PBSSnapshot`), so it emits
its half additively in the existing host-report; the hub assembles it with the controller's app half.
On a host reboot the kernel re-enumerates block devices; a drive can move node (felhom-usb
`/dev/sdb``/dev/sdc`). Two things could leave an enrolled drive unmounted after a reboot:
## Implementation
1. A systemd `.mount` unit left **`disabled`** by a prior detach (`Unmount` = stop + disable)
never gets a `multi-user.target.wants` symlink, so it does **not** auto-mount at boot — and
nothing in the agent re-asserted it at startup.
2. Any remount that trusted a remembered `/dev/sdX` node would target the WRONG device after a
letter move.
- `internal/hub/dr_recipe.go` — wire types + the pure `BuildDRRecipeHostHalf(guests, targets, pbs)`:
- `guests[]` ← each `Guest.Spec` (cores/memory/disk), skipping status-unknown guests.
- `drives[]` ← user-data external drives only (`isUserDataDrive`: usb/local-dir with a non-empty
durable-id + mount path) → `{durable_id, role, mount_path, intent=enrolled, total_bytes}`.
- `pve_storage[]`**every** storage target `{name, type, content}` (rebuild `storage.cfg`).
- `pbs` ← the latest snapshot's `{repo_id (the pbs storage id), namespace, latest_snapshot_id}`.
- `HostReport.DRRecipe *DRRecipeHostHalf json:"dr_recipe"` (always set in `Collect()`, never null).
- No new reads — derived from the just-collected report facts.
Confirmed live: felhom-usb's mount unit was `disabled` (`systemctl is-enabled``disabled`,
no wants-symlink) while felhom-flash's was `enabled`. The `What=` line was already correct
(`/dev/disk/by-uuid/<UUID>`), so the fix is to **re-assert the unit by UUID at startup**, not to
rewrite the unit.
## Boundary (non-negotiable)
### Fix
Every field is an identifier / intent / size / coordinate. The PBS **key** stays in escrow, the access
**token** in identity-escrow, the restic **password** in escrow — the recipe names only the coordinates
the restore targets. This is the exact axis the retired infra-backup violated (it shipped
`encryption_key_b64`/`restic_password`/`cf_api_token`).
- `storage.ResolveStorageDevice(durableID)` (`durable_device.go`) — resolves the enrolled
`uuid:<fs-uuid>` storage scheme to its **current** backing `/dev` node by re-scanning
`/dev/disk/by-uuid` (`EvalSymlinks`); rejects any non-`uuid:` scheme (anti-node-binding) and
errors when the UUID is absent so a caller skips a gone drive instead of fail-mounting.
- `storage.parseFelhomMountUnit` (`mountunit.go`) — pure inverse of `renderMountUnit`, gated on a
`Managed by felhom-agent` marker; extracts Name/UUID/Where/Type/Options, ignores foreign units.
- `(*SudoHostOps).ReassertEnrolledMounts(ctx)` (`hostops.go`) — for each enrolled `.mount` unit,
re-resolve by UUID and re-run `EnsureMount` (idempotent `systemctl enable --now`). Skips ONLY the
durable steady state (mounted AND enabled), via the pure `shouldReassertMount`; a
**mounted-but-DISABLED** unit (the exact live felhom-usb state — serves now, but a reboot would not
auto-mount it) is still re-asserted so the wants-symlink is re-created. Enabled-state read with a
privilege-free `os.Lstat` of the `multi-user.target.wants` symlink (`unitEnabled`) — no
`systemctl is-enabled` subprocess / no new sudoers entry. Absent UUIDs skipped (retry next tick).
- `main.go` — runs `ReassertEnrolledMounts` at startup **before** `ReassertGuestBinds`, and again
ahead of the bind re-assert on the periodic 20s tick, so host mounts are live before the guest
binds re-establish.
## Tests (non-hollow)
### Tests (non-hollow, seam = device resolution; Linux)
- `TestBuildDRRecipeHostHalf` — drives = only user-data; pve_storage = all targets; pbs = latest
snapshot; guests skip nil-spec.
- `TestBuildDRRecipeHostHalf_NoPBS` — no snapshots → `pbs` omitted, slices non-nil.
- `TestDRRecipeHostHalf_NoSecrets` — the boundary mirror: serialized half has NO key matching
`(?i)(password|secret|token|hash|passphrase|api[_-]?key|\bkey\b|enc:)`. (The load-bearing boundary
test, with a synthetic-secret app + allowlist red-proof, lives on the controller emitter.)
- `dr_recipe` key-set + sub-array element key-sets added to `TestHostReport_ContractMatchesGolden`
(the cross-repo golden, byte-pinned with the hub's copy).
- `TestResolveStorageDevice_ToleratesDeviceLetterMove` — fake `/dev/disk/by-uuid/<uuid>` symlink
pointed at `sdc` (moved from the enroll-time `sdb`); asserts `ResolveStorageDevice` returns the
CURRENT node `sdc`. **Companion (red-proof):** asserts the cached enroll-time node (`sdb`) differs
from the freshly-resolved node — a node-trusting remount would mount the wrong device.
- `TestResolveStorageDevice_AbsentAndScheme` — absent UUID errors; only `uuid:` is resolvable
(`/dev/sdb1`, `store:`, `byid:`, `""` all rejected).
- `TestParseFelhomMountUnit``renderMountUnit``parseFelhomMountUnit` round-trip + rejects a
non-felhom unit.
- `TestShouldReassertMount` — the four mounted/enabled combinations; pins that mounted-but-disabled
(the live bug) re-asserts and only mounted+enabled is skipped.
- `TestUnitEnabled` — wants-symlink presence ⇒ enabled (privilege-free `Lstat`).
## Versioning
## Task A2 — verdict: enrolling a NEW drive does NOT need an LXC restart
**NO.** Source-verified the enroll path: `/disks/guest-attach``handleDiskGuestAttach`
`GuestBinder.AttachDrive` (intermediary-mount model, "no pct, no reboot") — a live bind under the
single shared parent `/mnt/felhom-drives`, which provides unbounded **named** live slots. It does
NOT take the legacy `RebootGuest` branch. The operator's pre-created-slot-pool idea is unnecessary.
`recipe_version=1`, carried in the section. Read is ignore-unknown (encoding/json default) for
forward-compat, mirroring `storage_manifest`. Golden discipline: `host-report.golden.json` here must stay
byte-identical to the hub's copy — manual checksum-diff on any wire change (the golden spans three repos).
## Gate
`go build ./...`, `go vet ./...`, `go test ./...` all green locally (Windows: the symlink tests
skip — they assert on Linux, the agent's OS). Linux green gate + companion red-proof run on the
build server (192.168.0.180, go1.26.0) — see live-acceptance section.
`go build`, `go vet`, `go test ./...` all green (local + build server). Deployed to felhom-pve.
## Live acceptance — real host reboot ×2 on felhom-pve (agent v0.37.0 deployed)
## Deferred (NOT in this slice)
**Pre-reboot live proof of the root cause + fix:** felhom-usb's `mnt-felhom\x2dusb.mount` was
`disabled` while felhom-flash's was `enabled`. After deploying v0.37.0 and restarting the agent,
`ReassertEnrolledMounts` logged `re-asserted enrolled mount by UUID (enable --now) … wasMounted=true`
and the unit went `enabled` (wants-symlink created) — the mounted-but-disabled case handled.
**The `/dev/sdX` reshuffle actually occurred and was tolerated as a no-op** (the headline result):
| Drive (UUID) | before #1 | after #1 | after #2 |
|---|---|---|---|
| felhom-usb (`da9e7089…`, 931G) | `sdb` | **`sdc`** | `sdb` |
| felhom-flash (`81a26531…`, 119G) | `sdc` | **`sdb`** | `sdc` |
The two drives **swapped letters** on reboot #1 and swapped back on reboot #2. Both times each
landed at its correct mountpoint by UUID — `findmnt /mnt/felhom-usb` → the `da9e7089` device,
`/mnt/felhom-flash` → the `81a26531` device — never the wrong device. Both units `enabled`+`active`,
no agent storage errors. On a clean boot the (now-enabled) units auto-mount, so `ReassertEnrolledMounts`
finds them mounted+enabled and skips (the intended no-churn steady state).
- **Ordering:** `felhom-shared-parent.service` Finished `18:02:16` / `18:06:01`; `pve-guests.service`
Starting `18:02:22` / `18:06:07` — shared parent re-establishes ~6s BEFORE the guests, both reboots.
- **Guest recovery:** guest 9201 came up and all drive-backed apps recovered (immich, paperless,
jellyfin, calibre, audiobookshelf, radarr, romm, komga) both reboots; romm's backing mount
`/mnt/felhom-drives/felhom-flash/userdata/roms` resolved correctly across the swap.
Secrets (local_api token / leaf fingerprint) handled out-of-band — none committed.
The agent-side `restore_directive` consumption / recovery-mode execution (`syncer.go:92`) — slice-10D.
This slice only EMITS the recipe.