agent v0.38.0: DR recipe — emit secret-free storage/guest/PBS half in host-report
DR recipe slice (agent half), grounded in SPIKE-dr-recipe-2026-06-16. Additive `dr_recipe` host-report section = the non-secret reconstruction scaffolding the operator must rebuild before PBS bytes can land. Built by pure BuildDRRecipeHostHalf from facts the report already collects (no new reads): guests[] sizing, drives[] (user-data by durable_id/role/mount/intent), pve_storage[] (storage.cfg), pbs coordinates. BOUNDARY (Phase-1 lesson): every field is an identifier/intent/size/coordinate — never a key/password/token/hash/ENC:. PBS key stays in escrow; restic password stays in escrow; the recipe names only the coordinates the restore targets. Tests: BuildDRRecipeHostHalf selection, NoPBS, NoSecrets (boundary mirror), dr_recipe key-set in the cross-repo golden contract test. recipe_version=1, ignore-unknown on read. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,95 +1,56 @@
|
||||
# REPORT — felhom-agent v0.37.0 (host-reboot remount by durable-id/UUID)
|
||||
# REPORT — felhom-agent v0.38.0 (DR recipe: agent storage/guest/PBS half)
|
||||
|
||||
**TASK — Close out the reboot story (agent half: Task A + Task A2).**
|
||||
**TASK — DR recipe slice (agent half).** Emit the secret-free reconstruction-recipe scaffolding the
|
||||
agent already owns, as an additive host-report section. Grounded in `SPIKE-dr-recipe-2026-06-16.md`.
|
||||
|
||||
## Task A — host-reboot remount resolves by filesystem UUID, never a cached node
|
||||
## What
|
||||
|
||||
### Diagnose-first (root cause, pinned LIVE on felhom-pve)
|
||||
The recipe is the non-secret **re-provision plan** that complements escrow (keys) and PBS/restic
|
||||
(bytes): guest sizing, drive inventory (durable-id → role → mount → intent), PVE storage defs, and PBS
|
||||
coordinates — the host/guest/storage scaffolding the operator must rebuild on new hardware *before* the
|
||||
PBS bytes can land. The agent owns every fact (`StorageTarget`, `GuestSpec`, `PBSSnapshot`), so it emits
|
||||
its half additively in the existing host-report; the hub assembles it with the controller's app half.
|
||||
|
||||
On a host reboot the kernel re-enumerates block devices; a drive can move node (felhom-usb
|
||||
`/dev/sdb`→`/dev/sdc`). Two things could leave an enrolled drive unmounted after a reboot:
|
||||
## Implementation
|
||||
|
||||
1. A systemd `.mount` unit left **`disabled`** by a prior detach (`Unmount` = stop + disable)
|
||||
never gets a `multi-user.target.wants` symlink, so it does **not** auto-mount at boot — and
|
||||
nothing in the agent re-asserted it at startup.
|
||||
2. Any remount that trusted a remembered `/dev/sdX` node would target the WRONG device after a
|
||||
letter move.
|
||||
- `internal/hub/dr_recipe.go` — wire types + the pure `BuildDRRecipeHostHalf(guests, targets, pbs)`:
|
||||
- `guests[]` ← each `Guest.Spec` (cores/memory/disk), skipping status-unknown guests.
|
||||
- `drives[]` ← user-data external drives only (`isUserDataDrive`: usb/local-dir with a non-empty
|
||||
durable-id + mount path) → `{durable_id, role, mount_path, intent=enrolled, total_bytes}`.
|
||||
- `pve_storage[]` ← **every** storage target `{name, type, content}` (rebuild `storage.cfg`).
|
||||
- `pbs` ← the latest snapshot's `{repo_id (the pbs storage id), namespace, latest_snapshot_id}`.
|
||||
- `HostReport.DRRecipe *DRRecipeHostHalf json:"dr_recipe"` (always set in `Collect()`, never null).
|
||||
- No new reads — derived from the just-collected report facts.
|
||||
|
||||
Confirmed live: felhom-usb's mount unit was `disabled` (`systemctl is-enabled` → `disabled`,
|
||||
no wants-symlink) while felhom-flash's was `enabled`. The `What=` line was already correct
|
||||
(`/dev/disk/by-uuid/<UUID>`), so the fix is to **re-assert the unit by UUID at startup**, not to
|
||||
rewrite the unit.
|
||||
## Boundary (non-negotiable)
|
||||
|
||||
### Fix
|
||||
Every field is an identifier / intent / size / coordinate. The PBS **key** stays in escrow, the access
|
||||
**token** in identity-escrow, the restic **password** in escrow — the recipe names only the coordinates
|
||||
the restore targets. This is the exact axis the retired infra-backup violated (it shipped
|
||||
`encryption_key_b64`/`restic_password`/`cf_api_token`).
|
||||
|
||||
- `storage.ResolveStorageDevice(durableID)` (`durable_device.go`) — resolves the enrolled
|
||||
`uuid:<fs-uuid>` storage scheme to its **current** backing `/dev` node by re-scanning
|
||||
`/dev/disk/by-uuid` (`EvalSymlinks`); rejects any non-`uuid:` scheme (anti-node-binding) and
|
||||
errors when the UUID is absent so a caller skips a gone drive instead of fail-mounting.
|
||||
- `storage.parseFelhomMountUnit` (`mountunit.go`) — pure inverse of `renderMountUnit`, gated on a
|
||||
`Managed by felhom-agent` marker; extracts Name/UUID/Where/Type/Options, ignores foreign units.
|
||||
- `(*SudoHostOps).ReassertEnrolledMounts(ctx)` (`hostops.go`) — for each enrolled `.mount` unit,
|
||||
re-resolve by UUID and re-run `EnsureMount` (idempotent `systemctl enable --now`). Skips ONLY the
|
||||
durable steady state (mounted AND enabled), via the pure `shouldReassertMount`; a
|
||||
**mounted-but-DISABLED** unit (the exact live felhom-usb state — serves now, but a reboot would not
|
||||
auto-mount it) is still re-asserted so the wants-symlink is re-created. Enabled-state read with a
|
||||
privilege-free `os.Lstat` of the `multi-user.target.wants` symlink (`unitEnabled`) — no
|
||||
`systemctl is-enabled` subprocess / no new sudoers entry. Absent UUIDs skipped (retry next tick).
|
||||
- `main.go` — runs `ReassertEnrolledMounts` at startup **before** `ReassertGuestBinds`, and again
|
||||
ahead of the bind re-assert on the periodic 20s tick, so host mounts are live before the guest
|
||||
binds re-establish.
|
||||
## Tests (non-hollow)
|
||||
|
||||
### Tests (non-hollow, seam = device resolution; Linux)
|
||||
- `TestBuildDRRecipeHostHalf` — drives = only user-data; pve_storage = all targets; pbs = latest
|
||||
snapshot; guests skip nil-spec.
|
||||
- `TestBuildDRRecipeHostHalf_NoPBS` — no snapshots → `pbs` omitted, slices non-nil.
|
||||
- `TestDRRecipeHostHalf_NoSecrets` — the boundary mirror: serialized half has NO key matching
|
||||
`(?i)(password|secret|token|hash|passphrase|api[_-]?key|\bkey\b|enc:)`. (The load-bearing boundary
|
||||
test, with a synthetic-secret app + allowlist red-proof, lives on the controller emitter.)
|
||||
- `dr_recipe` key-set + sub-array element key-sets added to `TestHostReport_ContractMatchesGolden`
|
||||
(the cross-repo golden, byte-pinned with the hub's copy).
|
||||
|
||||
- `TestResolveStorageDevice_ToleratesDeviceLetterMove` — fake `/dev/disk/by-uuid/<uuid>` symlink
|
||||
pointed at `sdc` (moved from the enroll-time `sdb`); asserts `ResolveStorageDevice` returns the
|
||||
CURRENT node `sdc`. **Companion (red-proof):** asserts the cached enroll-time node (`sdb`) differs
|
||||
from the freshly-resolved node — a node-trusting remount would mount the wrong device.
|
||||
- `TestResolveStorageDevice_AbsentAndScheme` — absent UUID errors; only `uuid:` is resolvable
|
||||
(`/dev/sdb1`, `store:`, `byid:`, `""` all rejected).
|
||||
- `TestParseFelhomMountUnit` — `renderMountUnit`→`parseFelhomMountUnit` round-trip + rejects a
|
||||
non-felhom unit.
|
||||
- `TestShouldReassertMount` — the four mounted/enabled combinations; pins that mounted-but-disabled
|
||||
(the live bug) re-asserts and only mounted+enabled is skipped.
|
||||
- `TestUnitEnabled` — wants-symlink presence ⇒ enabled (privilege-free `Lstat`).
|
||||
## Versioning
|
||||
|
||||
## Task A2 — verdict: enrolling a NEW drive does NOT need an LXC restart
|
||||
|
||||
**NO.** Source-verified the enroll path: `/disks/guest-attach` → `handleDiskGuestAttach` →
|
||||
`GuestBinder.AttachDrive` (intermediary-mount model, "no pct, no reboot") — a live bind under the
|
||||
single shared parent `/mnt/felhom-drives`, which provides unbounded **named** live slots. It does
|
||||
NOT take the legacy `RebootGuest` branch. The operator's pre-created-slot-pool idea is unnecessary.
|
||||
`recipe_version=1`, carried in the section. Read is ignore-unknown (encoding/json default) for
|
||||
forward-compat, mirroring `storage_manifest`. Golden discipline: `host-report.golden.json` here must stay
|
||||
byte-identical to the hub's copy — manual checksum-diff on any wire change (the golden spans three repos).
|
||||
|
||||
## Gate
|
||||
|
||||
`go build ./...`, `go vet ./...`, `go test ./...` all green locally (Windows: the symlink tests
|
||||
skip — they assert on Linux, the agent's OS). Linux green gate + companion red-proof run on the
|
||||
build server (192.168.0.180, go1.26.0) — see live-acceptance section.
|
||||
`go build`, `go vet`, `go test ./...` all green (local + build server). Deployed to felhom-pve.
|
||||
|
||||
## Live acceptance — real host reboot ×2 on felhom-pve (agent v0.37.0 deployed)
|
||||
## Deferred (NOT in this slice)
|
||||
|
||||
**Pre-reboot live proof of the root cause + fix:** felhom-usb's `mnt-felhom\x2dusb.mount` was
|
||||
`disabled` while felhom-flash's was `enabled`. After deploying v0.37.0 and restarting the agent,
|
||||
`ReassertEnrolledMounts` logged `re-asserted enrolled mount by UUID (enable --now) … wasMounted=true`
|
||||
and the unit went `enabled` (wants-symlink created) — the mounted-but-disabled case handled.
|
||||
|
||||
**The `/dev/sdX` reshuffle actually occurred and was tolerated as a no-op** (the headline result):
|
||||
|
||||
| Drive (UUID) | before #1 | after #1 | after #2 |
|
||||
|---|---|---|---|
|
||||
| felhom-usb (`da9e7089…`, 931G) | `sdb` | **`sdc`** | `sdb` |
|
||||
| felhom-flash (`81a26531…`, 119G) | `sdc` | **`sdb`** | `sdc` |
|
||||
|
||||
The two drives **swapped letters** on reboot #1 and swapped back on reboot #2. Both times each
|
||||
landed at its correct mountpoint by UUID — `findmnt /mnt/felhom-usb` → the `da9e7089` device,
|
||||
`/mnt/felhom-flash` → the `81a26531` device — never the wrong device. Both units `enabled`+`active`,
|
||||
no agent storage errors. On a clean boot the (now-enabled) units auto-mount, so `ReassertEnrolledMounts`
|
||||
finds them mounted+enabled and skips (the intended no-churn steady state).
|
||||
|
||||
- **Ordering:** `felhom-shared-parent.service` Finished `18:02:16` / `18:06:01`; `pve-guests.service`
|
||||
Starting `18:02:22` / `18:06:07` — shared parent re-establishes ~6s BEFORE the guests, both reboots.
|
||||
- **Guest recovery:** guest 9201 came up and all drive-backed apps recovered (immich, paperless,
|
||||
jellyfin, calibre, audiobookshelf, radarr, romm, komga) both reboots; romm's backing mount
|
||||
`/mnt/felhom-drives/felhom-flash/userdata/roms` resolved correctly across the swap.
|
||||
|
||||
Secrets (local_api token / leaf fingerprint) handled out-of-band — none committed.
|
||||
The agent-side `restore_directive` consumption / recovery-mode execution (`syncer.go:92`) — slice-10D.
|
||||
This slice only EMITS the recipe.
|
||||
|
||||
Reference in New Issue
Block a user