v0.5.0-rc1: slice 5 Phase A — storage observe/report + watchdog (read-only, live)
Fill the slice-3 storage_targets stub and add the fast-poll storage watchdog. Read-only this phase; the host-root surface (mounts/SMART/grow/destructive gate) is Phase B. Hub-owned desired manifest is slice 10, so reconcile against it is built-but-unfed. - internal/storage: StorageTarget wire contract, durable_id derivation per type, HostReader seam (procfs/sysfs, root-free), Observer (storage_targets from ListStorage/NodeStorage + host reads, lvmthin thin-pool fill), and the watchdog (third daemon goroutine; debounced out-of-band report on a known target's attach/disconnect transition). - proxmox.Storage: additive parse-only config fields (durable_id sources). - collector StorageObserver seam; Loop.SetTrigger out-of-band report; daemon runs the watchdog as a third goroutine; StorageConfig knobs. - cross-repo golden kept byte-identical with felhom.eu/hub; bidirectional key-set test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,95 +1,81 @@
|
||||
# REPORT — Slice 4: reconcile engine + the reversibility gate (v0.4.0) (2026-06-08)
|
||||
# REPORT — Slice 5 Phase A: storage observe + report + watchdog (v0.5.0-rc1) (2026-06-09)
|
||||
|
||||
> Overwrite-latest report (most recent significant work only). Cumulative history lives in [CHANGELOG.md](CHANGELOG.md).
|
||||
|
||||
## Outcome
|
||||
|
||||
**Slice 4 is complete and pushed as `v0.4.0`.** Both phases landed:
|
||||
**Slice 5 Phase A is complete and pushed as `v0.5.0-rc1`** — the read-only, live half of the
|
||||
storage slice (doc 03 §7). The agent now observes every host storage target, reports it into
|
||||
the host-report (the slice-3 `storage_targets` stub is filled), and runs a fast-poll storage
|
||||
watchdog that pushes a disconnect to the hub out-of-band in seconds. **No host-root writes**
|
||||
this phase — mounts/SMART/disk-grow/destructive-gate are Phase B, which earns isolated review
|
||||
of the new privileged surface. This is the hard checkpoint: Phase A awaits validation before
|
||||
Phase B begins.
|
||||
|
||||
- **Phase A** (structural, pushed earlier as `v0.4.0-rc1`): the reconcile engine, the
|
||||
per-guest serializer (doc 03 §10), the desired-state model + provider seam, the
|
||||
field-normalization layer, the plan/diff engine, and the durable op journal +
|
||||
idempotency store. Runs **live but unfed** — `EmptyProvider` → zero mutations until
|
||||
slice 10 serves desired state.
|
||||
- **Phase B** (this push, the security core): the benign/destructive **classifier**,
|
||||
the **reversibility gate**, and the **signed-op consuming layer** over `internal/authz`
|
||||
— with role-scoping, op-to-action binding, idempotency/journaling, audit, and the
|
||||
crash-recovery consumer. The gate sits in front of the per-guest queue's executor, so
|
||||
**every mutation passes it**.
|
||||
## What landed
|
||||
|
||||
The whole module is **race-clean and vet-clean** on the Linux build server; 62 reconcile
|
||||
tests pass (the adversarial matrix runs against the real `authz.Verifier`).
|
||||
New package **`internal/storage`**:
|
||||
|
||||
## The security model (Phase B)
|
||||
- **`StorageTarget` wire contract** (`internal/hub/report.go`) — the slice-3 `struct{}` stub
|
||||
is now the full reported shape: identity (`name`/`type`/`durable_id`), `state`
|
||||
(`attached`/`disconnected`/`decommissioned`) + `reachable`, usage (`total`/`used`/`avail`/
|
||||
`used_fraction`), `content`, `mount_path`/`backing_device`, a `class_hint` (a rotational
|
||||
HINT — **never** authoritative; class is hub-owned), `role` (empty until slice 10), a
|
||||
`thin_pool` sub-object (lvmthin DATA fill), and a `smart` sub-object (`UNKNOWN` until Phase B).
|
||||
- **`durable_id` derivation** — deterministic per type, the DR-load-bearing re-attach key:
|
||||
fs-UUID (usb/local-dir), `server:export` (nfs/cifs), `repo+fingerprint` (pbs), `vg/pool`
|
||||
(lvmthin). Never empty — falls back to a stable store id so the hub's re-attach lookup
|
||||
always has something.
|
||||
- **`Observer`** — joins `ListStorage` (config) + `NodeStorage` (usage/active) with
|
||||
non-privileged host reads (`/proc/mounts`, `/dev/disk/by-uuid`, `/sys/.../rotational` +
|
||||
`removable`) behind a `HostReader` seam. Surfaces the lvmthin **thin-pool data fill**
|
||||
prominently (a full pool corrupts every guest on it) and warns at ≥85%.
|
||||
- **Storage watchdog** — the third daemon goroutine (alongside the hub loop + reconcile
|
||||
engine). Fast-polls (default 8s) the *known* target set for `attached↔disconnected`
|
||||
transitions and fires a **debounced** (default 30s) out-of-band host-report. Flags only a
|
||||
*known* target's change (never a never-attached device), coalesces flaps (leading +
|
||||
trailing edge). `CachingKnownTargets` rate-limits the Proxmox-derived known set;
|
||||
`HostLiveness` does device/mount-presence (local) + a reachability dial (network).
|
||||
|
||||
Hub-supplied intent is no longer trusted for destructive change — **by provenance +
|
||||
data-bearing-ness, not by verb** (doc 03 §4):
|
||||
Wiring + supporting changes:
|
||||
|
||||
- **Benign** (unsigned): start/stop/restart/create, and destroying a resource the agent
|
||||
created in the **same journaled transaction** (compensating rollback) or **tagged
|
||||
scratch**. That scratch/same-txn provenance is **agent-internal, journal-recorded, and
|
||||
never accepted from the hub** — a compromised hub cannot relabel a data-bearing guest
|
||||
as scratch to walk the gate.
|
||||
- **Destructive** (signature required): destroy/overwrite of the only/primary copy of
|
||||
customer data — **regardless of whether it arrives as a job or a desired-state delta**.
|
||||
Absent/invalid signature → refused **`pending_signature`**, never executed.
|
||||
- `proxmox.Storage` gained additive parse-only config fields (server/export/share/datastore/
|
||||
fingerprint/vgname/thinpool) — the durable_id sources. The API/root fence is untouched.
|
||||
- The collector gained a `StorageObserver` seam (hub does **not** import storage); `Loop`
|
||||
gained `SetTrigger` for the out-of-band report; the daemon runs the watchdog as a third
|
||||
goroutine; `StorageConfig` exposes the watchdog knobs.
|
||||
- **Hub** (`felhom.eu/hub`): `hostReportPayload` parses `storage_targets`, persists them via
|
||||
`report_json`, counts/warns on disconnected targets, and carries its half of the
|
||||
bidirectional golden key-set test. Both repos' goldens are byte-identical.
|
||||
|
||||
The signed-op consuming layer calls `authz.Verifier.Verify` (the locked
|
||||
namespace→allow-list→crypto→target→time→nonce pipeline, untouched) and then enforces
|
||||
the slice-4 policy on the `VerifiedOp`: **role-scoping** (recovery key = key-rotation
|
||||
only; operational key = ordinary destructive + planned rotation, doc 04 §4) and
|
||||
**op-to-action binding** (the verified op + host + guest + params must name the exact
|
||||
gated action). Idempotency keys the journal by the op nonce; every decision is audited
|
||||
(a signal, never the guard).
|
||||
## Tests
|
||||
|
||||
## Inert by design (slice-4 scope)
|
||||
`go test ./...` is green across both repos. New tests: observer build (incl. lvmthin
|
||||
thin-pool fill, USB-unplugged→disconnected, Proxmox-error fatal, mount-read-failure
|
||||
degrade), `durable_id` per-type table, watchdog transitions + debounce coalescing +
|
||||
never-attached suppression + caching TTL + stale-on-error + HostLiveness mount/network,
|
||||
collector seam (populate + error-degrade), loop out-of-band trigger, and the cross-repo
|
||||
golden contract on both sides.
|
||||
|
||||
There is **no live destructive execution** this slice: nothing serves destructive deltas
|
||||
until slice 10, and the guest-destroy/storage-wipe/restore-overwrite executors land in
|
||||
6/7. So the destructive path is fully **classified, gated, and adversarially tested**,
|
||||
but `RunSignedJob`'s executor is nil in production — an authorized destructive op is
|
||||
journaled as authorized-but-not-executed. Reconcile itself only produces the benign
|
||||
Start/Stop/SetConfig set, all allowed through the gate unsigned.
|
||||
**`go test -race`** must be run on the build server (192.168.0.180, has cgo/gcc); the local
|
||||
Windows toolchain lacks gcc. To run after pull: `CGO_ENABLED=1 go test -race ./...`.
|
||||
|
||||
## Adversarial proof (each case independently rejected)
|
||||
## Out of scope (deferred)
|
||||
|
||||
Run against the **real** `authz.Verifier` with in-test-minted SSHSIGs (the ~40-line
|
||||
framing is replicated in reconcile's test binary — production `authz` is untouched and
|
||||
gains no signing capability; live minting is required because the verifier's clock is
|
||||
not cross-package injectable):
|
||||
- **Phase B** (next, after this checkpoint validates): the host-root surface — systemd
|
||||
`.mount` units + sudoers allowlist behind a `HostOps` seam, benign re-mount-by-UUID,
|
||||
SMART (SATA + NVMe), the disk-grow executor (`pct resize`, grow-only), and routing
|
||||
destructive storage ops (detach/wipe/resize-shrink) through the slice-4 gate. Built +
|
||||
tested but inert live until slice 10 serves destructive deltas.
|
||||
- Reconcile against the hub manifest (attach new targets, enforce class/role/policy,
|
||||
`decommissioned`) → slice 10. Backup/restore orchestration → slice 6.
|
||||
|
||||
unsigned destructive **job** → pending_signature · unsigned destructive **desired-state
|
||||
delta** → pending_signature (distrusts hub desired state, not just jobs) · forged /
|
||||
unknown signer → `ErrUnknownSigner` · expired → `ErrExpired` · **replayed nonce across an
|
||||
agent restart** (durable `FileNonceStore`) → `ErrReplay` · wrong host → `ErrTarget` ·
|
||||
wrong guest / wrong op / wrong params → binding_mismatch · **recovery key on ordinary
|
||||
destructive** → role_denied · **hub-supplied "scratch" tag** on a data-bearing guest →
|
||||
ignored, still destructive → refused · **valid + correct role + correct target + fresh
|
||||
nonce → accepted**, and a second presentation → `ErrReplay`.
|
||||
## Validation notes for the reviewer
|
||||
|
||||
## The two forward-looking notes
|
||||
|
||||
- **Note 1 (carried in)** — the `InFlight()` **resume-or-rollback** startup consumer
|
||||
(`Engine.Recover`) landed **together with** the signed-op executor, as required. An op
|
||||
that crashed after the Proxmox POST but before its terminal record (`OpTaskRunning`,
|
||||
nonce already consumed) is not covered by idempotency dedupe — only this consumer
|
||||
resolves it (re-read the task via the new `TaskStatusOnce`, record the real outcome; a
|
||||
no-task-id op is abandoned fail-safe). Wired into daemon startup and tested.
|
||||
- **Note 2 (addressed)** — the memory comparison is canonicalized (`desiredMemoryMiB`):
|
||||
desired and actual compare in the same MiB unit that is then written, so a
|
||||
non-MiB-aligned `MemoryBytes` converges in one pass rather than re-issuing SetConfig
|
||||
every cycle. A test proves convergence. Recommendation stands that slice 10 serve
|
||||
MiB-aligned specs at the source.
|
||||
|
||||
## Verification
|
||||
|
||||
- `go test -race -count=1 ./...` and `go vet ./...` clean on the Linux build server
|
||||
(go1.26); all tests green locally and there.
|
||||
- No live Proxmox needed — Phase A is unfed and Phase B's destructive path is inert this
|
||||
slice. The gate's crypto path is proven end-to-end against the real verifier.
|
||||
|
||||
## Conventions
|
||||
|
||||
Version → **v0.4.0**. CHANGELOG has a per-phase entry (newest on top). No secrets in any
|
||||
committed file. Pushed to `main`. Per the task, I stop at this checkpoint and await the
|
||||
validation pass.
|
||||
- The reachability heuristic for dir storages: a Felhom usb/local-dir target is realized as
|
||||
its **own** mountpoint, so reachability = it is currently an exact mount + its device node
|
||||
exists (we deliberately do not fall through to PVE's `active` flag, which reads stale-
|
||||
attached because the mountpoint directory survives on the root fs after an unplug). Builtin
|
||||
`local` and network/block targets use the `active` flag. Worth confirming against the demo
|
||||
host's actual storage set during live validation.
|
||||
- A live `--selftest=hub` against the demo host will print the populated `storage_targets`.
|
||||
|
||||
Reference in New Issue
Block a user