docs: REPORT — v0.60.0 proof-of-launch gating shipped, live-proven, published
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,69 +1,98 @@
|
||||
# REPORT — raw-drive first-class support (agent v0.55.0 → v0.59.0)
|
||||
# REPORT — F1 (HIGH) + F2: never destroy a guest this transaction didn't create (v0.60.0)
|
||||
|
||||
**Date:** 2026-07-01 · **Repo:** `felhom-agent` · **Class:** Risky/supervised (drive health/enroll path).
|
||||
|
||||
Surfaced by the first live raw-drive enrollment (controller Impl-2b): the agent's enroll + tracking
|
||||
machinery assumed every drive is a PVE storage (visible in `Observe()`). A raw (non-PVE-storage) drive
|
||||
enrolled via the new wizard was therefore mounted+bound but not intent-tracked, not health-tracked, and
|
||||
shown "disconnected". This slice makes raw drives first-class end-to-end.
|
||||
**Date:** 2026-07-02 · **Repo:** `felhom-agent` · **Class:** Risky/supervised (destructive
|
||||
rollback/destroy path) — strictly SAFER. Fixes the pool-effects campaign's F1/F2
|
||||
(`felhom.eu/documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md`); F3 is the companion doc in
|
||||
felhom.eu (`documentation/runbooks/provisioning.md`, commit `6dae9d7`).
|
||||
|
||||
## 1. Baseline → target
|
||||
|
||||
felhom-agent `91f6a26` (v0.55.0, Impl-2a) → **v0.58.0**. Companion: controller v0.95.0 (Impl-2b).
|
||||
agent **v0.59.0** (`55ade9e`) → **v0.60.0** (commit **`b9356d6`** on `main`). felhom.eu doc commit
|
||||
`6dae9d7` (no bump).
|
||||
|
||||
## 2. Changes (all `internal/localapi/disks.go` + wiring), newest first
|
||||
## 2. Root cause + what changed
|
||||
|
||||
- **v0.59.0** — the `/disks` registry row also reports `BackingDevice` (resolved to the real /dev node via
|
||||
`ResolveStorageDevice`) + capacity (`statfsCapacity` — build-tagged `syscall.Statfs`) — the agent-view
|
||||
showed "—" for the device and no size bar for a raw drive. Commits `5ff5f8e` + `39ec76d`.
|
||||
- **v0.58.0** — the `/disks` registry-union row (Impl-2a) now reports `GuestPath` (`StablePathForRaw`) +
|
||||
`BoundUnderParent` (`boundUnderParent`), like the Observe path — a registry-only drive read as
|
||||
"Leválasztva" despite being bound live. Commit `3825664`.
|
||||
- **v0.57.0** — `ReassertGuestBinds` builds its durable-id→mount map from `Observe()` only → a raw drive's
|
||||
guest-bind was never re-asserted (post-reboot/re-mount). Augment from the mount table (raw `/mnt/<name>`
|
||||
→ device fs-UUID via `HostReader`, skipping the `/mnt/felhom-drives` bind); Observe failure no longer
|
||||
aborts. **Plus a latent wiring fix:** `buildLocalAPIServer` never passed `Options.HostReader`, so
|
||||
`s.host` was nil in production — the v0.56.0 fallback + the role gate's host classification silently
|
||||
no-op'd; now wired to `NewProcHostReader()`. Commits `67e2144` + `6448b80`.
|
||||
- **v0.56.0** — `durableIDForMount` was `Observe()`-only, so a raw drive's enroll/eject **intent** +
|
||||
**guest-bind** went unrecorded ("durable-id unresolved"). Fall back to resolving the mount's device
|
||||
fs-UUID (`HostReader.Mounts`+`ResolveUUID`) → `uuid:<fs-uuid>` (same scheme as Observe). Commit `e593fb2`.
|
||||
The compensating destroy ran on ASSUMED provenance (`Rollback:true`/`SameTxnCreated`), never verified
|
||||
against proof-of-launch. A `RestoreLXC` that failed synchronously (no UPID — e.g. PVE refusing a
|
||||
pre-existing vmid the pool-blind ListLXC couldn't see) still armed `DestroyLXC(vmid)` — destroying a
|
||||
guest the transaction never created. The pool ACL's 403 only saved the non-pool subset; an in-pool
|
||||
pre-existing guest would have been destroyed, in-process AND by Recover. Now **a RestoreLXC UPID is the
|
||||
sole destroy authorization**, in all three destroy paths (the pool ACL is defense-in-depth again):
|
||||
|
||||
## 3. Green gate + tests
|
||||
- **F1a `internal/reconcile/bringup.go`** — `runBringUp`'s rollback defer gated on `launched` (set only
|
||||
after the restore POST is accepted); a no-launch failure closes the owning entry terminal-failed with
|
||||
NO destroy. Pre-restore `OpStarted` append kept (crash-safety). `rollbackBringUp` doc updated: only
|
||||
ever called launch-proven.
|
||||
- **F1b `internal/reconcile/restoretest.go`** — `runScratchTest`'s `teardownScratch` defer gated the
|
||||
same way.
|
||||
- **F1c `internal/reconcile/recover.go`** — the no-UPID "POST never confirmed → abandon" check now runs
|
||||
BEFORE the Scratch/Rollback dispatch: a no-UPID guest-creating entry is abandoned (marked failed, NO
|
||||
destroy) instead of destroy-by-vmid-existence. Recover safe by DESIGN, not by the pool-blind
|
||||
"already gone" accident.
|
||||
- **F2 `restoretest.go`** — `RunRestoreTest` band-advance: an "already exists" refusal (new
|
||||
`pveAlreadyExists`, APIError-typed like `pveConfigLock` — a real restore failure is never
|
||||
misclassified) advances to the next free band vmid (`pickScratchVMID` + exclude set, bounded by the
|
||||
band width); band exhausted → `Skipped` (the scheduler raises no record/alert), never FAIL.
|
||||
- **Accepted residual (rule 4, by design):** a crash in the one-statement window between obtaining the
|
||||
UPID and journaling it leaks a half-built guest Recover won't destroy — cleanable, preferable to
|
||||
destroying an innocent guest.
|
||||
|
||||
`go build ./...`, `go vet ./...`, `go test ./...` — clean at each version. New tests:
|
||||
`TestDurableIDForMount_RawFallback` (+ red-proof: Observe-only → ""). The `ReassertGuestBinds` +
|
||||
`/disks`-row changes are covered by the live end-to-end (§5).
|
||||
## 3. Green gate
|
||||
|
||||
## 4. Published + deployed
|
||||
`go build ./... && go vet ./... && go test ./...` — clean, 17 packages ok (build server go1.26.0 build
|
||||
of the deploy binary also clean).
|
||||
|
||||
Gitea `felhom-agent/0.58.0` (sha `5ea7f63bb1617fa8412c1977b57369cd184bc1d6c1e6d6cf4f723835ebc06078`);
|
||||
deployed to felhom-pve (backup `.bak-0.57.0`), active. (0.56/0.57 also published; rollback binaries kept.)
|
||||
## 4. Tests (incl. the red-proofs)
|
||||
|
||||
## 5. Live validation (felhom-pve + the real controller UI)
|
||||
- **F1a red-proof `TestRunBringUp_NoLaunchNoDestroy`** (PVE "already exists" APIError + plain
|
||||
synchronous error): asserts `DestroyLXC` NOT called + entry terminal. **Companion verified:** with
|
||||
the `launched` gate reverted it fails `destroys=[8000]` — the exact destroy-innocent-guest bug.
|
||||
- **F1b `TestRunRestoreTest_RestoreNoLaunchNoTeardown`** (replaces the old
|
||||
`…RestoreFailureStillTearsDown`, which asserted the BUGGY behavior): no teardown on a no-launch
|
||||
failure. Red-proof verified (`destroys=[990000]` pre-fix).
|
||||
- **No-regression (Scenario B):** `TestRunRestoreTest_LaunchedTaskFailureStillTearsDown` + a new
|
||||
"restore-task failure after launch" case in `TestRunBringUp_CompensatingRollback` — a LAUNCHED
|
||||
restore whose task fails still destroys the half-built guest.
|
||||
- **F1c `TestRecover_BringUpNoUPIDAbandoned` + `TestRecover_ScratchNoUPIDAbandoned`:** a no-UPID
|
||||
Rollback/Scratch entry with a guest PRESENT at the vmid → abandoned via the fail-safe path, no
|
||||
destroy. Red-proof verified. Existing leaked-guest Recover tests updated to carry the UPID (the
|
||||
launch proof) their scenario implies.
|
||||
- **F2 `TestRunRestoreTest_AdvancesPastOccupiedBandVMID`** (squatter at 990000 → PASS at 990001,
|
||||
squatter never destroyed) + **`…BandFullOfSquattersSkips`** (Skipped, not FAIL, bounded — one try per
|
||||
band vmid) + `TestPickScratchVMID` exclude-set coverage.
|
||||
|
||||
The raw `/dev/sdd` SD card, enrolled via the wizard (controller Impl-2b), ends up fully first-class:
|
||||
## 5. Live proofs (felhom-pve, v0.60.0 deployed)
|
||||
|
||||
| Property | Evidence |
|
||||
|----------|----------|
|
||||
| Mounted + bound | `/mnt/teszt_enroll` (raw) + `/mnt/felhom-drives/teszt_enroll` (bind) live. |
|
||||
| Intent recorded | `drive-intents.json` → `uuid:f2236136… = enrolled`; journal `drive intent recorded … durable_id=uuid:f2236136…` (no "unresolved" — the v0.56.0 fallback fired once HostReader was wired). |
|
||||
| Guest-bind recorded | `guest-binds.json` → `9201: […, uuid:f2236136…]`; journal `guest-bind recorded`. |
|
||||
| Health-tracked | watchdog (RegistryKnownTargets) detected a drop + self-healed (re-mounted). |
|
||||
| `/disks` status | row reports `state=attached`, `guest_path=/mnt/felhom-drives/teszt_enroll`, `bound_under_parent=true`. |
|
||||
| Controller UI | shows **"Aktív"** (USB), alongside felhom-usb + felhom-flash — all Active, no false detach. |
|
||||
- **Probe A (headline):** `--selftest=bring-up -vmid 9001` (existing non-pool guest) → PVE refused
|
||||
("CT 9001 already exists"); **no gate decision, no DestroyLXC attempt of any kind** (pre-fix run had
|
||||
`gate decision class=guest_destroy` + a 403'd DELETE); 9001 untouched; journal drained in-process
|
||||
(next run: `recover: examined=0`).
|
||||
- **Probe D (F2):** decoy parked at 990000 → restore-test logged "band VMID occupied … advancing",
|
||||
**passed at 990001** (restored+booted+verified+torn-down in 28s), decoy survived, decoy removed.
|
||||
- **Regression:** forced post-launch failure (`-cores 10000` → config PUT 400 after restore) →
|
||||
compensating rollback destroyed the half-built 9310 ("bring-up: rolled back"); a normal provision of
|
||||
9310 then passed (boot+running 22s) and tore down cleanly.
|
||||
- STOP condition (any destroy attempted against 9001 or the decoy) did not trigger.
|
||||
|
||||
## 6. NOT done / follow-ups
|
||||
## 6. Deployed + published
|
||||
|
||||
- The raw drive's `.mount` unit is named by its UUID (assign sets `MountSpec.Name = uuid`), so the
|
||||
agent-view Name is the UUID; the controller shows its own label ("teszt_enroll"). Cosmetic.
|
||||
- Init slow-format completion is a controller concern (`runStorageInit` should poll
|
||||
`/disks/format/status`) — see controller REPORT.
|
||||
- **Impl-3** — shared-box operator gate for self-serve format.
|
||||
- felhom-pve: `/usr/local/bin/felhom-agent` = **0.60.0** (backup `felhom-agent.bak-0.59.0`), unit
|
||||
`active`, clean logs. Demo left healthy: pool `felhom` = {9201}, guests 9001/9100/9201/9999 intact.
|
||||
- Gitea generic package `felhom-agent/0.60.0` published (HTTP 201), download sha256 round-trip verified:
|
||||
`45d19b8799cc764333cd95bb64e0515f2246cdb08c0fb36d9b23409051725831`.
|
||||
- **Operator follow-up:** bump the hub Day-0 artifact manifest (password-gated UI) to agent **0.60.0**
|
||||
— until then, fresh Day-0 installs still fetch the manifest-pinned older version.
|
||||
|
||||
## 7. Observations
|
||||
## 7. F3 doc
|
||||
|
||||
- Root theme: making a drive first-class without a PVE storage touches every seam that keyed off
|
||||
`Observe()` — candidates (Impl-2a), watchdog Known (Impl-2a), UUID resolution, intent + guest-bind
|
||||
recording, guest-bind re-assert, and the `/disks` status row. All now resolve raw mounts via
|
||||
`HostReader` (mount table + by-uuid), consistent with the `uuid:<fs-uuid>` scheme Observe derives.
|
||||
`felhom.eu/documentation/runbooks/provisioning.md` (commit `6dae9d7`): pick the provision `-vmid` from
|
||||
the host's FULL guest list (`pct list` + `qm list`), never the pool-only hub inventory; reserved bands
|
||||
documented (9999, 990000–990009 fleet-wide). Campaign doc updated with the F1/F2/F3 resolution banner.
|
||||
|
||||
## 8. Observations (not acted on)
|
||||
|
||||
- The hub artifact-manifest bump remains operator-gated (hub v0.29.0 dropdowns make it one click).
|
||||
- Campaign F4 (restore-test can't read a NON-pool source guest's config for bind-mount neutralization)
|
||||
and F5 (pool-blind eject dependent-scan) remain open as LOW, per the campaign doc.
|
||||
- The bring-up duplicate-guard (`bringup.go` ListLXC scan) is still pool-blind — acceptable now that a
|
||||
collision is non-destructive and PVE refuses authoritatively; noted in the F3 runbook.
|
||||
|
||||
Reference in New Issue
Block a user