diff --git a/REPORT.md b/REPORT.md index fe3fd76..f0b1ffe 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,111 +1,100 @@ -# REPORT — DR-recipe completion: live PBS coord + drop the two unfillable drive fields +# REPORT — v0.40.0: SSD user-data volume (`/mnt/sys_drive`, mp1) — the third CT-volume split -**Repo:** `felhom-agent` · **Version:** v0.38.0 → **v0.39.0** · **Date:** 2026-06-16 -**Companion repo:** `felhom.eu/hub` v0.13.0 → **v0.13.1** (test-data + regression test only) -**Controller:** untouched. +**Task:** add a third CT volume (`mp1` @ `/mnt/sys_drive`, `backup=1`) baked into the golden + grown +at provision (`-sysdata-grow`), so the controller's `system_data_path` becomes a separate, PBS-covered +mount and the "not a separate drive" warning clears with **zero controller change**. Option A (static +CT volume, NOT the enrolled-drive intent machinery). -## 1. Confirmed baselines + commit hashes +## 1. Baselines +- felhom-agent `main` @ `b908b9a` (= spec's `b908b9a8e`), version **v0.39.0** → **v0.40.0**. +- Currently-deployed in-guest controller: **`felhom-controller:0.77.0`** (spec guessed 0.76.0; actual + is 0.77.0 — this is the `CONTROLLER_IMAGE` the golden must bake). +- felhom-controller **untouched** (it already expects `/mnt/sys_drive` + warns via + `system.IsMountPoint`). -| Repo | Baseline | Target | Commit | -|---|---|---|---| -| felhom-agent | v0.38.0 (`8abc1b8`) | **v0.39.0** | **`fbe1130`** | -| felhom-hub (`felhom.eu`) | v0.13.0 (`5f5e3c5`) | **v0.13.1** | **`149a3b0`** | +## 2. Files modified (felhom-agent only) +- `configs/build-golden.sh` — `--mp1 ...,mp=/mnt/sys_drive,backup=1` (env `GOLDEN_SYSDATA_GB=8`); + `findmnt /mnt/sys_drive` separate-mount guard; vzdump-inclusion guard now aborts if **mp0 OR mp1** + excluded (the B3 trap), with the mirrored WARN-if-unconfirmed. +- `internal/reconcile/bringup.go` — `const DefaultSysDataMount = "mp1"`; `BringUpSpec.{SysDataGrowGB, + SysDataMount}`; new **"4c"** online grow-only `ResizeLXC` block mirroring the "4b" Docker-data grow. +- `cmd/felhom-agent/main.go` — `-sysdata-grow` / `-sysdata-mount` flags; `bringUpSizing.{SysDataGrowGB, + SysDataMount}`; wired at all three sites (the two `--selftest=bring-up` dispatch/spec paths + the + `--selftest=provision` spec); version `0.40.0`; `--selftest=provision` help text updated. +- `internal/reconcile/bringup_test.go` — `TestRunBringUp_StorageSplit_SysDataGrow` + + `…_SysDataGrowZeroNoResize`. +- `CHANGELOG.md` (v0.40.0 entry, newest on top), `RUNBOOK-provisioning-storage.md` (three-volume + layout), this `REPORT.md`. -Both pushed to `main`. Green gate (`go build ./... && go vet ./... && go test ./...`) passed in BOTH repos. +## 3. Commits pushed to `main` +- `459dad9` — v0.40.0: third CT volume + `-sysdata-grow` (code + tests + CHANGELOG + RUNBOOK). +- (REPORT.md commit follows.) -## 2. What changed +## 4. Tests + red-proofs +- `go build ./... && go vet ./... && go test ./...` — **green** (reconcile + cmd + all packages). + - Pre-existing unrelated flake: `internal/escrow/TestGenerateRecoveryCode_EntropyAndFormat` + (random 10/11-word count) — passes on re-run; **not touched by this change**. +- `TestRunBringUp_StorageSplit_SysDataGrow`: PASS — fake api records `ResizeLXC(8051,"mp1","+42G")` + alongside `mp0 +240G`. +- `TestRunBringUp_StorageSplit_SysDataGrowZeroNoResize`: PASS — `SysDataGrowGB=0` ⇒ **no** mp1 resize. +- **Red-proof #1 (Go "4c" block):** removed the "4c" grow block → `…_SysDataGrow` FAILED + (`got [{mp0 +240G}]`, no mp1 resize) → reverted → green. ✅ +- reconcile test count: 127 RUN entries (subtests included), all pass. +- **Red-proof #2 (shell vzdump guard):** to be run live during the golden rebuild — temporarily flip + `mp1` to `backup=0`, confirm the guard ABORTS, revert. **PENDING (needs golden rebuild — §6).** -### Part 1 — live PBS coord (agent) -- **`internal/pbs/report.go`** — added `SnapshotStore.Get(datastore) []hub.PBSSnapshot` (mutex-guarded copy; per-datastore last-known-good). The ONLY `SnapshotStore` change. -- **`internal/pbs/live_reporter.go` (new)** — `LiveSnapshotReporter` implements `hub.PBSReporter`. Each collect: derive an 8 s child ctx (`DefaultLiveSnapshotTimeout`), resolve targets, and for each datastore do the cheap `Client.Snapshots()` list, convert via `Snapshot.ToHub()`, `store.Record(...)` (authoritative incl. empty), append. Per-datastore live error → append `store.Get(ds)` (last-known-good, store not clobbered). Targets-resolution error → `store.PBSSnapshots(ctx)` (full LKG aggregate). **List only — never triggers a verify.** A test seam (`listSnapshots` func field, default `liveListSnapshots`) lets tests inject a fake with no live PBS. -- **`cmd/felhom-agent/main.go`** — hoisted `pbsTargets := pbsTargetsFromPVE(...)` and wired `pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger)` into the collector in **both** `runDaemon` (verify loop reuses the SAME `pbsStore` + `pbsTargets`) **and** `runSelftestHub` (which previously passed a `nil` reporter — that is precisely why the live `--selftest=hub` showed `pbs_snapshots:[]`/no pbs coord). Intended side effect: `report.pbs_snapshots` is now live too. +## 5. Live deploy DONE — agent binary (§13-A) +- Built v0.40.0 on the build server (192.168.0.180, go1.26.0, `-ldflags -X main.version=0.40.0`). +- Deployed to felhom-pve: backed up prior binary to `/usr/local/bin/felhom-agent.bak-0.39.0`, + installed, `systemctl restart felhom-agent`. +- **Verified:** `felhom-agent --version` → `0.40.0`; service `active`; clean restart (local-api + listening on 192.168.0.162:8443; only WARN is the expected intent-gated skip of the lingering ejected + `ce9d…` drive on 9201 — documented prior behavior, not an error). -### Part 2 — drop `role` + `restic_repo_coord` from the v1 host-half drive shape (agent) -- **`internal/hub/dr_recipe.go`** — removed `Role` and `ResticRepoCoord` from `DRDrive` and the `Role: t.Role` literal. v1 drive shape is now `{durable_id, mount_path, intent, fs_type?, total_bytes}`. Header + `DRDrive` doc rewritten: role deferred (hub/operator manifest concept), restic coord reserved for a future offsite tier (none exists today), pbs coord resolved live. `isUserDataDrive`/`latestPBSCoord`/`DRPBSCoord` unchanged. +## 6. Live acceptance — PENDING (blocked on operator-held secrets + an irreversible CHECKPOINT) +The golden rebuild (§13-B), the `pct destroy 9201` CHECKPOINT (§13-C), and the re-provision (§13-D/E) +were **not** executed because they require inputs CC does not hold: +1. **`REGISTRY_USER` + `REGISTRY_TOKEN`** — for the golden's one-time controller-image pull inside the + build guest. Confirmed **absent on-host** (no `/root/.docker/config.json`, no saved build env). +2. **`DEMO_RETRIEVAL_PASSPHRASE`** (the `-hub-password`) — for the 9201 re-provision (the controller's + hub config pull). +3. **Operator confirmation** for the irreversible `pct stop 9201; pct destroy 9201 --purge` (§13-C + CHECKPOINT). Operator pre-authorized a clean reinstall in principle, but the destroy is gated. -### Part 3 — goldens (agent + hub) -- Agent golden `internal/hub/testdata/host-report.golden.json` — dropped `role` from `dr_recipe.drives[0]` (hand-maintained; no generator script exists — `configs/build-golden.sh` builds the LXC archive, not this JSON; the contract test validates key-sets only). -- Hub `internal/api/testdata/host-report.golden.json` — re-synced **byte-identical** with the agent golden (it previously lacked the `dr_recipe` section entirely). -- Hub `internal/store/testdata/dr-recipe.golden.json` + `drHostHalf` fixture — dropped `role` from `drives[0]`. - -## 3. Tests — results + demonstrated pre-fix companion failures - -All run with the mutation applied → confirmed FAIL → reverted → green. - -| Test | Result | Companion mutation → demonstrated failure | -|---|---|---| -| `pbs.TestLiveReporter_CoordPresentWithoutPriorVerify` (T1, load-bearing) | PASS | `PBSSnapshots` mutated to `return r.store.PBSSnapshots(ctx)` (no live read) → `want 2 live snapshots, got 0` + recipe `pbs` nil. Reverted. | -| `pbs.TestLiveReporter_ErrorFallsBackToLastKnownGood` (T2) | PASS | dropped the LKG-append on error → `want the last-known-good snapshot on live error, got []`. Reverted. | -| `pbs.TestLiveReporter_SuccessUpdatesStore` (T3) | PASS | (same no-live-read mutation) → `store not warmed`. Reverted. | -| `pbs.TestLiveReporter_TargetsErrorReturnsAggregate` (T4) | PASS | — | -| `pbs.TestLiveReporter_BoundedByTimeout` (T5) | PASS | dropped the LKG-append → blocked-lister path returned `[]` instead of LKG. Reverted. | -| `pbs.TestLiveReporter_EmptySuccessIsAuthoritative` (T6) | PASS | (same no-live-read mutation) → stale snapshot survived instead of being overwritten. Reverted. | -| `hub.TestDRRecipeHostHalf_V1DriveShape` (T7) | PASS | re-added `Role string \`json:"role"\`` to `DRDrive` → `v1 drive must NOT carry "role" key` AND `TestHostReport_ContractMatchesGolden` drift `struct keys = [... role ...]`. Reverted. | -| `hub.TestDRRecipeHostHalf_NoSecrets` (T8) | PASS (unchanged) | — | -| `hub.TestBuildDRRecipeHostHalf` / `..._NoPBS` / `TestHostReport_ContractMatchesGolden` | PASS (updated to v1 shape) | — | -| `store.TestAssembleDRRecipe_V1DriveShape` (T9, hub) | PASS | fixture re-added `role` to `drives[0]` → `v1 drives must not carry role/restic_repo_coord`. Reverted. | -| hub `store`/`api` full suites incl. golden byte-match | PASS | — | - -The companion mutations confirmed the live read (T1/T3/T6), the LKG fallback (T2/T5), and both repos' v1-shape guards (T7/T9) are each load-bearing — not hollow assertions. - -## 4. Deployed versions (proof) - -- Build: `go build -ldflags "-X main.version=0.39.0"` on 192.168.0.180 (go1.26.0) → sha256 `04b7a410…6bd6bd7`; same sha verified after scp to felhom-pve. -- `felhom-pve`: backed up `/usr/local/bin/felhom-agent` → `.bak-0.38.0`, `install -m0755`, `systemctl restart`. `felhom-agent --version` → **`felhom-agent 0.39.0`**; `systemctl is-active` → **active**. -- Controller: `gitea.dooplex.hu/admin/felhom-controller:0.73.0` (untouched). - -## 5. Live `--selftest=hub` `dr_recipe` block (post-deploy, immediately after restart — NO verify ran) - -```json -"dr_recipe": { - "recipe_version": 1, - "guests": [ - { "vmid": 9001, "cores": 2, "memory_bytes": 2147483648, "disk_bytes": 10737418240 }, - { "vmid": 9999, "cores": 1, "memory_bytes": 268435456, "disk_bytes": 2147483648 }, - { "vmid": 9201, "cores": 2, "memory_bytes": 12884901888, "disk_bytes": 33501757440 } - ], - "pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" }, - "drives": [ - { "durable_id": "uuid:81a26531-62d8-408d-812f-a178b1d35310", "mount_path": "/mnt/felhom-flash", "intent": "enrolled", "total_bytes": 125704151040 }, - { "durable_id": "uuid:da9e7089-cf8e-4617-adcb-a377743fae00", "mount_path": "/mnt/felhom-usb", "intent": "enrolled", "total_bytes": 983349346304 } - ], - "pve_storage": [ - { "name": "local-lvm", "type": "lvmthin", "content": "images,rootdir" }, - { "name": "felhom-pbs", "type": "pbs", "content": "backup" }, - { "name": "felhom-flash", "type": "usb", "content": "backup" }, - { "name": "felhom-usb", "type": "local-dir", "content": "backup" }, - { "name": "local", "type": "local", "content": "backup,vztmpl,iso,import" } - ] -} +Planned live run once provided (locked sizing per RUNBOOK = 32 rootfs / 200 docker-data / 50 user-data): ``` - -- `pbs` **present** — `repo_id=felhom-pbs`, `namespace=root`, `latest_snapshot_id=9201` (matches the latest `ct/9201 @ 2026-06-16T17:00:10Z` per `pvesm list felhom-pbs`). **This is the gap closed: present seconds after restart, no verify cycle.** -- `drives[]` carry **no** `role` / `restic_repo_coord` — only `{durable_id, mount_path, intent, total_bytes}` for both externals. -- `report.pbs_snapshots` is now **live-populated (6)**, was `[]`. - -## 6. Hub assembled recipe (post-push, the real `AssembleDRRecipe` over the live `hub.db`) - -Daemon's startup host-report landed in the hub DB (`dr_recipe` row `demo-felhom` / host `demo-felhom-01`, `updated_at 2026-06-16 18:28 UTC`). Assembled via a throwaway `store`-package test on 180 against a copy of `/data/hub.db` (dashboard-auth blocks curl; no password available): - -- Assembled recipe = **3329 B**, `recipe_version 1`, customer `demo-felhom` (Demo Ügyfél / demo-felhom.eu), **3 guests w/ sizing**, `pbs` present (`felhom-pbs/root/9201`), **2 drives clean** (no role/restic), 5 pve_storage, **14 apps** (app half from controller v0.73.0). -- **Secret scan over the FULL assembled recipe AND both raw stored halves: 0 secret-shaped key hits** (regex `(?i)(password|secret|token|hash|passphrase|api[_-]?key|\bkey\b|enc:)`). Nothing to redact — the boundary this whole arc protects. - -> NOTE: the assembled-recipe operator PANEL (`customer_unified.html` / `GET /customers/{id}/dr-recipe.json`) was NOT browser-rendered (no dashboard password in this env). Verified the exact server pipeline + stored/assembled bytes, not the rendered HTML. - -## 7. Cross-repo golden checksum match - -`internal/hub/testdata/host-report.golden.json` (agent) and `hub/internal/api/testdata/host-report.golden.json` (hub) are **byte-identical**: - -``` -57f2a5e7154ece1be94a79fd327f2924762e70c3ee388d6e375df7c918b2f2b5 (both) +# B. rebuild golden (root@felhom-pve) — bake controller 0.77.0 +GOLDEN_SYSDATA_GB=8 REGISTRY_USER=… REGISTRY_TOKEN=… \ + ./build-golden.sh 9100