From cbaa53f5651f951407af31f28de97e5fa7526dad Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 14 Jun 2026 20:17:47 +0200 Subject: [PATCH] docs: README migrate/decommission section + REPORT for B1+B2 (v0.65.0) Co-Authored-By: Claude Opus 4.8 (1M context) --- REPORT.md | 145 ++++++++++++++++++------------------------- controller/README.md | 21 ++++++- 2 files changed, 80 insertions(+), 86 deletions(-) diff --git a/REPORT.md b/REPORT.md index 5c97070..d7499f9 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,94 +1,69 @@ -# REPORT — TASK A storage-lifecycle cleanups → controller v0.64.0 (2026-06-14) +# REPORT — TASK B1+B2 data migration + self-serve decommission → controller v0.65.0 (2026-06-14) -Two settings-layer cleanups (A1, A2) + two doc-only changes (A3, A4) from the F9 storage-registration -diagnosis. Trunk-based, all on `main`, no branches. Each code change has a table-driven test that fails on -the pre-fix code. Built, pushed, and deployed to demo guest 9201; verified `:0.64.0 Up (healthy)` with a -clean startup (`controller_started (0.64.0)`, no errors). +Customer-self-serve storage **migration** (move app data between drives) and **decommission** (retire a +drive), implemented trunk-based on `main` against the locked spike design +(`felhom.eu/documentation/audits/SPIKE-decommission-migration-2026-06-14.md`). Pairs with **agent +v0.32.0**. Each change carries non-hollow tests; the load-bearing companions were mutation-proven to +fail on the pre-fix code. Built + deployed both images to demo guest 9201 / felhom-pve. -> Verify against pushed source, not this report. +> Validate against pushed source, not this report. **Live decommission/migration of real data is NOT +> yet validated — that is the supervised B3 session (see the end).** -## Pre-flight +## Confirmed baselines +- Controller `main` was `c37ee6d` = **v0.64.0** → shipped **v0.65.0**. +- Agent `main` was `8e6d00a` = **v0.31.0** → shipped **v0.32.0**. -`main` is the only branch in felhom-controller; HEAD was `688ba0d v0.63.0` (CHANGELOG top + live container -both `:0.63.0`). Confirmed before starting. +## Commits (controller repo unless noted) +| # | Commit | Scope | +|---|---|---| +| 1 | `b57150e` | B1 migration engine (`internal/stacks/migrate.go`, MigrateAll+MigrateApp) + tests + main wiring + backup mutual-exclusion (Change 3) | +| 2 | `16a4c3e` | B1 UI — `/api/storage/migrate{,-app,/status}` + settings.html & app_info.html + progress panel | +| 3 | `f43697c` (felhom-agent) | v0.32.0 — `POST /disks/decommission` + intent-aware `ReassertGuestBinds` + `GuestBindStore.Remove` + tests | +| 4 | `f2596ea` | B2b decommission orchestration + "Hiányzó tárhely" indicator + Change 4 re-enroll fix + tests | -## Per-item results +## What shipped +- **B1 engine** — crash-safe, resumable, journaled (`/migration.json`). Pipeline validate → stop + → copy (`rsync -a --checksum`, additive, **no `--delete`**; conflict-merge walk for non-app content) → + verify (`rsync -ani --checksum`, zero pending) → flip+redeploy (`RedeployFromEnv`) → **cleanup gated on + all-verified AND all-redeployed**. Two entry points (whole-namespace + per-app incl. SSD→drive) over one + pipeline. Single-flight; mutually exclusive with the backup orchestrator (migration refuses while a + backup runs; scheduled DB-dump/Tier-2 skip while a migration runs). +- **B1 UI** — migrate-all (settings page) + per-app "Áthelyezés másik tárhelyre" (app-info page) + shared + status poll/progress panel. +- **B2a (agent)** — self-serve `POST /disks/decommission` mirroring eject (user-data role gate, no + signature, never formats): intent=decommissioned + bind-prune + unmount. **`ReassertGuestBinds` is now + intent-aware** (skips non-enrolled) — the F9-reconnect correctness fix. Operator-signed path untouched. +- **B2b** — `handleStorageDecommission` (migrate-all-or-none): migrate-then-decommission via the migration + done-hook, or decommission-anyway (stop apps, keep HDD_PATH). `agentapi.Decommission` added. Soft-marker + retained. **Change 4:** re-enroll clears the marker (`ClearDecommissioned` had zero callers before). + **"Hiányzó tárhely"** badge for apps on a missing drive. -### A1 — `AutoDiscoverStoragePaths` made ADDITIVE -- **File:** `controller/internal/settings/settings.go` (the function previously at L584). -- **Change:** removed the `if len(s.StoragePaths) > 0 { return }` early-return. It now registers only - discovered paths NOT already present. Invariants enforced: - - Never removes/modifies a manually-added path. - - SKIPS any path already in the registry IN ANY STATE — including a `Decommissioned` soft-marked entry — - so it is never re-added or reactivated (`if existing[path] { continue }`, keyed on `filepath.Clean`). - - Never flips `IsDefault`: a newly-discovered path becomes default ONLY if the registry currently has no - default at all (and then only the first such new path) — `IsDefault: !hasDefault`, with `hasDefault` - seeded from existing entries and set true after the first new default. -- **Tests:** new `controller/internal/settings/storage_discovery_test.go` - (`TestAutoDiscoverStoragePaths_Additive`, table-driven, + `..._DecommissionedNotReactivated`): - - (a) non-empty registry + a deployed-app path missing from registry → exactly that path registered; - pre-existing entry byte-identical via `reflect.DeepEqual` (IsDefault asserted unchanged); new path not - promoted to default. - - (b) Decommissioned path present + a deployed app still pointing at it → NOT reactivated; entry - byte-identical; path count unchanged. - - extra cases: empty registry (first becomes default), registry-with-no-default (first new becomes - default), fallback path pickup. -- **Non-hollow proof:** temporarily removing the `if existing[path] { continue }` guard makes - `..._DecommissionedNotReactivated` FAIL (`path count changed: got 4 want 2`); restored → green. -- **Commit:** `2d4d432` (shared with A2). **Test result:** `ok internal/settings`. - -### A2 — internal-SSD label disambiguation -- **File:** `controller/internal/settings/settings.go` (`InferStorageLabel`). -- **Change:** when `filepath.Base(path) == appbackup.FelhomDataDir` (the `"felhom-data"` namespace dir), the - label is now `Belső SSD (rendszer)` instead of the colliding `Tárhely (felhom-data)`. Imported - `internal/appbackup` for the constant (no import cycle — appbackup only depends on `internal/util`). - Still overridable via `SetStorageLabel`. -- **User-drive safety confirmation:** Model-A user drives register their MOUNT ROOT (e.g. `/mnt/felhom-usb`) - — confirmed live on 9201, where the user drive is registered as `/mnt/felhom-usb`, never - `.../felhom-data` — so the `base == felhom-data` match cannot mislabel a user drive. -- **Untouched:** the separate host-metrics label in `web/agent_host_metrics_handler.go:91` was left as-is - (display-only host-metrics path; A2 is the storage-registry label only). -- **Tests:** `TestInferStorageLabel` covers `/mnt/sys_drive/felhom-data` → `Belső SSD (rendszer)`, a - second felhom-data path, plus the existing hdd/ssd/usb/`Tárhely` branches (note: `felhom-usb` does NOT - start with `usb`, so it correctly labels `Tárhely (felhom-usb)` — pre-existing behavior). -- **Demo seeded-label update (before/after):** the seeded `settings.json` on 9201 had - `"/mnt/sys_drive/felhom-data" → "Tárhely (felhom-data)"`. Updated **through the real UI endpoint** - (`POST /settings/storage/label` → `SetStorageLabel`, the same handler the dashboard form posts to) to - `"Belső SSD (rendszer)"`; verified persisted in `settings.json`. (Note: the public URL - `https://felhom.demo-felhom.eu` returned 200 without applying the change — the Cloudflare/Traefik path - mangled the POST body; posting to the container's own `127.0.0.1:8080` returned 302 success and - persisted. Same handler either way.) Before: `Tárhely (felhom-data)`. After: `Belső SSD (rendszer)`. -- **Commit:** `2d4d432` (shared with A1). **Test result:** `ok internal/settings`. - -### A3 — doc: reject auto-register-on-attach (no deploy) -- **File:** `felhom.eu/documentation/backlog/DIAGNOSIS-f9-storage-registration-gap-2026-06-14.md` (the doc - lives in `backlog/`, not `audits/` as the task said). -- **Change:** recommendation #1 annotated **REJECTED (2026-06-14)** — contradicts the new→enrolled - manual-enrollment model; manual enroll is by design. Noted that the "make discovery additive" sub-point - WAS adopted (v0.64.0 A1) but only for paths deployed apps reference, not auto-register of agent-attached - drives. -- **Commit:** `b5a9614` (felhom.eu repo). - -### A4 — doc: live-validation rule (no deploy) -- **Files:** `felhom-controller/CLAUDE.md` and `felhom-agent/CLAUDE.md` — added the verbatim "Live - validation" paragraph to both. -- **Commits:** `e850402` (felhom-controller), `8e6d00a` (felhom-agent). +## Tests (per-commit green: `go build && go vet && go test ./...`) +- `internal/stacks/migrate_test.go` — siblingName, conflict-merge (differ→`(1)`, identical→dedup, re-run + idempotent no `(1)(1)`, 3rd→`(2)`), assert-only verify, skip-dirs, cleanup-only-after-redeploy + (+companion: redeploy-fail → source intact), verify-catches-corruption, resume (no re-copy), + collision-refused, backup-exclusion, single-flight, SSD→drive resolution, cleanup gate. +- `internal/backup/migration_exclusion_test.go` — scheduled DB-dump **skipped** while migrating. +- `internal/web/decommission_test.go` — finalize soft-marks + calls agent (entry retained), re-enroll + clears marker (Change 4), missing-storage label, unflagged-migration no-op. +- **Mutation-proven companions** (fail on pre-fix, pass after revert): migration collision guard, cleanup + gate, Change-4 re-enroll clearing, and (agent) the intent-aware reassert (`disks.go` guard removed → + decommissioned drive re-bound → test FAILS). ## Build / deploy / verify +- Built+pushed `gitea.dooplex.hu/admin/felhom-controller:0.65.0` (192.168.0.180); deployed to guest 9201 + via golden/bootstrap → `docker ps`: `:0.65.0 Up (healthy)`, clean startup (`controller_started (0.65.0)`, + no errors), `GET /api/storage/migrate/status` → `{"job":null}` (idle, wired). +- Built agent v0.32.0 on 180; installed on felhom-pve (`/usr/local/bin/felhom-agent`, backup + `.bak-0.31.0`), `systemctl restart felhom-agent` → active, `--version` 0.32.0. +- **Live-proven (incidental, safe):** on agent restart the new intent-aware re-assert logged + `F9 re-assert: skipping non-enrolled drive (intent-gated) vmid=9201 durable_id=uuid:ce9d… intent=ejected` + — the exact drive that lingered in `guest-binds.json` (B0 spike finding) is now correctly NOT re-bound. -- `go build ./... && go vet ./internal/settings/ && go test ./...` → all green locally. -- Built + pushed `gitea.dooplex.hu/admin/felhom-controller:0.64.0` on 192.168.0.180. -- Deployed to guest 9201 via the golden/bootstrap mechanism (`docker pull` → write - `/etc/felhom-controller-image` → restart `felhom-controller-bootstrap.service`). -- **Verify:** `docker ps` → `gitea.dooplex.hu/admin/felhom-controller:0.64.0 Up (healthy)`; logs show - `Event pushed: controller_started (info) — Controller elindult (0.64.0)` and no startup errors. No - rollback needed. - -## Commit hashes (per item) - -| Item | Repo | Commit | -|---|---|---| -| A1 + A2 (code + tests + CHANGELOG + README) | felhom-controller | `2d4d432` | -| A4 (CLAUDE.md) | felhom-controller | `e850402` | -| A4 (CLAUDE.md) | felhom-agent | `8e6d00a` | -| A3 (diagnosis annotation) | felhom.eu | `b5a9614` | +## NOT yet live-validated — awaiting supervised B3 +The real end-to-end UI flow with actual data was deliberately NOT run (per the task STOP): +- put data on a drive → decommission (migrate-to-another OR decommission-anyway) → verify data integrity + end-to-end → drive un-enrolled and does NOT auto-mount/re-bind on reconnect → orphaned apps show + "Hiányzó tárhely" → re-enroll clears it. Operator-present, real UI (no agent/CLI shortcuts, per the A4 + rule). The engine/endpoint logic is unit-proven and the intent-aware reassert is already proven live by + the restart log above; the data-bearing round-trip is what B3 covers. diff --git a/controller/README.md b/controller/README.md index 7c4997a..0428d45 100644 --- a/controller/README.md +++ b/controller/README.md @@ -677,7 +677,26 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an > **"Belső SSD (rendszer)"** instead of the colliding "Tárhely (felhom-data)". Model-A user drives > register their MOUNT ROOT (e.g. `/mnt/felhom-usb`), never `.../felhom-data`, so this can't mislabel a > user drive. Still overridable via `SetStorageLabel`. -> - **Migration** (drive + per-stack) is **deferred** to its own slice (buttons disabled "Hamarosan"). +> - **(v0.65.0) Data migration** (`internal/stacks/migrate.go`) — move app data between drives, in-process +> over the controller's `/mnt:/mnt:rslave` RW mount; crash-safe + resumable via a journal +> (`/migration.json`). `MigrateAll(source,target)` moves the whole felhom-data namespace (every +> app + a conflict-merge walk for non-app/customer content); `MigrateApp(app,target)` moves one app's +> subtree (drive→drive AND SSD→drive). Pipeline: validate → stop → copy (`rsync -a --checksum`, additive, +> **no `--delete`**) → verify (`rsync -ani --checksum`, zero pending) → flip+redeploy (`RedeployFromEnv`) +> → cleanup. **CLEANUP — the only destructive step — is gated on every unit verified AND every app +> redeployed.** Conflict-merge: skip-identical (checksum vs target + its `(N)` siblings), rename-on-differ +> to lowest-free `(N)`, never overwrite; idempotent. Single-flight + mutually exclusive with the +> backup orchestrator. UI: `POST /api/storage/migrate{,-app}` + `GET /api/storage/migrate/status` (poll); +> migrate-all on the settings page, per-app on the app-info page, shared Hungarian progress panel. +> - **(v0.65.0) Self-serve decommission** (`handleStorageDecommission`, `POST /api/storage/decommission`) — +> retire a drive, non-destructive (never formats). Two choices (no partial): migrate-all-then-decommission +> (runs `MigrateAll`; the migration done-hook soft-marks the source + calls the agent on success), or +> decommission-anyway (type-to-confirm; stops the apps, keeps their `HDD_PATH` so they show "Hiányzó +> tárhely"). Both end at `settings.SetDecommissioned` (soft marker retained — blocks A1 resurrection) + +> `agentapi.Decommission` (agent v0.32.0: `POST /disks/decommission`, role-gated user-data, intent + +> bind-prune + unmount). Re-enrolling a decommissioned drive (`registerStoragePath`) clears the marker + +> restores `Schedulable`. A deployed app whose drive is decommissioned/disconnected/absent shows the +> **"Hiányzó tárhely"** warning badge on the dashboard/stacks/app card. > > The privileged controller-side disk subsections **below are historical** (the `internal/storage/*` scan/ > format code was removed in 8C — execution is the agent's now).