docs(spike): decommission/migration feasibility (TASK B0)
Report-only feasibility spike: data-move layer (controller, proven live), vestigial migrate button, self-serve POST /disks/decommission design, re-bind gap (intent-unaware ReassertGuestBinds + no GuestBindStore.Remove), soft-marker registry. Gates B1/B2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,261 @@
|
||||
# SPIKE — Decommission / migration feasibility (2026-06-14)
|
||||
|
||||
**Report-only. No feature code, no version bump.** GATE: B1/B2 do not start until the product owner
|
||||
reviews these findings and picks the forks at the end.
|
||||
|
||||
**Product decision (locked, restated):** storage decommission is **self-serve — NO operator signature**.
|
||||
It is the customer's own drive; we warn (and offer migration) but do not gate on operator authorization.
|
||||
Decommission is **non-destructive** — un-enroll + drop the guest bind; **data stays on the drive**. It must
|
||||
**NOT** format/wipe — that remains the separate customer-confirmed wipe flow.
|
||||
|
||||
Evidence is `file:line` + (live) where empirically checked on demo guest **9201** (host `felhom-pve`,
|
||||
192.168.0.162). The legacy bare-metal `controller/docker-compose.yml` was NOT trusted — the real deploy is
|
||||
the golden/bootstrap `docker run` in `felhom-agent/configs/build-golden.sh`.
|
||||
|
||||
---
|
||||
|
||||
## Q1 — Data-move layer (controller vs agent)
|
||||
|
||||
**The controller's actual /mnt mount** — `felhom-agent/configs/build-golden.sh:179-186`:
|
||||
|
||||
```
|
||||
docker run -d --name felhom-controller --restart unless-stopped ... \
|
||||
-v /mnt:/mnt:rslave \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
"$IMAGE"
|
||||
```
|
||||
|
||||
De-privileged (`build-golden.sh:176-178`: NO `--privileged`, no `/dev`, no `/etc/fstab`). The host pre-shares
|
||||
/mnt so agent binds propagate in (`build-golden.sh:173-175`: `mkdir -p /mnt; mount --rbind /mnt /mnt;
|
||||
mount --make-rshared /mnt`).
|
||||
|
||||
**`rslave` is read-WRITE for file content.** `rslave` governs *mount-event propagation direction* (host→
|
||||
container only — the container can't mutate the guest mount tree), NOT file permissions. The bind has **no
|
||||
`:ro`**, so reading/writing file *content* under /mnt is allowed.
|
||||
|
||||
**Live proof (inside the `felhom-controller` container on 9201):**
|
||||
|
||||
```
|
||||
$ docker exec felhom-controller sh -c '... write/read/delete a marker under each enrolled path ...'
|
||||
== mount opts for /mnt ==
|
||||
/dev/mapper/pve-vm--9201--disk--0 /mnt ext4 rw,relatime,stripe=16 0 0
|
||||
-- writing /mnt/felhom-usb/.felhom-spike-marker --
|
||||
WRITE_OK / spike-1781452108 / DELETE_OK
|
||||
-- writing /mnt/sys_drive/felhom-data/.felhom-spike-marker --
|
||||
WRITE_OK / spike-1781452108 / DELETE_OK
|
||||
```
|
||||
|
||||
Both enrolled storage paths (`/mnt/felhom-usb`, `/mnt/sys_drive/felhom-data`) are writable+deletable from
|
||||
inside the controller container.
|
||||
|
||||
**Conclusion — chosen layer = CONTROLLER.** The controller already holds RW access to every enrolled drive
|
||||
under /mnt, so it can perform an app's data move (`cp -a /mnt/X/... → /mnt/Y/...`) **itself**, in-process,
|
||||
no agent round-trip for the bytes. The agent is needed only for the privileged side-effects that bracket the
|
||||
move: **unmount** (eject/decommission) and **intent** recording. This matches the existing app-data backup
|
||||
design (the controller already rsyncs between drives for Tier-2 off-drive copies).
|
||||
|
||||
---
|
||||
|
||||
## Q2 — "Migrate" button state
|
||||
|
||||
`controller/internal/web/templates/settings.html:337`:
|
||||
|
||||
```html
|
||||
<span class="btn btn-xs btn-outline" style="opacity:.45;cursor:not-allowed" title="Hamarosan">📦 Összes adat átköltöztetése</span>
|
||||
```
|
||||
|
||||
**Vestigial.** It is a `<span>`, not a `<button>`/`<form>`; greyed (`opacity:.45`), `cursor:not-allowed`,
|
||||
`title="Hamarosan"` ("Coming soon"), **no `onclick`, no handler, no JS**. It renders only when
|
||||
`AppCount>0 && HasOtherPaths` (`settings.html:336`). There is **no** route, no controller handler, no
|
||||
`agentapi` method behind it. `Settings.SetDecommissioned` exists (`settings.go:821`) but is **never called**
|
||||
anywhere in the controller — the whole migrate/decommission round-trip is unwired.
|
||||
|
||||
**Net-new wiring required (all new):**
|
||||
1. **Data-move executor** (controller) — per app on the source drive: quiesce/stop → `cp -a` source→target
|
||||
under /mnt → verify → restart. (Layer proven in Q1.)
|
||||
2. **Deploy-path rewrite** — rewrite each moved app's `HDD_PATH` in its `app.yaml` to the new drive +
|
||||
redeploy (`docker compose up -d`), so the app uses the new location. (Mirrors the v0.52 catalog
|
||||
HDD_PATH work.)
|
||||
3. **Controller endpoint** — `POST /api/storage/migrate` (+ JS handler + progress UI replacing the span).
|
||||
4. **Source decommission** — after a successful move, call the new self-serve `POST /disks/decommission`
|
||||
(Q3) on the now-empty source drive.
|
||||
5. **Registry mark** — `Settings.SetDecommissioned(source, target)` (today uncalled) to soft-mark the
|
||||
source with `MigratedTo` (Q5).
|
||||
|
||||
---
|
||||
|
||||
## Q3 — Make decommission self-serve (the design)
|
||||
|
||||
### Current path (operator-signed only)
|
||||
|
||||
- **Class** — `internal/reconcile/classify.go:36` `ClassDecommission OpClass = "decommission"`.
|
||||
- **Gate verdict** — `classify.go:103-107`:
|
||||
```go
|
||||
case ClassGuestDestroy, ClassStorageWipe, ClassRestoreOverwrite, ClassDecommission:
|
||||
if prov.internalEvidence() { return Benign }
|
||||
return Destructive
|
||||
```
|
||||
i.e. decommission is **Destructive** → requires a verified, role-authorized, action-bound **operator
|
||||
signature** (`internal/reconcile/gate.go:144-177`, `ReasonPendingSignature` without one).
|
||||
- **Executor** — `internal/signedjobs/decommission.go:54-79` `DecommissionExecutor.Execute`: the gate has
|
||||
already burned the operator signature/nonce; it then only calls `intent.SetDecommissioned(durableID)`
|
||||
(`decommission.go:75`). It does **not** unmount and does **not** touch the guest bind.
|
||||
- **Wiring** — `cmd/felhom-agent/main.go:437` `signedjobs.NewDecommissionExecutor(...)`, reachable **only**
|
||||
via the signed-jobs runner (hub-queued operator op). `IntentDecommissioned` is reachable today **only**
|
||||
this way. The controller has **no** endpoint and **no** `agentapi` method for it
|
||||
(`internal/agentapi/client.go` has `EjectDisk` at :359 but no decommission).
|
||||
|
||||
### The self-serve precedent to mirror: eject
|
||||
|
||||
`internal/localapi/server.go:239` `mux.HandleFunc("POST /disks/eject", s.withGuest(s.handleDiskEject))` →
|
||||
`internal/localapi/disks.go:209 handleDiskEject`: **no signature** — `withGuest` per-guest-token self-scoping
|
||||
(`server.go:292-315`), role-gated to user-data (`disks.go:230-235`), records intent `"ejected"`
|
||||
(`disks.go:240`) then `Unmount` (`disks.go:241`). Confirmed self-serve, per-guest, sets intent without a
|
||||
signature.
|
||||
|
||||
### New endpoint contract — `POST /disks/decommission` (NO CODE, contract only)
|
||||
|
||||
| Aspect | Spec |
|
||||
|---|---|
|
||||
| **Route** | `mux.HandleFunc("POST /disks/decommission", s.withGuest(s.handleDiskDecommission))` in `internal/localapi/server.go` |
|
||||
| **Auth** | `withGuest` (per-guest bearer token → token's VMID is the only authority); `scopedFromBody` rejects a mismatched body `vmid` (403) — a guest acts only on its own drive. **No operator signature.** |
|
||||
| **Params** | `{vmid, where}` where `where` = the host mount path (e.g. `/mnt/felhom-usb`). The durable-id is **resolved server-side** via `durableIDForMount(where)` (`disks.go:697`) — never caller-supplied (same as `recordIntent`/eject). |
|
||||
| **Role gate** | user-data only: `roleForMountPath(where) != storage.RoleUserData` → 403 (mirror `disks.go:230-235`). Cannot decommission system/backup. Fails safe (unresolvable → protected → refused). |
|
||||
| **Actions** | 1) resolve durable-id; 2) `IntentStore.SetDecommissioned(id)` (already exists, `internal/storage/intent.go:84`); 3) `GuestBindStore.Remove(vmid, id)` (**NET-NEW method** — see Q4); 4) `Unmount(where)` (mirror eject — addresses the Q4 re-bind gap by leaving the drive un-host-mounted); 5) return dependent guests + decommissioned path. **NON-DESTRUCTIVE: no format/mkfs.** |
|
||||
| **Files touched (agent)** | `internal/localapi/server.go` (route), `internal/localapi/disks.go` (handler + a `recordDecommissionIntent` helper paralleling `recordIntent`), `internal/localapi/guestbindstore.go` (add `Remove`). |
|
||||
| **Files touched (controller)** | `internal/agentapi/client.go` (new `Decommission(where)` method), `internal/web/storage_handlers.go` (+ `server.go` route) for the customer flow, `internal/web/templates/settings.html` (real button + JS), and call `Settings.SetDecommissioned` on success (Q5). |
|
||||
|
||||
### Security check (explicit)
|
||||
|
||||
- **The hub cannot reach the agent local API.** The local API binds `local_api.listen_addr =
|
||||
192.168.0.162:8443` (live `agent.json`) — the host's LAN/bridge address — over TLS with a leaf cert and
|
||||
**per-guest bearer tokens** (`server.go:67-68,287-315`). The hub is remote (k3s) and the topology is
|
||||
**outbound-only**: the agent POSTs host-reports to the hub; the hub "holds no data-plane role and **never
|
||||
connects inbound to a box**" (`felhom.eu/CLAUDE.md`). The hub has no token. → A compromised hub **cannot**
|
||||
issue a self-serve decommission. (It also can't issue the *operator-signed* one beyond queuing a blob the
|
||||
gate must still cryptographically verify.)
|
||||
- **A compromised controller gains no new capability.** It already can `EjectDisk` (unmount its own
|
||||
user-data drive, `agentapi/client.go:359`) and drive the **customer-confirmed wipe** which *formats* a
|
||||
user-data drive (`storage_handlers.go:357` → agent `FormatDisk` → `gate.AuthorizeStorageWipe`). A
|
||||
non-destructive decommission (unmount + intent + drop bind, **data left intact**) is strictly **less**
|
||||
than the wipe it can already perform. The role-gate still blocks system/backup; self-scoping still blocks
|
||||
other guests.
|
||||
|
||||
### Bright line (does NOT wipe)
|
||||
|
||||
Decommission as designed calls **no format/mkfs path** — only `Unmount` + `SetDecommissioned` +
|
||||
`GuestBindStore.Remove`. Drive data is left intact. The customer-confirmed **wipe** stays separate and
|
||||
untouched: `controller handleStorageWipe` (`storage_handlers.go:357`) → `agent FormatDisk` →
|
||||
`gate.AuthorizeStorageWipe` (`gate.go:209`). **How the wipe gate differs:** a **USER-DATA** wipe is
|
||||
**customer-confirmable today** — `gate.go:210-226`: allowed iff `Confirmed` *and* `ConfirmDurableID ==
|
||||
DeviceDurableID` (durable-id-bound), `Disposition: CustomerConfirmable`, **no operator signature**. Only
|
||||
**system/backup** wipe takes the operator-signature path (`gate.go:227-229`, `pending_signature` / 409). So
|
||||
a user-data wipe is *not* operator-409 — it is customer-confirmable; decommission is even softer (no
|
||||
destruction at all).
|
||||
|
||||
### Recommendation — keep the operator-signed `DecommissionExecutor` (secondary / DR path)
|
||||
|
||||
**KEEP it.** Trade-off: the self-serve endpoint covers the customer's normal "I'm taking my drive back" on a
|
||||
**reachable, healthy** guest. The operator-signed path is the **only** way to decommission a drive that is
|
||||
physically **ABSENT** — `DecommissionExecutor` needs no `/dev` resolution and explicitly supports
|
||||
decommissioning an absent drive (`decommission.go:37-39`) — or when the in-guest controller is
|
||||
unreachable/dead (operator/DR). Removing it would drop the absent-drive + DR capability for **zero** security
|
||||
gain (it's already gate-verified). The two are complementary: self-serve = live happy path, signed =
|
||||
absent/DR.
|
||||
|
||||
---
|
||||
|
||||
## Q4 — Re-bind gap (needed regardless of signed vs self-serve)
|
||||
|
||||
- **`GuestBindStore` has no `Remove`.** `internal/localapi/guestbindstore.go`: `Record` (:58), `Guests`
|
||||
(:75) — and nothing else. Confirmed.
|
||||
- **`ReassertGuestBinds` ignores intent.** `internal/localapi/disks.go:643`: it replays
|
||||
`s.guestBinds.Guests()` and gates each durable-id only on presence in `mountByDurable`, which is built from
|
||||
`s.storage.Observe(ctx)` — **mounted drives only** (`disks.go:649-654`, skip-if-`!present` at
|
||||
`disks.go:673-677`). It never consults the `IntentStore`.
|
||||
- **Watchdog auto-mount is intent-gated.** `internal/storage/watchdog.go:174-178 reconcileAllowed`:
|
||||
```go
|
||||
func (w *Watchdog) reconcileAllowed(t KnownTarget) bool {
|
||||
if w.intent == nil { return true }
|
||||
return w.intent.Get(t.DurableID) == IntentEnrolled
|
||||
}
|
||||
```
|
||||
So self-heal **host**-mounts only `enrolled` drives — an ejected *or* decommissioned drive is never host
|
||||
auto-mounted. On a host **reboot** a decommissioned drive stays unmounted → absent from `storage.Observe`'s
|
||||
mounted set → absent from `ReassertGuestBinds`' `mountByDurable` → not re-bound. The watchdog is the
|
||||
primary guard.
|
||||
- **Live evidence of the latent gap.** On host `felhom-pve`:
|
||||
```
|
||||
drive-intents.json: { "uuid:277a…914":"enrolled", "uuid:ce9d…0f9":"ejected", "uuid:da9e…e00":"enrolled" }
|
||||
guest-binds.json: { "9201": [ "uuid:ce9d…0f9", "uuid:da9e…e00" ] }
|
||||
pct config 9201: mp1: /mnt/felhom-usb/felhom-data,mp=/mnt/felhom-usb (only one user-data mount)
|
||||
```
|
||||
`uuid:ce9d…0f9` is **`ejected`** in the intent map yet **still recorded in `guest-binds.json`** — direct
|
||||
proof that `GuestBindStore` never prunes. It is *not* currently re-bound only because it is unmounted
|
||||
(eject ran `Unmount`); `ReassertGuestBinds` skips it on `!present`.
|
||||
- **The residual gap (the invariant at risk).** `ReassertGuestBinds` runs on **agent startup** and trusts
|
||||
the *current mount state*, not intent. The planned decommission is **non-destructive** — the drive stays
|
||||
put. If decommission set `IntentDecommissioned` **without** unmounting and the agent then restarted while
|
||||
the drive was still host-mounted, `storage.Observe` would still see it → `ReassertGuestBinds` would
|
||||
**re-bind it into the guest**, violating "a decommissioned drive must never auto-mount on reconnect."
|
||||
(Could not be forced live without risking RomM's drive, and there is no self-serve intent-setter yet —
|
||||
conclusion is by trace + the ejected-lingering-record above.)
|
||||
|
||||
**Recommended fix — belt & suspenders:**
|
||||
- **Suspenders (must-have, single robust guard):** make `ReassertGuestBinds` **intent-aware** — skip any
|
||||
durable-id whose `IntentStore.Get(id) != IntentEnrolled`. One guard closes **both** the decommissioned
|
||||
case *and* the latent lingering-`ejected` record. (`s.intent` is already available to the local-API
|
||||
server.)
|
||||
- **Belt (defense-in-depth + hygiene):** the decommission endpoint also (a) `Unmount`s the drive (mirror
|
||||
eject — so it isn't host-mounted at reassert time), and (b) calls the new `GuestBindStore.Remove(vmid,id)`
|
||||
so the record doesn't linger (also worth calling from `handleDiskEject` to fix the eject hygiene shown
|
||||
above).
|
||||
|
||||
Minimal-correct = the intent-aware `ReassertGuestBinds`. Recommend shipping **both** layers.
|
||||
|
||||
---
|
||||
|
||||
## Q5 — Registry on decommission
|
||||
|
||||
`controller/internal/settings/settings.go:821-839 SetDecommissioned` **keeps** the entry (soft marker):
|
||||
|
||||
```go
|
||||
s.StoragePaths[i].Decommissioned = true
|
||||
s.StoragePaths[i].DecommissionedAt = time.Now()...
|
||||
s.StoragePaths[i].MigratedTo = migratedTo
|
||||
s.StoragePaths[i].IsDefault = false
|
||||
s.StoragePaths[i].Schedulable = false
|
||||
```
|
||||
|
||||
It is not removed. Report/UI already render it: `handlers.go:855-857` sets `MigratedToLabel`, and
|
||||
`settings.html:242-256` shows the decommissioned branch ("Adatok átköltöztetve ide: …" + an explicit
|
||||
"Eltávolítás a rendszerből" hard-remove via `/settings/storage/remove`). (Note: `SetDecommissioned` is not
|
||||
yet *called* anywhere — wiring is the net-new work in Q2/Q3.)
|
||||
|
||||
**Recommendation — KEEP the soft marker.** Decisive given **A1 is now additive** (v0.64.0): A1 re-registers
|
||||
any deployed-app path missing from the registry **but skips any path present in ANY state, including
|
||||
`Decommissioned`** (proven by its companion test). The soft marker is therefore exactly what **blocks A1
|
||||
from resurrecting** a just-decommissioned drive. **Resurrection risk:** soft marker → **none** (skip-by-
|
||||
presence); hard-remove → **real** — if any app still referenced the old path (or a stale `app.yaml`
|
||||
`HDD_PATH` lingered), additive discovery would re-add it as a fresh, active, default-eligible path. The soft
|
||||
marker also preserves the `MigratedTo`/`At` audit trail the UI renders; the user's explicit "Eltávolítás a
|
||||
rendszerből" remains the deliberate hard-remove when they're truly done.
|
||||
|
||||
---
|
||||
|
||||
## Summary table
|
||||
|
||||
| Q | Finding | file:line | Recommendation |
|
||||
|---|---|---|---|
|
||||
| Q1 | Controller mount is `-v /mnt:/mnt:rslave`, de-privileged; `rslave` is RW for content. **Live:** controller wrote+deleted markers under both enrolled /mnt paths. | `build-golden.sh:184` (`:173-178`); live 9201 | **Data move = CONTROLLER** (in-process `cp -a` between /mnt drives); agent only for unmount + intent. |
|
||||
| Q2 | Migrate button is a vestigial greyed `<span>`, no handler/route/JS; `SetDecommissioned` never called. | `settings.html:337`; `settings.go:821` | Build the whole round-trip: data-move executor, HDD_PATH rewrite+redeploy, `POST /api/storage/migrate`, source decommission, registry mark. |
|
||||
| Q3 | Decommission today = operator-signed only (Destructive in `Classify`); executor only sets intent; controller has no endpoint/agentapi method. Eject is the self-serve precedent. | `classify.go:36,103-107`; `decommission.go:54-79`; `main.go:437`; eject `server.go:239`+`disks.go:209` | Add self-serve `POST /disks/decommission` (withGuest, user-data role-gate, durable-id resolved server-side, intent+bind-remove+unmount, **no wipe**). Hub can't reach local API (8443 LAN, per-guest token, outbound-only); compromised controller gains nothing beyond eject/wipe. **Keep** operator-signed path for absent-drive/DR. |
|
||||
| Q4 | `GuestBindStore` has no `Remove`; `ReassertGuestBinds` ignores intent; watchdog gates host-mount to `enrolled`. **Live:** an `ejected` drive still sits in `guest-binds.json`. Gap: a decommissioned-but-still-mounted drive would be re-bound on agent restart. | `guestbindstore.go` (no Remove); `disks.go:643,649-654,673-677`; `watchdog.go:174-178`; live intent/bind state | **Both:** intent-aware `ReassertGuestBinds` (skip non-`enrolled`) [must-have] + decommission endpoint `Unmount` + new `GuestBindStore.Remove` [hygiene/DiD]. |
|
||||
| Q5 | `SetDecommissioned` keeps the entry as a soft marker (Decommissioned=true, Schedulable/IsDefault=false, MigratedTo); UI renders it. | `settings.go:821-839`; `handlers.go:855-857`; `settings.html:242-256` | **Keep soft marker** — it blocks A1-additive resurrection (skip-by-presence) + preserves audit. Hard-remove would risk re-registration if any reference lingers. |
|
||||
|
||||
## Forks needing the product owner's call (GATE)
|
||||
|
||||
1. **Operator-signed decommission** — keep as DR/absent-drive path (**recommended**) vs remove. → *recommend keep.*
|
||||
2. **Q4 fix shape** — intent-aware `ReassertGuestBinds` only, vs `GuestBindStore.Remove` only, vs **both** (**recommended**). → *recommend both.*
|
||||
3. **Data-move layer** — controller-side `cp -a` (**recommended**, proven live) vs agent-side mover. → *recommend controller.*
|
||||
4. **Decommission = unmount?** — endpoint unmounts the drive (mirror eject, so it's physically removable; **recommended**) vs leaves it host-mounted. → *recommend unmount.*
|
||||
Reference in New Issue
Block a user