felhom.eu/documentation/audits/SPIKE-decommission-migration-2026-06-14.md

# SPIKE — Decommission / migration feasibility (2026-06-14)

**Report-only. No feature code, no version bump.** GATE: B1/B2 do not start until the product owner
reviews these findings and picks the forks at the end.

**Product decision (locked, restated):** storage decommission is **self-serve — NO operator signature**.
It is the customer's own drive; we warn (and offer migration) but do not gate on operator authorization.
Decommission is **non-destructive** — un-enroll + drop the guest bind; **data stays on the drive**. It must
**NOT** format/wipe — that remains the separate customer-confirmed wipe flow.

Evidence is `file:line` + (live) where empirically checked on demo guest **9201** (host `felhom-pve`,
192.168.0.162). The legacy bare-metal `controller/docker-compose.yml` was NOT trusted — the real deploy is
the golden/bootstrap `docker run` in `felhom-agent/configs/build-golden.sh`.

---

## Q1 — Data-move layer (controller vs agent)

**The controller's actual /mnt mount** — `felhom-agent/configs/build-golden.sh:179-186`:

```
docker run -d --name felhom-controller --restart unless-stopped ... \
  -v /mnt:/mnt:rslave \
  -v /var/run/docker.sock:/var/run/docker.sock \
  "$IMAGE"
```

De-privileged (`build-golden.sh:176-178`: NO `--privileged`, no `/dev`, no `/etc/fstab`). The host pre-shares
/mnt so agent binds propagate in (`build-golden.sh:173-175`: `mkdir -p /mnt; mount --rbind /mnt /mnt;
mount --make-rshared /mnt`).

**`rslave` is read-WRITE for file content.** `rslave` governs *mount-event propagation direction* (host→
container only — the container can't mutate the guest mount tree), NOT file permissions. The bind has **no
`:ro`**, so reading/writing file *content* under /mnt is allowed.

**Live proof (inside the `felhom-controller` container on 9201):**

```
$ docker exec felhom-controller sh -c '... write/read/delete a marker under each enrolled path ...'
== mount opts for /mnt ==
/dev/mapper/pve-vm--9201--disk--0 /mnt ext4 rw,relatime,stripe=16 0 0
-- writing /mnt/felhom-usb/.felhom-spike-marker --
WRITE_OK / spike-1781452108 / DELETE_OK
-- writing /mnt/sys_drive/felhom-data/.felhom-spike-marker --
WRITE_OK / spike-1781452108 / DELETE_OK
```

Both enrolled storage paths (`/mnt/felhom-usb`, `/mnt/sys_drive/felhom-data`) are writable+deletable from
inside the controller container.

**Conclusion — chosen layer = CONTROLLER.** The controller already holds RW access to every enrolled drive
under /mnt, so it can perform an app's data move **itself**, in-process, no agent round-trip for the bytes.
The agent is needed only for the privileged side-effects that bracket the move: **unmount**
(eject/decommission) and **intent** recording. **The move primitive is `rsync`, not `cp -a`** (see locked
decision #3): B1 needs copy→verify→remove that is idempotent/resumable — `rsync -a --checksum` (copy +
verify) → confirm → only then remove the source; a crash mid-copy just re-runs. rsync is already the
controller's cross-drive primitive (Tier-2 off-drive backup).

---

## Q2 — "Migrate" button state

`controller/internal/web/templates/settings.html:337`:

```html
<span class="btn btn-xs btn-outline" style="opacity:.45;cursor:not-allowed" title="Hamarosan">📦 Összes adat átköltöztetése</span>
```

**Vestigial.** It is a `<span>`, not a `<button>`/`<form>`; greyed (`opacity:.45`), `cursor:not-allowed`,
`title="Hamarosan"` ("Coming soon"), **no `onclick`, no handler, no JS**. It renders only when
`AppCount>0 && HasOtherPaths` (`settings.html:336`). There is **no** route, no controller handler, no
`agentapi` method behind it. `Settings.SetDecommissioned` exists (`settings.go:821`) but is **never called**
anywhere in the controller — the whole migrate/decommission round-trip is unwired.

**Net-new wiring required (all new):**
1. **Data-move executor** (controller) — per app on the source drive: quiesce/stop → `cp -a` source→target
   under /mnt → verify → restart. (Layer proven in Q1.)
2. **Deploy-path rewrite** — rewrite each moved app's `HDD_PATH` in its `app.yaml` to the new drive +
   redeploy (`docker compose up -d`), so the app uses the new location. (Mirrors the v0.52 catalog
   HDD_PATH work.)
3. **Controller endpoint** — `POST /api/storage/migrate` (+ JS handler + progress UI replacing the span).
4. **Source decommission** — after a successful move, call the new self-serve `POST /disks/decommission`
   (Q3) on the now-empty source drive.
5. **Registry mark** — `Settings.SetDecommissioned(source, target)` (today uncalled) to soft-mark the
   source with `MigratedTo` (Q5).

---

## Q3 — Make decommission self-serve (the design)

### Current path (operator-signed only)

- **Class** — `internal/reconcile/classify.go:36` `ClassDecommission OpClass = "decommission"`.
- **Gate verdict** — `classify.go:103-107`:
  ```go
  case ClassGuestDestroy, ClassStorageWipe, ClassRestoreOverwrite, ClassDecommission:
      if prov.internalEvidence() { return Benign }
      return Destructive
  ```
  i.e. decommission is **Destructive** → requires a verified, role-authorized, action-bound **operator
  signature** (`internal/reconcile/gate.go:144-177`, `ReasonPendingSignature` without one).
- **Executor** — `internal/signedjobs/decommission.go:54-79` `DecommissionExecutor.Execute`: the gate has
  already burned the operator signature/nonce; it then only calls `intent.SetDecommissioned(durableID)`
  (`decommission.go:75`). It does **not** unmount and does **not** touch the guest bind.
- **Wiring** — `cmd/felhom-agent/main.go:437` `signedjobs.NewDecommissionExecutor(...)`, reachable **only**
  via the signed-jobs runner (hub-queued operator op). `IntentDecommissioned` is reachable today **only**
  this way. The controller has **no** endpoint and **no** `agentapi` method for it
  (`internal/agentapi/client.go` has `EjectDisk` at :359 but no decommission).

### The self-serve precedent to mirror: eject

`internal/localapi/server.go:239` `mux.HandleFunc("POST /disks/eject", s.withGuest(s.handleDiskEject))` →
`internal/localapi/disks.go:209 handleDiskEject`: **no signature** — `withGuest` per-guest-token self-scoping
(`server.go:292-315`), role-gated to user-data (`disks.go:230-235`), records intent `"ejected"`
(`disks.go:240`) then `Unmount` (`disks.go:241`). Confirmed self-serve, per-guest, sets intent without a
signature.

### New endpoint contract — `POST /disks/decommission` (NO CODE, contract only)

| Aspect | Spec |
|---|---|
| **Route** | `mux.HandleFunc("POST /disks/decommission", s.withGuest(s.handleDiskDecommission))` in `internal/localapi/server.go` |
| **Auth** | `withGuest` (per-guest bearer token → token's VMID is the only authority); `scopedFromBody` rejects a mismatched body `vmid` (403) — a guest acts only on its own drive. **No operator signature.** |
| **Params** | `{vmid, where}` where `where` = the host mount path (e.g. `/mnt/felhom-usb`). The durable-id is **resolved server-side** via `durableIDForMount(where)` (`disks.go:697`) — never caller-supplied (same as `recordIntent`/eject). |
| **Role gate** | user-data only: `roleForMountPath(where) != storage.RoleUserData` → 403 (mirror `disks.go:230-235`). Cannot decommission system/backup. Fails safe (unresolvable → protected → refused). |
| **Actions** | 1) resolve durable-id; 2) `IntentStore.SetDecommissioned(id)` (already exists, `internal/storage/intent.go:84`); 3) `GuestBindStore.Remove(vmid, id)` (**NET-NEW method** — see Q4); 4) `Unmount(where)` (mirror eject — addresses the Q4 re-bind gap by leaving the drive un-host-mounted); 5) return dependent guests + decommissioned path. **NON-DESTRUCTIVE: no format/mkfs.** |
| **Files touched (agent)** | `internal/localapi/server.go` (route), `internal/localapi/disks.go` (handler + a `recordDecommissionIntent` helper paralleling `recordIntent`), `internal/localapi/guestbindstore.go` (add `Remove`). |
| **Files touched (controller)** | `internal/agentapi/client.go` (new `Decommission(where)` method), `internal/web/storage_handlers.go` (+ `server.go` route) for the customer flow, `internal/web/templates/settings.html` (real button + JS), and call `Settings.SetDecommissioned` on success (Q5). |

### Security check (explicit)

- **The hub cannot reach the agent local API.** The local API binds `local_api.listen_addr =
  192.168.0.162:8443` (live `agent.json`) — the host's LAN/bridge address — over TLS with a leaf cert and
  **per-guest bearer tokens** (`server.go:67-68,287-315`). The hub is remote (k3s) and the topology is
  **outbound-only**: the agent POSTs host-reports to the hub; the hub "holds no data-plane role and **never
  connects inbound to a box**" (`felhom.eu/CLAUDE.md`). The hub has no token. → A compromised hub **cannot**
  issue a self-serve decommission. (It also can't issue the *operator-signed* one beyond queuing a blob the
  gate must still cryptographically verify.)
- **A compromised controller gains no new capability.** It already can `EjectDisk` (unmount its own
  user-data drive, `agentapi/client.go:359`) and drive the **customer-confirmed wipe** which *formats* a
  user-data drive (`storage_handlers.go:357` → agent `FormatDisk` → `gate.AuthorizeStorageWipe`). A
  non-destructive decommission (unmount + intent + drop bind, **data left intact**) is strictly **less**
  than the wipe it can already perform. The role-gate still blocks system/backup; self-scoping still blocks
  other guests.

### Bright line (does NOT wipe)

Decommission as designed calls **no format/mkfs path** — only `Unmount` + `SetDecommissioned` +
`GuestBindStore.Remove`. Drive data is left intact. The customer-confirmed **wipe** stays separate and
untouched: `controller handleStorageWipe` (`storage_handlers.go:357`) → `agent FormatDisk` →
`gate.AuthorizeStorageWipe` (`gate.go:209`). **How the wipe gate differs:** a **USER-DATA** wipe is
**customer-confirmable today** — `gate.go:210-226`: allowed iff `Confirmed` *and* `ConfirmDurableID ==
DeviceDurableID` (durable-id-bound), `Disposition: CustomerConfirmable`, **no operator signature**. Only
**system/backup** wipe takes the operator-signature path (`gate.go:227-229`, `pending_signature` / 409). So
a user-data wipe is *not* operator-409 — it is customer-confirmable; decommission is even softer (no
destruction at all).

### Recommendation — keep the operator-signed `DecommissionExecutor` (secondary / DR path)

**KEEP it.** Trade-off: the self-serve endpoint covers the customer's normal "I'm taking my drive back" on a
**reachable, healthy** guest. The operator-signed path is the **only** way to decommission a drive that is
physically **ABSENT** — `DecommissionExecutor` needs no `/dev` resolution and explicitly supports
decommissioning an absent drive (`decommission.go:37-39`) — or when the in-guest controller is
unreachable/dead (operator/DR). Removing it would drop the absent-drive + DR capability for **zero** security
gain (it's already gate-verified). The two are complementary: self-serve = live happy path, signed =
absent/DR.

---

## Q4 — Re-bind gap (needed regardless of signed vs self-serve)

- **`GuestBindStore` has no `Remove`.** `internal/localapi/guestbindstore.go`: `Record` (:58), `Guests`
  (:75) — and nothing else. Confirmed.
- **`ReassertGuestBinds` ignores intent.** `internal/localapi/disks.go:643`: it replays
  `s.guestBinds.Guests()` and gates each durable-id only on presence in `mountByDurable`, which is built from
  `s.storage.Observe(ctx)` — **mounted drives only** (`disks.go:649-654`, skip-if-`!present` at
  `disks.go:673-677`). It never consults the `IntentStore`.
- **Watchdog auto-mount is intent-gated.** `internal/storage/watchdog.go:174-178 reconcileAllowed`:
  ```go
  func (w *Watchdog) reconcileAllowed(t KnownTarget) bool {
      if w.intent == nil { return true }
      return w.intent.Get(t.DurableID) == IntentEnrolled
  }
  ```
  So self-heal **host**-mounts only `enrolled` drives — an ejected *or* decommissioned drive is never host
  auto-mounted. On a host **reboot** a decommissioned drive stays unmounted → absent from `storage.Observe`'s
  mounted set → absent from `ReassertGuestBinds`' `mountByDurable` → not re-bound. The watchdog is the
  primary guard.
- **Live evidence of the latent gap.** On host `felhom-pve`:
  ```
  drive-intents.json: { "uuid:277a…914":"enrolled", "uuid:ce9d…0f9":"ejected", "uuid:da9e…e00":"enrolled" }
  guest-binds.json:   { "9201": [ "uuid:ce9d…0f9", "uuid:da9e…e00" ] }
  pct config 9201:    mp1: /mnt/felhom-usb/felhom-data,mp=/mnt/felhom-usb   (only one user-data mount)
  ```
  `uuid:ce9d…0f9` is **`ejected`** in the intent map yet **still recorded in `guest-binds.json`** — direct
  proof that `GuestBindStore` never prunes. It is *not* currently re-bound only because it is unmounted
  (eject ran `Unmount`); `ReassertGuestBinds` skips it on `!present`.
- **The residual gap (the invariant at risk).** `ReassertGuestBinds` runs on **agent startup** and trusts
  the *current mount state*, not intent. The planned decommission is **non-destructive** — the drive stays
  put. If decommission set `IntentDecommissioned` **without** unmounting and the agent then restarted while
  the drive was still host-mounted, `storage.Observe` would still see it → `ReassertGuestBinds` would
  **re-bind it into the guest**, violating "a decommissioned drive must never auto-mount on reconnect."
  (Could not be forced live without risking RomM's drive, and there is no self-serve intent-setter yet —
  conclusion is by trace + the ejected-lingering-record above.)

**Recommended fix — belt & suspenders:**
- **Suspenders (must-have, single robust guard):** make `ReassertGuestBinds` **intent-aware** — skip any
  durable-id whose `IntentStore.Get(id) != IntentEnrolled`. One guard closes **both** the decommissioned
  case *and* the latent lingering-`ejected` record. (`s.intent` is already available to the local-API
  server.)
- **Belt (defense-in-depth + hygiene):** the decommission endpoint also (a) `Unmount`s the drive (mirror
  eject — so it isn't host-mounted at reassert time), and (b) calls the new `GuestBindStore.Remove(vmid,id)`
  so the record doesn't linger (also worth calling from `handleDiskEject` to fix the eject hygiene shown
  above).

Minimal-correct = the intent-aware `ReassertGuestBinds`. Recommend shipping **both** layers.

---

## Q5 — Registry on decommission

`controller/internal/settings/settings.go:821-839 SetDecommissioned` **keeps** the entry (soft marker):

```go
s.StoragePaths[i].Decommissioned = true
s.StoragePaths[i].DecommissionedAt = time.Now()...
s.StoragePaths[i].MigratedTo = migratedTo
s.StoragePaths[i].IsDefault = false
s.StoragePaths[i].Schedulable = false
```

It is not removed. Report/UI already render it: `handlers.go:855-857` sets `MigratedToLabel`, and
`settings.html:242-256` shows the decommissioned branch ("Adatok átköltöztetve ide: …" + an explicit
"Eltávolítás a rendszerből" hard-remove via `/settings/storage/remove`). (Note: `SetDecommissioned` is not
yet *called* anywhere — wiring is the net-new work in Q2/Q3.)

**Recommendation — KEEP the soft marker.** Decisive given **A1 is now additive** (v0.64.0): A1 re-registers
any deployed-app path missing from the registry **but skips any path present in ANY state, including
`Decommissioned`** (proven by its companion test). The soft marker is therefore exactly what **blocks A1
from resurrecting** a just-decommissioned drive. **Resurrection risk:** soft marker → **none** (skip-by-
presence); hard-remove → **real** — if any app still referenced the old path (or a stale `app.yaml`
`HDD_PATH` lingered), additive discovery would re-add it as a fresh, active, default-eligible path. The soft
marker also preserves the `MigratedTo`/`At` audit trail the UI renders; the user's explicit "Eltávolítás a
rendszerből" remains the deliberate hard-remove when they're truly done.

---

## Summary table

| Q | Finding | file:line | Recommendation |
|---|---|---|---|
| Q1 | Controller mount is `-v /mnt:/mnt:rslave`, de-privileged; `rslave` is RW for content. **Live:** controller wrote+deleted markers under both enrolled /mnt paths. | `build-golden.sh:184` (`:173-178`); live 9201 | **Data move = CONTROLLER** (in-process **`rsync -a --checksum`**, copy→verify→remove, between /mnt drives); agent only for unmount + intent. |
| Q2 | Migrate button is a vestigial greyed `<span>`, no handler/route/JS; `SetDecommissioned` never called. | `settings.html:337`; `settings.go:821` | Build the whole round-trip: data-move executor, HDD_PATH rewrite+redeploy, `POST /api/storage/migrate`, source decommission, registry mark. |
| Q3 | Decommission today = operator-signed only (Destructive in `Classify`); executor only sets intent; controller has no endpoint/agentapi method. Eject is the self-serve precedent. | `classify.go:36,103-107`; `decommission.go:54-79`; `main.go:437`; eject `server.go:239`+`disks.go:209` | Add self-serve `POST /disks/decommission` (withGuest, user-data role-gate, durable-id resolved server-side, intent+bind-remove+unmount, **no wipe**). Hub can't reach local API (8443 LAN, per-guest token, outbound-only); compromised controller gains nothing beyond eject/wipe. **Keep** operator-signed path for absent-drive/DR. |
| Q4 | `GuestBindStore` has no `Remove`; `ReassertGuestBinds` ignores intent; watchdog gates host-mount to `enrolled`. **Live:** an `ejected` drive still sits in `guest-binds.json`. Gap: a decommissioned-but-still-mounted drive would be re-bound on agent restart. | `guestbindstore.go` (no Remove); `disks.go:643,649-654,673-677`; `watchdog.go:174-178`; live intent/bind state | **Both:** intent-aware `ReassertGuestBinds` (skip non-`enrolled`) = **load-bearing correctness fix, sufficient alone** (covers self-serve + signed paths) + `GuestBindStore.Remove` = hygiene/DiD (re-commission re-`Record`s via `recordGuestBind` — B2 must state). |
| Q5 | `SetDecommissioned` keeps the entry as a soft marker (Decommissioned=true, Schedulable/IsDefault=false, MigratedTo); UI renders it. | `settings.go:821-839`; `handlers.go:855-857`; `settings.html:242-256` | **Keep soft marker** — it blocks A1-additive resurrection (skip-by-presence) + preserves audit. Hard-remove would risk re-registration if any reference lingers. |

## Decisions — LOCKED by product owner (2026-06-14). Gate cleared; B1/B2 may proceed.

1. **Operator-signed decommission — KEEP.** Retained as the DR / absent-drive path alongside the new
   self-serve endpoint. (Q3.)

2. **Q4 fix — BOTH, with a clear load-bearing split:**
   - **Load-bearing (correctness):** make `ReassertGuestBinds` **intent-aware** — skip any durable-id whose
     intent is not `enrolled`. **This alone is sufficient** for the invariant: it covers BOTH the self-serve
     decommission AND the operator-signed (absent-drive) path, because both routes set
     `IntentDecommissioned` and the guard keys on intent, not on who set it.
   - **Hygiene (not correctness):** `GuestBindStore.Remove` keeps `guest-binds.json` honest for *present*
     drives (no lingering records like the live `ejected` one). It is defense-in-depth, not the guard.
   - **Nuance the B2 spec MUST state:** if `Remove` is implemented, **re-commission must re-`Record`** the
     bind. This is already covered — `recordGuestBind` runs on every enroll/guest-attach
     (`disks.go:290,309`) — but B2 must call it out explicitly so a future change doesn't break it.

3. **Data-move layer — CONTROLLER, via `rsync` (NOT `cp -a`).** This is a **B1 implementation choice, not a
   fork.** B1 requires **copy → verify → remove**, idempotent/resumable; `cp -a` gives neither resumability
   nor verification. `rsync` is already an existing controller primitive (cross-drive Tier-2 backup, per
   CONTEXT). B1 contract: `rsync -a --checksum` (copy + verify) → confirm success → **only then** remove the
   source. A crash mid-copy simply re-runs (idempotent). [Supersedes the `cp -a` wording above in Q1 / the
   summary table — same layer, better primitive.]

4. **Decommission UNMOUNTS — agreed.** The endpoint performs a **role-gated** unmount (mirror eject) so the
   drive is physically removable. (Q3 / Q4 belt.)