felhom.eu/documentation/audits/SPIKE-decommission-migration-2026-06-14.md

SPIKE — Decommission / migration feasibility (2026-06-14)

Report-only. No feature code, no version bump. GATE: B1/B2 do not start until the product owner reviews these findings and picks the forks at the end.

Product decision (locked, restated): storage decommission is self-serve — NO operator signature. It is the customer's own drive; we warn (and offer migration) but do not gate on operator authorization. Decommission is non-destructive — un-enroll + drop the guest bind; data stays on the drive. It must NOT format/wipe — that remains the separate customer-confirmed wipe flow.

Evidence is file:line + (live) where empirically checked on demo guest 9201 (host felhom-pve, 192.168.0.162). The legacy bare-metal controller/docker-compose.yml was NOT trusted — the real deploy is the golden/bootstrap docker run in felhom-agent/configs/build-golden.sh.

---

Q1 — Data-move layer (controller vs agent)

The controller's actual /mnt mount — felhom-agent/configs/build-golden.sh:179-186:

docker run -d --name felhom-controller --restart unless-stopped ... \
  -v /mnt:/mnt:rslave \
  -v /var/run/docker.sock:/var/run/docker.sock \
  "$IMAGE"

De-privileged (build-golden.sh:176-178: NO --privileged, no /dev, no /etc/fstab). The host pre-shares /mnt so agent binds propagate in (build-golden.sh:173-175: mkdir -p /mnt; mount --rbind /mnt /mnt; mount --make-rshared /mnt).

rslave is read-WRITE for file content. rslave governs mount-event propagation direction (host→ container only — the container can't mutate the guest mount tree), NOT file permissions. The bind has no :ro, so reading/writing file content under /mnt is allowed.

Live proof (inside the felhom-controller container on 9201):

$ docker exec felhom-controller sh -c '... write/read/delete a marker under each enrolled path ...'
== mount opts for /mnt ==
/dev/mapper/pve-vm--9201--disk--0 /mnt ext4 rw,relatime,stripe=16 0 0
-- writing /mnt/felhom-usb/.felhom-spike-marker --
WRITE_OK / spike-1781452108 / DELETE_OK
-- writing /mnt/sys_drive/felhom-data/.felhom-spike-marker --
WRITE_OK / spike-1781452108 / DELETE_OK

Both enrolled storage paths (/mnt/felhom-usb, /mnt/sys_drive/felhom-data) are writable+deletable from inside the controller container.

Conclusion — chosen layer = CONTROLLER. The controller already holds RW access to every enrolled drive under /mnt, so it can perform an app's data move itself, in-process, no agent round-trip for the bytes. The agent is needed only for the privileged side-effects that bracket the move: unmount (eject/decommission) and intent recording. The move primitive is rsync, not cp -a (see locked decision #3): B1 needs copy→verify→remove that is idempotent/resumable — rsync -a --checksum (copy + verify) → confirm → only then remove the source; a crash mid-copy just re-runs. rsync is already the controller's cross-drive primitive (Tier-2 off-drive backup).

---

Q2 — "Migrate" button state

controller/internal/web/templates/settings.html:337:

<span class="btn btn-xs btn-outline" style="opacity:.45;cursor:not-allowed" title="Hamarosan">📦 Összes adat átköltöztetése</span>

Vestigial. It is a <span>, not a <button>/<form>; greyed (opacity:.45), cursor:not-allowed, title="Hamarosan" ("Coming soon"), no onclick, no handler, no JS. It renders only when AppCount>0 && HasOtherPaths (settings.html:336). There is no route, no controller handler, no agentapi method behind it. Settings.SetDecommissioned exists (settings.go:821) but is never called anywhere in the controller — the whole migrate/decommission round-trip is unwired.

Net-new wiring required (all new):

1. Data-move executor (controller) — per app on the source drive: quiesce/stop → cp -a source→target under /mnt → verify → restart. (Layer proven in Q1.)
2. Deploy-path rewrite — rewrite each moved app's HDD_PATH in its app.yaml to the new drive + redeploy (docker compose up -d), so the app uses the new location. (Mirrors the v0.52 catalog HDD_PATH work.)
3. Controller endpoint — POST /api/storage/migrate (+ JS handler + progress UI replacing the span).
4. Source decommission — after a successful move, call the new self-serve POST /disks/decommission (Q3) on the now-empty source drive.
5. Registry mark — Settings.SetDecommissioned(source, target) (today uncalled) to soft-mark the source with MigratedTo (Q5).

---

Q3 — Make decommission self-serve (the design)

Current path (operator-signed only)

• Class — internal/reconcile/classify.go:36 ClassDecommission OpClass = "decommission".
• Gate verdict — classify.go:103-107:

  case ClassGuestDestroy, ClassStorageWipe, ClassRestoreOverwrite, ClassDecommission:
      if prov.internalEvidence() { return Benign }
      return Destructive
  

  i.e. decommission is Destructive → requires a verified, role-authorized, action-bound operator signature (internal/reconcile/gate.go:144-177, ReasonPendingSignature without one).
• Executor — internal/signedjobs/decommission.go:54-79 DecommissionExecutor.Execute: the gate has already burned the operator signature/nonce; it then only calls intent.SetDecommissioned(durableID) (decommission.go:75). It does not unmount and does not touch the guest bind.
• Wiring — cmd/felhom-agent/main.go:437 signedjobs.NewDecommissionExecutor(...), reachable only via the signed-jobs runner (hub-queued operator op). IntentDecommissioned is reachable today only this way. The controller has no endpoint and no agentapi method for it (internal/agentapi/client.go has EjectDisk at :359 but no decommission).

The self-serve precedent to mirror: eject

internal/localapi/server.go:239 mux.HandleFunc("POST /disks/eject", s.withGuest(s.handleDiskEject)) → internal/localapi/disks.go:209 handleDiskEject: no signature — withGuest per-guest-token self-scoping (server.go:292-315), role-gated to user-data (disks.go:230-235), records intent "ejected" (disks.go:240) then Unmount (disks.go:241). Confirmed self-serve, per-guest, sets intent without a signature.

New endpoint contract — POST /disks/decommission (NO CODE, contract only)

Aspect	Spec
Route	mux.HandleFunc("POST /disks/decommission", s.withGuest(s.handleDiskDecommission)) in internal/localapi/server.go
Auth	withGuest (per-guest bearer token → token's VMID is the only authority); scopedFromBody rejects a mismatched body vmid (403) — a guest acts only on its own drive. No operator signature.
Params	{vmid, where} where where = the host mount path (e.g. /mnt/felhom-usb). The durable-id is resolved server-side via durableIDForMount(where) (disks.go:697) — never caller-supplied (same as recordIntent/eject).
Role gate	user-data only: roleForMountPath(where) != storage.RoleUserData → 403 (mirror disks.go:230-235). Cannot decommission system/backup. Fails safe (unresolvable → protected → refused).
Actions	1) resolve durable-id; 2) IntentStore.SetDecommissioned(id) (already exists, internal/storage/intent.go:84); 3) GuestBindStore.Remove(vmid, id) (NET-NEW method — see Q4); 4) Unmount(where) (mirror eject — addresses the Q4 re-bind gap by leaving the drive un-host-mounted); 5) return dependent guests + decommissioned path. NON-DESTRUCTIVE: no format/mkfs.
Files touched (agent)	internal/localapi/server.go (route), internal/localapi/disks.go (handler + a recordDecommissionIntent helper paralleling recordIntent), internal/localapi/guestbindstore.go (add Remove).
Files touched (controller)	internal/agentapi/client.go (new Decommission(where) method), internal/web/storage_handlers.go (+ server.go route) for the customer flow, internal/web/templates/settings.html (real button + JS), and call Settings.SetDecommissioned on success (Q5).

Security check (explicit)

• The hub cannot reach the agent local API. The local API binds local_api.listen_addr = 192.168.0.162:8443 (live agent.json) — the host's LAN/bridge address — over TLS with a leaf cert and per-guest bearer tokens (server.go:67-68,287-315). The hub is remote (k3s) and the topology is outbound-only: the agent POSTs host-reports to the hub; the hub "holds no data-plane role and never connects inbound to a box" (felhom.eu/CLAUDE.md). The hub has no token. → A compromised hub cannot issue a self-serve decommission. (It also can't issue the operator-signed one beyond queuing a blob the gate must still cryptographically verify.)
• A compromised controller gains no new capability. It already can EjectDisk (unmount its own user-data drive, agentapi/client.go:359) and drive the customer-confirmed wipe which formats a user-data drive (storage_handlers.go:357 → agent FormatDisk → gate.AuthorizeStorageWipe). A non-destructive decommission (unmount + intent + drop bind, data left intact) is strictly less than the wipe it can already perform. The role-gate still blocks system/backup; self-scoping still blocks other guests.

Bright line (does NOT wipe)

Decommission as designed calls no format/mkfs path — only Unmount + SetDecommissioned + GuestBindStore.Remove. Drive data is left intact. The customer-confirmed wipe stays separate and untouched: controller handleStorageWipe (storage_handlers.go:357) → agent FormatDisk → gate.AuthorizeStorageWipe (gate.go:209). How the wipe gate differs: a USER-DATA wipe is customer-confirmable today — gate.go:210-226: allowed iff Confirmed and ConfirmDurableID == DeviceDurableID (durable-id-bound), Disposition: CustomerConfirmable, no operator signature. Only system/backup wipe takes the operator-signature path (gate.go:227-229, pending_signature / 409). So a user-data wipe is not operator-409 — it is customer-confirmable; decommission is even softer (no destruction at all).

Recommendation — keep the operator-signed DecommissionExecutor (secondary / DR path)

KEEP it. Trade-off: the self-serve endpoint covers the customer's normal "I'm taking my drive back" on a reachable, healthy guest. The operator-signed path is the only way to decommission a drive that is physically ABSENT — DecommissionExecutor needs no /dev resolution and explicitly supports decommissioning an absent drive (decommission.go:37-39) — or when the in-guest controller is unreachable/dead (operator/DR). Removing it would drop the absent-drive + DR capability for zero security gain (it's already gate-verified). The two are complementary: self-serve = live happy path, signed = absent/DR.

---

Q4 — Re-bind gap (needed regardless of signed vs self-serve)

• GuestBindStore has no Remove. internal/localapi/guestbindstore.go: Record (:58), Guests (:75) — and nothing else. Confirmed.
• ReassertGuestBinds ignores intent. internal/localapi/disks.go:643: it replays s.guestBinds.Guests() and gates each durable-id only on presence in mountByDurable, which is built from s.storage.Observe(ctx) — mounted drives only (disks.go:649-654, skip-if-!present at disks.go:673-677). It never consults the IntentStore.
• Watchdog auto-mount is intent-gated. internal/storage/watchdog.go:174-178 reconcileAllowed:

  func (w *Watchdog) reconcileAllowed(t KnownTarget) bool {
      if w.intent == nil { return true }
      return w.intent.Get(t.DurableID) == IntentEnrolled
  }
  

  So self-heal host-mounts only enrolled drives — an ejected or decommissioned drive is never host auto-mounted. On a host reboot a decommissioned drive stays unmounted → absent from storage.Observe's mounted set → absent from ReassertGuestBinds' mountByDurable → not re-bound. The watchdog is the primary guard.
• Live evidence of the latent gap. On host felhom-pve:

  drive-intents.json: { "uuid:277a…914":"enrolled", "uuid:ce9d…0f9":"ejected", "uuid:da9e…e00":"enrolled" }
  guest-binds.json:   { "9201": [ "uuid:ce9d…0f9", "uuid:da9e…e00" ] }
  pct config 9201:    mp1: /mnt/felhom-usb/felhom-data,mp=/mnt/felhom-usb   (only one user-data mount)
  

  uuid:ce9d…0f9 is ejected in the intent map yet still recorded in guest-binds.json — direct proof that GuestBindStore never prunes. It is not currently re-bound only because it is unmounted (eject ran Unmount); ReassertGuestBinds skips it on !present.
• The residual gap (the invariant at risk). ReassertGuestBinds runs on agent startup and trusts the current mount state, not intent. The planned decommission is non-destructive — the drive stays put. If decommission set IntentDecommissioned without unmounting and the agent then restarted while the drive was still host-mounted, storage.Observe would still see it → ReassertGuestBinds would re-bind it into the guest, violating "a decommissioned drive must never auto-mount on reconnect." (Could not be forced live without risking RomM's drive, and there is no self-serve intent-setter yet — conclusion is by trace + the ejected-lingering-record above.)

Recommended fix — belt & suspenders:

• Suspenders (must-have, single robust guard): make ReassertGuestBinds intent-aware — skip any durable-id whose IntentStore.Get(id) != IntentEnrolled. One guard closes both the decommissioned case and the latent lingering-ejected record. (s.intent is already available to the local-API server.)
• Belt (defense-in-depth + hygiene): the decommission endpoint also (a) Unmounts the drive (mirror eject — so it isn't host-mounted at reassert time), and (b) calls the new GuestBindStore.Remove(vmid,id) so the record doesn't linger (also worth calling from handleDiskEject to fix the eject hygiene shown above).

Minimal-correct = the intent-aware ReassertGuestBinds. Recommend shipping both layers.

---

Q5 — Registry on decommission

controller/internal/settings/settings.go:821-839 SetDecommissioned keeps the entry (soft marker):

s.StoragePaths[i].Decommissioned = true
s.StoragePaths[i].DecommissionedAt = time.Now()...
s.StoragePaths[i].MigratedTo = migratedTo
s.StoragePaths[i].IsDefault = false
s.StoragePaths[i].Schedulable = false

It is not removed. Report/UI already render it: handlers.go:855-857 sets MigratedToLabel, and settings.html:242-256 shows the decommissioned branch ("Adatok átköltöztetve ide: …" + an explicit "Eltávolítás a rendszerből" hard-remove via /settings/storage/remove). (Note: SetDecommissioned is not yet called anywhere — wiring is the net-new work in Q2/Q3.)

Recommendation — KEEP the soft marker. Decisive given A1 is now additive (v0.64.0): A1 re-registers any deployed-app path missing from the registry but skips any path present in ANY state, including Decommissioned (proven by its companion test). The soft marker is therefore exactly what blocks A1 from resurrecting a just-decommissioned drive. Resurrection risk: soft marker → none (skip-by- presence); hard-remove → real — if any app still referenced the old path (or a stale app.yaml HDD_PATH lingered), additive discovery would re-add it as a fresh, active, default-eligible path. The soft marker also preserves the MigratedTo/At audit trail the UI renders; the user's explicit "Eltávolítás a rendszerből" remains the deliberate hard-remove when they're truly done.

---

Summary table

Q	Finding	file:line	Recommendation
Q1	Controller mount is -v /mnt:/mnt:rslave, de-privileged; rslave is RW for content. Live: controller wrote+deleted markers under both enrolled /mnt paths.	build-golden.sh:184 (:173-178); live 9201	Data move = CONTROLLER (in-process rsync -a --checksum, copy→verify→remove, between /mnt drives); agent only for unmount + intent.
Q2	Migrate button is a vestigial greyed <span>, no handler/route/JS; SetDecommissioned never called.	settings.html:337; settings.go:821	Build the whole round-trip: data-move executor, HDD_PATH rewrite+redeploy, POST /api/storage/migrate, source decommission, registry mark.
Q3	Decommission today = operator-signed only (Destructive in Classify); executor only sets intent; controller has no endpoint/agentapi method. Eject is the self-serve precedent.	classify.go:36,103-107; decommission.go:54-79; main.go:437; eject server.go:239+disks.go:209	Add self-serve POST /disks/decommission (withGuest, user-data role-gate, durable-id resolved server-side, intent+bind-remove+unmount, no wipe). Hub can't reach local API (8443 LAN, per-guest token, outbound-only); compromised controller gains nothing beyond eject/wipe. Keep operator-signed path for absent-drive/DR.
Q4	GuestBindStore has no Remove; ReassertGuestBinds ignores intent; watchdog gates host-mount to enrolled. Live: an ejected drive still sits in guest-binds.json. Gap: a decommissioned-but-still-mounted drive would be re-bound on agent restart.	guestbindstore.go (no Remove); disks.go:643,649-654,673-677; watchdog.go:174-178; live intent/bind state	Both: intent-aware ReassertGuestBinds (skip non-enrolled) = load-bearing correctness fix, sufficient alone (covers self-serve + signed paths) + GuestBindStore.Remove = hygiene/DiD (re-commission re-Records via recordGuestBind — B2 must state).
Q5	SetDecommissioned keeps the entry as a soft marker (Decommissioned=true, Schedulable/IsDefault=false, MigratedTo); UI renders it.	settings.go:821-839; handlers.go:855-857; settings.html:242-256	Keep soft marker — it blocks A1-additive resurrection (skip-by-presence) + preserves audit. Hard-remove would risk re-registration if any reference lingers.

Decisions — LOCKED by product owner (2026-06-14). Gate cleared; B1/B2 may proceed.

1. Operator-signed decommission — KEEP. Retained as the DR / absent-drive path alongside the new self-serve endpoint. (Q3.)

2. Q4 fix — BOTH, with a clear load-bearing split:

   • Load-bearing (correctness): make ReassertGuestBinds intent-aware — skip any durable-id whose intent is not enrolled. This alone is sufficient for the invariant: it covers BOTH the self-serve decommission AND the operator-signed (absent-drive) path, because both routes set IntentDecommissioned and the guard keys on intent, not on who set it.
   • Hygiene (not correctness): GuestBindStore.Remove keeps guest-binds.json honest for present drives (no lingering records like the live ejected one). It is defense-in-depth, not the guard.
   • Nuance the B2 spec MUST state: if Remove is implemented, re-commission must re-Record the bind. This is already covered — recordGuestBind runs on every enroll/guest-attach (disks.go:290,309) — but B2 must call it out explicitly so a future change doesn't break it.

3. Data-move layer — CONTROLLER, via rsync (NOT cp -a). This is a B1 implementation choice, not a fork. B1 requires copy → verify → remove, idempotent/resumable; cp -a gives neither resumability nor verification. rsync is already an existing controller primitive (cross-drive Tier-2 backup, per CONTEXT). B1 contract: rsync -a --checksum (copy + verify) → confirm success → only then remove the source. A crash mid-copy simply re-runs (idempotent). [Supersedes the cp -a wording above in Q1 / the summary table — same layer, better primitive.]

4. Decommission UNMOUNTS — agreed. The endpoint performs a role-gated unmount (mirror eject) so the drive is physically removable. (Q3 / Q4 belt.)