From d42d90fed7162e6e9fbffa37632102219b89514a Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 30 Jul 2026 14:21:44 +0200 Subject: [PATCH] =?UTF-8?q?R-108=20CLOSED=20=E2=80=94=20D5's=20preconditio?= =?UTF-8?q?n=20is=20met=20(controller=20v0.187.0)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four-artifact update per the coupling rule, plus the audit. 07-backup-architecture.md: §10.1 retitled CLOSED with the ruling and the D5 sentence; the FileBrowser network-share row flipped YES->NO, closed at the PLACEMENT rather than at the bind; the exposure chain annotated with the fifth surface (decommission-with-migrate guarded only its source) and the correction that the boundary is the deploy POST, not the dropdown; §7.3 retitled UNBLOCKED; register row collapsed; open question F answered. 00-capability-map.md: new §D row PROVEN-LIVE, with the un-exercised legs named — the deploy-POST and decommission refusals are unit-tested, not live-fired. OPEN-ITEMS.md: R-108 dispositioned; D5 given its OWN row as READY/UNBLOCKED (it had existed only inside other rows' prose — the R-123 thread-loss pattern); R-126 registered. ROADMAP.md: R-108 collapsed to a shipped one-liner; R-126 added. R-126 filed not fixed: a .fab bundle (plaintext secrets, optional password) can be exported ONTO a NAS. Split out of R-108 rather than folded in — it is an explicit customer-chosen export destination, not a browsing surface reaching a backup tree, so it was never part of D5's precondition. Live evidence: same-box before/after on demo-felhom through the real authenticated endpoint, the network-specific refusal on demo-hp, non-effect verified in the registry, and R-67's share-root bind diffed byte-identical across the deploy. --- .../architecture/00-capability-map.md | 1 + .../architecture/07-backup-architecture.md | 77 +++++- .../R108-network-app-namespace-2026-07-30.md | 251 ++++++++++++++++++ documentation/backlog/OPEN-ITEMS.md | 4 +- documentation/backlog/ROADMAP.md | 3 +- 5 files changed, 323 insertions(+), 13 deletions(-) create mode 100644 documentation/audits/R108-network-app-namespace-2026-07-30.md diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index b36d19a..5362c6d 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -95,6 +95,7 @@ | Drive wizard: scan/format/mount/enroll, incl. legacy-boot LVM-root hosts | controller, agent v0.87 | **PROVEN-LIVE** | `DISPOSITION-ia-finding2-systemdisks-2026-07-13` (legacy EFI+LVM host, root not offered, byte-identical); enroll/format live in `storage-lifecycle-acceptance-2026-06-15` (E10 re-enroll, data intact); agent fence self-test refuses `/dev/sda` | (Cited `CAMPAIGN-2` T-STG-ENROLL/SEC-FORMAT were auth-hollow CSRF-403.) **Fresh-USB wizard enroll+format through the customer UI PROVEN-LIVE (controller v0.141.0, 2026-07-17):** a 64 GB scratch USB driven through the real `/api/storage/init` endpoints (login+CSRF) → confirm → detached format (~27 s mkfs) → mount → register → mounted+registered at `/mnt/felhom-drives/scratch1`. **F6 (initialize-to-usable) now covered:** the wizard runs the chain as a detached, disconnect-safe, pollable job (3-step progress) with an agent format-status poll for a slow mkfs **2026-07-26 — a SILENT failure class on the channel every agent-backed capability depends on (this row, data migration, USB enrollment, guest RAM, quiesce/PBS) is now DETECTED (controller v0.173.0, R-77). No row status flips.** `controller.yaml` and `bootstrap.json` could disagree on `local_api.endpoint` indefinitely with no signal: the R-50 island migration rewrote the latter, the fleet kept dialling the former, and for 17.5 h the only alert was a generic "agent unreachable" that read as an infrastructure blip. Drift now raises its own event type (`local_api_endpoint_drift`) naming both values. It is DETECTION ONLY — the authority ruling is R-78 — so the class is now loud, not prevented. Evidence: `audits/DIAG-agent-channel-2026-07-26.md`. | | Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | `CAMPAIGN-6C` 4P-5 (scope=app round-trip, byte-identical); `storage-lifecycle-acceptance-2026-06-15` (two migrate-all runs via dashboard UI, sha256 byte-identical) | (Cited `CAMPAIGN-2` T-STG-MIGRATE-* were auth-hollow.) "crash-safe" is design-level (copy→verify→remove) — no clean live crash-during-migration PASS | | NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | `SPIKE-nas-verify-2026-07-11`, `SPIKE-nas-dsm-2026-07-11`, `CAMPAIGN-3-2026-07-11` (boot/reassert fixes) | | +| **Network storage (NAS) is browse + bulk-media only — it may NOT host an app's data namespace** | controller v0.187.0 | **PROVEN-LIVE** (2026-07-30) | `audits/R108-network-app-namespace-2026-07-30.md`. Same-box before/after on demo-felhom through the exact endpoint the UI invokes (`POST /api/storage/migrate-app`, authenticated + CSRF): **pre-fix v0.186.0** the target was never examined — both a NAS-shaped and an unregistered path passed straight into `MigrateApp` and failed only on the app name (409); **post-fix v0.187.0** both are refused 400 with a Hungarian reason, while a real local drive still reaches `MigrateApp` (409) proving the guard is not over-broad. On demo-hp (the box with a REGISTERED share) the network-specific refusal fires. Non-effect verified in the registry: no `migrated_to`, nothing decommissioned, app `HDD_PATH` unchanged, no `backups/` on the share | **Closes R-108 and UNBLOCKS D5** (`07-backup-architecture.md` §7.3, §10.1). The share-root `:rslave` FileBrowser bind is deliberately UNCHANGED — load-bearing for automount wake, and unscopable — verified byte-identical by diffing demo-hp's generated compose before/after. Fails closed: `/mnt/felhom-drives` holds both kinds, so an unregistered path under it is un-classifiable and refused. **Not exercised:** the deploy POST and decommission-migrate refusals are unit-tested (non-effect, nil `stackMgr`) but were NOT live-fired — only migrate-app was. `.fab`-export-onto-NAS remains open (→ R-126) | | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | `storage-lifecycle-acceptance-2026-06-15` E4 (yanked-while-running → agent auto-rebind) + E10 (re-enroll, data intact); `CAMPAIGN-4`/`6A` (3 USB re-establish across device-letter reshuffle) | (Cited `RUNBOOK-usb` could NOT complete a wizard enrollment; `CAMPAIGN-2` legs were auth-hollow.) Fresh-USB **wizard enrollment** specifically still unproven | | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | `storage-lifecycle-acceptance-2026-06-15` E9 (decommission-anyway → bind detached, parent mp untouched, reboot-safe) + E12 (eject drive holding all apps) | (Cited `CAMPAIGN-2` T-STG-DECOM-* were auth-hollow; `SPIKE-decommission` was report-only, button still vestigial.) | | Boot ordering: automount + networking survive reboot; appliance self-heal watchdog | agent v0.85 | **PROVEN-LIVE** | `CAMPAIGN-4-2026-07-13` (F12 fix HOLDS: demo-host reboot + 5-boot storm, 0 ordering cycles, caps 63/63, WG re-handshake) + `CAMPAIGN-6A-2026-07-14` 1D (re-arm reboot-survival across 9 guest + 1 host reboots) | (`CAMPAIGN-3` F10/F11/F12 were the CRITICAL/HIGH *failures*; fixes shipped in agent v0.85 and were re-validated live in 4/6A — cite the validation, not the finding.) Residual: `skip-active` on `pct reboot` carried by the heal path; a NAS outage spanning a guest reboot can strand the share until agent restart (6A) | diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 4f1f806..5ee8153 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -372,18 +372,28 @@ that. requires the guest's secrets. So offsite alone cannot rebuild an app onto a fresh guest. → **R-107** -### 7.3 What D5 would change — and why it is blocked +### 7.3 What D5 would change — and why it was blocked (PRECONDITION NOW MET, 2026-07-30) -**[DESIGN, TARGET — BLOCKED]** The intended fix is to make app secrets travel with the **local** +**[DESIGN, TARGET — UNBLOCKED]** The intended fix is to make app secrets travel with the **local** recovery unit, so Tier-1 and Tier-2 restore work **without the guest and without R**. Offsite already encrypts everything, so secrets travelling offsite would be covered by the escrowed repo password. R would then be required for **offsite recovery and host identity only** — losing R would cost the offsite route, not local recovery. -**This is not adopted.** §2 of the task that produced this document required the premise to be -established, not assumed: the backup tree must be unreachable from every browsing, download and -export surface. **It is not.** The verification and the exposure are in §10.1. Until that is closed -(**R-108**), D5 stays a target and §7.1's chain stands as the model. +**The premise D5 rests on is now established.** §2 of the task that produced this document required +it to be proven, not assumed: the backup tree must be unreachable from every browsing, download and +export surface. When this document was written it was **not** — the FileBrowser network-share bind +reached it. **R-108 closed that on 2026-07-30** (controller v0.187.0) by refusing app namespaces on +network storage, so no `backups/` tree can exist under the share-root bind; every other surface was +already clear (§10.1's table). **D5's precondition is therefore MET and D5 may be adopted.** + +**Still true, and not part of D5's precondition:** a `.fab` bundle carries plaintext secrets by +design with an optional password, and `storageDriveList()` does not filter network paths, so a bundle +can be **exported onto** a NAS (§5, → **R-126**). That is an export destination the customer chooses +explicitly, not a browsing surface reaching a backup tree, and it is unchanged by D5 — D5 moves +secrets into the local recovery unit, not into `.fab`. It is tracked separately rather than folded in. + +Until D5 is actually implemented, §7.1's chain stands as the model. --- @@ -467,7 +477,39 @@ uplink — **no customer has ever driven a restore**. Every divergence between the model above and the system as it is, each with an ID. -### 10.1 D5 is BLOCKED — the backup tree is reachable from a browsing surface +### 10.1 ~~D5 is BLOCKED~~ — CLOSED 2026-07-30 by R-108 (controller v0.187.0) + +> **D5's PRECONDITION IS MET.** An app's data namespace can no longer be placed on network storage, so +> no `backups/` tree can exist inside FileBrowser's share-root bind, and the browsing surface therefore +> cannot reach the backup tree on ANY storage class. Every other read surface in the table below was +> already NO. **D5 may be adopted** — nothing in this section blocks it. +> +> **The fix inverted the obvious one, and that is the durable lesson here.** The share-root bind was +> not narrowed, because it **cannot** be: (a) the `:rslave` share-ROOT bind is load-bearing — a +> Phase-0 probe (2026-07-22) proved an in-container access through it wakes the idle automount +> trigger, so narrowing it breaks NAS access itself; (b) there is no `userdata/` layer to scope to, +> since apps on a share store at `/`; and (c) creating one would write Felhom's directory +> convention onto a customer's own NAS, which R-67 forbids outright. The browsing surface being +> immovable is precisely *why* the backup tree must never be placed under it. Tier 2 had already +> reached the same conclusion for its own targets (`F-6C-1`); R-108 closes the PRIMARY namespace, +> which was the last remaining route. +> +> **Operator ruling 2026-07-30: refuse the placement, keep the browse bind.** `RefuseAsAppNamespace` +> (`internal/settings/settings.go`) is the single predicate; all placement surfaces consult it. It +> **fails closed** — `/mnt/felhom-drives` holds both storage kinds in-guest, so a path prefix cannot +> classify and `Kind` exists only on a REGISTERED path; an unregistered path under that root is +> therefore un-classifiable and is refused rather than assumed to be a drive. +> +> **Nothing was stranded:** zero apps on network storage across all six hub customers including Peti. +> R-67's browse capability is byte-for-byte unchanged (verified by diffing demo-hp's generated +> compose before and after the deploy). This also **supersedes** the controller README's "NAS backup +> locality — decision A" (v0.118.0), which deliberately kept a NAS-resident app's Tier-1 artifacts on +> the NAS: that case can no longer arise. +> +> Evidence: `audits/R108-network-app-namespace-2026-07-30.md`. The pre-fix analysis below is retained +> verbatim as the record of what was wrong. + +#### 10.1 (historical) The exposure as it stood before v0.187.0 **[FACT] The verification and its result.** Every surface that can read a file was checked: @@ -477,7 +519,7 @@ Every divergence between the model above and the system as it is, each with an I | SMB browse (folder picker) | **NO** | same deny set applied per child (`sharing_handlers.go:521-537`) | | SMB `ensureImportShare` (the store-direct bypass) | **NO** | writes one controller-generated constant, `GetImportRoot()` = `/userdata/import` (`sharing_handlers.go:564-587`) | | FileBrowser — **local drives** | **NO** | the bind is `appbackup.UserdataDir(sp.Path)` only, and the comment says why (`internal/web/handlers.go:2450-2460`) | -| **FileBrowser — network shares** | **YES** | the bind is the share **ROOT**: `- %s:/srv/%s:rslave` (`handlers.go:2432`), and the path joins the config source list (`:2433`) | +| **FileBrowser — network shares** | ~~**YES**~~ → **NO** (R-108, v0.187.0) | the bind is still the share **ROOT** (`- %s:/srv/%s:rslave`, `handlers.go:2437`) and deliberately so — but **no app namespace, hence no `backups/` tree, can exist on a share**, so the root bind reaches only the customer's own files. The reachability is closed at the PLACEMENT, not at the bind | | `.fab` import path validation | **NO** | confined to `/exports` (`handler_export.go:400-408`, `estimate.go:215-217`) | | `.fab` browser download | **NO** | name-pattern + parent-must-be-the-staging-dir double guard (`handler_export_download.go:36-45,120-140`) | | `/api/debug/*` | **NO** | no file-serving branch (`handler_debug.go:46-92`) | @@ -514,6 +556,15 @@ Every divergence between the model above and the system as it is, each with an I (`recovery_unit.go:73`). **D5 would make it one.** That is exactly the test §2 set, and D5 therefore does **not** hold as written. → **R-108** +> **CLOSED (v0.187.0).** Links 2, 3 and 4 of the chain above are now guarded, and a **fifth** surface +> the chain did not list was found and guarded too: `handleStorageDecommission` mode=`migrate` checked +> only `req.Where` (the SOURCE) via `refuseNetworkLifecycle`, so a whole namespace could be +> decommissioned ONTO a NAS. Link 2's framing also understated the problem — the deploy **dropdown** is +> only a UI list; the boundary is the deploy **POST** (`internal/api/router.go`), which accepts any +> caller-supplied `HDD_PATH` and whose only other validation is `os.Stat` existence +> (`internal/stacks/deploy.go`). Filtering the list alone would have left the surface open. Links 5 and +> 6 are unchanged and still true — they simply can no longer be reached. + ### 10.2 The gap register | ID | Gap | Consequence | @@ -524,7 +575,8 @@ does **not** hold as written. → **R-108** | **R-105** | Three hub-held DR records are empty on the whole live fleet: `hosts.dr_record_json`, `host_escrow.directive_json`, `dr_recipe.host_half.drives` | the Recipe (§4) is incomplete in exactly the fields host-loss recovery reads. Causes may differ per field | | **R-106** | `dr_recipe.host_half.pbs.namespace` records `"root"` on every box | the recorded restore coordinate is wrong; real namespaces are per-customer | | **R-107** | No offsite action unpacks the named-volume tars Tier-3 captures on every run | offsite alone cannot rebuild a named-volume app (§7.2) | -| **R-108** | Network storage can host an app's namespace, and FileBrowser binds a network share at its **root** | **blocks D5** (§10.1); today it also lets a `.fab` with plaintext secrets be exported to a NAS (§5) | +| ~~**R-108**~~ | ~~Network storage can host an app's namespace~~ | **CLOSED 2026-07-30, controller v0.187.0 — D5 UNBLOCKED.** An app namespace may no longer be placed on network storage (5 surfaces guarded by one fail-closed predicate); the share-root bind is deliberately UNCHANGED because it is load-bearing and unscopable (§10.1). `audits/R108-network-app-namespace-2026-07-30.md` | +| **R-126** | A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS: `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | split out of R-108, which closed without it. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (§5, §7.3) | | R-95 (open) | The restic offsite credential **can delete** — the box can `forget --prune` its own repo | the tier holding the customer's documents and photos is the one whose credential can destroy it (matrix row 10) | | R-86 (open) | Restore-tests are interval-scheduled, not backup-aligned | a tier's proof cadence is unrelated to when its archives are written | | R-87 (open) | The restic tier is never restore-tested | matrix row 4's route has no unattended proof | @@ -567,8 +619,11 @@ mountpoints) are both on `/dev/sda3` → VG `pve`. The tier therefore protects a and operator error only**, never against disk failure. Accept and name it honestly in the customer- facing description, or move the target. -**F (added by §10.1, not in the original list).** D5 cannot be adopted until R-108 closes. Is -closing R-108 the intended path, or is D5 withdrawn? +**F (added by §10.1, not in the original list). — ANSWERED 2026-07-30.** D5 could not be adopted +until R-108 closed. Closing R-108 *was* the intended path, and it is done (controller v0.187.0): the +operator ruled to refuse app namespaces on network storage rather than narrow the browsing surface, +because the share-root bind is load-bearing and cannot be scoped. **D5 is no longer blocked.** Whether +to now *implement* D5 remains an open scheduling decision, not a blocked one. --- diff --git a/documentation/audits/R108-network-app-namespace-2026-07-30.md b/documentation/audits/R108-network-app-namespace-2026-07-30.md new file mode 100644 index 0000000..5f58c24 --- /dev/null +++ b/documentation/audits/R108-network-app-namespace-2026-07-30.md @@ -0,0 +1,251 @@ +# AUDIT — R-108: network storage may not host an app's data namespace (2026-07-30) + +Shipped: **controller v0.187.0** (`felhom-controller` `2f27a36`), live on **both** demo boxes. +**D5's precondition is met.** + +--- + +## 1. Baselines, established separately + +| component | value | how | +|---|---|---| +| controller on `main` | **0.186.0** → shipped 0.187.0 | top of `CHANGELOG.md` | +| controller on demo-felhom 9201 | **0.186.0** (pre-fix — used as the BEFORE) | `docker ps --filter name=felhom-controller` | +| controller on demo-hp 9201 | **0.185.1** (was drifted one release behind) | same | +| **golden-baked controller** | **0.186.0** | `hub_settings.artifact_golden_version` | +| agent (both demo boxes) | **0.118.1** | `hosts.agent_version`, hub-side | +| hub | **0.83.0** | `deploy/hub` image tag | +| `felhom.eu` HEAD | `70f8494` | — | + +**R-120's gate holds:** golden `0.186.0` == newest fleet-reported controller `0.186.0` at session start, so +`compareVersions(golden, newest) == 0`, not `< 0`. **It will now REFUSE** until the golden is re-baked to +0.187.0 — deliberate and correct: the fleet has moved past the golden, which is exactly the drift the gate +exists to catch. **No golden re-bake was performed** (out of scope; not required for this fix). + +**R-121 (agent-on-box drift): CLOSED for the demo fleet.** Both boxes report 0.118.1 and the manifest +vouches 0.118.1 — no drift. `sess-f` (0.116.0) and `drill-r50` (0.113.0) remain behind and were not touched. + +--- + +## 2. §2 citation reconfirmation — all six hold, line numbers moved + +| register citation | now at | verdict | +|---|---|---| +| `handlers.go:2450-2460` local userdata-scoped | `:2464-2465` | **holds** | +| `handlers.go:2432` network share-ROOT, `download: true` | bind `:2437`; `download: true` at `internal/infra/infra.go:328` | **holds** | +| `settings.go:904-914` no `IsNetwork()` filter | `GetSchedulableStoragePaths`, `:926-936` | **holds** | +| `handlers.go:674-679` migrate targets unfiltered | `:673-678` | **holds** | +| `storage_handlers.go:410-424` migrate-app unguarded | `:416-425` | **holds** | +| `storage_handlers.go:397` sibling refuses — the precedent | `handleStorageMigrate`, `:404` — refuses BOTH source and target | **holds, and is stronger than cited** | + +--- + +## 3. The §3.1 ruling — REFUSE the placement, KEEP the browse bind + +**"Scope the bind" is not implementable.** Establishing that is what inverted the fix: + +1. **The share-root `:rslave` bind is load-bearing, not an oversight.** Its own doc comment records a + Phase-0 probe (2026-07-22) proving an in-container access through an `rslave` bind wakes the idle + automount trigger. Narrow it and NAS access itself breaks. +2. **There is no `userdata/` layer to scope to.** `deployedAppsOnPath` + (`internal/web/netstorage_handlers.go`) documents that *apps on a share store `/`* — + the local `/userdata|appdata|backups` convention does not apply to shares at all. +3. **Creating one is forbidden:** *"Never a skeleton, never any write toward the NAS — Felhom conventions + must not be written onto a customer's own NAS."* + +The browsing surface being **immovable** is precisely why the backup tree must never be placed under it. +Tier 2 had already reached that conclusion for its own targets (`F-6C-1`, `internal/backup/tier2.go`); +R-108 closes the PRIMARY namespace, the last remaining route. + +**This needed an operator decision** (it removes a capability going forward) and got one. Evidence supplied +with the question: **zero apps on network storage across all six hub customers including Peti** — verified +from every stored DR-recipe app half; only `hdd_1` and `nvme-1tb`, both local drives. demo-hp's +`Felhom-Share` holds only the customer's own files (`.DS_Store`, `test file.txt`, `teszt/`) and **no +`backups/`**. Nothing stranded, no migration owed. + +Two capabilities were separable and only one was removed: +- **browse a NAS through FileBrowser** (R-67) — **KEPT, byte-identical.** +- **host an app's data namespace on a NAS** — **REFUSED.** + +--- + +## 4. The exposure chain, confirmed at source + +`namespaceRoot(drivePath)` returns any non-system drive path AS-IS +(`internal/backup/backup.go:262-263`), so an app's namespace root **is** its `HDD_PATH`; the recovery unit +is `/backups/primary//` (`internal/appbackup/paths.go`). On a NAS that lands inside the +share, which FileBrowser binds whole with `download: true`. + +**This was documented, deliberate behaviour** — not an accident. `controller/README.md` carried *"NAS +backup locality (v0.118.0, CAMPAIGN-3 Part 4 — decision A): a NAS-resident app's tier-1 artifacts live on +the NAS itself"*. R-108 supersedes decision A; the README now says so, and the NAS-outage window that +block warned about is gone with it. + +--- + +## 5. §3.2 — the surface set is FIVE, not four + +| # | surface | state before | in the register? | +|---|---|---|---| +| 1 | **deploy POST** (`internal/api/router.go`) | accepted any caller-supplied `HDD_PATH`; only other validation is `os.Stat` existence (`internal/stacks/deploy.go`) | **understated** — the row says "the dropdown has no filter", but the dropdown is a UI list; this is the boundary | +| 2 | per-app migrate target list (`internal/web/handlers.go`) | unfiltered | yes | +| 3 | `handleStorageMigrateApp` | no guard | yes | +| 4 | **`handleStorageDecommission` mode=`migrate`, the TARGET** | `refuseNetworkLifecycle` guarded only `req.Where` (the SOURCE) — a whole namespace could be decommissioned ONTO a NAS | **NO — found by enumerating** | +| 5 | the FileBrowser bind | share ROOT + `download: true` | yes — deliberately unchanged, now pinned by a test | + +Also found and **filed, not fixed**: `storageDriveList()` (`internal/web/handler_export.go`) does not +filter network paths, so a `.fab` bundle (plaintext secrets, optional password) can be **exported onto** a +NAS → **R-126**. The register already noted this as R-108's "second effect, independent of D5"; it is now +its own row so it did not vanish with R-108's closure. It is **not** a D5 precondition: an explicit +customer-chosen export destination is not a browsing surface reaching a backup tree. + +**No gap was found in the §10.1 swept set.** Every other row in that table was re-read and still holds. + +--- + +## 6. Fail-closed, and why it needed a function rather than an `IsNetwork()` call + +`/mnt/felhom-drives` holds **both** storage kinds in-guest — `.../nvme-1tb` is a local drive, +`.../Felhom-Share` is a NAS — so a **path prefix cannot classify**. `Kind` is the only discriminator and it +exists only on a **registered** path. Therefore an unregistered path under that root is un-classifiable, +and un-classifiable **refuses**. Every share is registered under that root by construction, so the network +set is completely covered without touching drives. Empty `HDD_PATH` (SSD-resident) stays allowed; a nil +registry refuses. + +--- + +## 7. Live evidence — same box, before and after, through the real endpoint + +**Method:** `POST /api/storage/migrate-app` — the exact endpoint the storage UI's migrate button invokes — +authenticated with the session cookie and the `X-CSRF-Token` the page carries, against the controller's +container IP with the mandatory `Host` header. No browser exists on DooPlex; this is the endpoint-level +method, and no server logic is skipped. The app name is deliberately **non-existent**, so the call is safe +on both code paths (pre-fix it fails on the app name; post-fix it is refused earlier). + +### demo-felhom — the same box across the upgrade + +``` +BEFORE (controller 0.186.0) the target was NEVER EXAMINED + target=/mnt/felhom-drives/Felhom-Share -> 409 {"error":"alkalmazás nem található: r108-probe-nonexistent"} + target=/mnt/felhom-drives/not-registered -> 409 {"error":"alkalmazás nem található: r108-probe-nonexistent"} + +AFTER (controller 0.187.0) refused at the target + target=/mnt/felhom-drives/Felhom-Share -> 400 "A megadott tárhely nem azonosítható regisztrált + meghajtóként, ezért alkalmazás adatkönyvtáraként nem + használható. Válasszon a listából csatlakoztatott meghajtót." + target=/mnt/felhom-drives/not-registered -> 400 (same — fail-closed) + target=/mnt/felhom-drives/hdd_1 -> 409 {"error":"alkalmazás nem található"} <- guard PASSED THROUGH +``` + +Pre-fix, both paths sailed into `MigrateApp` and failed only on the app name — **with a real app name the +migration would have started**. That is the defect, live. Post-fix both are refused, and the real local +drive still reaches `MigrateApp` (**409, a different code and message**), which is the not-over-broad proof. + +### demo-hp — the box with a REGISTERED network share + +``` +target=/mnt/felhom-drives/Felhom-Share -> 400 "Hálózati tárhelyen (NAS) nem futtatható alkalmazás + adatkönyvtára — a NAS megosztás tallózásra és + médiatárolásra használható. Válasszon csatlakoztatott + meghajtót." +target=/mnt/felhom-drives/not-registered -> 400 (undeterminable reason — fail-closed) +target=/mnt/felhom-drives/nvme-1tb -> 409 app not found <- guard PASSED THROUGH +``` + +The **network-specific** reason fires here because the share is registered with `kind=network`; on +demo-felhom the same path is unregistered and correctly gets the fail-closed reason instead. The two +messages differing per box is the predicate demonstrably reading real state, not a constant. + +### Non-effect — the state after the refused calls (demo-hp) + +``` +storage registry: + path=/mnt/felhom-drives/nvme-1tb kind=drive migrated_to=- decommissioned=False + path=/mnt/felhom-drives/Felhom-Share kind=network migrated_to=- decommissioned=False +app HDD_PATH: /mnt/felhom-drives/nvme-1tb (unchanged) +share backups/: ls: cannot access '.../Felhom-Share/backups': No such file or directory +``` + +No `migrated_to`, nothing decommissioned, no job id or `started` flag returned, `HDD_PATH` unchanged, and +no `backups/` tree on the share. + +### R-67 preserved — the FileBrowser compose, diffed across the deploy + +``` +- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb <- drive: userdata-scoped +- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave <- network: ROOT (deliberate) +- /mnt/sys_drive/felhom-data/userdata/import:/srv/beolvasas +``` + +`diff` of demo-hp's generated compose before vs after 0.187.0: **IDENTICAL**. The capability the ruling +preserves is preserved byte-for-byte, not merely "probably unaffected". + +--- + +## 8. Tests + +**9 new**, all asserting the **non-effect**. The refusal tests run against a Server with a deliberately +**nil `stackMgr`**: a guard that fails to fire reaches the mutation and **panics** rather than passing +quietly. Asserted: no job id, no `started` flag, no `MigratedTo`, and for decommission that the source was +**not** soft-marked. Fixtures are demo-hp's real two-class storage set (both paths under the same mount +root — the trap). + +**Seams named (R-125).** The FileBrowser tests inject at `fbPathDeps` (`isMount`/`classify`/ +`ensureSkeleton`) and run the real `buildFileBrowserPaths`; the bind-string construction — what the +assertion is about — is **not** injected. **Not covered by that span:** `RenderFileBrowserConfig` and the +compose template downstream, closed by the live compose diff in §7. The handler tests inject **nothing** — +real handler, real settings store, `httptest`. + +**Red-proofs: 4**, each mutation asserted to have landed *before* the run: + +| # | mutation | caught by | +|---|---|---| +| 1 | drop the migrate-app target guard | panic on nil `stackMgr` — execution reached the mutation | +| 2 | drop the decommission migrate-target guard | panic, same shape | +| 3 | let an unregistered path under the mount root fall open | 3 tests, incl. the live-shaped handler case | +| 4 | userdata-scope the share (break R-67) | the regression guard, quoting the broken bind | + +Suite **rc=0, 27 packages, 0 FAIL** (run separately from every commit); `go vet` rc=0; +`template_id_gate.py` and `emoji_gate.py` both OK. + +--- + +## 9. §7 — the sentence this task exists to produce + +**D5's precondition IS met.** An app's data namespace can no longer be placed on network storage, so no +`backups/` tree can exist inside FileBrowser's share-root bind; every other read surface was already clear +in the §10.1 sweep. **Nothing in `07-backup-architecture.md` §10.1 blocks D5 any more.** + +What remains is scheduling, not blocking. `.fab`-export-onto-NAS (**R-126**) is open but is not a D5 +precondition — it is an export destination the customer chooses, and D5 moves secrets into the local +recovery unit, not into `.fab`. + +--- + +## 10. Four-artifact update + +| artifact | what changed | +|---|---| +| `architecture/07-backup-architecture.md` | §10.1 retitled **CLOSED**, with the ruling, the reasoning and the D5 sentence; the network-share table row flipped YES→NO *with the reason it is closed at the placement rather than the bind*; the exposure chain annotated with the fifth surface and the deploy-POST correction; §7.3 retitled **UNBLOCKED**; §10.2 register row collapsed + R-126 added; open question **F answered** | +| `architecture/00-capability-map.md` | new §D row, **PROVEN-LIVE**, citing this audit — with the un-exercised legs named explicitly (deploy POST + decommission refusals are unit-tested, not live-fired) | +| `backlog/ROADMAP.md` | R-108 collapsed to a shipped one-liner; **R-126** added | +| `backlog/OPEN-ITEMS.md` | R-108 dispositioned; **D5 given its own row** as READY/UNBLOCKED (it had existed only inside other rows' prose — the R-123 pattern); **R-126** registered | + +--- + +## 11. What did NOT happen, and why + +- **D5 not implemented** — explicitly fenced; this unblocks it only. +- **No customer data migrated** — none needed it (zero apps on network storage). +- **The share-root bind not narrowed** — deliberate, and §3 is the argument. +- **R-126 filed, not fixed** — it is a separate surface with its own decision (filter the destination, or + force the bundle password), and folding it in would have widened a security fix mid-flight. +- **No golden re-bake** — out of scope. **Consequence to be aware of:** the golden bakes 0.186.0 while the + fleet now runs 0.187.0, so R-120's vouch gate will REFUSE the next artifact vouch until the golden is + re-baked. That is the gate working as designed, not a regression. +- **`sess-f` and `drill-r50` not upgraded** — neither was named as a venue; `drill-r50` is fenced. +- **No live-fire of the deploy-POST or decommission refusals** — both are unit-tested with non-effect + assertions; live-firing them needs a real app deploy and a real decommission on a box with a registered + share, which is a destructive drill this task did not authorise. Named in the capability-map row. +- **Nothing written to demo-hp's `local-lvm`**; `drill-r50`, guest 9201 on other hosts and Peti's cluster + untouched; neither demo box re-targeted; no `AttachDrive` call. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 40e2cfa..e453171 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -47,7 +47,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **C9-F3** → **R-104** | An **interrupted offsite run leaves an exclusive restic lock the self-heal cannot reach**: `resticStep` (`offbox.go:634-648`) has `unlock --remove-all`, but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`offbox.go:77-93`) has no lock case → `"other"` → fail-fast. Tier dead until a human unlocks; `ClassifyOffsiteFailure` likewise has no lock case so the operator is told **„A távoli mentés ismeretlen okból nem sikerült"** for a precisely-known, self-healable condition | **READY (MEDIUM)** | — | Add a lock case to both classifiers and let the probe path escalate to `unlock --remove-all`. Answers Phase C item 8: the repo is NOT usable after a killed run. Cleared manually this run; tier proven working again (`ok`, 1m35s). Reachable by any interruption — container restart, OOM, **host reboot mid-backup** | CC | | **C9-F1b** → **R-103** | Tier-2's restore cannot cover 43 of 53 apps; the action that CAN is the keep-side unit restore (`POST /backup/restore` → `RestoreFromRecoveryUnit`, replays volume tars + DB dumps). v0.183.0 NAMES it in the refusal text but does not route to it | **READY** | — | Put the working action in the card the customer already opened. **Deliberately its own task:** it places a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference — the reason it was not folded into v0.183.0 | CC | | **C9-F4** → **R-102** | **Nothing reads the Tier-2 copy's `recovery-unit/` mirror.** It is written by EVERY Tier-2 run (`tier2.go:369`, „Unit leg (always)") and read by no code path: `RecoveryUnitPath` resolves to `backups/**primary**/` (`appbackup/paths.go:46-48`), and the only reader of the secondary tree is `tier2_restore.go:79`, which reads `hdd/`+`userdata/` only | **READY (potentially > C9-F1)** | — | Tier-2 exists for the case where the PRIMARY drive is lost — and in exactly that case the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action, leaving offsite as the only route. Verified by enumeration: 6 references to `"secondary"` in the tree, one writer, one reader, one wipe-warning lister | CC | -| **R-108** | **Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT.** Local drives are `userdata`-scoped (`web/handlers.go:2450-2460`); network paths are bound at the share root (`:2432`) and served with `download: true`. No `IsNetwork()` filter guards the deploy dropdown (`settings.go:904-914`), the per-app migrate targets (`handlers.go:674-679`), or `handleStorageMigrateApp` (`storage_handlers.go:410-424` — its whole-namespace sibling DOES refuse, `:397`) | **READY — BLOCKS an architectural target** | — | **This is why D5 was not adopted** in the 2026-07-28 `07-backup-architecture.md` rewrite: D5 moves app secrets into the local recovery unit so Tier-1/Tier-2 restore stop needing the guest, and that is safe only if no browsing surface can reach the backup tree. Every other surface was verified clean (SMB both namespace shapes, FileBrowser for drives, `.fab` import + download, `/api/debug/*`, all three `ServeFile` sites, storage-path add) — `07` §10.1 has the full sweep. **Not a leak today** (the unit's `app.yaml` is secret-stripped). Verified LIVE in demo-hp's generated FileBrowser compose | CC | +| **R-108** | ~~Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT~~ | **SHIPPED + PROVEN-LIVE** (controller v0.187.0, 2026-07-30) | — | **D5 IS UNBLOCKED — that is this item's whole point.** `RefuseAsAppNamespace` (`internal/settings/settings.go`) is one fail-closed predicate consulted by every placement surface, so no `backups/` tree can exist inside the share-root bind. **The fix inverted the obvious one:** the bind was NOT narrowed, because it cannot be — the `:rslave` share-ROOT bind is load-bearing (a 2026-07-22 probe proved an in-container access through it wakes the idle automount trigger), there is no `userdata/` layer to scope to (apps on a share store at `/`), and creating one would write Felhom convention onto a customer's own NAS, which R-67 forbids. Operator ruling: refuse the placement, keep the browse bind. **FIVE surfaces, not the four this row named** — `handleStorageDecommission` mode=migrate guarded only its SOURCE, so a whole namespace could be decommissioned ONTO a NAS; and the *boundary* is the deploy **POST**, not the dropdown (it takes any caller-supplied `HDD_PATH`, validated only by `os.Stat`). **Nothing stranded:** zero apps on network storage across all six customers incl. Peti. R-67's bind verified byte-identical before/after. Supersedes the controller README's "NAS backup locality — decision A". `audits/R108-network-app-namespace-2026-07-30.md` | — | +| **D5** | **Move app secrets into the LOCAL recovery unit** so Tier-1/Tier-2 restore stop needing the guest and stop needing R (R would then be required for offsite recovery + host identity only) | **READY — UNBLOCKED 2026-07-30** (was blocked by R-108) | — | **The precondition is now met and stated in `07-backup-architecture.md` §7.3 + §10.1:** no browsing surface can reach the backup tree on any storage class. Every other read surface was already clear in the §10.1 sweep (SMB both namespace shapes, FileBrowser for drives, `.fab` import + download, `/api/debug/*`, all three `ServeFile` sites, storage-path add); the network-share bind was the one exception and R-108 closed it at the PLACEMENT rather than at the bind. **Registered as its own row 2026-07-30** — it previously existed only inside R-108's and E-2's prose, which is the R-123 thread-loss pattern. **NOT started:** unblocking is not implementing, and the task that closed R-108 was explicitly fenced from starting D5. Scope note: `.fab` bundles carry plaintext secrets independently of D5 (→ **R-126**) and are not part of its precondition | CC | +| **R-126** | **A `.fab` bundle — plaintext secrets, OPTIONAL password — can be exported ONTO a NAS.** `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | READY (S) | — | Split out of R-108, which closed without it: this is an explicit customer-chosen **export destination**, not a browsing surface reaching a backup tree, so it was never part of D5's precondition (`07` §7.3 records that reasoning). Was recorded inside R-108's row as its "second effect, independent of D5"; promoted to its own row so it does not vanish with R-108's closure. Fix = filter network paths out of the export destination list, or require the bundle password when the destination is a share | CC | | **F-DIAG** | ~~Four distinct offsite failure causes collapse into two operator-visible strings~~ | **SHIPPED** (controller v0.182.0, 2026-07-28) | — | `ClassifyOffsiteFailure` → quota / orphaned / no_repo / no_units / transport / **unknown**, each with its own Hungarian message. Unclassifiable says so rather than being folded into a neighbour. **Secrets:** the old message was a raw `err.Error()` passthrough carrying `sftp:@:`; redaction is now by the target's **actual** host/user/path (a first regex-only attempt leaked on a bare hostname and its own test caught it). Unit-proven; **not** yet exercised by a live offsite failure of each class | — | | **F-OPS** | ~~A manual `pct restore` inherits the source guest's bind mounts — during a real DR, on a different host, under pressure~~ | **DOCUMENTED** (2026-07-28) | — | `documentation/runbooks/RUNBOOK-manual-guest-restore.md`: which `mpN` are volumes vs host binds, the `mp9` source-VMID trap (it can bind **another guest's bootstrap credentials**), strip-and-re-add before first boot, and a positive pre-start verification. Docs only by design — the agent already neutralises binds on its own restore paths, and a second implementation would drift | — | | **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index d14f49c..74ff556 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -108,7 +108,8 @@ | R-105 | **Three hub-held DR records are empty on the entire live fleet.** `hosts.dr_record_json` = `{}` on all 3 hosts; `host_escrow.directive_json` = `{}` on both escrowed hosts; `dr_recipe.host_half.drives` = `[]` on every customer **including two with enrolled data drives** (916 GB USB on demo-felhom, 938 GB NVMe on demo-hp) | M | READY — 2026-07-28 | These are exactly the fields a host-loss recovery reads: `05-hub-architecture.md:175-176,186` names the slim DR record as one of four durable sources; `06-offsite-connectivity.md:148-150` says the escrow upload carried the DR directive; `felhom-agent/internal/dr/plan.go:34-35` makes `PlannedDrive` the re-attach-by-`durable_id` wrong-disk guard. **The three may have different causes** — `isUserDataDrive` (`internal/hub/dr_recipe.go:129-136`) requires type `usb`/`local-dir` **and** a non-empty `DurableID` **and** `MountPath`, and which of the three fails was not traced. Evidence: `architecture/_recovery-inventory-2026-07-28.md` Part D2.3. **UPDATE 2026-07-28 (vzdump-target move): the `drives` third is TRACED and now POPULATED on both demo boxes.** Cause: the enrolled data drives were never PVE storages at all — only agent-generated systemd mounts — so they never entered `report.StorageTargets` and `isUserDataDrive` never saw them. Giving each drive a `dir` storage at its own mountpoint supplied all three required fields at once (type `local-dir`, fs-UUID durable id, mount path), and the recipe now emits `uuid:91d2dc2d-…`/`/mnt/nvme-1tb` on demo-hp and `uuid:47a3361a-…`/`/mnt/hdd_1` on demo-felhom. **The other two fields (`hosts.dr_record_json`, `host_escrow.directive_json`) are untouched and still `{}`** — this narrows R-105, it does not close it. See `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §5.4. Flips: `07` §4 | | R-106 | ~~The DR recipe records the PBS namespace as `"root"` on every box~~ | XS | **SHIPPED** — agent v0.118.1, 2026-07-30 | Namespace resolves from the pbs STORAGE (storage.cfg `namespace`) — the field `vzdump --storage ` makes PVE read — not from the listed snapshot, whose `ns` PBS omits once the list is `?ns=`-scoped. `namespace_state` distinguishes a genuine root namespace from not knowing. Live on both boxes. Follow-on: **R-124** (the `"root"` spelling is not `--ns`-pasteable). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 | | R-107 | **No offsite action unpacks the named-volume tars Tier-3 captures on every run.** `ReconstituteFromOffsite` skips the unit outright (`internal/backup/offbox_reconstitute.go:284-289`); `PlaceOffsiteRestore` places it only when the live unit is ABSENT (`internal/backup/offbox_restore.go:352-356`). The DB **is** replayed, from the scratch unit's `db-dumps/` | M | READY — 2026-07-28 | 52 of 53 catalog templates keep data in Docker named volumes, so for most apps the offsite tier carries the volume data and cannot itself put it back. A two-step route exists (full restore → place → Tier-1 restore, which does unpack tars) but **no single action does it and no UI routes it** — the R-103 shape one tier over. Sibling of R-102: both are "captured and never read". Flips: the offsite-restore row in map §C; `07` §6.3, §7.2 | -| R-108 | **Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT — this BLOCKS D5.** For local drives FileBrowser binds `/userdata` only (`internal/web/handlers.go:2450-2460`); for network paths it binds the share **root** with `:rslave` (`:2432`) and serves it with `download: true` (`internal/infra/infra.go:326`). Nothing stops an app's namespace landing there: `GetSchedulableStoragePaths()` has no `IsNetwork()` filter (`internal/settings/settings.go:904-914`, feeds the deploy dropdown at `handlers.go:462-473`), the per-app migrate target list has none (`handlers.go:674-679`), and `handleStorageMigrateApp` does not call `refuseNetworkLifecycle` though its whole-namespace sibling does (`internal/web/storage_handlers.go:397` vs `:410-424`) | M | **READY — blocks an architectural target** — 2026-07-28 | **Today this is not a secret leak** — the recovery unit's `app.yaml` is secret-stripped (`internal/backup/recovery_unit.go:73`). It becomes one under **D5**, which would put app secrets in the local unit precisely so Tier-1/Tier-2 restore stop needing the guest. Verified LIVE in demo-hp's generated compose, where the asymmetry is visible rather than inferred: `- /mnt/felhom-drives/nvme-1tb/userdata:/srv/nvme-1tb` beside `- /mnt/felhom-drives/Felhom-Share:/srv/Felhom-Share:rslave`. **Second effect, independent of D5:** `.fab` bundles already carry **plaintext** secrets by design with an OPTIONAL password (`internal/appexport/export.go:484,506-511,307`) and `storageDriveList()` does not filter network paths (`internal/web/handler_export.go:377-387`), so one can be exported onto a NAS today. Full sweep of every read surface: `07-backup-architecture.md` §10.1. Flips: `07` §7.3 (D5 target → adopted) once closed | +| R-108 | ~~Network storage can host an app's namespace, and FileBrowser binds a network share at its ROOT — this BLOCKS D5~~ | M | **SHIPPED** — controller v0.187.0, 2026-07-30 | **D5 UNBLOCKED.** One fail-closed predicate (`Settings.RefuseAsAppNamespace`) consulted by every placement surface, so no `backups/` tree can exist inside the share-root bind. The bind was deliberately NOT narrowed — it is load-bearing (`:rslave` automount wake, 2026-07-22 probe) and unscopable (apps on a share store at `/`; creating a `userdata/` layer would write Felhom convention onto a customer's NAS). **Five surfaces, not the four this row named** — decommission-with-migrate guarded only its SOURCE; and the boundary is the deploy POST, not the dropdown. Zero apps on network storage fleet-wide, so nothing stranded. The `.fab`-onto-NAS "second effect" is split out as **R-126**. Flipped: `07` §7.3 (D5 → unblocked), §10.1 (CLOSED), `00-capability-map.md` §D (new PROVEN-LIVE row). `audits/R108-network-app-namespace-2026-07-30.md` | +| R-126 | **A `.fab` bundle — plaintext secrets, optional password — can be exported ONTO a NAS.** `storageDriveList()` (`internal/web/handler_export.go`) does not filter network paths | S | READY — 2026-07-30 | Split out of R-108 on its closure. NOT a D5 precondition: an explicit customer-chosen export destination, not a browsing surface reaching a backup tree (`07` §7.3 records the reasoning). Fix = filter network paths from the export destination list, or force the bundle password when the destination is a share. Flips: `07` §5 | | R-109 | ~~The DR recipe records no backup target~~ | XS | **SHIPPED** — agent v0.118.1 + hub v0.83.0, 2026-07-30 | `backup_target` resolves from the PRIMARY tier of `cfg.Backup.BackupTiers()` (the scheduler's own function) + the mountpoint that separates `/mnt/hdd_1` from `/var/lib/vz`; unresolvable is recorded as unresolvable, never defaulted. Required an unscoped HUB half: `AssembleDRRecipe` allow-lists top-level keys → **R-122**, a fourth recipe-completeness defect that had already shipped (`offsite_restic` dropped for the feature's whole life). `audits/R106-R109-recipe-completeness-2026-07-30.md`; flipped `07` §4 | | E-2 | **Drive-role machinery around the moved vzdump target.** The 2026-07-28 runbook proved the architecture change by hand on both demo boxes; this is the machinery: a **backup-target role** on `StoragePath` beside `Schedulable`/`IsDefault`/`Kind`; **assignment in the storage wizard** (suggest by attribute, refuse the absurd, never decide by transport or `removable` — on the reference hardware demo-felhom's target IS a USB HDD and BOTH drives report `removable=0`); **unassigned drives do nothing automatically**; **stickiness** (never silently retarget); `felhom-host-install.sh` creating the target with `--is_mountpoint 1` **and** issuing the `FelhomAgentStore` ACL; **absent-target policy**; **retention/space accounting** on a drive the customer shares; the honest **single-drive label**; remaining fleet migration | M | READY — 2026-07-28 | Full scope + rationale in `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §7. Two traps already paid for live: the storage `path` must BE the mountpoint or the agent reports the target `disconnected` forever (`internal/storage/observe.go:321`), and the per-storage `FelhomAgentStore` grant is mandatory or every backup 403s. Absent-drive behaviour today is **fail-loudly, no silent retarget** (`is_mountpoint 1` proven live) — which is NOT the intended fall-back-and-alarm design. Flips: matrix row 4 |