diff --git a/REPORT.md b/REPORT.md index 5242542..bfce613 100644 --- a/REPORT.md +++ b/REPORT.md @@ -4,6 +4,31 @@ --- +# REPORT — B3: live unattended storage-lifecycle exercise + feature sweep (2026-06-15) + +## Type +Live, unattended, full-UI exercise of migration / decommission / re-enroll / eject + a broad feature sweep on shipped **controller v0.66.2 / agent v0.32.0** (demo guest 9201). No production code changed. Full doc: [documentation/audits/B3-storage-lifecycle-2026-06-15.md](documentation/audits/B3-storage-lifecycle-2026-06-15.md). + +## Result +Migration engine is **solid** (every move checksum-verified identical, sources cleaned, HDD_PATH flipped, apps healthy). The decommission/eject/re-enroll **lifecycle has serious gaps**, incl. one critical guest-bricking bug and three unrouted endpoints the UI calls. End-state is sane: all apps healthy on **felhom-flash (now default)**, both data drives Aktív, sda/PBS/controller untouched. + +UI-driven per A4 where the dashboard exposes the action (per-app migrate + migrate-all driven via the real buttons, native `confirm()` auto-accepted); operations the UI lacks a button for / calls an unrouted endpoint for were driven via the controller's own JSON API in-guest (correct `Host` header, bypassing the body-mangling proxy) — each gap documented as a finding. + +## Findings (→ next-slice fixes) +- 🔴 **C1 (critical):** decommission/eject never `pct set --delete mpN`, so the stale LXC bind **bricks the guest on next reboot** (pre-start hook exit 255, ALL apps down). Recovered live via manual `pct set --delete mp1`. +- 🟠 **H1:** `/api/storage/disconnect`, `/api/storage/reconnect`, `/api/storage/restart-apps` are **404 (unrouted)** though settings.js calls them. +- 🟠 **H2:** no UI button for `/api/storage/decommission` (B2b flow unreachable from the dashboard). +- 🟠 **H3:** no one-click re-enroll for a decommissioned drive (only "Eltávolítás"). +- 🟡 **M1:** decommissioning the default drive **doesn't reassign the default** (orphaned, no failover). +- 🟡 **M2:** eject does **NOT auto-remount** on reconnect — intent-gated-skipped like decommission (premise contradicted); difference is bind-record retention + soft-marker. +- 🟡 **M3:** `userdata/import/calibre` created **755** at enroll (skeleton gap) and migrate-all doesn't re-assert setgid on pre-existing dirs (16/17 dirs preserved 2775). +- 🟡 **M4:** felhom-usb's systemd mount unit had a **stale UUID** (couldn't auto-mount); fixed in restore. **M6:** test-email fails (no SMTP). + +## Passes +Per-app migration, migrate-all (7 apps, checksums identical, setgid 16/17), decommission state (intent/unmount/prune/Kiváltva), no-auto-rebind invariant (intent-gated), missing-storage badges (8×), re-enroll marker-clear, encrypted backup→PBS (snapshot landed), FileBrowser umask (PID1 0002 → 2775 group-writable + collab write), fresh-app deploy+lifecycle+logs, monitoring page. + +--- + # REPORT — SPIKE: self-serve raw-drive plug-and-enroll (GATE) (2026-06-15) ## Type diff --git a/documentation/audits/B3-storage-lifecycle-2026-06-15.md b/documentation/audits/B3-storage-lifecycle-2026-06-15.md new file mode 100644 index 0000000..14de112 --- /dev/null +++ b/documentation/audits/B3-storage-lifecycle-2026-06-15.md @@ -0,0 +1,114 @@ +# B3 — Live unattended exercise of the storage lifecycle (migration / decommission / re-enroll / eject) + feature sweep + +**Date:** 2026-06-15 +**Target:** shipped **controller v0.66.2 + agent v0.32.0**, demo guest **9201** on **felhom-pve** (node `demo-felhom`). +**Mode:** live, unattended, all 8 phases run without pausing. No production code changed. +**Drive method:** the real operations were driven through the **dashboard UI via the browser tool** wherever the UI exposes them (per the A4 strict-UI requirement). Where the UI has **no button** or calls an **unrouted endpoint** (documented findings below), the operation was driven via the controller's own JSON API hit **in-guest at the container IP with the correct `Host` header** (bypassing the cloudflared/traefik proxy that mangles bodies) — never a raw shortcut around the controller's logic. Host-level captures/simulations used SSH (`root@felhom-pve`). +**Safety rails honoured:** `/dev/sda`, the felhom-controller container, the bootstrap unit, `felhom-pbs`, `local`/`local-lvm`, and guest 9201's root disk were never touched. All operations confined to the two user-data drives (felhom-usb `/dev/sdb1`, felhom-flash `/dev/sdc1`) + the demo apps. + +--- + +## VERDICT + +The migration engine is **solid** (checksums verified identical across every move, sources cleaned, HDD_PATH flipped, apps healthy). The **decommission / eject / re-enroll lifecycle has serious gaps**, including **one critical guest-bricking bug** and **three unrouted endpoints the UI calls**. The B3 migration target (felhom-flash) now holds all apps, healthy, and is the default drive. + +### 🔴 Critical (fix before any customer uses decommission) +- **C1 — Decommission bricks the guest on next reboot.** The agent decommission removes its bind-tracking JSON + host-unmounts, but **never runs `pct set --delete mpN`**, so the stale `mp` bind entry survives in the LXC config. On the next guest reboot the LXC **pre-start hook fails (exit 255)** because the bind source (`/mnt//felhom-data`) no longer exists → **the whole guest fails to start — ALL apps down, not just the decommissioned drive's.** Proven live (Phase 3): guest 9201 went down on reboot; only `pct set 9201 --delete mp1` + `pct start` recovered it. + +### 🟠 High (UI lifecycle largely non-functional) +- **H1 — Three endpoints the settings JS calls are unrouted → HTTP 404:** `/api/storage/disconnect`, `/api/storage/reconnect`, `/api/storage/restart-apps`. The "Leválasztás" (disconnect), "Csatlakoztatás" (reconnect), and "Alkalmazások indítása" (restart-apps) buttons are dead. +- **H2 — No UI trigger for `/api/storage/decommission`.** The B2b decommission flow (Phases 2 & 4 of this exercise) has **no dashboard button** — `settings.html` only exposes migrate-all (`/api/storage/migrate`), eject (`/api/storage/eject`), and remove. The routed `/api/storage/decommission` (mode=migrate|anyway) is reachable only by API. +- **H3 — No one-click re-enroll** for a decommissioned drive; its only UI action is "Eltávolítás a rendszerből" (remove). The Change-4 marker-clear fires only on re-register, which has no button on the decommissioned entry. + +### 🟡 Medium +- **M1 — Decommissioning the default drive does not reassign the default.** After decommissioning felhom-usb (the default), **no drive held the default badge** (both remaining drives showed "Legyen alapértelmezett") — the default was orphaned on the decommissioned drive, no failover. New HDD-app deploys would have had no valid default target. +- **M2 — Eject does NOT auto-remount on reconnect** (contradicts the expected eject≠decommission distinction). An ejected drive is intent-gated-skipped by `ReassertGuestBinds` exactly like a decommissioned one; its systemd unit is disabled and it stays unmounted. The real distinction is: eject **keeps** the bind record (reassert logs a skip line) + sets **no** soft-marker; decommission **removes** the bind record + sets the "Kiváltva" soft-marker. Neither auto-remounts, and the only "reconnect" path (`/api/storage/reconnect`) is 404 (H1). +- **M3 — `import/calibre` userdata dir loses setgid.** After migrate-all, `userdata/import/calibre` was **755** on the target while it was **2775** on the source (and 16/17 other dirs preserved 2775). Root cause: the **enroll-time userdata skeleton created `import/calibre` as 755** (mtime = enroll time), and the migrate-all merge-walk does not re-assert setgid on a pre-existing target dir (it correctly creates *new* dirs like `import/paperless` as 2775). calibre-web binds that dir — group-write/setgid is broken there. +- **M4 — felhom-usb's systemd mount unit had a stale UUID** (`277a2179…` vs the real `da9e7089…` from the 2026-06-14 reformat), so it could not auto-mount and a role-gated eject 403'd. Fixed in restore. +- **M5 — Decommission's host-unmount does not release the drive from the running guest.** The unprivileged LXC bind keeps `/dev/sdX1` live-mounted inside the guest until reboot, so "safely removable" is only true post-reboot. +- **M6 — Test notification email fails** (returns "Hiba"), likely no SMTP transport configured on the demo. + +--- + +## Per-phase results (PASS/FAIL + evidence) + +### Phase 0 — baseline + seed — PASS +- Baseline: 8 HDD apps on felhom-usb (audiobookshelf, calibre-web, immich, jellyfin, komga, paperless-ngx, radarr, romm); felhom-usb = default; flash = empty skeleton. +- Userdata ownership baseline: **all 17 dirs `2775` setgid, gid 101000** (= guest gid 1000). ✓ +- Seeded checksummed files (source of truth): + - `userdata/media/movies/b3-movie.txt` → `6155f6df…833e8` + - `userdata/documents/b3-document.txt` → `1f2bb9a5…3764` + - `userdata/media/photos/b3-photo.txt` → `162a7fa1…6e4f` + - `appdata/romm/config/b3-romm.txt` → `3643c7cc…60f1` + +### Phase 1 — per-app migration (romm, felhom-usb → felhom-flash) — PASS (UI-driven) +Driven via the **romm app page → "Áthelyezés másik tárhelyre" → felhom-flash → Áthelyezés** button (native `confirm()` auto-accepted to clear the blocking dialog; the real `appMigrate → POST /api/storage/migrate-app` flow ran). Job `mig-20260615-094757`. +- stop → copy → verify → flip → redeploy **healthy** ✓ (romm `Up … (healthy)`). +- Migrated appdata checksum on flash = `3643c7cc…60f1` — **identical** ✓. +- Source subtree `…/felhom-usb/…/appdata/romm` **removed** ✓. +- HDD_PATH **flipped**: `docker inspect romm` mounts now `/mnt/felhom-flash/...`; `app.yaml HDD_PATH: /mnt/felhom-flash` ✓. +- UI badge updated to **"Adattároló: felhom-flash"** ✓. + +### Phase 2 — migrate-all + decommission felhom-usb → felhom-flash — PASS (migrate via UI; decommission via API per H2) +Migrate-all driven via **settings → felhom-usb → "Összes adat áthelyezése" → felhom-flash** (real `storageMigrateAll → POST /api/storage/migrate`). Job `mig-20260615-095545`, all 7 apps `cleaned`. +- All 7 apps **healthy on flash** ✓ (komga `unhealthy` = pre-existing healthcheck quirk, still serving). +- All 3 seeded userdata checksums **identical** on flash (`6155f6df` / `1f2bb9a5` / `162a7fa1`) ✓. +- Source userdata files + `appdata/` **removed** ✓. +- **#8 setgid proof:** 16/17 migrated userdata dirs `2775` gid 101000; **1 exception → `import/calibre` = 755** (finding M3). +- **Decommission** (no UI button → routed `POST /api/storage/decommission {mode:anyway}`): `decommissioned:true`; agent **intent=decommissioned**; host **unmounted**; **guest-binds.json pruned** (da9e7089 removed); controller badge **"Kiváltva"** ✓. But **`pct config` still showed `mp1`** (stale — see C1). +- **Default edge (M1):** felhom-usb *was* default; afterwards **no drive held the default badge** — not reassigned. + +### Phase 3 — reconnect felhom-usb → NO auto-rebind — PASS (invariant holds) + 🔴 C1 surfaced +- Agent restart: host `/mnt/felhom-usb` **not remounted**; reassert log `F9 re-assert: skipping non-enrolled drive (intent-gated) intent=ejected` (demonstrated on the lingering ce9d drive; felhom-usb was pruned from the bind list entirely so it isn't even reattempted). No auto-rebind ✓. +- **Guest reboot → C1 fired:** `lxc_init … Failed to run lxc.hook.pre-start … startup for container '9201' failed` (exit 255) due to stale `mp1`. **Recovery:** `pct set 9201 --delete mp1` → `pct start 9201` → booted first try, felhom-usb correctly absent, 24→25 containers back. + +### Phase 4 — decommission-anyway felhom-flash → missing-storage — PASS (decommission via API per H2) +`POST /api/storage/decommission {mode:anyway, mount_name:felhom-flash}` → `stopped_apps` = all 8. +- Apps **stopped**, **keep HDD_PATH=/mnt/felhom-flash** ✓; flash **intent=decommissioned + unmounted** ✓. +- Missing-storage badge **"Hiányzó tárhely: felhom-flash"** rendered **8×** on dashboard, **8×** on stacks, on the app card ✓. Dashboard screenshot: 9 running / 8 stopped, Audiobookshelf "Leállítva" + the orange badge. + +### Phase 5 — re-enroll felhom-flash clears decommissioned + badge — PASS (re-mount via SSH + re-register via API per H3) +Re-mounted flash (data **intact**, seed checksum `6155f6df` still matched ✓), `POST /api/storage/register` → reEnrollClearMarker. +- Agent **intent → enrolled** (Change 4 marker cleared) ✓; flash badge no longer Kiváltva; **0 "Hiányzó tárhely"** on dashboard ✓. +- Apps restarted via `POST /api/stacks/{name}/start` (because **`/api/storage/restart-apps` returned 404** — H1); **25 containers healthy** ✓. + +### Phase 6 — eject vs decommission — PASS (finding M2: premise contradicted) +Eject flash via `POST /api/storage/eject`: intent→**ejected**, host **unmounted**, **systemd unit disabled**, bind record **kept**. Agent restart (reconnect sim): **NOT auto-remounted**; reassert logged `skipping non-enrolled drive (intent-gated) intent=ejected`. Apps stayed up (guest bind persisted). Flash restored (re-mount + register → enrolled, 25 containers). + +### Phase 7 — feature sweep +| Feature | Result | +|---|---| +| Manual backup (whole-guest → felhom-pbs) | **PASS** — real encrypted vzdump→proxmox-backup-client; phase reached `done`; new PBS snapshot `ct/9201/2026-06-15T10:29:13Z` landed (4 total). A 2nd trigger correctly returned 409 "already in progress". | +| Fresh app deploy + lifecycle + logs (Glance) | **PASS** — deploy 202 → healthy; stop/start/restart all HTTP 200; logs endpoint returned app output. | +| FileBrowser umask fix | **PASS** — PID1 `Umask: 0002`, entrypoint `[sh -c umask 002; exec …]`; umask-002 folder = **2775 group-writable** (gid 101000); a uid-1000 process **wrote into it** ("WRITE OK") → FileBrowser/app group collaboration confirmed. (Note: FileBrowser web session had expired; folder-creation via the UI was not re-driven because logging in would require entering a password — prohibited. The fix's effect was verified by replicating PID1's exact umask-002 environment + the group-write test.) | +| Monitoring page | **PASS** — renders host CPU/mem/temp/uptime + storage bars; felhom-usb correctly shown "Nem elérhető" while decommissioned. | +| Test notification email | **FAIL (M6)** — `/settings/notifications/test` returned "Hiba" (likely no SMTP configured). | +| Mem-gate | Gate **allows within budget** (Glance deployed cleanly); a **refusal was not reproducible** with ~10 GB free and no catalog app large enough to exceed 12 GB — would require artificial RAM exhaustion (not done, would destabilise the demo). | + +### Phase 8 — restore — DONE (sane end-state) +- **felhom-flash**: `Alapértelmezett | Aktív | USB` — **default**, all apps healthy. +- **felhom-usb**: `Aktív` (stale-UUID unit fixed → da9e7089, re-mounted, soft-marker cleared; mp1 re-bound and valid). Note: its **agent intent remains `decommissioned`** (a latent cosmetic inconsistency — harmless because the drive is mounted and its mp is valid; the reassert would merely skip re-adding an already-present bind). +- **Belső SSD**: `Aktív`. No "Kiváltva" anywhere; **no stale mps** (mp1=usb, mp2=flash both mounted → reboot-safe); **25 containers running**; Glance test app removed. sda / PBS / controller / local / local-lvm untouched throughout. + +--- + +## Code-fix candidates for the next slice + +1. **(C1, critical) On decommission/eject, the agent must `pct set --delete mpN`** for the drive's bind (or the controller must, post-finalize) so the stale bind cannot brick the guest on reboot. Add a regression test: decommission a data drive → reboot guest → assert it starts and the drive is absent. Alternatively, make the agent's pre-start hook **self-heal** by dropping config binds whose source path is missing. +2. **(H1) Route `/api/storage/disconnect`, `/api/storage/reconnect`, `/api/storage/restart-apps`** (or remove the dead buttons). These are called by `settings.html` JS but absent from `ServeStorageAPI`. +3. **(H2) Wire a decommission button** (migrate-all-then-decommission / decommission-anyway with type-to-confirm) into the settings storage actions → `/api/storage/decommission`. +4. **(H3) Add a re-enroll action** on a decommissioned drive entry (re-mount + re-register in one click), instead of only "Eltávolítás a rendszerből". +5. **(M1) Reassign the default** to another schedulable drive when the current default is decommissioned/removed (or block decommissioning the default until reassigned). +6. **(M2) Decide eject semantics:** either make eject auto-remount on reconnect (and route `/api/storage/reconnect`), or relabel the UI so "eject" reads as the permanent-until-re-enroll action it actually is. +7. **(M3) Fix the userdata skeleton** so every nested dir (incl. `import/calibre`) is created `2775` setgid at enroll, and have the migrate-all merge-walk **re-assert setgid** on pre-existing target dirs. +8. **(M4) Update the systemd mount unit's `What=` UUID when a drive is reformatted** (or key the unit on a durable id), so re-mount after reformat works. + +--- + +## Evidence index (selected) +- Checksums (source-of-truth → verified-identical-on-target): movie `6155f6df…`, document `1f2bb9a5…`, photo `162a7fa1…`, romm-appdata `3643c7cc…`. +- Ownership: source 17/17 `2775` gid 101000; post-migrate-all target 16/17 `2775`, `import/calibre` `755` (M3). +- C1 brick: `__lxc_start: 2208 Failed to initialize container "9201"`; recovery via `pct set 9201 --delete mp1`. +- Intent transitions (agent `drive-intents.json`): usb da9e7089 enrolled→decommissioned→(restore, controller-cleared); flash 81a26531 enrolled→ejected→enrolled, →decommissioned→enrolled. +- PBS snapshot landed: `felhom-pbs:backup/ct/9201/2026-06-15T10:29:13Z`. +- Screenshots saved (browser): romm-on-flash badge, Phase-4 dashboard missing-storage badges, monitoring page, settings storage states.