felhom-controller/RUNBOOK-e2e-live-drive.md

# RUNBOOK — End-to-end live drive (SUPERVISED) — felhom-controller

**Purpose:** a hands-on end-to-end exercise of the demo stack that deliberately stresses the fixes
shipped 2026-06-13 — **CTRL-T2-1** (crash-safe deploy state), **CTRL-001** (import path traversal),
**AGENT-001** (anti-retarget wipe), plus backup/restore — and surfaces UX/latency/error-handling
friction. **Execute SUPERVISED (operator present).** Do the non-destructive sections (1–6) first; the
destructive section (7) is last and gated.

**Status:** NOT executed yet — written 2026-06-13 for the supervised session.

## Environment & conventions

- **Demo controller:** guest **9201** (`demo-felhom`) on Proxmox host `felhom-pve` (192.168.0.162),
  bootstrap-managed. Public dashboard/API: **https://felhom.demo-felhom.eu** (no dashboard password →
  the API is open; drive it via the PUBLIC URL, not the container IP).
- **Versions at writing:** controller **v0.60.0**, agent **v0.30.0**, hub v0.11.0.
- Run from DooPlex (192.168.0.180); host root via SSH alias `felhom-pve` — plain `ssh felhom-pve`.
  (Legacy Windows workstation: needed `SSH=/c/Windows/System32/OpenSSH/ssh.exe` and
  `export MSYS_NO_PATHCONV=1` for `pct exec`.)
- **Findings log:** record every observation (✓/✗ + notes on UX friction, latency, confusing labels,
  error handling) in a new `REPORT-e2e-live-drive-<date>.md`. Each step says what "good" looks like and
  what to watch for.
- **Markers:** **[DESTRUCTIVE — operator confirm]** = needs operator eyes + explicit go-ahead.
  **[HUMAN]** = cannot be done by the agent (physical / real-decision).

---

## 1. Baseline (read-only)

1. Controller healthy + version:
   - `curl -s https://felhom.demo-felhom.eu/api/health` → `{"ok":true,...}`.
   - `ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"` → `:0.60.0 Up ... (healthy)`.
   - Dashboard loads (Hungarian UI), no error banners.
2. Agent healthy + version: `ssh felhom-pve "systemctl is-active felhom-agent; /usr/local/bin/felhom-agent --version"` → `active`, `0.30.0`.
3. Headroom (deploys pull images — **bound to ≤3 small apps**):
   - Docker-data volume free: dashboard storage bars, or `ssh felhom-pve "pct exec 9201 -- df -h /var/lib/docker /"`. Need comfortably above the v0.58 reserve (`max(5GB,10%)`) or deploys will be gated **507**.
   - RAM: `ssh felhom-pve "pct exec 9201 -- free -h"`.
   - Disk list sane: `curl -s https://felhom.demo-felhom.eu/api/disks` → felhom-usb (user-data, data_bearing), local/local-lvm (system), felhom-pbs (backup).
4. Record current deployed apps (so cleanup is unambiguous): dashboard "Alkalmazások", or `curl -s https://felhom.demo-felhom.eu/api/stacks/rescan` then the stacks list. (actualbudget is expected already deployed.)
   - **Good:** all green/healthy; free space well above reserve. **Watch for:** any app stuck "Telepítés alatt" (deploying) from a prior run — note and resolve before starting.

---

## 2. Deploy flow — happy path (stresses CTRL-T2-1, normal case)

Pick **two small apps** not currently deployed (suggest: `vikunja`, `mealie` — small images; avoid immich/nextcloud/paperless which are large pulls). UI: **Alkalmazások → <app> → Telepítés**, or API.

1. Deploy app #1 via UI (fill any required fields, generate passwords where prompted), click **Telepítés**.
   - Watch the **3-step progress panel** (config → containers → health) and the live polling
     (`GET /api/stacks/<app>` every 3s). API equivalent: `curl -s -X POST https://felhom.demo-felhom.eu/api/stacks/<app>/deploy -H 'Content-Type: application/json' -d '{"values":{}}'` then poll `GET /api/stacks/<app>`.
   - **Good:** progresses config→containers→health; ends `running`/healthy within ~120s; the card flips to deployed; no "Telepítés" button reappears mid-pull (in-memory Deployed=true during pull).
   - **Watch for:** stuck at a step, health-probe never going green (check the app's healthcheck tool exists), confusing Hungarian labels, the deploy gate returning **507** (insufficient Docker-data headroom — expected if low on space; note the banner wording).
2. Deploy app #2; same checks.
3. Confirm on disk the durable record is correct (CTRL-T2-1, happy case): `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller cat /opt/docker/stacks/<app>/app.yaml | grep deployed"` → `deployed: true` (only after success).
   - **Good:** `deployed: true` on disk after a successful deploy. **Watch for:** secrets appearing in plaintext in app.yaml (they must be `enc:`-prefixed — H10/encryption check).

---

## 3. Deploy crash-window probe (THE CTRL-T2-1 test)

Goal: prove a crash during the image-pull window leaves the stack **NOT-deployed and redeployable**, not ghost-stuck.

1. Pick a **third app with a non-trivial image pull** (so the pull window is a few seconds — e.g. `paperless-ngx` if space allows, else `mealie`). Start the deploy (UI Telepítés or API POST), and **immediately** — while it is still pulling (status `deploying`, before `running`) — kill the controller:
   - `ssh felhom-pve "pct exec 9201 -- docker kill felhom-controller"`  **[operator: time this during the pull]**
   - The bootstrap service (`felhom-controller-bootstrap.service`) restarts it within seconds. Confirm back up: `curl -s https://felhom.demo-felhom.eu/api/health`.
2. After restart, check the stack state:
   - On disk: `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller cat /opt/docker/stacks/<app>/app.yaml | grep deployed"` → **`deployed: false`** (transitional — the fix).
   - UI/API: `GET /api/stacks/<app>` → state `not_deployed` (the card shows **Telepítés**, not a ghost "deployed").
3. **Redeploy** the same app — it must be **allowed** (no "already deployed; use update instead" refusal) and complete normally.
   - **Good:** post-crash the app reads not-deployed and redeploys cleanly. **PRE-FIX behaviour (must NOT occur):** app.yaml `deployed: true` with no containers, and redeploy refused — that's the ghost-stuck regression the fix removes.
   - **Watch for:** any orphaned containers from the killed pull (`docker ps -a`); the half-pulled image is fine (compose re-pulls).

---

## 4. Import flow — round-trip + negative (THE CTRL-001 test)

1. **Export** a deployed app to a `.fab` bundle. UI: app page → **Exportálás** (or API `POST /api/export/start` with `{"stack_name":"<app>","drive":"<drive>"}`, poll `GET /api/export/status`). The `.fab` lands under the chosen drive's `exports/`.
   - **Good:** export completes; a `.fab` appears; manifest readable via `POST /api/export/manifest`.
2. **Re-import** the same `.fab`: UI import flow (or `POST /api/export/import` with the file path), poll `GET /api/export/import/status`.
   - **Good:** imports, recreates the stack, data restored; fail-closed data-key gate honored if the app has a data-encrypting key.
3. **Negative — path traversal (CTRL-001):** craft a hostile `.fab` and confirm it is **rejected at parse**, not written.
   - Build a minimal bundle whose `manifest.json` has `"app_name":"../evil"` (and/or an `hdd_subdirs` / `volume_names` entry with `../`). Place it under a registered `exports/` dir on the host:
     `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller sh -c 'ls /mnt/felhom-usb/exports/'"` to find the dir.
   - Attempt import of the hostile bundle.
   - **Good (the fix):** import **fails immediately** with a manifest/validation error; **no directory is created outside the stacks dir** (verify: `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller ls -la /opt/docker/evil /etc/evil 2>/dev/null"` → nothing). **PRE-FIX (must NOT occur):** a dir/file written outside `/opt/docker/stacks/`.
   - **Watch for:** the error message clarity (does the UI explain why it was rejected?).

---

## 5. Backup — per-app + full (3-2-1 tiers)

1. **Per-app / on-demand DB-dump + recovery-unit refresh:** trigger `POST /api/backup/run` (or the dashboard "Biztonsági mentés" action). Watch `GET /api/backup/status`.
   - **Good:** completes; DB dumps written under `<drive>/backups/primary/<app>/db-dumps/`; recovery unit refreshed (`backups/primary/<app>/` has compose + manifest.json).
2. **Tier-2 off-drive copy:** `POST /api/backup/tier2`. (Single-drive demo → Tier-2 goes to the internal SSD for SMALL units only, DB/config-only, with the rootfs-headroom guard.)
   - **Good:** the per-app "2. mentés" card in the UI populates (success → target drive, or an honest "needs 2nd HDD" no-target reason). **Watch for:** the rootfs-headroom guard correctly **refusing** rather than filling rootfs when the unit is too big for the SSD.
3. **Whole-guest (PBS) tier** is the agent's: confirm via the monitoring page that the PBS/offsite tier shows recent snapshots (the controller surfaces agent/PBS status; it does not run vzdump itself).
   - **Good:** 3-2-1 visible — primary (on-drive recovery unit) + secondary (Tier-2 off-drive) + offsite (PBS). **Watch for:** any tier showing stale/never — note it.
4. Verify completion is reflected, not just started: re-check `GET /api/backup/status` returns `done` and the UI timestamps update.

---

## 6. Wipe gate re-check (AGENT-001 — NON-destructive)

Re-confirm the two refusals proven on 2026-06-13. **Do NOT send a matching confirmation for a real data drive.** Target the real data-bearing device `/dev/sdb1` (felhom-usb) — both calls REFUSE, nothing is formatted.

1. **Refusal A — no confirmation:** `curl -s -w '\n%{http_code}\n' -X POST https://felhom.demo-felhom.eu/api/disks/format -H 'Content-Type: application/json' -d '{"device":"/dev/sdb1","fstype":"ext4"}'`
   - **Good:** `formatted:false`, `needs_confirmation:true`, **HTTP 409**; no mkfs.
2. **Refusal B — wrong durable_id:** same call with `"confirmed":true,"durable_id":"byid:wwn-0xDEADBEEF-DOES-NOT-EXIST"`.
   - **Good:** `formatted:false`, refused **409**; a non-matching confirmation does not authorize a wipe.
3. **Data-safety assertion:** `ssh felhom-pve "findmnt /mnt/felhom-usb -o TARGET,SOURCE,FSTYPE; pct exec 9201 -- docker exec felhom-controller sh -c 'df -h /mnt/felhom-usb'"` → still mounted, used space unchanged.
4. **Happy-path destructive wipe** = **[HUMAN]** — never wipe a real/customer drive to test; covered by the agent unit test `retarget-mismatch-refused`. Only on a genuinely disposable blank device, supervised. **[DESTRUCTIVE — operator confirm]**

---

## 7. Restore — DESTRUCTIVE **[DESTRUCTIVE — operator confirm]**

Do this LAST, on a NON-critical app (e.g. one deployed in §2, not customer RomM data). Restore overwrites the app's current data.

1. Pick an app with a backup/snapshot from §5. Note its current data state (log in, note an item) so you can confirm the restore round-trip.
2. **[operator confirm]** Restore: `POST /backup/restore` (web form: `stack_name=<app>&snapshot_id=<id>`), or the dashboard restore UI.
   - **Good:** the app stops, restores volume tars / recovery unit, restarts; data returns to the snapshot state; the fail-closed data-key gate refuses (with a clear message) if a data-encrypting key can't be recovered — in which case STOP and do NOT force.
   - **Watch for:** data loss vs the snapshot, secret-regeneration (must recover from the guest's own app.yaml, regenerate nothing), confusing progress/labels.
3. **[operator]** Any **delete/remove** of a stack to clean up the §2/§3 test apps is **[DESTRUCTIVE — operator confirm]** — protected stacks (traefik/cloudflared/filebrowser/controller) must remain unstoppable server-side; confirm a delete attempt on one of those is refused.

---

## 8. Wrap-up

- Tear down the test apps deployed in §2/§3 (operator-confirmed deletes), leaving the demo as found (actualbudget + RomM intact).
- Write `REPORT-e2e-live-drive-<date>.md`: per-section ✓/✗, every UX/latency/error-handling finding, and any regression (especially: did §3 leave a ghost-deployed stack? did §4 write outside the stacks dir? did any wipe call format?). File new bugs against the audit record.
- Confirm final: controller `:0.60.0` healthy, agent `0.30.0` active, felhom-usb intact, no orphaned containers.

### Order summary (non-destructive first)
1 Baseline → 2 Deploy happy → 3 Crash-window probe → 4 Import + traversal-negative → 5 Backup → 6 Wipe-gate refusals → **7 Restore [DESTRUCTIVE]** → 8 Wrap-up.