From 29eda5d86e50bd2e1bdcada85758819354974863 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 19 Jul 2026 12:27:42 +0200 Subject: [PATCH] docs(runbook): e2e live-drive commands are DooPlex-local (plain ssh felhom-pve) --- RUNBOOK-e2e-live-drive.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/RUNBOOK-e2e-live-drive.md b/RUNBOOK-e2e-live-drive.md index d187744..cd68f96 100644 --- a/RUNBOOK-e2e-live-drive.md +++ b/RUNBOOK-e2e-live-drive.md @@ -29,12 +29,12 @@ destructive section (7) is last and gated. 1. Controller healthy + version: - `curl -s https://felhom.demo-felhom.eu/api/health` → `{"ok":true,...}`. - - `$SSH felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"` → `:0.60.0 Up ... (healthy)`. + - `ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"` → `:0.60.0 Up ... (healthy)`. - Dashboard loads (Hungarian UI), no error banners. -2. Agent healthy + version: `$SSH felhom-pve "systemctl is-active felhom-agent; /usr/local/bin/felhom-agent --version"` → `active`, `0.30.0`. +2. Agent healthy + version: `ssh felhom-pve "systemctl is-active felhom-agent; /usr/local/bin/felhom-agent --version"` → `active`, `0.30.0`. 3. Headroom (deploys pull images — **bound to ≤3 small apps**): - - Docker-data volume free: dashboard storage bars, or `$SSH felhom-pve "pct exec 9201 -- df -h /var/lib/docker /"`. Need comfortably above the v0.58 reserve (`max(5GB,10%)`) or deploys will be gated **507**. - - RAM: `$SSH felhom-pve "pct exec 9201 -- free -h"`. + - Docker-data volume free: dashboard storage bars, or `ssh felhom-pve "pct exec 9201 -- df -h /var/lib/docker /"`. Need comfortably above the v0.58 reserve (`max(5GB,10%)`) or deploys will be gated **507**. + - RAM: `ssh felhom-pve "pct exec 9201 -- free -h"`. - Disk list sane: `curl -s https://felhom.demo-felhom.eu/api/disks` → felhom-usb (user-data, data_bearing), local/local-lvm (system), felhom-pbs (backup). 4. Record current deployed apps (so cleanup is unambiguous): dashboard "Alkalmazások", or `curl -s https://felhom.demo-felhom.eu/api/stacks/rescan` then the stacks list. (actualbudget is expected already deployed.) - **Good:** all green/healthy; free space well above reserve. **Watch for:** any app stuck "Telepítés alatt" (deploying) from a prior run — note and resolve before starting. @@ -51,7 +51,7 @@ Pick **two small apps** not currently deployed (suggest: `vikunja`, `mealie` — - **Good:** progresses config→containers→health; ends `running`/healthy within ~120s; the card flips to deployed; no "Telepítés" button reappears mid-pull (in-memory Deployed=true during pull). - **Watch for:** stuck at a step, health-probe never going green (check the app's healthcheck tool exists), confusing Hungarian labels, the deploy gate returning **507** (insufficient Docker-data headroom — expected if low on space; note the banner wording). 2. Deploy app #2; same checks. -3. Confirm on disk the durable record is correct (CTRL-T2-1, happy case): `$SSH felhom-pve "pct exec 9201 -- docker exec felhom-controller cat /opt/docker/stacks//app.yaml | grep deployed"` → `deployed: true` (only after success). +3. Confirm on disk the durable record is correct (CTRL-T2-1, happy case): `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller cat /opt/docker/stacks//app.yaml | grep deployed"` → `deployed: true` (only after success). - **Good:** `deployed: true` on disk after a successful deploy. **Watch for:** secrets appearing in plaintext in app.yaml (they must be `enc:`-prefixed — H10/encryption check). --- @@ -61,10 +61,10 @@ Pick **two small apps** not currently deployed (suggest: `vikunja`, `mealie` — Goal: prove a crash during the image-pull window leaves the stack **NOT-deployed and redeployable**, not ghost-stuck. 1. Pick a **third app with a non-trivial image pull** (so the pull window is a few seconds — e.g. `paperless-ngx` if space allows, else `mealie`). Start the deploy (UI Telepítés or API POST), and **immediately** — while it is still pulling (status `deploying`, before `running`) — kill the controller: - - `$SSH felhom-pve "pct exec 9201 -- docker kill felhom-controller"` **[operator: time this during the pull]** + - `ssh felhom-pve "pct exec 9201 -- docker kill felhom-controller"` **[operator: time this during the pull]** - The bootstrap service (`felhom-controller-bootstrap.service`) restarts it within seconds. Confirm back up: `curl -s https://felhom.demo-felhom.eu/api/health`. 2. After restart, check the stack state: - - On disk: `$SSH felhom-pve "pct exec 9201 -- docker exec felhom-controller cat /opt/docker/stacks//app.yaml | grep deployed"` → **`deployed: false`** (transitional — the fix). + - On disk: `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller cat /opt/docker/stacks//app.yaml | grep deployed"` → **`deployed: false`** (transitional — the fix). - UI/API: `GET /api/stacks/` → state `not_deployed` (the card shows **Telepítés**, not a ghost "deployed"). 3. **Redeploy** the same app — it must be **allowed** (no "already deployed; use update instead" refusal) and complete normally. - **Good:** post-crash the app reads not-deployed and redeploys cleanly. **PRE-FIX behaviour (must NOT occur):** app.yaml `deployed: true` with no containers, and redeploy refused — that's the ghost-stuck regression the fix removes. @@ -80,9 +80,9 @@ Goal: prove a crash during the image-pull window leaves the stack **NOT-deployed - **Good:** imports, recreates the stack, data restored; fail-closed data-key gate honored if the app has a data-encrypting key. 3. **Negative — path traversal (CTRL-001):** craft a hostile `.fab` and confirm it is **rejected at parse**, not written. - Build a minimal bundle whose `manifest.json` has `"app_name":"../evil"` (and/or an `hdd_subdirs` / `volume_names` entry with `../`). Place it under a registered `exports/` dir on the host: - `$SSH felhom-pve "pct exec 9201 -- docker exec felhom-controller sh -c 'ls /mnt/felhom-usb/exports/'"` to find the dir. + `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller sh -c 'ls /mnt/felhom-usb/exports/'"` to find the dir. - Attempt import of the hostile bundle. - - **Good (the fix):** import **fails immediately** with a manifest/validation error; **no directory is created outside the stacks dir** (verify: `$SSH felhom-pve "pct exec 9201 -- docker exec felhom-controller ls -la /opt/docker/evil /etc/evil 2>/dev/null"` → nothing). **PRE-FIX (must NOT occur):** a dir/file written outside `/opt/docker/stacks/`. + - **Good (the fix):** import **fails immediately** with a manifest/validation error; **no directory is created outside the stacks dir** (verify: `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller ls -la /opt/docker/evil /etc/evil 2>/dev/null"` → nothing). **PRE-FIX (must NOT occur):** a dir/file written outside `/opt/docker/stacks/`. - **Watch for:** the error message clarity (does the UI explain why it was rejected?). --- @@ -107,7 +107,7 @@ Re-confirm the two refusals proven on 2026-06-13. **Do NOT send a matching confi - **Good:** `formatted:false`, `needs_confirmation:true`, **HTTP 409**; no mkfs. 2. **Refusal B — wrong durable_id:** same call with `"confirmed":true,"durable_id":"byid:wwn-0xDEADBEEF-DOES-NOT-EXIST"`. - **Good:** `formatted:false`, refused **409**; a non-matching confirmation does not authorize a wipe. -3. **Data-safety assertion:** `$SSH felhom-pve "findmnt /mnt/felhom-usb -o TARGET,SOURCE,FSTYPE; pct exec 9201 -- docker exec felhom-controller sh -c 'df -h /mnt/felhom-usb'"` → still mounted, used space unchanged. +3. **Data-safety assertion:** `ssh felhom-pve "findmnt /mnt/felhom-usb -o TARGET,SOURCE,FSTYPE; pct exec 9201 -- docker exec felhom-controller sh -c 'df -h /mnt/felhom-usb'"` → still mounted, used space unchanged. 4. **Happy-path destructive wipe** = **[HUMAN]** — never wipe a real/customer drive to test; covered by the agent unit test `retarget-mismatch-refused`. Only on a genuinely disposable blank device, supervised. **[DESTRUCTIVE — operator confirm]** ---