docs(runbook): e2e live-drive commands are DooPlex-local (plain ssh felhom-pve)

This commit is contained in:
2026-07-19 12:27:42 +02:00
parent 062357f778
commit 29eda5d86e
+10 -10
View File
@@ -29,12 +29,12 @@ destructive section (7) is last and gated.
1. Controller healthy + version: 1. Controller healthy + version:
- `curl -s https://felhom.demo-felhom.eu/api/health``{"ok":true,...}`. - `curl -s https://felhom.demo-felhom.eu/api/health``{"ok":true,...}`.
- `$SSH felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"``:0.60.0 Up ... (healthy)`. - `ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"``:0.60.0 Up ... (healthy)`.
- Dashboard loads (Hungarian UI), no error banners. - Dashboard loads (Hungarian UI), no error banners.
2. Agent healthy + version: `$SSH felhom-pve "systemctl is-active felhom-agent; /usr/local/bin/felhom-agent --version"``active`, `0.30.0`. 2. Agent healthy + version: `ssh felhom-pve "systemctl is-active felhom-agent; /usr/local/bin/felhom-agent --version"``active`, `0.30.0`.
3. Headroom (deploys pull images — **bound to ≤3 small apps**): 3. Headroom (deploys pull images — **bound to ≤3 small apps**):
- Docker-data volume free: dashboard storage bars, or `$SSH felhom-pve "pct exec 9201 -- df -h /var/lib/docker /"`. Need comfortably above the v0.58 reserve (`max(5GB,10%)`) or deploys will be gated **507**. - Docker-data volume free: dashboard storage bars, or `ssh felhom-pve "pct exec 9201 -- df -h /var/lib/docker /"`. Need comfortably above the v0.58 reserve (`max(5GB,10%)`) or deploys will be gated **507**.
- RAM: `$SSH felhom-pve "pct exec 9201 -- free -h"`. - RAM: `ssh felhom-pve "pct exec 9201 -- free -h"`.
- Disk list sane: `curl -s https://felhom.demo-felhom.eu/api/disks` → felhom-usb (user-data, data_bearing), local/local-lvm (system), felhom-pbs (backup). - Disk list sane: `curl -s https://felhom.demo-felhom.eu/api/disks` → felhom-usb (user-data, data_bearing), local/local-lvm (system), felhom-pbs (backup).
4. Record current deployed apps (so cleanup is unambiguous): dashboard "Alkalmazások", or `curl -s https://felhom.demo-felhom.eu/api/stacks/rescan` then the stacks list. (actualbudget is expected already deployed.) 4. Record current deployed apps (so cleanup is unambiguous): dashboard "Alkalmazások", or `curl -s https://felhom.demo-felhom.eu/api/stacks/rescan` then the stacks list. (actualbudget is expected already deployed.)
- **Good:** all green/healthy; free space well above reserve. **Watch for:** any app stuck "Telepítés alatt" (deploying) from a prior run — note and resolve before starting. - **Good:** all green/healthy; free space well above reserve. **Watch for:** any app stuck "Telepítés alatt" (deploying) from a prior run — note and resolve before starting.
@@ -51,7 +51,7 @@ Pick **two small apps** not currently deployed (suggest: `vikunja`, `mealie` —
- **Good:** progresses config→containers→health; ends `running`/healthy within ~120s; the card flips to deployed; no "Telepítés" button reappears mid-pull (in-memory Deployed=true during pull). - **Good:** progresses config→containers→health; ends `running`/healthy within ~120s; the card flips to deployed; no "Telepítés" button reappears mid-pull (in-memory Deployed=true during pull).
- **Watch for:** stuck at a step, health-probe never going green (check the app's healthcheck tool exists), confusing Hungarian labels, the deploy gate returning **507** (insufficient Docker-data headroom — expected if low on space; note the banner wording). - **Watch for:** stuck at a step, health-probe never going green (check the app's healthcheck tool exists), confusing Hungarian labels, the deploy gate returning **507** (insufficient Docker-data headroom — expected if low on space; note the banner wording).
2. Deploy app #2; same checks. 2. Deploy app #2; same checks.
3. Confirm on disk the durable record is correct (CTRL-T2-1, happy case): `$SSH felhom-pve "pct exec 9201 -- docker exec felhom-controller cat /opt/docker/stacks/<app>/app.yaml | grep deployed"``deployed: true` (only after success). 3. Confirm on disk the durable record is correct (CTRL-T2-1, happy case): `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller cat /opt/docker/stacks/<app>/app.yaml | grep deployed"``deployed: true` (only after success).
- **Good:** `deployed: true` on disk after a successful deploy. **Watch for:** secrets appearing in plaintext in app.yaml (they must be `enc:`-prefixed — H10/encryption check). - **Good:** `deployed: true` on disk after a successful deploy. **Watch for:** secrets appearing in plaintext in app.yaml (they must be `enc:`-prefixed — H10/encryption check).
--- ---
@@ -61,10 +61,10 @@ Pick **two small apps** not currently deployed (suggest: `vikunja`, `mealie` —
Goal: prove a crash during the image-pull window leaves the stack **NOT-deployed and redeployable**, not ghost-stuck. Goal: prove a crash during the image-pull window leaves the stack **NOT-deployed and redeployable**, not ghost-stuck.
1. Pick a **third app with a non-trivial image pull** (so the pull window is a few seconds — e.g. `paperless-ngx` if space allows, else `mealie`). Start the deploy (UI Telepítés or API POST), and **immediately** — while it is still pulling (status `deploying`, before `running`) — kill the controller: 1. Pick a **third app with a non-trivial image pull** (so the pull window is a few seconds — e.g. `paperless-ngx` if space allows, else `mealie`). Start the deploy (UI Telepítés or API POST), and **immediately** — while it is still pulling (status `deploying`, before `running`) — kill the controller:
- `$SSH felhom-pve "pct exec 9201 -- docker kill felhom-controller"` **[operator: time this during the pull]** - `ssh felhom-pve "pct exec 9201 -- docker kill felhom-controller"` **[operator: time this during the pull]**
- The bootstrap service (`felhom-controller-bootstrap.service`) restarts it within seconds. Confirm back up: `curl -s https://felhom.demo-felhom.eu/api/health`. - The bootstrap service (`felhom-controller-bootstrap.service`) restarts it within seconds. Confirm back up: `curl -s https://felhom.demo-felhom.eu/api/health`.
2. After restart, check the stack state: 2. After restart, check the stack state:
- On disk: `$SSH felhom-pve "pct exec 9201 -- docker exec felhom-controller cat /opt/docker/stacks/<app>/app.yaml | grep deployed"`**`deployed: false`** (transitional — the fix). - On disk: `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller cat /opt/docker/stacks/<app>/app.yaml | grep deployed"`**`deployed: false`** (transitional — the fix).
- UI/API: `GET /api/stacks/<app>` → state `not_deployed` (the card shows **Telepítés**, not a ghost "deployed"). - UI/API: `GET /api/stacks/<app>` → state `not_deployed` (the card shows **Telepítés**, not a ghost "deployed").
3. **Redeploy** the same app — it must be **allowed** (no "already deployed; use update instead" refusal) and complete normally. 3. **Redeploy** the same app — it must be **allowed** (no "already deployed; use update instead" refusal) and complete normally.
- **Good:** post-crash the app reads not-deployed and redeploys cleanly. **PRE-FIX behaviour (must NOT occur):** app.yaml `deployed: true` with no containers, and redeploy refused — that's the ghost-stuck regression the fix removes. - **Good:** post-crash the app reads not-deployed and redeploys cleanly. **PRE-FIX behaviour (must NOT occur):** app.yaml `deployed: true` with no containers, and redeploy refused — that's the ghost-stuck regression the fix removes.
@@ -80,9 +80,9 @@ Goal: prove a crash during the image-pull window leaves the stack **NOT-deployed
- **Good:** imports, recreates the stack, data restored; fail-closed data-key gate honored if the app has a data-encrypting key. - **Good:** imports, recreates the stack, data restored; fail-closed data-key gate honored if the app has a data-encrypting key.
3. **Negative — path traversal (CTRL-001):** craft a hostile `.fab` and confirm it is **rejected at parse**, not written. 3. **Negative — path traversal (CTRL-001):** craft a hostile `.fab` and confirm it is **rejected at parse**, not written.
- Build a minimal bundle whose `manifest.json` has `"app_name":"../evil"` (and/or an `hdd_subdirs` / `volume_names` entry with `../`). Place it under a registered `exports/` dir on the host: - Build a minimal bundle whose `manifest.json` has `"app_name":"../evil"` (and/or an `hdd_subdirs` / `volume_names` entry with `../`). Place it under a registered `exports/` dir on the host:
`$SSH felhom-pve "pct exec 9201 -- docker exec felhom-controller sh -c 'ls /mnt/felhom-usb/exports/'"` to find the dir. `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller sh -c 'ls /mnt/felhom-usb/exports/'"` to find the dir.
- Attempt import of the hostile bundle. - Attempt import of the hostile bundle.
- **Good (the fix):** import **fails immediately** with a manifest/validation error; **no directory is created outside the stacks dir** (verify: `$SSH felhom-pve "pct exec 9201 -- docker exec felhom-controller ls -la /opt/docker/evil /etc/evil 2>/dev/null"` → nothing). **PRE-FIX (must NOT occur):** a dir/file written outside `/opt/docker/stacks/`. - **Good (the fix):** import **fails immediately** with a manifest/validation error; **no directory is created outside the stacks dir** (verify: `ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller ls -la /opt/docker/evil /etc/evil 2>/dev/null"` → nothing). **PRE-FIX (must NOT occur):** a dir/file written outside `/opt/docker/stacks/`.
- **Watch for:** the error message clarity (does the UI explain why it was rejected?). - **Watch for:** the error message clarity (does the UI explain why it was rejected?).
--- ---
@@ -107,7 +107,7 @@ Re-confirm the two refusals proven on 2026-06-13. **Do NOT send a matching confi
- **Good:** `formatted:false`, `needs_confirmation:true`, **HTTP 409**; no mkfs. - **Good:** `formatted:false`, `needs_confirmation:true`, **HTTP 409**; no mkfs.
2. **Refusal B — wrong durable_id:** same call with `"confirmed":true,"durable_id":"byid:wwn-0xDEADBEEF-DOES-NOT-EXIST"`. 2. **Refusal B — wrong durable_id:** same call with `"confirmed":true,"durable_id":"byid:wwn-0xDEADBEEF-DOES-NOT-EXIST"`.
- **Good:** `formatted:false`, refused **409**; a non-matching confirmation does not authorize a wipe. - **Good:** `formatted:false`, refused **409**; a non-matching confirmation does not authorize a wipe.
3. **Data-safety assertion:** `$SSH felhom-pve "findmnt /mnt/felhom-usb -o TARGET,SOURCE,FSTYPE; pct exec 9201 -- docker exec felhom-controller sh -c 'df -h /mnt/felhom-usb'"` → still mounted, used space unchanged. 3. **Data-safety assertion:** `ssh felhom-pve "findmnt /mnt/felhom-usb -o TARGET,SOURCE,FSTYPE; pct exec 9201 -- docker exec felhom-controller sh -c 'df -h /mnt/felhom-usb'"` → still mounted, used space unchanged.
4. **Happy-path destructive wipe** = **[HUMAN]** — never wipe a real/customer drive to test; covered by the agent unit test `retarget-mismatch-refused`. Only on a genuinely disposable blank device, supervised. **[DESTRUCTIVE — operator confirm]** 4. **Happy-path destructive wipe** = **[HUMAN]** — never wipe a real/customer drive to test; covered by the agent unit test `retarget-mismatch-refused`. Only on a genuinely disposable blank device, supervised. **[DESTRUCTIVE — operator confirm]**
--- ---