admin/felhom-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
70cb21b05878ece6de13a3b1c7bdfcf30c0aa57d
felhom-controller/RUNBOOK-e2e-live-drive.md
T

12 KiB
Raw Blame History

RUNBOOK — End-to-end live drive (SUPERVISED) — felhom-controller

Purpose: a hands-on end-to-end exercise of the demo stack that deliberately stresses the fixes shipped 2026-06-13 — CTRL-T2-1 (crash-safe deploy state), CTRL-001 (import path traversal), AGENT-001 (anti-retarget wipe), plus backup/restore — and surfaces UX/latency/error-handling friction. Execute SUPERVISED (operator present). Do the non-destructive sections (1–6) first; the destructive section (7) is last and gated.

Status: NOT executed yet — written 2026-06-13 for the supervised session.

Environment & conventions

  • Demo controller: guest 9201 (demo-felhom) on Proxmox host felhom-pve (192.168.0.162), bootstrap-managed. Public dashboard/API: https://felhom.demo-felhom.eu (no dashboard password → the API is open; drive it via the PUBLIC URL, not the container IP).
  • Versions at writing: controller v0.60.0, agent v0.30.0, hub v0.11.0.
  • Run from DooPlex (192.168.0.180); host root via SSH alias felhom-pve — plain ssh felhom-pve. (Legacy Windows workstation: needed SSH=/c/Windows/System32/OpenSSH/ssh.exe and export MSYS_NO_PATHCONV=1 for pct exec.)
  • Findings log: record every observation (✓/✗ + notes on UX friction, latency, confusing labels, error handling) in a new REPORT-e2e-live-drive-<date>.md. Each step says what "good" looks like and what to watch for.
  • Markers: [DESTRUCTIVE — operator confirm] = needs operator eyes + explicit go-ahead. [HUMAN] = cannot be done by the agent (physical / real-decision).

1. Baseline (read-only)

  1. Controller healthy + version:
    • curl -s https://felhom.demo-felhom.eu/api/health → {"ok":true,...}.
    • ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'" → :0.60.0 Up ... (healthy).
    • Dashboard loads (Hungarian UI), no error banners.
  2. Agent healthy + version: ssh felhom-pve "systemctl is-active felhom-agent; /usr/local/bin/felhom-agent --version" → active, 0.30.0.
  3. Headroom (deploys pull images — bound to ≤3 small apps):
    • Docker-data volume free: dashboard storage bars, or ssh felhom-pve "pct exec 9201 -- df -h /var/lib/docker /". Need comfortably above the v0.58 reserve (max(5GB,10%)) or deploys will be gated 507.
    • RAM: ssh felhom-pve "pct exec 9201 -- free -h".
    • Disk list sane: curl -s https://felhom.demo-felhom.eu/api/disks → felhom-usb (user-data, data_bearing), local/local-lvm (system), felhom-pbs (backup).
  4. Record current deployed apps (so cleanup is unambiguous): dashboard "Alkalmazások", or curl -s https://felhom.demo-felhom.eu/api/stacks/rescan then the stacks list. (actualbudget is expected already deployed.)
    • Good: all green/healthy; free space well above reserve. Watch for: any app stuck "Telepítés alatt" (deploying) from a prior run — note and resolve before starting.

2. Deploy flow — happy path (stresses CTRL-T2-1, normal case)

Pick two small apps not currently deployed (suggest: vikunja, mealie — small images; avoid immich/nextcloud/paperless which are large pulls). UI: Alkalmazások → → Telepítés, or API.

  1. Deploy app #1 via UI (fill any required fields, generate passwords where prompted), click Telepítés.
    • Watch the 3-step progress panel (config → containers → health) and the live polling (GET /api/stacks/<app> every 3s). API equivalent: curl -s -X POST https://felhom.demo-felhom.eu/api/stacks/<app>/deploy -H 'Content-Type: application/json' -d '{"values":{}}' then poll GET /api/stacks/<app>.
    • Good: progresses config→containers→health; ends running/healthy within ~120s; the card flips to deployed; no "Telepítés" button reappears mid-pull (in-memory Deployed=true during pull).
    • Watch for: stuck at a step, health-probe never going green (check the app's healthcheck tool exists), confusing Hungarian labels, the deploy gate returning 507 (insufficient Docker-data headroom — expected if low on space; note the banner wording).
  2. Deploy app #2; same checks.
  3. Confirm on disk the durable record is correct (CTRL-T2-1, happy case): ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller cat /opt/docker/stacks/<app>/app.yaml | grep deployed" → deployed: true (only after success).
    • Good: deployed: true on disk after a successful deploy. Watch for: secrets appearing in plaintext in app.yaml (they must be enc:-prefixed — H10/encryption check).

3. Deploy crash-window probe (THE CTRL-T2-1 test)

Goal: prove a crash during the image-pull window leaves the stack NOT-deployed and redeployable, not ghost-stuck.

  1. Pick a third app with a non-trivial image pull (so the pull window is a few seconds — e.g. paperless-ngx if space allows, else mealie). Start the deploy (UI Telepítés or API POST), and immediately — while it is still pulling (status deploying, before running) — kill the controller:
    • ssh felhom-pve "pct exec 9201 -- docker kill felhom-controller" [operator: time this during the pull]
    • The bootstrap service (felhom-controller-bootstrap.service) restarts it within seconds. Confirm back up: curl -s https://felhom.demo-felhom.eu/api/health.
  2. After restart, check the stack state:
    • On disk: ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller cat /opt/docker/stacks/<app>/app.yaml | grep deployed" → deployed: false (transitional — the fix).
    • UI/API: GET /api/stacks/<app> → state not_deployed (the card shows Telepítés, not a ghost "deployed").
  3. Redeploy the same app — it must be allowed (no "already deployed; use update instead" refusal) and complete normally.
    • Good: post-crash the app reads not-deployed and redeploys cleanly. PRE-FIX behaviour (must NOT occur): app.yaml deployed: true with no containers, and redeploy refused — that's the ghost-stuck regression the fix removes.
    • Watch for: any orphaned containers from the killed pull (docker ps -a); the half-pulled image is fine (compose re-pulls).

4. Import flow — round-trip + negative (THE CTRL-001 test)

  1. Export a deployed app to a .fab bundle. UI: app page → Exportálás (or API POST /api/export/start with {"stack_name":"<app>","drive":"<drive>"}, poll GET /api/export/status). The .fab lands under the chosen drive's exports/.
    • Good: export completes; a .fab appears; manifest readable via POST /api/export/manifest.
  2. Re-import the same .fab: UI import flow (or POST /api/export/import with the file path), poll GET /api/export/import/status.
    • Good: imports, recreates the stack, data restored; fail-closed data-key gate honored if the app has a data-encrypting key.
  3. Negative — path traversal (CTRL-001): craft a hostile .fab and confirm it is rejected at parse, not written.
    • Build a minimal bundle whose manifest.json has "app_name":"../evil" (and/or an hdd_subdirs / volume_names entry with ../). Place it under a registered exports/ dir on the host: ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller sh -c 'ls /mnt/felhom-usb/exports/'" to find the dir.
    • Attempt import of the hostile bundle.
    • Good (the fix): import fails immediately with a manifest/validation error; no directory is created outside the stacks dir (verify: ssh felhom-pve "pct exec 9201 -- docker exec felhom-controller ls -la /opt/docker/evil /etc/evil 2>/dev/null" → nothing). PRE-FIX (must NOT occur): a dir/file written outside /opt/docker/stacks/.
    • Watch for: the error message clarity (does the UI explain why it was rejected?).

5. Backup — per-app + full (3-2-1 tiers)

  1. Per-app / on-demand DB-dump + recovery-unit refresh: trigger POST /api/backup/run (or the dashboard "Biztonsági mentés" action). Watch GET /api/backup/status.
    • Good: completes; DB dumps written under <drive>/backups/primary/<app>/db-dumps/; recovery unit refreshed (backups/primary/<app>/ has compose + manifest.json).
  2. Tier-2 off-drive copy: POST /api/backup/tier2. (Single-drive demo → Tier-2 goes to the internal SSD for SMALL units only, DB/config-only, with the rootfs-headroom guard.)
    • Good: the per-app "2. mentés" card in the UI populates (success → target drive, or an honest "needs 2nd HDD" no-target reason). Watch for: the rootfs-headroom guard correctly refusing rather than filling rootfs when the unit is too big for the SSD.
  3. Whole-guest (PBS) tier is the agent's: confirm via the monitoring page that the PBS/offsite tier shows recent snapshots (the controller surfaces agent/PBS status; it does not run vzdump itself).
    • Good: 3-2-1 visible — primary (on-drive recovery unit) + secondary (Tier-2 off-drive) + offsite (PBS). Watch for: any tier showing stale/never — note it.
  4. Verify completion is reflected, not just started: re-check GET /api/backup/status returns done and the UI timestamps update.

6. Wipe gate re-check (AGENT-001 — NON-destructive)

Re-confirm the two refusals proven on 2026-06-13. Do NOT send a matching confirmation for a real data drive. Target the real data-bearing device /dev/sdb1 (felhom-usb) — both calls REFUSE, nothing is formatted.

  1. Refusal A — no confirmation: curl -s -w '\n%{http_code}\n' -X POST https://felhom.demo-felhom.eu/api/disks/format -H 'Content-Type: application/json' -d '{"device":"/dev/sdb1","fstype":"ext4"}'
    • Good: formatted:false, needs_confirmation:true, HTTP 409; no mkfs.
  2. Refusal B — wrong durable_id: same call with "confirmed":true,"durable_id":"byid:wwn-0xDEADBEEF-DOES-NOT-EXIST".
    • Good: formatted:false, refused 409; a non-matching confirmation does not authorize a wipe.
  3. Data-safety assertion: ssh felhom-pve "findmnt /mnt/felhom-usb -o TARGET,SOURCE,FSTYPE; pct exec 9201 -- docker exec felhom-controller sh -c 'df -h /mnt/felhom-usb'" → still mounted, used space unchanged.
  4. Happy-path destructive wipe = [HUMAN] — never wipe a real/customer drive to test; covered by the agent unit test retarget-mismatch-refused. Only on a genuinely disposable blank device, supervised. [DESTRUCTIVE — operator confirm]

7. Restore — DESTRUCTIVE [DESTRUCTIVE — operator confirm]

Do this LAST, on a NON-critical app (e.g. one deployed in §2, not customer RomM data). Restore overwrites the app's current data.

  1. Pick an app with a backup/snapshot from §5. Note its current data state (log in, note an item) so you can confirm the restore round-trip.
  2. [operator confirm] Restore: POST /backup/restore (web form: stack_name=<app>&snapshot_id=<id>), or the dashboard restore UI.
    • Good: the app stops, restores volume tars / recovery unit, restarts; data returns to the snapshot state; the fail-closed data-key gate refuses (with a clear message) if a data-encrypting key can't be recovered — in which case STOP and do NOT force.
    • Watch for: data loss vs the snapshot, secret-regeneration (must recover from the guest's own app.yaml, regenerate nothing), confusing progress/labels.
  3. [operator] Any delete/remove of a stack to clean up the §2/§3 test apps is [DESTRUCTIVE — operator confirm] — protected stacks (traefik/cloudflared/filebrowser/controller) must remain unstoppable server-side; confirm a delete attempt on one of those is refused.

8. Wrap-up

  • Tear down the test apps deployed in §2/§3 (operator-confirmed deletes), leaving the demo as found (actualbudget + RomM intact).
  • Write REPORT-e2e-live-drive-<date>.md: per-section ✓/✗, every UX/latency/error-handling finding, and any regression (especially: did §3 leave a ghost-deployed stack? did §4 write outside the stacks dir? did any wipe call format?). File new bugs against the audit record.
  • Confirm final: controller :0.60.0 healthy, agent 0.30.0 active, felhom-usb intact, no orphaned containers.

Order summary (non-destructive first)

1 Baseline → 2 Deploy happy → 3 Crash-window probe → 4 Import + traversal-negative → 5 Backup → 6 Wipe-gate refusals → 7 Restore [DESTRUCTIVE] → 8 Wrap-up.

Reference in New Issue View Git Blame Copy Permalink