From cf1a9402f22290bd4c77882dfe3f502712e066f4 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 4 Jul 2026 22:53:07 +0200 Subject: [PATCH] docs(audit): app-data restore drill 2026-07-04 (guest 9201) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Keep-side restore validated live: class-A DB replay, fail-closed data-key gate + proven non-destruction, resettable-secret discrimination, Tier-2 class-C copy, and guardrails (traversal/missing/single-flight/restart). Findings: F1 (HIGH) UI restore dead — /api/backup/snapshots has no handler; F2 no one-click in-place class-C restore; F3 named-volume data never backed up (DumpAppVolumes* has no caller). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- .../DRILL-appdata-restore-2026-07-04.md | 182 ++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100644 documentation/audits/DRILL-appdata-restore-2026-07-04.md diff --git a/documentation/audits/DRILL-appdata-restore-2026-07-04.md b/documentation/audits/DRILL-appdata-restore-2026-07-04.md new file mode 100644 index 0000000..3bcaf8c --- /dev/null +++ b/documentation/audits/DRILL-appdata-restore-2026-07-04.md @@ -0,0 +1,182 @@ +# DRILL — App-data restore (keep-side), guest 9201 / felhom-pve — 2026-07-04 + +**Class:** Runbook-style validation (unsupervised, throwaway demo box). No production code changed. +**Executed by:** Claude Code (CC), live SSH `root@felhom-pve` + in-guest `pct exec 9201`, real controller +endpoints via authenticated in-namespace `curl` (method (b) per the runbook). +**Hard rail honoured:** every destructive step was scoped to guest **9201** on host **felhom-pve** +(`demo-felhom`). `sajatfelhom.hu` / Peti's box and every other guest were never touched. `wg show dump` +never run. + +--- + +## 1. Confirmed baselines (reconfirmed against live Gitea + live host) + +| Repo | `main` @ commit | Version | Reconfirm | +|------|-----------------|---------|-----------| +| felhom-controller | `a518023` | **v0.98.3** | local HEAD == runbook commit; `HEAD..origin/main = 0` | +| felhom-agent | `72d7f05` | **v0.69.0** | local HEAD == runbook commit; `HEAD..origin/main = 0` | +| app-catalog-felhom.eu | `13eedb1` | — | HEAD == runbook commit | + +- **Live controller image on 9201:** `gitea.dooplex.hu/admin/felhom-controller:0.98.3` (healthy). +- **Live agent on felhom-pve:** `felhom-agent 0.69.0`. Node `demo-felhom`, `pve-manager/9.2.2`. +- Identity rail re-asserted before every destructive step: host `felhom-pve` (192.168.0.162), guest `9201`. + +--- + +## 2. Environment ground truth (Phase 0) + +**Deployed stacks on 9201:** sparkyfitness (+ -server, -db postgres15), rallly (+ postgres16), nextcloud +(+ mariadb 11.6, + redis), gitea, calcom (+ postgres16), filebrowser; infra: felhom-controller, cloudflared, +traefik. **AdventureLog was NOT deployed** — see §2.1 for the substitute. + +**Enrolled user-data drives (in-guest mounts = namespace roots):** + +| Drive | Device | In-guest mount = `nsRoot` | Schedulable | +|-------|--------|---------------------------|-------------| +| felhom-flash | /dev/sdb1 | /mnt/felhom-drives/felhom-flash | yes | +| felhom-usb | /dev/sdd1 | /mnt/felhom-drives/felhom-usb | yes | +| teszt_enroll | /dev/sdc | /mnt/felhom-drives/teszt_enroll | yes | + +No `felhom-data/felhom-data` double-nesting observed. Apps with **no** dedicated user-data drive fall back to +the SSD system path `/mnt/sys_drive/felhom-data` (`inGuestDrive=false`, so `felhom-data` is appended there). + +**Per-app data-class map + `nsRoot`:** + +| App | nsRoot | Class A (DB) | Class B (named vol) | Class C (HDD bind) | data_key env vars | +|-----|--------|--------------|---------------------|--------------------|-------------------| +| **sparkyfitness** | /mnt/sys_drive/felhom-data | postgres ✓ | none (`volume_dumps: null`) | none | **API_ENCRYPTION_KEY, BETTER_AUTH_SECRET** | +| **nextcloud** | /mnt/felhom-drives/felhom-usb | mariadb ✓ | `nextcloud_nextcloud_html` (→ /var/www/html) | `appdata/nextcloud` (→ /var/www/html/data, 37 MB) | none | +| rallly / calcom / gitea | /mnt/sys_drive/felhom-data | postgres ✓ | — | — | none | + +**Pre-existing backup artifacts:** recovery units (`compose/ + db-dumps/ + manifest.json`) present for the four +sys_drive apps and nextcloud; **no `volume-dumps/` anywhere** (see Finding F3). felhom-flash holds stale +recovery units for undeployed apps (immich, paperless-ngx, jellyfin, …). Offbox target configured but +**disabled** (`enabled:false`, host 192.168.0.128). + +### 2.1 Substitute for AdventureLog + +AdventureLog (the runbook's fatal-gate app) was not deployed. **sparkyfitness is a faithful substitute**: its +recovery-unit manifest carries `data_key_env_vars: [API_ENCRYPTION_KEY, BETTER_AUTH_SECRET]` and resettable +`secret_env_vars` incl. `DB_PASSWORD`, `APP_DB_PASSWORD` — exactly the `data_key` / resettable split the drill +needs, plus a Postgres DB for class A. All gate phases ran on it, avoiding a fresh deploy (no DNS/subdomain +churn on the shared box). + +### 2.2 How the real endpoints were driven (method (b)) + +Controller binds `8080/tcp` only on the docker bridge (not host-published) and uses **host-based routing**. +CC hit the real handler chain from the guest netns: `curl -H 'Host: felhom.demo-felhom.eu' http://172.17.0.2:8080…`. +No server logic skipped. **CSRF is not populated in this unauth deployment** (``, +no Set-Cookie), so POSTs carry an empty `_csrf` — matching what the live UI would send. + +--- + +## 3. Per-phase results + +| Phase | Result | Evidence (sentinel / flash / log) | +|-------|--------|-----------------------------------| +| **1 — Class A DB rows** (sparkyfitness) | **PASS** | Sentinel `DRILL-d2788b3c…` inserted → `POST /api/backup/run` → dump `sparkyfitness-postgres.sql` fresh (20:38:44) & contains UUID → `DROP TABLE` → `POST /backup/restore` → `302 flash=sparkyfitness visszaállítva`; log `replayed 1 DB dump(s)`; **row present again**. | +| **1 companion** (red-proof) | **PASS** | New row `DRILL2-04a98c1d…` inserted, **no** backup, restore → row **absent** afterwards, original UUID **present**. Proves restore replays the on-disk dump, not a cache/no-op. | +| **2 — Class B named-volume** | **NOT EXERCISED (Finding F3)** | No live affordance produces `volume-dumps/`. `POST /api/backup/run` = DB dumps + recovery-unit capture only. `DumpAppVolumes`/`DumpAppVolumesSafe` have **no caller**. Live-confirmed: nextcloud HAS a named volume but its unit has **no `volume-dumps/` dir**. `restoreDockerVolumes` is a no-op when no tar exists. | +| **3 — Class C HDD bind-mount** (nextcloud) | **PASS (expected-negative confirmed)** | 1 MiB sentinel `776a1d3f….bin` (sha `571f6090…`) in `appdata/nextcloud/drill` → `POST /api/backup/tier2` copied it to `felhom-flash/backups/secondary/nextcloud/appdata/drill` (sha matches) → deleted from live → **normal `POST /backup/restore`** (302 success, nextcloud healthy) → **sentinel STILL missing** from live appdata. Real recovery = operator copy-back from Tier-2 (sha `571f6090…` matched). See Finding F2. | +| **4 — Fatal data-key gate** (sparkyfitness) — SAFETY CENTREPIECE | **PASS** | Removed `API_ENCRYPTION_KEY` + `BETTER_AUTH_SECRET` from guest app.yaml (backed up) → restore **REFUSED**: `flash_error=Visszaállítás sikertelen: refusing to restore: data-encrypting key(s) [API_ENCRYPTION_KEY BETTER_AUTH_SECRET] could not be recovered…`; log `[ERROR] [backup] Restore REFUSED`. **Non-destruction proven:** container IDs unchanged (`f4609d4be80d` etc.), not restarted; sentinel row intact — gate aborts *before* `StopStack`. | +| **4 positive + reversibility** | **PASS** | With keys present: `secrets recovered=4/4, data_keys=2` + `Restore-from-unit completed` (seen in Phase 1 and again after restoring the original app.yaml). Box fully recovered; change reversible. | +| **5 — Resettable-secret path** (sparkyfitness) | **PASS (gate discriminates) + Observation O4** | Removed only `DB_PASSWORD` → restore **PROCEEDS not refused**: `[WARN] 1 resettable secret(s) unrecoverable [DB_PASSWORD] — proceeding … no data-key affected`, `recovered=3/4, data_keys=2`. It then **failed at compose-up** (`DB_PASSWORD … Defaulting to a blank string`, exit 1) — see O4. Healed by restoring app.yaml + re-running restore (`recovered=4/4`, healthy). | +| **6 — Offbox restore-to-verify** | **SKIPPED** | Offbox not configured/enabled. `POST /backup/offbox/restore` → `flash_error=A NAS mentési cél nincs beállítva.` (correct guard). | +| **7 — Real app-UI round-trip** | **PARTIAL** | Class-C round-trip proven at the filesystem layer (Phase 3, sha-verified). In-app artifact-render round-trip via the app's own UI not performed (no browser session in this run). | +| **8.1 single-flight** | **PASS** | Backup running + restore → `flash_error=…backup or restore already in progress`. | +| **8.2 path traversal (F2)** | **PASS** | `stack_name=../../etc` → `flash_error=Érvénytelen alkalmazásnév`; log `restore rejected: invalid stack_name` — **no** restore work reached `RestoreFromRecoveryUnit`. | +| **8.3 missing params** | **PASS** | empty `stack_name`/`snapshot_id` → `flash_error=Hiányzó paraméterek`. | +| **8.4 disconnected-drive skip** | **VERIFIED BY SOURCE (not live-run)** | Skip guard present `backup.go:229-238` (`IsDisconnected` / `IsDecommissioned` → `SKIP`). Not exercised live to avoid mutating drive state. | +| **8.5 restart resilience** | **PASS** | `docker restart felhom-controller` → `GET /backups` = 200, `/api/backup/status` healthy, state not wedged. | + +--- + +## 4. Headline finding — Class C recovery (the drill's core question) + +**Does the normal "Visszaállítás" button recover HDD bind-mount (class-C) data? NO — by design, and it is +now live-proven.** `RestoreFromRecoveryUnit` (and `restoreDockerVolumes`) never reference `appdata/`; +after a normal restore the deleted class-C sentinel stayed gone while the app returned healthy. + +**The real class-C recovery paths that exist:** +1. **Tier-2 cross-drive copy** (`POST /api/backup/tier2` → `felhom-flash/backups/secondary//appdata/…`) — + the data survives (sha-verified) but recovery is an **operator manual copy-back**; there is **no + "restore from Tier-2" button** in the UI (the Tier-2 row only offers *Beállítás*). +2. **Offbox restore-to-verify** (`POST /backup/offbox/restore`) — restores to a **verification folder** + `/offbox-restore/`, never in-place, and requires a configured NAS (disabled here). +3. **`.fab` bundle import** (`/import` → `appexport.restoreHDDData`) — requires a prior manual bundle export. + +**Verdict / top pilot gap:** there is **no discoverable, one-click, in-place class-C restore** for a +non-technical customer. The nightly protection (Tier-2) is sound, but a customer who deletes their own files +cannot get them back from the dashboard — recovery needs an operator. **This is the headline gap for pilot.** + +--- + +## 5. Volume-dump-trigger finding (class B) + +**No user action produces `volume-dumps/` on this build.** `POST /api/backup/run` runs `runDBDumpsInternal` +(DB dumps + `captureAllRecoveryUnits`) and never calls `DumpAppVolumes`. `DumpAppVolumes` / +`DumpAppVolumesSafe` (`backup.go:303 / 375`) have **no live caller** anywhere in the tree; `captureAllRecoveryUnits` +only *enumerates* pre-existing `.tar`s. Consequence: **Docker named-volume data is not backed up by any current +trigger** — even nextcloud's `nextcloud_nextcloud_html` volume has no dump on disk. The restore side +(`restoreDockerVolumes`) is implemented and would work, but has nothing to restore. See Finding F3. + +--- + +## 6. Gate proof (safety centrepiece) + +- **Fatal refusal:** unrecoverable `data_key` → `RestoreFromRecoveryUnit` returns the refusal error and the + handler redirects `?flash_error=…refusing to restore: data-encrypting key(s) […]…`. ✔ +- **Proven non-destruction:** the gate returns at `restore_unit.go:113-116`, *before* `StopStack` + (`:127`) / `restoreDockerVolumes` (`:130`). Live: container IDs unchanged, app not stopped, sentinel intact. ✔ +- **Correct discrimination:** a missing **resettable** secret (`DB_PASSWORD`) is non-fatal — restore proceeds + with a WARN naming it. ✔ (`recovered=3/4, data_keys=2`). + +--- + +## 7. Bugs / observations found (NOT fixed — candidates for follow-up TASKs) + +### F1 — HIGH — UI restore is unreachable: `/api/backup/snapshots` has no handler +The restore panel (`backups.html:759`) fetches `GET /api/backup/snapshots?stack=` to populate the +snapshot dropdown. **No such route is registered** — the API router's catch-all returns +`{"ok":false,"error":"endpoint not found"}` (`internal/api/router.go:316`; only `/api/backup/{status,run,tier2}` +exist). Live-confirmed for nextcloud and sparkyfitness. Because the JS then shows *"Nincs elérhető mentés"* +and leaves the **"Visszaállítás indítása" button permanently disabled**, **a customer cannot restore any app +through the dashboard.** The server-side `POST /backup/restore` handler itself works (this drill drove it +directly), so the fix is small (register the snapshots endpoint, or repopulate the dropdown from +`AppDataInfo`/`/api/backup/status`) — but as shipped the whole keep-side restore UI is dead. +*Files:* `internal/web/templates/backups.html:759`, `internal/api/router.go:251-316`. + +### F2 — MEDIUM/gap — no one-click in-place class-C restore (see §4) +Design boundary, but a real customer-facing gap for pilot. Candidate: a "restore user files from Tier-2/offbox +in place" affordance, guarded by confirmation. + +### F3 — MEDIUM — named-volume data never backed up (see §5) +`DumpAppVolumes*` is dead code with no trigger; apps with named volumes (e.g. nextcloud html) have no +`volume-dumps/`. Either wire a trigger (scheduler/Tier-1) or document that named-volume data is out of scope +and steer catalog apps to bind-mounts. *Files:* `internal/backup/backup.go:303,375`. + +### O4 — LOW/observation — resettable-secret "proceed" path deploys a *blank* secret +When a resettable secret is genuinely unrecoverable, the gate correctly proceeds — but `RecreateStackFromUnit` +then redeploys with the secret **unset/blank** (compose `Defaulting to a blank string` → exit 1 for +sparkyfitness). "Proceeding (may need a credential reset)" is accurate, but nothing performs the reset, so the +redeploy can fail. Consider generating/resetting the resettable secret on restore rather than passing empty. +*Files:* `internal/backup/restore_unit.go:117-120,134`. + +--- + +## 8. Cleanup confirmation + +- `felhom_drill` table **dropped** from `sparkyfitness_db`; a final `POST /api/backup/run` refreshed all dumps + — **no `felhom_drill` remains in any on-disk dump** (grep clean). +- Class-C sentinel removed from **both** live (`felhom-usb`) and Tier-2 (`felhom-flash`) — no residue. +- All Phase-4/5 `app.yaml` edits reverted; **no `*.drillbak` files remain**. +- **Final health:** all app containers `healthy` (sparkyfitness, nextcloud + db/redis, rallly, calcom, gitea, + filebrowser); rallly/calcom/gitea untouched. Controller `/backups` = 200. No drill residue left to confuse a + later session. + +--- + +*Method note:* restore/backup actions went through the real endpoints (`/backup/restore`, `/api/backup/run`, +`/api/backup/tier2`, `/backup/offbox/restore`) via authenticated in-namespace curl (runbook method (b)); the +UI equivalent (method (a)) is blocked by Finding F1. Fixture creation/verification used `pct exec` / `docker +exec` (plumbing, not the feature under test).