admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(audit): app-data restore drill 2026-07-04 (guest 9201)

Browse Source 
Keep-side restore validated live: class-A DB replay, fail-closed data-key
gate + proven non-destruction, resettable-secret discrimination, Tier-2
class-C copy, and guardrails (traversal/missing/single-flight/restart).

Findings: F1 (HIGH) UI restore dead — /api/backup/snapshots has no handler;
F2 no one-click in-place class-C restore; F3 named-volume data never backed
up (DumpAppVolumes* has no caller).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
authentik Default Admin
2026-07-04 22:53:07 +02:00
parent 87f2841692
commit cf1a9402f2
1 changed files with 182 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/DRILL-appdata-restore-2026-07-04.md
+182
View File
@@ -0,0 +1,182 @@
# DRILL — App-data restore (keep-side), guest 9201 / felhom-pve — 2026-07-04
**Class:** Runbook-style validation (unsupervised, throwaway demo box). No production code changed.
**Executed by:** Claude Code (CC), live SSH `root@felhom-pve` + in-guest `pct exec 9201`, real controller
endpoints via authenticated in-namespace `curl` (method (b) per the runbook).
**Hard rail honoured:** every destructive step was scoped to guest **9201** on host **felhom-pve**
(`demo-felhom`). `sajatfelhom.hu` / Peti's box and every other guest were never touched. `wg show dump`
never run.
---
## 1. Confirmed baselines (reconfirmed against live Gitea + live host)
| Repo | `main` @ commit | Version | Reconfirm |
|------|-----------------|---------|-----------|
| felhom-controller | `a518023` | **v0.98.3** | local HEAD == runbook commit; `HEAD..origin/main = 0` |
| felhom-agent | `72d7f05` | **v0.69.0** | local HEAD == runbook commit; `HEAD..origin/main = 0` |
| app-catalog-felhom.eu | `13eedb1` | — | HEAD == runbook commit |
- **Live controller image on 9201:** `gitea.dooplex.hu/admin/felhom-controller:0.98.3` (healthy).
- **Live agent on felhom-pve:** `felhom-agent 0.69.0`. Node `demo-felhom`, `pve-manager/9.2.2`.
- Identity rail re-asserted before every destructive step: host `felhom-pve` (192.168.0.162), guest `9201`.
---
## 2. Environment ground truth (Phase 0)
**Deployed stacks on 9201:** sparkyfitness (+ -server, -db postgres15), rallly (+ postgres16), nextcloud
(+ mariadb 11.6, + redis), gitea, calcom (+ postgres16), filebrowser; infra: felhom-controller, cloudflared,
traefik. **AdventureLog was NOT deployed** — see §2.1 for the substitute.
**Enrolled user-data drives (in-guest mounts = namespace roots):**
| Drive | Device | In-guest mount = `nsRoot` | Schedulable |
|-------|--------|---------------------------|-------------|
| felhom-flash | /dev/sdb1 | /mnt/felhom-drives/felhom-flash | yes |
| felhom-usb | /dev/sdd1 | /mnt/felhom-drives/felhom-usb | yes |
| teszt_enroll | /dev/sdc | /mnt/felhom-drives/teszt_enroll | yes |
No `felhom-data/felhom-data` double-nesting observed. Apps with **no** dedicated user-data drive fall back to
the SSD system path `/mnt/sys_drive/felhom-data` (`inGuestDrive=false`, so `felhom-data` is appended there).
**Per-app data-class map + `nsRoot`:**
| App | nsRoot | Class A (DB) | Class B (named vol) | Class C (HDD bind) | data_key env vars |
|-----|--------|--------------|---------------------|--------------------|-------------------|
| **sparkyfitness** | /mnt/sys_drive/felhom-data | postgres ✓ | none (`volume_dumps: null`) | none | **API_ENCRYPTION_KEY, BETTER_AUTH_SECRET** |
| **nextcloud** | /mnt/felhom-drives/felhom-usb | mariadb ✓ | `nextcloud_nextcloud_html` (→ /var/www/html) | `appdata/nextcloud` (→ /var/www/html/data, 37 MB) | none |
| rallly / calcom / gitea | /mnt/sys_drive/felhom-data | postgres ✓ | — | — | none |
**Pre-existing backup artifacts:** recovery units (`compose/ + db-dumps/ + manifest.json`) present for the four
sys_drive apps and nextcloud; **no `volume-dumps/` anywhere** (see Finding F3). felhom-flash holds stale
recovery units for undeployed apps (immich, paperless-ngx, jellyfin, …). Offbox target configured but
**disabled** (`enabled:false`, host 192.168.0.128).
### 2.1 Substitute for AdventureLog
AdventureLog (the runbook's fatal-gate app) was not deployed. **sparkyfitness is a faithful substitute**: its
recovery-unit manifest carries `data_key_env_vars: [API_ENCRYPTION_KEY, BETTER_AUTH_SECRET]` and resettable
`secret_env_vars` incl. `DB_PASSWORD`, `APP_DB_PASSWORD` — exactly the `data_key` / resettable split the drill
needs, plus a Postgres DB for class A. All gate phases ran on it, avoiding a fresh deploy (no DNS/subdomain
churn on the shared box).
### 2.2 How the real endpoints were driven (method (b))
Controller binds `8080/tcp` only on the docker bridge (not host-published) and uses **host-based routing**.
CC hit the real handler chain from the guest netns: `curl -H 'Host: felhom.demo-felhom.eu' http://172.17.0.2:8080…`.
No server logic skipped. **CSRF is not populated in this unauth deployment** (`<meta csrf-token content="">`,
no Set-Cookie), so POSTs carry an empty `_csrf` — matching what the live UI would send.
---
## 3. Per-phase results
| Phase | Result | Evidence (sentinel / flash / log) |
|-------|--------|-----------------------------------|
| **1 — Class A DB rows** (sparkyfitness) | **PASS** | Sentinel `DRILL-d2788b3c…` inserted → `POST /api/backup/run` → dump `sparkyfitness-postgres.sql` fresh (20:38:44) & contains UUID → `DROP TABLE` → `POST /backup/restore` → `302 flash=sparkyfitness visszaállítva`; log `replayed 1 DB dump(s)`; **row present again**. |
| **1 companion** (red-proof) | **PASS** | New row `DRILL2-04a98c1d…` inserted, **no** backup, restore → row **absent** afterwards, original UUID **present**. Proves restore replays the on-disk dump, not a cache/no-op. |
| **2 — Class B named-volume** | **NOT EXERCISED (Finding F3)** | No live affordance produces `volume-dumps/`. `POST /api/backup/run` = DB dumps + recovery-unit capture only. `DumpAppVolumes`/`DumpAppVolumesSafe` have **no caller**. Live-confirmed: nextcloud HAS a named volume but its unit has **no `volume-dumps/` dir**. `restoreDockerVolumes` is a no-op when no tar exists. |
| **3 — Class C HDD bind-mount** (nextcloud) | **PASS (expected-negative confirmed)** | 1 MiB sentinel `776a1d3f….bin` (sha `571f6090…`) in `appdata/nextcloud/drill` → `POST /api/backup/tier2` copied it to `felhom-flash/backups/secondary/nextcloud/appdata/drill` (sha matches) → deleted from live → **normal `POST /backup/restore`** (302 success, nextcloud healthy) → **sentinel STILL missing** from live appdata. Real recovery = operator copy-back from Tier-2 (sha `571f6090…` matched). See Finding F2. |
| **4 — Fatal data-key gate** (sparkyfitness) — SAFETY CENTREPIECE | **PASS** | Removed `API_ENCRYPTION_KEY` + `BETTER_AUTH_SECRET` from guest app.yaml (backed up) → restore **REFUSED**: `flash_error=Visszaállítás sikertelen: refusing to restore: data-encrypting key(s) [API_ENCRYPTION_KEY BETTER_AUTH_SECRET] could not be recovered…`; log `[ERROR] [backup] Restore REFUSED`. **Non-destruction proven:** container IDs unchanged (`f4609d4be80d` etc.), not restarted; sentinel row intact — gate aborts *before* `StopStack`. |
| **4 positive + reversibility** | **PASS** | With keys present: `secrets recovered=4/4, data_keys=2` + `Restore-from-unit completed` (seen in Phase 1 and again after restoring the original app.yaml). Box fully recovered; change reversible. |
| **5 — Resettable-secret path** (sparkyfitness) | **PASS (gate discriminates) + Observation O4** | Removed only `DB_PASSWORD` → restore **PROCEEDS not refused**: `[WARN] 1 resettable secret(s) unrecoverable [DB_PASSWORD] — proceeding … no data-key affected`, `recovered=3/4, data_keys=2`. It then **failed at compose-up** (`DB_PASSWORD … Defaulting to a blank string`, exit 1) — see O4. Healed by restoring app.yaml + re-running restore (`recovered=4/4`, healthy). |
| **6 — Offbox restore-to-verify** | **SKIPPED** | Offbox not configured/enabled. `POST /backup/offbox/restore` → `flash_error=A NAS mentési cél nincs beállítva.` (correct guard). |
| **7 — Real app-UI round-trip** | **PARTIAL** | Class-C round-trip proven at the filesystem layer (Phase 3, sha-verified). In-app artifact-render round-trip via the app's own UI not performed (no browser session in this run). |
| **8.1 single-flight** | **PASS** | Backup running + restore → `flash_error=…backup or restore already in progress`. |
| **8.2 path traversal (F2)** | **PASS** | `stack_name=../../etc` → `flash_error=Érvénytelen alkalmazásnév`; log `restore rejected: invalid stack_name` — **no** restore work reached `RestoreFromRecoveryUnit`. |
| **8.3 missing params** | **PASS** | empty `stack_name`/`snapshot_id` → `flash_error=Hiányzó paraméterek`. |
| **8.4 disconnected-drive skip** | **VERIFIED BY SOURCE (not live-run)** | Skip guard present `backup.go:229-238` (`IsDisconnected` / `IsDecommissioned` → `SKIP`). Not exercised live to avoid mutating drive state. |
| **8.5 restart resilience** | **PASS** | `docker restart felhom-controller` → `GET /backups` = 200, `/api/backup/status` healthy, state not wedged. |
---
## 4. Headline finding — Class C recovery (the drill's core question)
**Does the normal "Visszaállítás" button recover HDD bind-mount (class-C) data? NO — by design, and it is
now live-proven.** `RestoreFromRecoveryUnit` (and `restoreDockerVolumes`) never reference `appdata/<stack>`;
after a normal restore the deleted class-C sentinel stayed gone while the app returned healthy.
**The real class-C recovery paths that exist:**
1. **Tier-2 cross-drive copy** (`POST /api/backup/tier2` → `felhom-flash/backups/secondary/<app>/appdata/…`) —
the data survives (sha-verified) but recovery is an **operator manual copy-back**; there is **no
"restore from Tier-2" button** in the UI (the Tier-2 row only offers *Beállítás*).
2. **Offbox restore-to-verify** (`POST /backup/offbox/restore`) — restores to a **verification folder**
`<dataDir>/offbox-restore/<app>`, never in-place, and requires a configured NAS (disabled here).
3. **`.fab` bundle import** (`/import` → `appexport.restoreHDDData`) — requires a prior manual bundle export.
**Verdict / top pilot gap:** there is **no discoverable, one-click, in-place class-C restore** for a
non-technical customer. The nightly protection (Tier-2) is sound, but a customer who deletes their own files
cannot get them back from the dashboard — recovery needs an operator. **This is the headline gap for pilot.**
---
## 5. Volume-dump-trigger finding (class B)
**No user action produces `volume-dumps/` on this build.** `POST /api/backup/run` runs `runDBDumpsInternal`
(DB dumps + `captureAllRecoveryUnits`) and never calls `DumpAppVolumes`. `DumpAppVolumes` /
`DumpAppVolumesSafe` (`backup.go:303 / 375`) have **no live caller** anywhere in the tree; `captureAllRecoveryUnits`
only *enumerates* pre-existing `.tar`s. Consequence: **Docker named-volume data is not backed up by any current
trigger** — even nextcloud's `nextcloud_nextcloud_html` volume has no dump on disk. The restore side
(`restoreDockerVolumes`) is implemented and would work, but has nothing to restore. See Finding F3.
---
## 6. Gate proof (safety centrepiece)
- **Fatal refusal:** unrecoverable `data_key` → `RestoreFromRecoveryUnit` returns the refusal error and the
handler redirects `?flash_error=…refusing to restore: data-encrypting key(s) […]…`. ✔
- **Proven non-destruction:** the gate returns at `restore_unit.go:113-116`, *before* `StopStack`
(`:127`) / `restoreDockerVolumes` (`:130`). Live: container IDs unchanged, app not stopped, sentinel intact. ✔
- **Correct discrimination:** a missing **resettable** secret (`DB_PASSWORD`) is non-fatal — restore proceeds
with a WARN naming it. ✔ (`recovered=3/4, data_keys=2`).
---
## 7. Bugs / observations found (NOT fixed — candidates for follow-up TASKs)
### F1 — HIGH — UI restore is unreachable: `/api/backup/snapshots` has no handler
The restore panel (`backups.html:759`) fetches `GET /api/backup/snapshots?stack=<app>` to populate the
snapshot dropdown. **No such route is registered** — the API router's catch-all returns
`{"ok":false,"error":"endpoint not found"}` (`internal/api/router.go:316`; only `/api/backup/{status,run,tier2}`
exist). Live-confirmed for nextcloud and sparkyfitness. Because the JS then shows *"Nincs elérhető mentés"*
and leaves the **"Visszaállítás indítása" button permanently disabled**, **a customer cannot restore any app
through the dashboard.** The server-side `POST /backup/restore` handler itself works (this drill drove it
directly), so the fix is small (register the snapshots endpoint, or repopulate the dropdown from
`AppDataInfo`/`/api/backup/status`) — but as shipped the whole keep-side restore UI is dead.
*Files:* `internal/web/templates/backups.html:759`, `internal/api/router.go:251-316`.
### F2 — MEDIUM/gap — no one-click in-place class-C restore (see §4)
Design boundary, but a real customer-facing gap for pilot. Candidate: a "restore user files from Tier-2/offbox
in place" affordance, guarded by confirmation.
### F3 — MEDIUM — named-volume data never backed up (see §5)
`DumpAppVolumes*` is dead code with no trigger; apps with named volumes (e.g. nextcloud html) have no
`volume-dumps/`. Either wire a trigger (scheduler/Tier-1) or document that named-volume data is out of scope
and steer catalog apps to bind-mounts. *Files:* `internal/backup/backup.go:303,375`.
### O4 — LOW/observation — resettable-secret "proceed" path deploys a *blank* secret
When a resettable secret is genuinely unrecoverable, the gate correctly proceeds — but `RecreateStackFromUnit`
then redeploys with the secret **unset/blank** (compose `Defaulting to a blank string` → exit 1 for
sparkyfitness). "Proceeding (may need a credential reset)" is accurate, but nothing performs the reset, so the
redeploy can fail. Consider generating/resetting the resettable secret on restore rather than passing empty.
*Files:* `internal/backup/restore_unit.go:117-120,134`.
---
## 8. Cleanup confirmation
- `felhom_drill` table **dropped** from `sparkyfitness_db`; a final `POST /api/backup/run` refreshed all dumps
— **no `felhom_drill` remains in any on-disk dump** (grep clean).
- Class-C sentinel removed from **both** live (`felhom-usb`) and Tier-2 (`felhom-flash`) — no residue.
- All Phase-4/5 `app.yaml` edits reverted; **no `*.drillbak` files remain**.
- **Final health:** all app containers `healthy` (sparkyfitness, nextcloud + db/redis, rallly, calcom, gitea,
filebrowser); rallly/calcom/gitea untouched. Controller `/backups` = 200. No drill residue left to confuse a
later session.
---
*Method note:* restore/backup actions went through the real endpoints (`/backup/restore`, `/api/backup/run`,
`/api/backup/tier2`, `/backup/offbox/restore`) via authenticated in-namespace curl (runbook method (b)); the
UI equivalent (method (a)) is blocked by Finding F1. Fixture creation/verification used `pct exec` / `docker
exec` (plumbing, not the feature under test).