felhom.eu/documentation/audits/DRILL-appdata-restore-2026-07-04.md

DRILL — App-data restore (keep-side), guest 9201 / felhom-pve — 2026-07-04

Class: Runbook-style validation (unsupervised, throwaway demo box). No production code changed. Executed by: Claude Code (CC), live SSH root@felhom-pve + in-guest pct exec 9201, real controller endpoints via authenticated in-namespace curl (method (b) per the runbook). Hard rail honoured: every destructive step was scoped to guest 9201 on host felhom-pve (demo-felhom). sajatfelhom.hu / Peti's box and every other guest were never touched. wg show dump never run.

---

1. Confirmed baselines (reconfirmed against live Gitea + live host)

Repo	main @ commit	Version	Reconfirm
felhom-controller	a518023	v0.98.3	local HEAD == runbook commit; HEAD..origin/main = 0
felhom-agent	72d7f05	v0.69.0	local HEAD == runbook commit; HEAD..origin/main = 0
app-catalog-felhom.eu	13eedb1	—	HEAD == runbook commit

• Live controller image on 9201: gitea.dooplex.hu/admin/felhom-controller:0.98.3 (healthy).
• Live agent on felhom-pve: felhom-agent 0.69.0. Node demo-felhom, pve-manager/9.2.2.
• Identity rail re-asserted before every destructive step: host felhom-pve (192.168.0.162), guest 9201.

---

2. Environment ground truth (Phase 0)

Deployed stacks on 9201: sparkyfitness (+ -server, -db postgres15), rallly (+ postgres16), nextcloud (+ mariadb 11.6, + redis), gitea, calcom (+ postgres16), filebrowser; infra: felhom-controller, cloudflared, traefik. AdventureLog was NOT deployed — see §2.1 for the substitute.

Enrolled user-data drives (in-guest mounts = namespace roots):

Drive	Device	In-guest mount = nsRoot	Schedulable
felhom-flash	/dev/sdb1	/mnt/felhom-drives/felhom-flash	yes
felhom-usb	/dev/sdd1	/mnt/felhom-drives/felhom-usb	yes
teszt_enroll	/dev/sdc	/mnt/felhom-drives/teszt_enroll	yes

No felhom-data/felhom-data double-nesting observed. Apps with no dedicated user-data drive fall back to the SSD system path /mnt/sys_drive/felhom-data (inGuestDrive=false, so felhom-data is appended there).

Per-app data-class map + nsRoot:

App	nsRoot	Class A (DB)	Class B (named vol)	Class C (HDD bind)	data_key env vars
sparkyfitness	/mnt/sys_drive/felhom-data	postgres ✓	none (volume_dumps: null)	none	API_ENCRYPTION_KEY, BETTER_AUTH_SECRET
nextcloud	/mnt/felhom-drives/felhom-usb	mariadb ✓	nextcloud_nextcloud_html (→ /var/www/html)	appdata/nextcloud (→ /var/www/html/data, 37 MB)	none
rallly / calcom / gitea	/mnt/sys_drive/felhom-data	postgres ✓	—	—	none

Pre-existing backup artifacts: recovery units (compose/ + db-dumps/ + manifest.json) present for the four sys_drive apps and nextcloud; no volume-dumps/ anywhere (see Finding F3). felhom-flash holds stale recovery units for undeployed apps (immich, paperless-ngx, jellyfin, …). Offbox target configured but disabled (enabled:false, host 192.168.0.128).

2.1 Substitute for AdventureLog

AdventureLog (the runbook's fatal-gate app) was not deployed. sparkyfitness is a faithful substitute: its recovery-unit manifest carries data_key_env_vars: [API_ENCRYPTION_KEY, BETTER_AUTH_SECRET] and resettable secret_env_vars incl. DB_PASSWORD, APP_DB_PASSWORD — exactly the data_key / resettable split the drill needs, plus a Postgres DB for class A. All gate phases ran on it, avoiding a fresh deploy (no DNS/subdomain churn on the shared box).

2.2 How the real endpoints were driven (method (b))

Controller binds 8080/tcp only on the docker bridge (not host-published) and uses host-based routing. CC hit the real handler chain from the guest netns: curl -H 'Host: felhom.demo-felhom.eu' http://172.17.0.2:8080…. No server logic skipped. CSRF is not populated in this unauth deployment (<meta csrf-token content="">, no Set-Cookie), so POSTs carry an empty _csrf — matching what the live UI would send.

---

3. Per-phase results

Phase	Result	Evidence (sentinel / flash / log)
1 — Class A DB rows (sparkyfitness)	PASS	Sentinel DRILL-d2788b3c… inserted → POST /api/backup/run → dump sparkyfitness-postgres.sql fresh (20:38:44) & contains UUID → DROP TABLE → POST /backup/restore → 302 flash=sparkyfitness visszaállítva; log replayed 1 DB dump(s); row present again.
1 companion (red-proof)	PASS	New row DRILL2-04a98c1d… inserted, no backup, restore → row absent afterwards, original UUID present. Proves restore replays the on-disk dump, not a cache/no-op.
2 — Class B named-volume	NOT EXERCISED (Finding F3)	No live affordance produces volume-dumps/. POST /api/backup/run = DB dumps + recovery-unit capture only. DumpAppVolumes/DumpAppVolumesSafe have no caller. Live-confirmed: nextcloud HAS a named volume but its unit has no volume-dumps/ dir. restoreDockerVolumes is a no-op when no tar exists.
3 — Class C HDD bind-mount (nextcloud)	PASS (expected-negative confirmed)	1 MiB sentinel 776a1d3f….bin (sha 571f6090…) in appdata/nextcloud/drill → POST /api/backup/tier2 copied it to felhom-flash/backups/secondary/nextcloud/appdata/drill (sha matches) → deleted from live → normal POST /backup/restore (302 success, nextcloud healthy) → sentinel STILL missing from live appdata. Real recovery = operator copy-back from Tier-2 (sha 571f6090… matched). See Finding F2.
4 — Fatal data-key gate (sparkyfitness) — SAFETY CENTREPIECE	PASS	Removed API_ENCRYPTION_KEY + BETTER_AUTH_SECRET from guest app.yaml (backed up) → restore REFUSED: flash_error=Visszaállítás sikertelen: refusing to restore: data-encrypting key(s) [API_ENCRYPTION_KEY BETTER_AUTH_SECRET] could not be recovered…; log [ERROR] [backup] Restore REFUSED. Non-destruction proven: container IDs unchanged (f4609d4be80d etc.), not restarted; sentinel row intact — gate aborts before StopStack.
4 positive + reversibility	PASS	With keys present: secrets recovered=4/4, data_keys=2 + Restore-from-unit completed (seen in Phase 1 and again after restoring the original app.yaml). Box fully recovered; change reversible.
5 — Resettable-secret path (sparkyfitness)	PASS (gate discriminates) + Observation O4	Removed only DB_PASSWORD → restore PROCEEDS not refused: [WARN] 1 resettable secret(s) unrecoverable [DB_PASSWORD] — proceeding … no data-key affected, recovered=3/4, data_keys=2. It then failed at compose-up (DB_PASSWORD … Defaulting to a blank string, exit 1) — see O4. Healed by restoring app.yaml + re-running restore (recovered=4/4, healthy).
6 — Offbox restore-to-verify	SKIPPED	Offbox not configured/enabled. POST /backup/offbox/restore → flash_error=A NAS mentési cél nincs beállítva. (correct guard).
7 — Real app-UI round-trip	PARTIAL	Class-C round-trip proven at the filesystem layer (Phase 3, sha-verified). In-app artifact-render round-trip via the app's own UI not performed (no browser session in this run).
8.1 single-flight	PASS	Backup running + restore → flash_error=…backup or restore already in progress.
8.2 path traversal (F2)	PASS	stack_name=../../etc → flash_error=Érvénytelen alkalmazásnév; log restore rejected: invalid stack_name — no restore work reached RestoreFromRecoveryUnit.
8.3 missing params	PASS	empty stack_name/snapshot_id → flash_error=Hiányzó paraméterek.
8.4 disconnected-drive skip	VERIFIED BY SOURCE (not live-run)	Skip guard present backup.go:229-238 (IsDisconnected / IsDecommissioned → SKIP). Not exercised live to avoid mutating drive state.
8.5 restart resilience	PASS	docker restart felhom-controller → GET /backups = 200, /api/backup/status healthy, state not wedged.

---

4. Headline finding — Class C recovery (the drill's core question)

Does the normal "Visszaállítás" button recover HDD bind-mount (class-C) data? NO — by design, and it is now live-proven. RestoreFromRecoveryUnit (and restoreDockerVolumes) never reference appdata/<stack>; after a normal restore the deleted class-C sentinel stayed gone while the app returned healthy.

The real class-C recovery paths that exist:

1. Tier-2 cross-drive copy (POST /api/backup/tier2 → felhom-flash/backups/secondary/<app>/appdata/…) — the data survives (sha-verified) but recovery is an operator manual copy-back; there is no "restore from Tier-2" button in the UI (the Tier-2 row only offers Beállítás).
2. Offbox restore-to-verify (POST /backup/offbox/restore) — restores to a verification folder <dataDir>/offbox-restore/<app>, never in-place, and requires a configured NAS (disabled here).
3. .fab bundle import (/import → appexport.restoreHDDData) — requires a prior manual bundle export.

Verdict / top pilot gap: there is no discoverable, one-click, in-place class-C restore for a non-technical customer. The nightly protection (Tier-2) is sound, but a customer who deletes their own files cannot get them back from the dashboard — recovery needs an operator. This is the headline gap for pilot.

---

5. Volume-dump-trigger finding (class B)

No user action produces volume-dumps/ on this build. POST /api/backup/run runs runDBDumpsInternal (DB dumps + captureAllRecoveryUnits) and never calls DumpAppVolumes. DumpAppVolumes / DumpAppVolumesSafe (backup.go:303 / 375) have no live caller anywhere in the tree; captureAllRecoveryUnits only enumerates pre-existing .tars. Consequence: Docker named-volume data is not backed up by any current trigger — even nextcloud's nextcloud_nextcloud_html volume has no dump on disk. The restore side (restoreDockerVolumes) is implemented and would work, but has nothing to restore. See Finding F3.

---

6. Gate proof (safety centrepiece)

• Fatal refusal: unrecoverable data_key → RestoreFromRecoveryUnit returns the refusal error and the handler redirects ?flash_error=…refusing to restore: data-encrypting key(s) […]…. ✔
• Proven non-destruction: the gate returns at restore_unit.go:113-116, before StopStack (:127) / restoreDockerVolumes (:130). Live: container IDs unchanged, app not stopped, sentinel intact. ✔
• Correct discrimination: a missing resettable secret (DB_PASSWORD) is non-fatal — restore proceeds with a WARN naming it. ✔ (recovered=3/4, data_keys=2).

---

7. Bugs / observations found (NOT fixed — candidates for follow-up TASKs)

F1 — HIGH — UI restore is unreachable: /api/backup/snapshots has no handler

The restore panel (backups.html:759) fetches GET /api/backup/snapshots?stack=<app> to populate the snapshot dropdown. No such route is registered — the API router's catch-all returns {"ok":false,"error":"endpoint not found"} (internal/api/router.go:316; only /api/backup/{status,run,tier2} exist). Live-confirmed for nextcloud and sparkyfitness. Because the JS then shows "Nincs elérhető mentés" and leaves the "Visszaállítás indítása" button permanently disabled, a customer cannot restore any app through the dashboard. The server-side POST /backup/restore handler itself works (this drill drove it directly), so the fix is small (register the snapshots endpoint, or repopulate the dropdown from AppDataInfo//api/backup/status) — but as shipped the whole keep-side restore UI is dead. Files: internal/web/templates/backups.html:759, internal/api/router.go:251-316.

F2 — MEDIUM/gap — no one-click in-place class-C restore (see §4)

Design boundary, but a real customer-facing gap for pilot. Candidate: a "restore user files from Tier-2/offbox in place" affordance, guarded by confirmation.

F3 — MEDIUM — named-volume data never backed up (see §5)

DumpAppVolumes* is dead code with no trigger; apps with named volumes (e.g. nextcloud html) have no volume-dumps/. Either wire a trigger (scheduler/Tier-1) or document that named-volume data is out of scope and steer catalog apps to bind-mounts. Files: internal/backup/backup.go:303,375.

O4 — LOW/observation — resettable-secret "proceed" path deploys a blank secret

When a resettable secret is genuinely unrecoverable, the gate correctly proceeds — but RecreateStackFromUnit then redeploys with the secret unset/blank (compose Defaulting to a blank string → exit 1 for sparkyfitness). "Proceeding (may need a credential reset)" is accurate, but nothing performs the reset, so the redeploy can fail. Consider generating/resetting the resettable secret on restore rather than passing empty. Files: internal/backup/restore_unit.go:117-120,134.

---

8. Cleanup confirmation

• felhom_drill table dropped from sparkyfitness_db; a final POST /api/backup/run refreshed all dumps — no felhom_drill remains in any on-disk dump (grep clean).
• Class-C sentinel removed from both live (felhom-usb) and Tier-2 (felhom-flash) — no residue.
• All Phase-4/5 app.yaml edits reverted; no *.drillbak files remain.
• Final health: all app containers healthy (sparkyfitness, nextcloud + db/redis, rallly, calcom, gitea, filebrowser); rallly/calcom/gitea untouched. Controller /backups = 200. No drill residue left to confuse a later session.

---

Method note: restore/backup actions went through the real endpoints (/backup/restore, /api/backup/run, /api/backup/tier2, /backup/offbox/restore) via authenticated in-namespace curl (runbook method (b)); the UI equivalent (method (a)) is blocked by Finding F1. Fixture creation/verification used pct exec / docker exec (plumbing, not the feature under test).