felhom.eu/documentation/controller/backup-architecture.md

Controller backup architecture

Source of truth: felhom-controller internal/appbackup/, internal/backup/, internal/quiesce/ at v0.59.0, updated for the v0.99.0 restore-path fixes (F1/F3/O4 — TASK C1); whole-guest backup is the agent's (felhom-agent). Line numbers are v0.59.0-era landmarks — re-grep the symbol.

This document describes what the in-guest controller actually does for backup and restore at v0.59.0. The controller was de-privileged in slice 8C: restic, cross-drive-to-arbitrary-disks execution, drive-recovery and infra-backup were removed from the controller and now live in the host agent + PBS. The restic string still appears in the backup package only inside comments that record its removal (backup.go:20, restore.go:14, restore.go:19); there is no restic execution in the controller.

1. The split: controller vs agent/PBS

The controller owns the app-data domain only. The agent owns the whole-LXC domain.

Domain	Owner	What
Per-app DB dumps	controller	pg_dump / mariadb-dump via docker exec
Per-app Docker-volume tars	controller	docker run alpine tar of named volumes
Per-app secret-free recovery unit	controller	compose + dumps + manifest.json on the app's drive
Tier-2 off-drive copy	controller	rsync mirror of an HDD app's unit + userdata to a different physical disk
Whole-LXC vzdump snapshot	agent	POST /backup, crash- or stop-consistent vzdump to local / PBS
Offsite / encrypted backup, verify, restore-test	agent + PBS	reported read-only to the controller
Whole-guest restore (rootfs, secrets, keys)	agent + PBS	the controller refuses any restore that needs this

The boundary is precise: app data that lives on an external user-data drive (an HDD bind mount) is NOT inside the PBS whole-guest snapshot — PBS cannot reach bind mounts. That data is protected only by the controller's recovery unit + Tier-2 copy. App data that lives on the rootfs (non-HDD apps) IS inside the PBS whole-guest snapshot, so it gets no Tier-2 copy (tier2.go:197-218). The encrypted app.yaml and the controller's encryption key live on the rootfs, so they are inside PBS too — which is exactly why the restore path recovers secrets from the guest rather than storing them off-rootfs (see §3, §4).

The agent surface the controller talks to is internal/agentapi/client.go: a TLS client pinning the agent's self-signed leaf by SHA-256 (client.go:69-82) and authenticating with a per-guest bearer token. Backup-relevant calls: BackupDue (GET /backup/due), StartBackup (POST /backup), BackupStatus (GET /backup/status), RestoreTestStatus (GET /restore-test/status). The whole-guest backup record (BackupRecord) and restore-test record are rendered read-only — the comment at client.go:135-136 states the controller does not own whole-guest backup.

2. Database dumps (internal/appbackup/dbdump.go)

Discovery is docker ps-driven. DiscoverDatabases (dbdump.go:66) runs docker ps --format {{.ID}}\t{{.Names}}\t{{.Image}} --filter status=running and classifies each container by image substring: postgres → Postgres, mariadb/mysql → MariaDB; everything else is skipped. The stack name is derived by stripping a known DB suffix from the container name (deriveStackName, dbdump.go:536 — postgres/db/mariadb/mysql/database/redis/cache). Connection details come from the container's own env (populateDBEnv, dbdump.go:477): POSTGRES_USER/POSTGRES_DB (defaults postgres), or MYSQL_DATABASE/MARIADB_DATABASE with root.

Per-DB dump is DumpOne (dbdump.go:162), one container at a time, 5-minute timeout each. It re-checks the container is still running, then:

• Postgres: docker exec <id> pg_dump -U <user> -d <db> --clean --if-exists --no-owner --no-privileges (dbdump.go:202).
• MariaDB: docker exec <id> mariadb-dump -u root -p<pw> --single-transaction --routines --triggers <db> (dbdump.go:228). The root password is read from the container env (MYSQL_ROOT_PASSWORD/MARIADB_ROOT_PASSWORD, dbdump.go:516); the password is never logged (a redacted -p*** form is built only for the debug log line).

The tmpfile safety (H8). The dump streams to <name>.sql.tmp, then before the rename to the final .sql: tmpFile.Sync() then tmpFile.Close() are called explicitly, each removing the tmp and failing the dump on error (dbdump.go:266-278). Only after a non-empty stat (dbdump.go:281) is the tmp atomically os.Renamed to the final path (dbdump.go:293). This guarantees the data is flushed to disk before the rename makes the dump "visible", so a crash never leaves a half-written .sql. Stale .tmp files older than one hour are reaped at the start of each run (cleanupTmpFiles, dbdump.go:553). Each finished dump is structurally validated (ValidateDump, dbdump.go:320): header line + at least one CREATE TABLE, scanned line-by-line with a bounded bufio.Reader so large data lines do not allocate (H1).

Where dumps are stored. Manager.GetAppDrivePath(stack) (backup.go:88) returns the app's HDD_PATH if it has one, else falls back to the configured systemDataPath (the internal SSD) for SSD-only apps. That drive path is mapped to its felhom-data namespace root by namespaceRoot (backup.go:105 — Model A: an in-guest drive mount IS the namespace root, so it is used as-is; only the system-data fallback gets the felhom-data subdir appended). Dumps land in AppDBDumpPath(nsRoot, stack) = <nsRoot>/backups/primary/<stack>/db-dumps/ (paths.go:58). The SSD-only fallback is the case that the C3 DR finding was about — and it is handled here: SSD-only apps get a correct path via systemDataPath, so there is no path gap. (The fallback only warns when systemDataPath itself is unconfigured, backup.go:75/backup.go:94.)

RunDBDumps (backup.go:143) acquires the running flag, discovers, dumps each DB to its app's path (skipping drives marked disconnected/decommissioned), persists each validation result to settings.json, then — v0.99.0 (F3) — runs the named-volume dumps (runVolumeDumps) and finally refreshes every recovery unit (captureAllRecoveryUnits) — even on partial failure, so units never go stale.

Volume dumps are real again (F3, v0.99.0). After the restic removal, DumpAppVolumesSafe had no caller — no trigger ever produced volume-dumps/, so named-volume app data was never captured (drill finding F3). runVolumeDumps now runs inside the same nightly/manual backup run, per deployed stack: skip if protected (IsProtectedStack), skip if it has no named volumes (this check deliberately precedes DumpAppVolumesSafe, which stops the stack before its own check), skip disconnected/decommissioned drives, else stop → tar each volume (docker run alpine tar) → restart. It runs BEFORE the unit capture so manifest.VolumeDumps enumerates the fresh tars. A per-stack failure lands in the run summary (FAIL <app> volumes:), flips the run's Success flag and fails the run — no silent partials. Note the operational consequence: volume-bearing apps are briefly stopped during each nightly backup (the locked stop-first policy — a live tar of a database volume would be torn).

Atomic volume dumps (F7, v0.118.0, CAMPAIGN-3). The volume dump now has the SAME crash-safety the DB dump has always had: DumpAppVolumes writes the tar to <vol>.tar.tmp (docker run … tar cf …/<vol>.tar.tmp), then atomicPromoteTar fsyncs the tmp (and best-effort the dir) and os.Renames it over the final .tar only on success. Before this, tar wrote the .tar IN PLACE, so a mid-write NFS cut left a 0-byte tar replacing the last good dump — and a tier-1 restore is replace-semantics, so the only "restore point" then restored an empty volume (the exact CAMPAIGN-3 F7 failure: a exportfs -u during a dump truncated a 247 MB calibre tar to 0 bytes). Now any tar error / timeout / dead-NFS EIO removes only the .tmp; the last good .tar is byte-untouched. The .tar.tmp name ends .tmp (not .tar), so it is invisible to the restore-point and stale scans; orphan .tar.tmp from a killed run is swept on the next dump. Live-proven 2026-07-12: an exportfs -u mid-volume-dump left all NAS volume tars byte-identical (sha unchanged), no 0-byte file, run success:false; the next run produced fresh good tars.

Stale-primary sweep (F5, v0.118.0). After the units are refreshed, pruneStalePrimaryDirs removes an orphaned backups/primary/<app> dir an app left on an OLD drive when its HDD_PATH moved (invisible disk residue). Guards: removes only when the app is deployed AND its current namespace root differs from the dir's drive; never the current-drive dir (the live restore point) or an undeployed app's dir; strictly under a backups/primary/ prefix.

3. Recovery units (internal/backup/recovery_unit.go)

A recovery unit is a per-app, secret-free, self-contained directory at <nsRoot>/backups/primary/<app>/ (paths.go:42). It contains:

• compose/ — docker-compose.yml + .felhom.yml + a secret-stripped app.yaml.
• db-dumps/ — the .sql dumps from §2.
• volume-dumps/ — named-volume .tar archives.
• manifest.json — the RecoveryManifest (recovery_unit.go:31).

What the unit excludes: it holds no secret values, no data-encrypting keys, and not the Docker image. The manifest stores only the pinned image tag(s) (ImagePins — re-pulled on restore), the names of the secret env vars (SecretEnvVars), and the names of the data-key env vars (DataKeyEnvVars); SecretSource records in plain text that the values come from "guest app.yaml (live rootfs) or PBS whole-guest snapshot — never stored in this unit" (recovery_unit.go:141). The stripped app.yaml carries only non-secret env, with a header naming the omitted secrets (buildStrippedAppYaml, recovery_unit.go:188).

CaptureRecoveryUnit (recovery_unit.go:68) pulls the app's RecoveryInfo from the stack provider (GetStackRecoveryInfo), builds the captured content in memory, and is idempotent: it skips all drive writes when the existing manifest matches the current controller version, config checksums (sha256 of each captured file), and the DB/volume dump set (recovery_unit.go:112-118). This is what lets it run on the 5-minute status refresh without thrashing a spinning USB drive. Writes are atomic (atomicWrite, recovery_unit.go:270 — tmp + rename). DataKeyEnvVars is a fail-closed restore annotation only (see §4); it does not affect capture.

The .fab portable export/import path (internal/appexport/) is a separate, operator-driven mechanism documented elsewhere — cross-reference that doc; it is not the periodic recovery unit.

4. Restore

Two keep-side restore entry points exist; neither does a whole-guest restore (that is the agent's).

RestoreApp(stack, snapshotID) (restore.go:21) — the legacy volume-only path. Stops the stack, re-imports the named-volume .tar dumps (restoreDockerVolumes, restore.go:85: docker volume rm -f + create + docker run alpine tar xf), restarts, and health-checks. snapshotID is retained only for signature/logging compatibility now that restic is gone (restore.go:19-20).

RestoreFromRecoveryUnit(stack) (restore_unit.go:74) — the recovery-unit path. It reads the unit manifest (falling back to RestoreApp if no unit exists, restore_unit.go:99), recovers the secret values from the guest's own live app.yaml via stackProvider.RecoverStackSecrets (never from the unit), reconciles them, restores the named-volume data, then RecreateStackFromUnit rebuilds the app's definition from compose/ and redeploys with the reconstructed env (re-pulling the pinned image). Nothing is regenerated; no secret is read from the unit.

The fail-closed data-key gate is reconcileRestoreSecrets (restore_unit.go:22) — a pure, unit-tested function. It merges non-secret env with recovered secrets, then:

• A missing resettable secret (DB password, admin password) is non-fatal: returned in missing. v0.99.0 (O4): the caller now generates a replacement from the field's catalog generate spec (stacks.GenerateSecretForField via the backup.SetSecretGenerator seam) instead of proceeding with the secret blank — pre-O4, the redeploy failed compose-up with "Defaulting to a blank string". The generated value rides fullEnv into RecreateStackFromUnit → SaveAppConfig, so it persists encrypted in the guest app.yaml and round-trips on later backups. Fields with no generate spec still proceed, with a WARN that the app may fail to start. Generation NEVER applies to data-keys (the gate below refuses first, and the generator itself rejects data_key fields). Residual case (honesty note): generation fully fixes the fresh-init path (empty DB volume → DB initialises with the new credential → dump replay restores the data). If a restored volume tar carries the OLD internal credential hash, the app may still fail auth until a manual in-DB credential reset — generation does not cover that.
• A missing data-encrypting key (DataKeyEnvVars) is fatal: the restore is refused with an explicit error directing the operator to do a PBS whole-guest restore first, because regenerating the key would render the stored data unreadable (restore_unit.go:45-50). This is the safety centerpiece: the controller never silently recreates an app whose data it can no longer decrypt.

The restore panel's data source (F1, v0.99.0). GET /api/backup/snapshots?stack=<app> is backed by backup.Manager.ListRestorePoints (internal/backup/restore_points.go). The route was a restic-era leftover the backups.html template still fetched; it was never registered, so the snapshot dropdown could never populate and the restore button never enabled (drill finding F1 — the whole keep-side restore UI was dead). It returns at most ONE entry — the current recovery unit (time = newest artifact mtime among manifest/db-dumps/volume-dumps, short_id:"helyi", tier:1, drive_label from the storage registry). Tier-2 entries are never emitted: POST /backup/restore only reads the primary unit, so a tier-2 listing would silently restore tier-1 data while claiming tier-2. Guards: invalid/empty stack name → 400, unknown stack → 404, no unit yet → ok:true, data:[].

Class-C in-place file restore (C2, v0.100.0 — closes drill finding F2). HDD bind-mount user files (appdata/<stack>) are outside both keep-side restore paths above (the drill's expected-negative). The customer path is now POST /backup/tier2/restore → RestoreTier2Files (internal/backup/tier2_restore.go), surfaced as the "Fájlok visszaállítása" button on the Tier-2 row. Semantics are deliberately additive-only (rsync -a --ignore-existing — rsyncRestoreMissing, the exec shape of rsyncMirror with the opposite-direction flags):

• a file missing live is copied back from the recorded Tier-2 copy (attrs preserved);
• an existing live file is never overwritten — a customer edit after the last nightly copy always wins;
• nothing is ever deleted — rsyncMirror's --delete in this direction would erase every file created since the last copy, which is why that helper must never be used tier2→live.

The source is the RECORDED CrossDriveBackup.DestinationPath (never a fresh selectTier2Target, which could re-pick an empty drive). Refusals — no Tier-2 copy / never ran / copy dir gone / either drive disconnected / live drive decommissioned — happen before the app is stopped, with customer-readable Hungarian reasons; the flow is stop → copy → restart → health-wait, single-flight with backup/restore. Zero files copied is a success ("Nincs hiányzó fájl — minden fájl megvan a helyén."), and re-running the button is idempotent.

What it deliberately does NOT do: no overwrite/point-in-time restore (corruption rollback stays with the offbox restore-to-verify + operator paths), no per-file selection, and it never touches recovery-unit/ under the Tier-2 dir (backup artifacts are not user files). App-reindex caveat: the restore is filesystem-level — apps that index their data dir (e.g. Nextcloud → occ files:scan) may need a rescan before restored files appear in the app's own UI.

4b. Security note — .fab import path validation (CTRL-001, v0.59.0)

The portable .fab import (internal/appexport/) validates every manifest path segment before it reaches a filepath.Join against a trusted base (appexport.ValidateSegment, validate.go:28; validateManifestPaths, validate.go:51, called from UnmarshalManifest). The attacker-controllable AppName / HDDSubdirs / VolumeNames are rejected on any .., path separator, or absolute path, closing the v0.59.0 path-traversal finding. This is cross-referenced here; the detail lives in the appexport doc and the v0.59.0 audit record.

5. Tier-2 cross-drive (internal/backup/tier2.go)

Tier 2 is the only off-drive protection browsable HDD userdata can get (PBS cannot reach bind mounts, tier2.go:17-23). It is an rsync -a --delete mirror (rsyncMirror, tier2.go:358) of an HDD app's recovery unit + bulk appdata/ to <target>/backups/secondary/<app>/{recovery-unit,appdata}/ on a different physical disk.

No single-copy backups (F6, v0.118.0, CAMPAIGN-3). RunAllTier2 used to continue on every non-HDD app, so a volume-only app (no HDD_PATH, its dumps on sys_drive) got NO tier-2 copy — a single controller-level copy on one device. It now flows through too: its recovery unit (holding the db/volume dumps) gets the cross-drive second copy like any HDD app (live-proven 2026-07-12: actualbudget/seerr appear under felhom-usb/backups/secondary/). A sys_drive app's restore-point drive label is now the clear "Belső SSD (rendszer)", never blank.

3-2-1 honesty on a single-drive box (F6). When NO off-drive target exists (Manager.hasOffDriveTarget false — only the system drive, no enrolled second disk), there is genuinely only ONE local copy. FullBackupStatus.SingleCopyWarning surfaces an honest Hungarian banner on the backup page ("Csak egy másolat készül (nincs második meghajtó) — a 3-2-1 mentéshez csatlakoztasson egy második meghajtót vagy offsite tárolót.") rather than implying a 3-2-1 guarantee the box cannot keep.

NAS backup locality (Part 4 — decision A, v0.118.0). A NAS-resident app's tier-1 dumps live on the NAS itself (nas-media/backups/primary/<app>), beside the data. During a NAS outage both the app data AND its freshest tier-1 dump are on the dead device — the tier-2 cross-drive copy to a local drive is the off-NAS leg that saves them, and only after it has run. This locality was kept deliberately in the operator fork (over retargeting tier-1 to a local drive, which would have moved restore-point resolution off the drive); the tier-2 copy is the mitigation. Documented here so the outage window is never a surprise.

Auto-target selection (selectTier2Target, tier2.go:54), in order:

1. A customer-pinned target (PreferredTarget from the config panel) if it is still registered, schedulable, and off-disk (tier2.go:62-83).
2. Another registered user-data drive on a different physical disk — can hold bulk userdata (tier2.go:86-102).
3. The internal SSD (system data path) — small units only, headroom-guarded.

Off-disk-ness is decided by system.SamePhysicalDevice (a Stat_t.Dev compare). If the only candidate is the same physical disk, errNoOffDiskTarget is returned.

The rootfs-headroom guard is the key safety. The internal SSD is the ~8 GB guest rootfs, so option 3 refuses rather than fills: tier2FitsSystemDrive (tier2.go:121) → tier2FitsHeadroom (tier2.go:43) requires the copy to leave a reserve of max(2 GB, 20% of total) free, else returns errSSDNoHeadroom. A non-fitting or single-drive case is recorded as an honest no_target status (recordTier2NoTarget, tier2.go:331) with a Hungarian "needs a 2nd HDD" reason — the rootfs is never filled. When the SSD target is used, it is labelled DB/config-only (tier2.go:116, log suffix [SSD: DB/config only]).

RunAllTier2 (tier2.go:199) iterates deployed stacks, processes only those with an HDD_PATH (non-HDD apps are skipped — they are already in PBS), and skips disconnected/decommissioned drives. Status is persisted into settings.CrossDriveBackup (method rsync), with the customer-preference fields (UserDisabled, PreferredTarget) preserved across runner writes by withTier2Prefs (tier2.go:290). The config-panel view is Tier2Info (tier2.go:245, read-only).

6. Concurrency, scheduling, and the quiesce loop

Single-flight / running mutex. Manager guards a running flag with a mutex; acquireRunning/releaseRunning (backup.go:373) reject a second backup or restore with "already in progress". RestoreApp and RestoreFromRecoveryUnit take the same flag (restore.go:31, restore_unit.go:79).

Scheduling (wired in cmd/controller/main.go, Europe/Budapest):

• db-dump daily at cfg.Backup.DBDumpSchedule (default 02:30, config.go:271) → RunDBDumps (which also refreshes recovery units) (main.go:328).
• tier2-backup daily at 03:30 → RunAllTier2 (main.go:360).
• RefreshCache runs on the 5-minute status refresh, re-scanning dump files and (idempotently) re-capturing recovery units (backup.go:477).

Manual triggers exist via the API router (RunDBDumps, RunAllTier2 launched as goroutines, router.go:764/781).

The quiesce loop (internal/quiesce/quiesce.go) drives the whole-guest backup app-consistently. The agent's vzdump is crash-consistent only (an LXC has no fsfreeze), so the controller stops the app stacks first. Loop.Run polls GET /backup/due; when due, quiesceAndPoll (quiesce.go:205):

1. Writes a persisted marker (atomic, 0600) listing the stacks it is about to stop — before stopping anything (quiesce.go:207).
2. Stops the running app stacks.
3. POST /backup, records the job id, polls GET /backup/status.
4. Resumes early at the snapshotted phase (8B.2 downtime optimization — the storage snapshot has captured the stopped state, so the app may come back up; the loop keeps polling to done/failed), or at done in stop/downgraded mode (quiesce.go:256-279).

Unquiesce is guaranteed: a deferred closure restarts exactly the stopped stacks and clears the marker on every exit path — backup error, status-poll error, the MaxQuiesce bound (default 30 min, restarts the app while the backup continues on the agent), or context cancellation (quiesce.go:213-225, quiesce.go:243-249). On startup, Recover (quiesce.go:113) restarts any stacks left stopped by a mid-quiesce crash, then clears the marker. Single-flight is enforced both within the process (a TryLock mutex shared by the scheduled loop and the manual TriggerNow, quiesce.go:149/quiesce.go:181) and across restarts (the active marker — a cycle refuses to start on top of one, quiesce.go:157). TriggerNow runs the same flow asynchronously for the manual "Mentés most" action, returning ErrBackupInProgress if a cycle is already running.

Restic is gone from the controller. All disk-tier backup (restic snapshots, cross-drive-to-other-disks, drive recovery, infra backup) is the agent's; the controller's only remaining off-drive copy is the Tier-2 rsync mirror in §5.