felhom.eu/documentation/controller/backup-architecture.md

Controller backup architecture

Source of truth: felhom-controller internal/appbackup/, internal/backup/, internal/quiesce/ at v0.59.0, updated for the v0.99.0 restore-path fixes (F1/F3/O4 — TASK C1); whole-guest backup is the agent's (felhom-agent). Line numbers are v0.59.0-era landmarks — re-grep the symbol.

This document describes what the in-guest controller actually does for backup and restore at v0.59.0. The controller was de-privileged in slice 8C: restic, cross-drive-to-arbitrary-disks execution, drive-recovery and infra-backup were removed from the controller and now live in the host agent + PBS. The restic string still appears in the backup package only inside comments that record its removal (backup.go:20, restore.go:14, restore.go:19); there is no restic execution in the controller.

1. The split: controller vs agent/PBS

The controller owns the app-data domain only. The agent owns the whole-LXC domain.

Domain	Owner	What
Per-app DB dumps	controller	pg_dump / mariadb-dump via docker exec
Per-app Docker-volume tars	controller	docker run alpine tar of named volumes
Per-app secret-free recovery unit	controller	compose + dumps + manifest.json on the app's drive
Tier-2 off-drive copy	controller	rsync mirror of an HDD app's unit + userdata to a different physical disk
Whole-LXC vzdump snapshot	agent	POST /backup, crash- or stop-consistent vzdump to local / PBS
Offsite / encrypted backup, verify, restore-test	agent + PBS	reported read-only to the controller
Whole-guest restore (rootfs, secrets, keys)	agent + PBS	the controller refuses any restore that needs this

The boundary is precise: app data that lives on an external user-data drive (an HDD bind mount) is NOT inside the PBS whole-guest snapshot — PBS cannot reach bind mounts. That data is protected only by the controller's recovery unit + Tier-2 copy. App data that lives on the rootfs (non-HDD apps) IS inside the PBS whole-guest snapshot, so it gets no Tier-2 copy (tier2.go:197-218). The encrypted app.yaml and the controller's encryption key live on the rootfs, so they are inside PBS too — which is exactly why the restore path recovers secrets from the guest rather than storing them off-rootfs (see §3, §4).

The agent surface the controller talks to is internal/agentapi/client.go: a TLS client pinning the agent's self-signed leaf by SHA-256 (client.go:69-82) and authenticating with a per-guest bearer token. Backup-relevant calls: BackupDue (GET /backup/due), StartBackup (POST /backup), BackupStatus (GET /backup/status), RestoreTestStatus (GET /restore-test/status). The whole-guest backup record (BackupRecord) and restore-test record are rendered read-only — the comment at client.go:135-136 states the controller does not own whole-guest backup.

2. Database dumps (internal/appbackup/dbdump.go)

Discovery is docker ps-driven. DiscoverDatabases (dbdump.go:66) runs docker ps --format {{.ID}}\t{{.Names}}\t{{.Image}} --filter status=running and classifies each container by image substring: postgres → Postgres, mariadb/mysql → MariaDB; everything else is skipped. The stack name is derived by stripping a known DB suffix from the container name (deriveStackName, dbdump.go:536 — postgres/db/mariadb/mysql/database/redis/cache). Connection details come from the container's own env (populateDBEnv, dbdump.go:477): POSTGRES_USER/POSTGRES_DB (defaults postgres), or MYSQL_DATABASE/MARIADB_DATABASE with root.

Per-DB dump is DumpOne (dbdump.go:162), one container at a time, 5-minute timeout each. It re-checks the container is still running, then:

• Postgres: docker exec <id> pg_dump -U <user> -d <db> --clean --if-exists --no-owner --no-privileges (dbdump.go:202).
• MariaDB: docker exec <id> mariadb-dump -u root -p<pw> --single-transaction --routines --triggers <db> (dbdump.go:228). The root password is read from the container env (MYSQL_ROOT_PASSWORD/MARIADB_ROOT_PASSWORD, dbdump.go:516); the password is never logged (a redacted -p*** form is built only for the debug log line).

The tmpfile safety (H8). The dump streams to <name>.sql.tmp, then before the rename to the final .sql: tmpFile.Sync() then tmpFile.Close() are called explicitly, each removing the tmp and failing the dump on error (dbdump.go:266-278). Only after a non-empty stat (dbdump.go:281) is the tmp atomically os.Renamed to the final path (dbdump.go:293). This guarantees the data is flushed to disk before the rename makes the dump "visible", so a crash never leaves a half-written .sql. Stale .tmp files older than one hour are reaped at the start of each run (cleanupTmpFiles, dbdump.go:553). Each finished dump is structurally validated (ValidateDump, dbdump.go:320): header line + at least one CREATE TABLE, scanned line-by-line with a bounded bufio.Reader so large data lines do not allocate (H1).

Where dumps are stored. Manager.GetAppDrivePath(stack) (backup.go:88) returns the app's HDD_PATH if it has one, else falls back to the configured systemDataPath (the internal SSD) for SSD-only apps. That drive path is mapped to its felhom-data namespace root by namespaceRoot (backup.go:105 — Model A: an in-guest drive mount IS the namespace root, so it is used as-is; only the system-data fallback gets the felhom-data subdir appended). Dumps land in AppDBDumpPath(nsRoot, stack) = <nsRoot>/backups/primary/<stack>/db-dumps/ (paths.go:58). The SSD-only fallback is the case that the C3 DR finding was about — and it is handled here: SSD-only apps get a correct path via systemDataPath, so there is no path gap. (The fallback only warns when systemDataPath itself is unconfigured, backup.go:75/backup.go:94.)

RunDBDumps (backup.go:143) acquires the running flag, discovers, dumps each DB to its app's path (skipping drives marked disconnected/decommissioned), persists each validation result to settings.json, then — v0.99.0 (F3) — runs the named-volume dumps (runVolumeDumps) and finally refreshes every recovery unit (captureAllRecoveryUnits) — even on partial failure, so units never go stale.

Volume dumps are real again (F3, v0.99.0). After the restic removal, DumpAppVolumesSafe had no caller — no trigger ever produced volume-dumps/, so named-volume app data was never captured (drill finding F3). runVolumeDumps now runs inside the same nightly/manual backup run, per deployed stack: skip if protected (IsProtectedStack), skip if it has no named volumes (this check deliberately precedes DumpAppVolumesSafe, which stops the stack before its own check), skip disconnected/decommissioned drives, else stop → tar each volume (docker run alpine tar) → restart. It runs BEFORE the unit capture so manifest.VolumeDumps enumerates the fresh tars. A per-stack failure lands in the run summary (FAIL <app> volumes:), flips the run's Success flag and fails the run — no silent partials. Note the operational consequence: volume-bearing apps are briefly stopped during each nightly backup (the locked stop-first policy — a live tar of a database volume would be torn).

3. Recovery units (internal/backup/recovery_unit.go)

A recovery unit is a per-app, secret-free, self-contained directory at <nsRoot>/backups/primary/<app>/ (paths.go:42). It contains:

• compose/ — docker-compose.yml + .felhom.yml + a secret-stripped app.yaml.
• db-dumps/ — the .sql dumps from §2.
• volume-dumps/ — named-volume .tar archives.
• manifest.json — the RecoveryManifest (recovery_unit.go:31).

What the unit excludes: it holds no secret values, no data-encrypting keys, and not the Docker image. The manifest stores only the pinned image tag(s) (ImagePins — re-pulled on restore), the names of the secret env vars (SecretEnvVars), and the names of the data-key env vars (DataKeyEnvVars); SecretSource records in plain text that the values come from "guest app.yaml (live rootfs) or PBS whole-guest snapshot — never stored in this unit" (recovery_unit.go:141). The stripped app.yaml carries only non-secret env, with a header naming the omitted secrets (buildStrippedAppYaml, recovery_unit.go:188).

CaptureRecoveryUnit (recovery_unit.go:68) pulls the app's RecoveryInfo from the stack provider (GetStackRecoveryInfo), builds the captured content in memory, and is idempotent: it skips all drive writes when the existing manifest matches the current controller version, config checksums (sha256 of each captured file), and the DB/volume dump set (recovery_unit.go:112-118). This is what lets it run on the 5-minute status refresh without thrashing a spinning USB drive. Writes are atomic (atomicWrite, recovery_unit.go:270 — tmp + rename). DataKeyEnvVars is a fail-closed restore annotation only (see §4); it does not affect capture.

The .fab portable export/import path (internal/appexport/) is a separate, operator-driven mechanism documented elsewhere — cross-reference that doc; it is not the periodic recovery unit.

4. Restore

Two keep-side restore entry points exist; neither does a whole-guest restore (that is the agent's).

RestoreApp(stack, snapshotID) (restore.go:21) — the legacy volume-only path. Stops the stack, re-imports the named-volume .tar dumps (restoreDockerVolumes, restore.go:85: docker volume rm -f + create + docker run alpine tar xf), restarts, and health-checks. snapshotID is retained only for signature/logging compatibility now that restic is gone (restore.go:19-20).

RestoreFromRecoveryUnit(stack) (restore_unit.go:74) — the recovery-unit path. It reads the unit manifest (falling back to RestoreApp if no unit exists, restore_unit.go:99), recovers the secret values from the guest's own live app.yaml via stackProvider.RecoverStackSecrets (never from the unit), reconciles them, restores the named-volume data, then RecreateStackFromUnit rebuilds the app's definition from compose/ and redeploys with the reconstructed env (re-pulling the pinned image). Nothing is regenerated; no secret is read from the unit.

The fail-closed data-key gate is reconcileRestoreSecrets (restore_unit.go:22) — a pure, unit-tested function. It merges non-secret env with recovered secrets, then:

• A missing resettable secret (DB password, admin password) is non-fatal: returned in missing. v0.99.0 (O4): the caller now generates a replacement from the field's catalog generate spec (stacks.GenerateSecretForField via the backup.SetSecretGenerator seam) instead of proceeding with the secret blank — pre-O4, the redeploy failed compose-up with "Defaulting to a blank string". The generated value rides fullEnv into RecreateStackFromUnit → SaveAppConfig, so it persists encrypted in the guest app.yaml and round-trips on later backups. Fields with no generate spec still proceed, with a WARN that the app may fail to start. Generation NEVER applies to data-keys (the gate below refuses first, and the generator itself rejects data_key fields). Residual case (honesty note): generation fully fixes the fresh-init path (empty DB volume → DB initialises with the new credential → dump replay restores the data). If a restored volume tar carries the OLD internal credential hash, the app may still fail auth until a manual in-DB credential reset — generation does not cover that.
• A missing data-encrypting key (DataKeyEnvVars) is fatal: the restore is refused with an explicit error directing the operator to do a PBS whole-guest restore first, because regenerating the key would render the stored data unreadable (restore_unit.go:45-50). This is the safety centerpiece: the controller never silently recreates an app whose data it can no longer decrypt.

The restore panel's data source (F1, v0.99.0). GET /api/backup/snapshots?stack=<app> is backed by backup.Manager.ListRestorePoints (internal/backup/restore_points.go). The route was a restic-era leftover the backups.html template still fetched; it was never registered, so the snapshot dropdown could never populate and the restore button never enabled (drill finding F1 — the whole keep-side restore UI was dead). It returns at most ONE entry — the current recovery unit (time = newest artifact mtime among manifest/db-dumps/volume-dumps, short_id:"helyi", tier:1, drive_label from the storage registry). Tier-2 entries are never emitted: POST /backup/restore only reads the primary unit, so a tier-2 listing would silently restore tier-1 data while claiming tier-2. Guards: invalid/empty stack name → 400, unknown stack → 404, no unit yet → ok:true, data:[].

Class-C in-place file restore (C2, v0.100.0 — closes drill finding F2). HDD bind-mount user files (appdata/<stack>) are outside both keep-side restore paths above (the drill's expected-negative). The customer path is now POST /backup/tier2/restore → RestoreTier2Files (internal/backup/tier2_restore.go), surfaced as the "Fájlok visszaállítása" button on the Tier-2 row. Semantics are deliberately additive-only (rsync -a --ignore-existing — rsyncRestoreMissing, the exec shape of rsyncMirror with the opposite-direction flags):

• a file missing live is copied back from the recorded Tier-2 copy (attrs preserved);
• an existing live file is never overwritten — a customer edit after the last nightly copy always wins;
• nothing is ever deleted — rsyncMirror's --delete in this direction would erase every file created since the last copy, which is why that helper must never be used tier2→live.

The source is the RECORDED CrossDriveBackup.DestinationPath (never a fresh selectTier2Target, which could re-pick an empty drive). Refusals — no Tier-2 copy / never ran / copy dir gone / either drive disconnected / live drive decommissioned — happen before the app is stopped, with customer-readable Hungarian reasons; the flow is stop → copy → restart → health-wait, single-flight with backup/restore. Zero files copied is a success ("Nincs hiányzó fájl — minden fájl megvan a helyén."), and re-running the button is idempotent.

What it deliberately does NOT do: no overwrite/point-in-time restore (corruption rollback stays with the offbox restore-to-verify + operator paths), no per-file selection, and it never touches recovery-unit/ under the Tier-2 dir (backup artifacts are not user files). App-reindex caveat: the restore is filesystem-level — apps that index their data dir (e.g. Nextcloud → occ files:scan) may need a rescan before restored files appear in the app's own UI.

4b. Security note — .fab import path validation (CTRL-001, v0.59.0)

The portable .fab import (internal/appexport/) validates every manifest path segment before it reaches a filepath.Join against a trusted base (appexport.ValidateSegment, validate.go:28; validateManifestPaths, validate.go:51, called from UnmarshalManifest). The attacker-controllable AppName / HDDSubdirs / VolumeNames are rejected on any .., path separator, or absolute path, closing the v0.59.0 path-traversal finding. This is cross-referenced here; the detail lives in the appexport doc and the v0.59.0 audit record.

5. Tier-2 cross-drive (internal/backup/tier2.go)

Tier 2 is the only off-drive protection browsable HDD userdata can get (PBS cannot reach bind mounts, tier2.go:17-23). It is an rsync -a --delete mirror (rsyncMirror, tier2.go:358) of an HDD app's recovery unit + bulk appdata/ to <target>/backups/secondary/<app>/{recovery-unit,appdata}/ on a different physical disk.

Auto-target selection (selectTier2Target, tier2.go:54), in order:

1. A customer-pinned target (PreferredTarget from the config panel) if it is still registered, schedulable, and off-disk (tier2.go:62-83).
2. Another registered user-data drive on a different physical disk — can hold bulk userdata (tier2.go:86-102).
3. The internal SSD (system data path) — small units only, headroom-guarded.

Off-disk-ness is decided by system.SamePhysicalDevice (a Stat_t.Dev compare). If the only candidate is the same physical disk, errNoOffDiskTarget is returned.

The rootfs-headroom guard is the key safety. The internal SSD is the ~8 GB guest rootfs, so option 3 refuses rather than fills: tier2FitsSystemDrive (tier2.go:121) → tier2FitsHeadroom (tier2.go:43) requires the copy to leave a reserve of max(2 GB, 20% of total) free, else returns errSSDNoHeadroom. A non-fitting or single-drive case is recorded as an honest no_target status (recordTier2NoTarget, tier2.go:331) with a Hungarian "needs a 2nd HDD" reason — the rootfs is never filled. When the SSD target is used, it is labelled DB/config-only (tier2.go:116, log suffix [SSD: DB/config only]).

RunAllTier2 (tier2.go:199) iterates deployed stacks, processes only those with an HDD_PATH (non-HDD apps are skipped — they are already in PBS), and skips disconnected/decommissioned drives. Status is persisted into settings.CrossDriveBackup (method rsync), with the customer-preference fields (UserDisabled, PreferredTarget) preserved across runner writes by withTier2Prefs (tier2.go:290). The config-panel view is Tier2Info (tier2.go:245, read-only).

6. Concurrency, scheduling, and the quiesce loop

Single-flight / running mutex. Manager guards a running flag with a mutex; acquireRunning/releaseRunning (backup.go:373) reject a second backup or restore with "already in progress". RestoreApp and RestoreFromRecoveryUnit take the same flag (restore.go:31, restore_unit.go:79).

Scheduling (wired in cmd/controller/main.go, Europe/Budapest):

• db-dump daily at cfg.Backup.DBDumpSchedule (default 02:30, config.go:271) → RunDBDumps (which also refreshes recovery units) (main.go:328).
• tier2-backup daily at 03:30 → RunAllTier2 (main.go:360).
• RefreshCache runs on the 5-minute status refresh, re-scanning dump files and (idempotently) re-capturing recovery units (backup.go:477).

Manual triggers exist via the API router (RunDBDumps, RunAllTier2 launched as goroutines, router.go:764/781).

The quiesce loop (internal/quiesce/quiesce.go) drives the whole-guest backup app-consistently. The agent's vzdump is crash-consistent only (an LXC has no fsfreeze), so the controller stops the app stacks first. Loop.Run polls GET /backup/due; when due, quiesceAndPoll (quiesce.go:205):

1. Writes a persisted marker (atomic, 0600) listing the stacks it is about to stop — before stopping anything (quiesce.go:207).
2. Stops the running app stacks.
3. POST /backup, records the job id, polls GET /backup/status.
4. Resumes early at the snapshotted phase (8B.2 downtime optimization — the storage snapshot has captured the stopped state, so the app may come back up; the loop keeps polling to done/failed), or at done in stop/downgraded mode (quiesce.go:256-279).

Unquiesce is guaranteed: a deferred closure restarts exactly the stopped stacks and clears the marker on every exit path — backup error, status-poll error, the MaxQuiesce bound (default 30 min, restarts the app while the backup continues on the agent), or context cancellation (quiesce.go:213-225, quiesce.go:243-249). On startup, Recover (quiesce.go:113) restarts any stacks left stopped by a mid-quiesce crash, then clears the marker. Single-flight is enforced both within the process (a TryLock mutex shared by the scheduled loop and the manual TriggerNow, quiesce.go:149/quiesce.go:181) and across restarts (the active marker — a cycle refuses to start on top of one, quiesce.go:157). TriggerNow runs the same flow asynchronously for the manual "Mentés most" action, returning ErrBackupInProgress if a cycle is already running.

Restic is gone from the controller. All disk-tier backup (restic snapshots, cross-drive-to-other-disks, drive recovery, infra backup) is the agent's; the controller's only remaining off-drive copy is the Tier-2 rsync mirror in §5.