felhom.eu/documentation/controller/backup-architecture.md

# Controller backup architecture

Source of truth: felhom-controller `internal/appbackup/`, `internal/backup/`, `internal/quiesce/` at v0.59.0, updated for the v0.99.0 restore-path fixes (F1/F3/O4 — TASK C1); whole-guest backup is the agent's (`felhom-agent`). Line numbers are v0.59.0-era landmarks — re-grep the symbol.

This document describes what the in-guest controller actually does for backup and restore at v0.59.0. The controller was de-privileged in slice 8C: restic, cross-drive-to-arbitrary-disks execution, drive-recovery and infra-backup were removed from the controller and now live in the host agent + PBS. The `restic` string still appears in the backup package only inside comments that record its removal (`backup.go:20`, `restore.go:14`, `restore.go:19`); there is no restic execution in the controller.

## 1. The split: controller vs agent/PBS

The controller owns the **app-data domain** only. The agent owns the **whole-LXC domain**.

| Domain | Owner | What |
|---|---|---|
| Per-app DB dumps | controller | `pg_dump` / `mariadb-dump` via `docker exec` |
| Per-app Docker-volume tars | controller | `docker run alpine tar` of named volumes |
| Per-app secret-free recovery unit | controller | compose + dumps + `manifest.json` on the app's drive |
| Tier-2 off-drive copy | controller | `rsync` mirror of an HDD app's unit + userdata to a different physical disk |
| Whole-LXC vzdump snapshot | agent | `POST /backup`, crash- or stop-consistent vzdump to local / PBS |
| Offsite / encrypted backup, verify, restore-test | agent + PBS | reported read-only to the controller |
| Whole-guest restore (rootfs, secrets, keys) | agent + PBS | the controller refuses any restore that needs this |

The boundary is precise: app data that lives on an **external user-data drive** (an HDD bind mount) is NOT inside the PBS whole-guest snapshot — PBS cannot reach bind mounts. That data is protected only by the controller's recovery unit + Tier-2 copy. App data that lives on the **rootfs** (non-HDD apps) IS inside the PBS whole-guest snapshot, so it gets no Tier-2 copy (`tier2.go:197-218`). The encrypted `app.yaml` and the controller's encryption key live on the rootfs, so they are inside PBS too — which is exactly why the restore path recovers secrets from the guest rather than storing them off-rootfs (see §3, §4).

The agent surface the controller talks to is `internal/agentapi/client.go`: a TLS client pinning the agent's self-signed leaf by SHA-256 (`client.go:69-82`) and authenticating with a per-guest bearer token. Backup-relevant calls: `BackupDue` (`GET /backup/due`), `StartBackup` (`POST /backup`), `BackupStatus` (`GET /backup/status`), `RestoreTestStatus` (`GET /restore-test/status`). The whole-guest backup record (`BackupRecord`) and restore-test record are rendered read-only — the comment at `client.go:135-136` states the controller does not own whole-guest backup.

## 2. Database dumps (`internal/appbackup/dbdump.go`)

**Discovery** is `docker ps`-driven. `DiscoverDatabases` (`dbdump.go:66`) runs `docker ps --format {{.ID}}\t{{.Names}}\t{{.Image}} --filter status=running` and classifies each container by image substring: `postgres` → Postgres, `mariadb`/`mysql` → MariaDB; everything else is skipped. The stack name is derived by stripping a known DB suffix from the container name (`deriveStackName`, `dbdump.go:536` — `postgres`/`db`/`mariadb`/`mysql`/`database`/`redis`/`cache`). Connection details come from the container's own env (`populateDBEnv`, `dbdump.go:477`): `POSTGRES_USER`/`POSTGRES_DB` (defaults `postgres`), or `MYSQL_DATABASE`/`MARIADB_DATABASE` with root.

**Per-DB dump** is `DumpOne` (`dbdump.go:162`), one container at a time, 5-minute timeout each. It re-checks the container is still running, then:

- Postgres: `docker exec <id> pg_dump -U <user> -d <db> --clean --if-exists --no-owner --no-privileges` (`dbdump.go:202`).
- MariaDB: `docker exec <id> mariadb-dump -u root -p<pw> --single-transaction --routines --triggers <db>` (`dbdump.go:228`). The root password is read from the container env (`MYSQL_ROOT_PASSWORD`/`MARIADB_ROOT_PASSWORD`, `dbdump.go:516`); the password is never logged (a redacted `-p***` form is built only for the debug log line).

**The tmpfile safety (H8).** The dump streams to `<name>.sql.tmp`, then before the rename to the final `.sql`: `tmpFile.Sync()` then `tmpFile.Close()` are called explicitly, each removing the tmp and failing the dump on error (`dbdump.go:266-278`). Only after a non-empty stat (`dbdump.go:281`) is the tmp atomically `os.Rename`d to the final path (`dbdump.go:293`). This guarantees the data is flushed to disk before the rename makes the dump "visible", so a crash never leaves a half-written `.sql`. Stale `.tmp` files older than one hour are reaped at the start of each run (`cleanupTmpFiles`, `dbdump.go:553`). Each finished dump is structurally validated (`ValidateDump`, `dbdump.go:320`): header line + at least one `CREATE TABLE`, scanned line-by-line with a bounded `bufio.Reader` so large data lines do not allocate (H1).

**Where dumps are stored.** `Manager.GetAppDrivePath(stack)` (`backup.go:88`) returns the app's `HDD_PATH` if it has one, else falls back to the configured `systemDataPath` (the internal SSD) for SSD-only apps. That drive path is mapped to its felhom-data namespace root by `namespaceRoot` (`backup.go:105` — Model A: an in-guest drive mount IS the namespace root, so it is used as-is; only the system-data fallback gets the `felhom-data` subdir appended). Dumps land in `AppDBDumpPath(nsRoot, stack)` = `<nsRoot>/backups/primary/<stack>/db-dumps/` (`paths.go:58`). The SSD-only fallback is the case that the C3 DR finding was about — and it is handled here: SSD-only apps get a correct path via `systemDataPath`, so there is no path gap. (The fallback only warns when `systemDataPath` itself is unconfigured, `backup.go:75`/`backup.go:94`.)

`RunDBDumps` (`backup.go:143`) acquires the running flag, discovers, dumps each DB to its app's path (skipping drives marked disconnected/decommissioned), persists each validation result to `settings.json`, then — **v0.99.0 (F3)** — runs the named-volume dumps (`runVolumeDumps`) and finally refreshes every recovery unit (`captureAllRecoveryUnits`) — even on partial failure, so units never go stale.

**Volume dumps are real again (F3, v0.99.0).** After the restic removal, `DumpAppVolumesSafe` had no caller — no trigger ever produced `volume-dumps/`, so named-volume app data was never captured (drill finding F3). `runVolumeDumps` now runs inside the same nightly/manual backup run, per deployed stack: skip if protected (`IsProtectedStack`), skip if it has **no named volumes** (this check deliberately precedes `DumpAppVolumesSafe`, which stops the stack before its own check), skip disconnected/decommissioned drives, else stop → tar each volume (`docker run alpine tar`) → restart. It runs BEFORE the unit capture so `manifest.VolumeDumps` enumerates the fresh tars. A per-stack failure lands in the run summary (`FAIL <app> volumes:`), flips the run's Success flag and fails the run — no silent partials. Note the operational consequence: volume-bearing apps are briefly stopped during each nightly backup (the locked stop-first policy — a live tar of a database volume would be torn).

**Atomic volume dumps (F7, v0.118.0, CAMPAIGN-3).** The volume dump now has the SAME crash-safety the DB dump has always had: `DumpAppVolumes` writes the tar to `<vol>.tar.tmp` (`docker run … tar cf …/<vol>.tar.tmp`), then `atomicPromoteTar` fsyncs the tmp (and best-effort the dir) and `os.Rename`s it over the final `.tar` **only on success**. Before this, tar wrote the `.tar` IN PLACE, so a mid-write NFS cut left a **0-byte tar replacing the last good dump** — and a tier-1 restore is replace-semantics, so the only "restore point" then restored an empty volume (the exact CAMPAIGN-3 F7 failure: a `exportfs -u` during a dump truncated a 247 MB calibre tar to 0 bytes). Now any tar error / timeout / dead-NFS EIO removes only the `.tmp`; the last good `.tar` is byte-untouched. The `.tar.tmp` name ends `.tmp` (not `.tar`), so it is invisible to the restore-point and stale scans; orphan `.tar.tmp` from a killed run is swept on the next dump. **Live-proven 2026-07-12:** an `exportfs -u` mid-volume-dump left all NAS volume tars byte-identical (sha unchanged), no 0-byte file, run `success:false`; the next run produced fresh good tars.

**Stale-primary sweep (F5, v0.118.0).** After the units are refreshed, `pruneStalePrimaryDirs` removes an orphaned `backups/primary/<app>` dir an app left on an OLD drive when its `HDD_PATH` moved (invisible disk residue). Guards: removes only when the app is deployed AND its current namespace root differs from the dir's drive; never the current-drive dir (the live restore point) or an undeployed app's dir; strictly under a `backups/primary/` prefix.

## 3. Recovery units (`internal/backup/recovery_unit.go`)

A recovery unit is a per-app, **secret-free**, self-contained directory at `<nsRoot>/backups/primary/<app>/` (`paths.go:42`). It contains:

- `compose/` — `docker-compose.yml` + `.felhom.yml` + a **secret-stripped** `app.yaml`.
- `db-dumps/` — the `.sql` dumps from §2.
- `volume-dumps/` — named-volume `.tar` archives.
- `manifest.json` — the `RecoveryManifest` (`recovery_unit.go:31`).

What the unit **excludes**: it holds no secret values, no data-encrypting keys, and not the Docker image. The manifest stores only the pinned image tag(s) (`ImagePins` — re-pulled on restore), the **names** of the secret env vars (`SecretEnvVars`), and the names of the data-key env vars (`DataKeyEnvVars`); `SecretSource` records in plain text that the values come from "guest app.yaml (live rootfs) or PBS whole-guest snapshot — never stored in this unit" (`recovery_unit.go:141`). The stripped `app.yaml` carries only non-secret env, with a header naming the omitted secrets (`buildStrippedAppYaml`, `recovery_unit.go:188`).

`CaptureRecoveryUnit` (`recovery_unit.go:68`) pulls the app's `RecoveryInfo` from the stack provider (`GetStackRecoveryInfo`), builds the captured content in memory, and is **idempotent**: it skips all drive writes when the existing manifest matches the current controller version, config checksums (sha256 of each captured file), and the DB/volume dump set (`recovery_unit.go:112-118`). This is what lets it run on the 5-minute status refresh without thrashing a spinning USB drive. Writes are atomic (`atomicWrite`, `recovery_unit.go:270` — tmp + rename). `DataKeyEnvVars` is a fail-closed restore annotation only (see §4); it does not affect capture.

The `.fab` portable export/import path (`internal/appexport/`) is a separate, operator-driven mechanism documented elsewhere — cross-reference that doc; it is not the periodic recovery unit.

## 4. Restore

Two keep-side restore entry points exist; neither does a whole-guest restore (that is the agent's).

**`RestoreApp(stack, snapshotID)`** (`restore.go:21`) — the legacy volume-only path. Stops the stack, re-imports the named-volume `.tar` dumps (`restoreDockerVolumes`, `restore.go:85`: `docker volume rm -f` + `create` + `docker run alpine tar xf`), restarts, and health-checks. `snapshotID` is retained only for signature/logging compatibility now that restic is gone (`restore.go:19-20`).

**`RestoreFromRecoveryUnit(stack)`** (`restore_unit.go:74`) — the recovery-unit path. It reads the unit manifest (falling back to `RestoreApp` if no unit exists, `restore_unit.go:99`), recovers the secret values from the **guest's own live `app.yaml`** via `stackProvider.RecoverStackSecrets` (never from the unit), reconciles them, restores the named-volume data, then `RecreateStackFromUnit` rebuilds the app's definition from `compose/` and redeploys with the reconstructed env (re-pulling the pinned image). Nothing is regenerated; no secret is read from the unit.

**The fail-closed data-key gate** is `reconcileRestoreSecrets` (`restore_unit.go:22`) — a pure, unit-tested function. It merges non-secret env with recovered secrets, then:

- A missing **resettable** secret (DB password, admin password) is non-fatal: returned in `missing`. **v0.99.0 (O4):** the caller now generates a replacement from the field's catalog `generate` spec (`stacks.GenerateSecretForField` via the `backup.SetSecretGenerator` seam) instead of proceeding with the secret blank — pre-O4, the redeploy failed compose-up with "Defaulting to a blank string". The generated value rides `fullEnv` into `RecreateStackFromUnit` → `SaveAppConfig`, so it persists encrypted in the guest `app.yaml` and round-trips on later backups. Fields with no `generate` spec still proceed, with a WARN that the app may fail to start. Generation NEVER applies to data-keys (the gate below refuses first, and the generator itself rejects `data_key` fields). **Residual case (honesty note):** generation fully fixes the fresh-init path (empty DB volume → DB initialises with the new credential → dump replay restores the data). If a restored volume tar carries the OLD internal credential hash, the app may still fail auth until a manual in-DB credential reset — generation does not cover that.
- A missing **data-encrypting key** (`DataKeyEnvVars`) is **fatal**: the restore is refused with an explicit error directing the operator to do a PBS whole-guest restore first, because regenerating the key would render the stored data unreadable (`restore_unit.go:45-50`). This is the safety centerpiece: the controller never silently recreates an app whose data it can no longer decrypt.

**The restore panel's data source (F1, v0.99.0).** `GET /api/backup/snapshots?stack=<app>` is backed by `backup.Manager.ListRestorePoints` (`internal/backup/restore_points.go`). The route was a restic-era leftover the backups.html template still fetched; it was never registered, so the snapshot dropdown could never populate and the restore button never enabled (drill finding F1 — the whole keep-side restore UI was dead). It returns at most ONE entry — the current recovery unit (`time` = newest artifact mtime among manifest/db-dumps/volume-dumps, `short_id:"helyi"`, `tier:1`, `drive_label` from the storage registry). Tier-2 entries are never emitted: `POST /backup/restore` only reads the primary unit, so a tier-2 listing would silently restore tier-1 data while claiming tier-2. Guards: invalid/empty stack name → 400, unknown stack → 404, no unit yet → `ok:true, data:[]`.

**Class-C in-place file restore (C2, v0.100.0 — closes drill finding F2).** HDD bind-mount user files (`appdata/<stack>`) are outside both keep-side restore paths above (the drill's expected-negative). The customer path is now **`POST /backup/tier2/restore`** → `RestoreTier2Files` (`internal/backup/tier2_restore.go`), surfaced as the **"Fájlok visszaállítása"** button on the Tier-2 row. Semantics are deliberately **additive-only** (`rsync -a --ignore-existing` — `rsyncRestoreMissing`, the exec shape of `rsyncMirror` with the opposite-direction flags):

- a file missing live is copied back from the recorded Tier-2 copy (attrs preserved);
- an existing live file is **never overwritten** — a customer edit after the last nightly copy always wins;
- **nothing is ever deleted** — `rsyncMirror`'s `--delete` in this direction would erase every file created since the last copy, which is why that helper must never be used tier2→live.

The source is the RECORDED `CrossDriveBackup.DestinationPath` (never a fresh `selectTier2Target`, which could re-pick an empty drive). Refusals — no Tier-2 copy / never ran / copy dir gone / either drive disconnected / live drive decommissioned — happen before the app is stopped, with customer-readable Hungarian reasons; the flow is stop → copy → restart → health-wait, single-flight with backup/restore. Zero files copied is a success ("Nincs hiányzó fájl — minden fájl megvan a helyén."), and re-running the button is idempotent.

**What it deliberately does NOT do:** no overwrite/point-in-time restore (corruption rollback stays with the offbox restore-to-verify + operator paths), no per-file selection, and it never touches `recovery-unit/` under the Tier-2 dir (backup artifacts are not user files). **App-reindex caveat:** the restore is filesystem-level — apps that index their data dir (e.g. Nextcloud → `occ files:scan`) may need a rescan before restored files appear in the app's own UI.

### 4b. Security note — `.fab` import path validation (CTRL-001, v0.59.0)

The portable `.fab` import (`internal/appexport/`) validates every manifest path segment before it reaches a `filepath.Join` against a trusted base (`appexport.ValidateSegment`, `validate.go:28`; `validateManifestPaths`, `validate.go:51`, called from `UnmarshalManifest`). The attacker-controllable `AppName` / `HDDSubdirs` / `VolumeNames` are rejected on any `..`, path separator, or absolute path, closing the v0.59.0 path-traversal finding. This is cross-referenced here; the detail lives in the appexport doc and the v0.59.0 audit record.

## 5. Tier-2 cross-drive (`internal/backup/tier2.go`)

Tier 2 is the only off-drive protection browsable HDD userdata can get (PBS cannot reach bind mounts, `tier2.go:17-23`). It is an `rsync -a --delete` **mirror** (`rsyncMirror`, `tier2.go:358`) of an HDD app's recovery unit + bulk `appdata/` to `<target>/backups/secondary/<app>/{recovery-unit,appdata}/` on a **different physical disk**.

**No single-copy backups (F6, v0.118.0, CAMPAIGN-3).** `RunAllTier2` used to `continue` on every non-HDD app, so a **volume-only** app (no `HDD_PATH`, its dumps on `sys_drive`) got NO tier-2 copy — a single controller-level copy on one device. It now flows through too: its recovery unit (holding the db/volume dumps) gets the cross-drive second copy like any HDD app (live-proven 2026-07-12: `actualbudget`/`seerr` appear under `felhom-usb/backups/secondary/`). A sys_drive app's restore-point drive label is now the clear "Belső SSD (rendszer)", never blank.

**3-2-1 honesty on a single-drive box (F6).** When NO off-drive target exists (`Manager.hasOffDriveTarget` false — only the system drive, no enrolled second disk), there is genuinely only ONE local copy. `FullBackupStatus.SingleCopyWarning` surfaces an honest Hungarian banner on the backup page ("Csak egy másolat készül (nincs második meghajtó) — a 3-2-1 mentéshez csatlakoztasson egy második meghajtót vagy offsite tárolót.") rather than implying a 3-2-1 guarantee the box cannot keep.

**NAS backup locality (Part 4 — decision A, v0.118.0).** A NAS-resident app's tier-1 dumps live **on the NAS itself** (`nas-media/backups/primary/<app>`), beside the data. During a NAS outage both the app data AND its freshest tier-1 dump are on the dead device — the tier-2 cross-drive copy to a local drive is the off-NAS leg that saves them, **and only after it has run**. This locality was kept deliberately in the operator fork (over retargeting tier-1 to a local drive, which would have moved restore-point resolution off the drive); the tier-2 copy is the mitigation. Documented here so the outage window is never a surprise.

**Auto-target selection** (`selectTier2Target`, `tier2.go:54`), in order:

1. A customer-pinned target (`PreferredTarget` from the config panel) if it is still registered, schedulable, and off-disk (`tier2.go:62-83`).
2. Another registered user-data drive on a different physical disk — can hold bulk userdata (`tier2.go:86-102`).
3. The internal SSD (system data path) — **small units only**, headroom-guarded.

Off-disk-ness is decided by `system.SamePhysicalDevice` (a `Stat_t.Dev` compare). If the only candidate is the same physical disk, `errNoOffDiskTarget` is returned.

**The rootfs-headroom guard** is the key safety. The internal SSD is the ~8 GB guest rootfs, so option 3 refuses rather than fills: `tier2FitsSystemDrive` (`tier2.go:121`) → `tier2FitsHeadroom` (`tier2.go:43`) requires the copy to leave a reserve of `max(2 GB, 20% of total)` free, else returns `errSSDNoHeadroom`. A non-fitting or single-drive case is recorded as an honest `no_target` status (`recordTier2NoTarget`, `tier2.go:331`) with a Hungarian "needs a 2nd HDD" reason — the rootfs is never filled. When the SSD target is used, it is labelled DB/config-only (`tier2.go:116`, log suffix `[SSD: DB/config only]`).

`RunAllTier2` (`tier2.go:199`) iterates deployed stacks, processes only those with an `HDD_PATH` (non-HDD apps are skipped — they are already in PBS), and skips disconnected/decommissioned drives. Status is persisted into `settings.CrossDriveBackup` (method `rsync`), with the customer-preference fields (`UserDisabled`, `PreferredTarget`) preserved across runner writes by `withTier2Prefs` (`tier2.go:290`). The config-panel view is `Tier2Info` (`tier2.go:245`, read-only).

## 6. Concurrency, scheduling, and the quiesce loop

**Single-flight / running mutex.** `Manager` guards a `running` flag with a mutex; `acquireRunning`/`releaseRunning` (`backup.go:373`) reject a second backup or restore with "already in progress". `RestoreApp` and `RestoreFromRecoveryUnit` take the same flag (`restore.go:31`, `restore_unit.go:79`).

**Scheduling** (wired in `cmd/controller/main.go`, Europe/Budapest):

- `db-dump` daily at `cfg.Backup.DBDumpSchedule` (default `02:30`, `config.go:271`) → `RunDBDumps` (which also refreshes recovery units) (`main.go:328`).
- `tier2-backup` daily at `03:30` → `RunAllTier2` (`main.go:360`).
- `RefreshCache` runs on the 5-minute status refresh, re-scanning dump files and (idempotently) re-capturing recovery units (`backup.go:477`).

Manual triggers exist via the API router (`RunDBDumps`, `RunAllTier2` launched as goroutines, `router.go:764`/`781`).

**The quiesce loop** (`internal/quiesce/quiesce.go`) drives the whole-guest backup app-consistently. The agent's vzdump is crash-consistent only (an LXC has no fsfreeze), so the controller stops the app stacks first. `Loop.Run` polls `GET /backup/due`; when due, `quiesceAndPoll` (`quiesce.go:205`):

1. Writes a persisted marker (atomic, `0600`) listing the stacks it is about to stop — **before** stopping anything (`quiesce.go:207`).
2. Stops the running app stacks.
3. `POST /backup`, records the job id, polls `GET /backup/status`.
4. Resumes early at the `snapshotted` phase (8B.2 downtime optimization — the storage snapshot has captured the stopped state, so the app may come back up; the loop keeps polling to `done`/`failed`), or at `done` in stop/downgraded mode (`quiesce.go:256-279`).

Unquiesce is **guaranteed**: a deferred closure restarts exactly the stopped stacks and clears the marker on every exit path — backup error, status-poll error, the `MaxQuiesce` bound (default 30 min, restarts the app while the backup continues on the agent), or context cancellation (`quiesce.go:213-225`, `quiesce.go:243-249`). On startup, `Recover` (`quiesce.go:113`) restarts any stacks left stopped by a mid-quiesce crash, then clears the marker. Single-flight is enforced both within the process (a `TryLock` mutex shared by the scheduled loop and the manual `TriggerNow`, `quiesce.go:149`/`quiesce.go:181`) and across restarts (the active marker — a cycle refuses to start on top of one, `quiesce.go:157`). `TriggerNow` runs the same flow asynchronously for the manual "Mentés most" action, returning `ErrBackupInProgress` if a cycle is already running.

**Restic is gone from the controller.** All disk-tier backup (restic snapshots, cross-drive-to-other-disks, drive recovery, infra backup) is the agent's; the controller's only remaining off-drive copy is the Tier-2 rsync mirror in §5.