felhom.eu/documentation/controller/backup-architecture.md

# Controller backup architecture

Source of truth: felhom-controller `internal/appbackup/`, `internal/backup/`, `internal/quiesce/` at v0.59.0; whole-guest backup is the agent's (`felhom-agent`).

This document describes what the in-guest controller actually does for backup and restore at v0.59.0. The controller was de-privileged in slice 8C: restic, cross-drive-to-arbitrary-disks execution, drive-recovery and infra-backup were removed from the controller and now live in the host agent + PBS. The `restic` string still appears in the backup package only inside comments that record its removal (`backup.go:20`, `restore.go:14`, `restore.go:19`); there is no restic execution in the controller.

## 1. The split: controller vs agent/PBS

The controller owns the **app-data domain** only. The agent owns the **whole-LXC domain**.

| Domain | Owner | What |
|---|---|---|
| Per-app DB dumps | controller | `pg_dump` / `mariadb-dump` via `docker exec` |
| Per-app Docker-volume tars | controller | `docker run alpine tar` of named volumes |
| Per-app secret-free recovery unit | controller | compose + dumps + `manifest.json` on the app's drive |
| Tier-2 off-drive copy | controller | `rsync` mirror of an HDD app's unit + userdata to a different physical disk |
| Whole-LXC vzdump snapshot | agent | `POST /backup`, crash- or stop-consistent vzdump to local / PBS |
| Offsite / encrypted backup, verify, restore-test | agent + PBS | reported read-only to the controller |
| Whole-guest restore (rootfs, secrets, keys) | agent + PBS | the controller refuses any restore that needs this |

The boundary is precise: app data that lives on an **external user-data drive** (an HDD bind mount) is NOT inside the PBS whole-guest snapshot — PBS cannot reach bind mounts. That data is protected only by the controller's recovery unit + Tier-2 copy. App data that lives on the **rootfs** (non-HDD apps) IS inside the PBS whole-guest snapshot, so it gets no Tier-2 copy (`tier2.go:197-218`). The encrypted `app.yaml` and the controller's encryption key live on the rootfs, so they are inside PBS too — which is exactly why the restore path recovers secrets from the guest rather than storing them off-rootfs (see §3, §4).

The agent surface the controller talks to is `internal/agentapi/client.go`: a TLS client pinning the agent's self-signed leaf by SHA-256 (`client.go:69-82`) and authenticating with a per-guest bearer token. Backup-relevant calls: `BackupDue` (`GET /backup/due`), `StartBackup` (`POST /backup`), `BackupStatus` (`GET /backup/status`), `RestoreTestStatus` (`GET /restore-test/status`). The whole-guest backup record (`BackupRecord`) and restore-test record are rendered read-only — the comment at `client.go:135-136` states the controller does not own whole-guest backup.

## 2. Database dumps (`internal/appbackup/dbdump.go`)

**Discovery** is `docker ps`-driven. `DiscoverDatabases` (`dbdump.go:66`) runs `docker ps --format {{.ID}}\t{{.Names}}\t{{.Image}} --filter status=running` and classifies each container by image substring: `postgres` → Postgres, `mariadb`/`mysql` → MariaDB; everything else is skipped. The stack name is derived by stripping a known DB suffix from the container name (`deriveStackName`, `dbdump.go:536` — `postgres`/`db`/`mariadb`/`mysql`/`database`/`redis`/`cache`). Connection details come from the container's own env (`populateDBEnv`, `dbdump.go:477`): `POSTGRES_USER`/`POSTGRES_DB` (defaults `postgres`), or `MYSQL_DATABASE`/`MARIADB_DATABASE` with root.

**Per-DB dump** is `DumpOne` (`dbdump.go:162`), one container at a time, 5-minute timeout each. It re-checks the container is still running, then:

- Postgres: `docker exec <id> pg_dump -U <user> -d <db> --clean --if-exists --no-owner --no-privileges` (`dbdump.go:202`).
- MariaDB: `docker exec <id> mariadb-dump -u root -p<pw> --single-transaction --routines --triggers <db>` (`dbdump.go:228`). The root password is read from the container env (`MYSQL_ROOT_PASSWORD`/`MARIADB_ROOT_PASSWORD`, `dbdump.go:516`); the password is never logged (a redacted `-p***` form is built only for the debug log line).

**The tmpfile safety (H8).** The dump streams to `<name>.sql.tmp`, then before the rename to the final `.sql`: `tmpFile.Sync()` then `tmpFile.Close()` are called explicitly, each removing the tmp and failing the dump on error (`dbdump.go:266-278`). Only after a non-empty stat (`dbdump.go:281`) is the tmp atomically `os.Rename`d to the final path (`dbdump.go:293`). This guarantees the data is flushed to disk before the rename makes the dump "visible", so a crash never leaves a half-written `.sql`. Stale `.tmp` files older than one hour are reaped at the start of each run (`cleanupTmpFiles`, `dbdump.go:553`). Each finished dump is structurally validated (`ValidateDump`, `dbdump.go:320`): header line + at least one `CREATE TABLE`, scanned line-by-line with a bounded `bufio.Reader` so large data lines do not allocate (H1).

**Where dumps are stored.** `Manager.GetAppDrivePath(stack)` (`backup.go:88`) returns the app's `HDD_PATH` if it has one, else falls back to the configured `systemDataPath` (the internal SSD) for SSD-only apps. That drive path is mapped to its felhom-data namespace root by `namespaceRoot` (`backup.go:105` — Model A: an in-guest drive mount IS the namespace root, so it is used as-is; only the system-data fallback gets the `felhom-data` subdir appended). Dumps land in `AppDBDumpPath(nsRoot, stack)` = `<nsRoot>/backups/primary/<stack>/db-dumps/` (`paths.go:58`). The SSD-only fallback is the case that the C3 DR finding was about — and it is handled here: SSD-only apps get a correct path via `systemDataPath`, so there is no path gap. (The fallback only warns when `systemDataPath` itself is unconfigured, `backup.go:75`/`backup.go:94`.)

`RunDBDumps` (`backup.go:143`) acquires the running flag, discovers, dumps each DB to its app's path (skipping drives marked disconnected/decommissioned), persists each validation result to `settings.json`, and finally refreshes every recovery unit (`captureAllRecoveryUnits`, `backup.go:247`) — even on partial DB failure, so units never go stale.

## 3. Recovery units (`internal/backup/recovery_unit.go`)

A recovery unit is a per-app, **secret-free**, self-contained directory at `<nsRoot>/backups/primary/<app>/` (`paths.go:42`). It contains:

- `compose/` — `docker-compose.yml` + `.felhom.yml` + a **secret-stripped** `app.yaml`.
- `db-dumps/` — the `.sql` dumps from §2.
- `volume-dumps/` — named-volume `.tar` archives.
- `manifest.json` — the `RecoveryManifest` (`recovery_unit.go:31`).

What the unit **excludes**: it holds no secret values, no data-encrypting keys, and not the Docker image. The manifest stores only the pinned image tag(s) (`ImagePins` — re-pulled on restore), the **names** of the secret env vars (`SecretEnvVars`), and the names of the data-key env vars (`DataKeyEnvVars`); `SecretSource` records in plain text that the values come from "guest app.yaml (live rootfs) or PBS whole-guest snapshot — never stored in this unit" (`recovery_unit.go:141`). The stripped `app.yaml` carries only non-secret env, with a header naming the omitted secrets (`buildStrippedAppYaml`, `recovery_unit.go:188`).

`CaptureRecoveryUnit` (`recovery_unit.go:68`) pulls the app's `RecoveryInfo` from the stack provider (`GetStackRecoveryInfo`), builds the captured content in memory, and is **idempotent**: it skips all drive writes when the existing manifest matches the current controller version, config checksums (sha256 of each captured file), and the DB/volume dump set (`recovery_unit.go:112-118`). This is what lets it run on the 5-minute status refresh without thrashing a spinning USB drive. Writes are atomic (`atomicWrite`, `recovery_unit.go:270` — tmp + rename). `DataKeyEnvVars` is a fail-closed restore annotation only (see §4); it does not affect capture.

The `.fab` portable export/import path (`internal/appexport/`) is a separate, operator-driven mechanism documented elsewhere — cross-reference that doc; it is not the periodic recovery unit.

## 4. Restore

Two keep-side restore entry points exist; neither does a whole-guest restore (that is the agent's).

**`RestoreApp(stack, snapshotID)`** (`restore.go:21`) — the legacy volume-only path. Stops the stack, re-imports the named-volume `.tar` dumps (`restoreDockerVolumes`, `restore.go:85`: `docker volume rm -f` + `create` + `docker run alpine tar xf`), restarts, and health-checks. `snapshotID` is retained only for signature/logging compatibility now that restic is gone (`restore.go:19-20`).

**`RestoreFromRecoveryUnit(stack)`** (`restore_unit.go:74`) — the recovery-unit path. It reads the unit manifest (falling back to `RestoreApp` if no unit exists, `restore_unit.go:99`), recovers the secret values from the **guest's own live `app.yaml`** via `stackProvider.RecoverStackSecrets` (never from the unit), reconciles them, restores the named-volume data, then `RecreateStackFromUnit` rebuilds the app's definition from `compose/` and redeploys with the reconstructed env (re-pulling the pinned image). Nothing is regenerated; no secret is read from the unit.

**The fail-closed data-key gate** is `reconcileRestoreSecrets` (`restore_unit.go:22`) — a pure, unit-tested function. It merges non-secret env with recovered secrets, then:

- A missing **resettable** secret (DB password, admin password) is non-fatal: returned in `missing`, the caller warns and proceeds (`restore_unit.go:117`).
- A missing **data-encrypting key** (`DataKeyEnvVars`) is **fatal**: the restore is refused with an explicit error directing the operator to do a PBS whole-guest restore first, because regenerating the key would render the stored data unreadable (`restore_unit.go:45-50`). This is the safety centerpiece: the controller never silently recreates an app whose data it can no longer decrypt.

### 4b. Security note — `.fab` import path validation (CTRL-001, v0.59.0)

The portable `.fab` import (`internal/appexport/`) validates every manifest path segment before it reaches a `filepath.Join` against a trusted base (`appexport.ValidateSegment`, `validate.go:28`; `validateManifestPaths`, `validate.go:51`, called from `UnmarshalManifest`). The attacker-controllable `AppName` / `HDDSubdirs` / `VolumeNames` are rejected on any `..`, path separator, or absolute path, closing the v0.59.0 path-traversal finding. This is cross-referenced here; the detail lives in the appexport doc and the v0.59.0 audit record.

## 5. Tier-2 cross-drive (`internal/backup/tier2.go`)

Tier 2 is the only off-drive protection browsable HDD userdata can get (PBS cannot reach bind mounts, `tier2.go:17-23`). It is an `rsync -a --delete` **mirror** (`rsyncMirror`, `tier2.go:358`) of an HDD app's recovery unit + bulk `appdata/` to `<target>/backups/secondary/<app>/{recovery-unit,appdata}/` on a **different physical disk**.

**Auto-target selection** (`selectTier2Target`, `tier2.go:54`), in order:

1. A customer-pinned target (`PreferredTarget` from the config panel) if it is still registered, schedulable, and off-disk (`tier2.go:62-83`).
2. Another registered user-data drive on a different physical disk — can hold bulk userdata (`tier2.go:86-102`).
3. The internal SSD (system data path) — **small units only**, headroom-guarded.

Off-disk-ness is decided by `system.SamePhysicalDevice` (a `Stat_t.Dev` compare). If the only candidate is the same physical disk, `errNoOffDiskTarget` is returned.

**The rootfs-headroom guard** is the key safety. The internal SSD is the ~8 GB guest rootfs, so option 3 refuses rather than fills: `tier2FitsSystemDrive` (`tier2.go:121`) → `tier2FitsHeadroom` (`tier2.go:43`) requires the copy to leave a reserve of `max(2 GB, 20% of total)` free, else returns `errSSDNoHeadroom`. A non-fitting or single-drive case is recorded as an honest `no_target` status (`recordTier2NoTarget`, `tier2.go:331`) with a Hungarian "needs a 2nd HDD" reason — the rootfs is never filled. When the SSD target is used, it is labelled DB/config-only (`tier2.go:116`, log suffix `[SSD: DB/config only]`).

`RunAllTier2` (`tier2.go:199`) iterates deployed stacks, processes only those with an `HDD_PATH` (non-HDD apps are skipped — they are already in PBS), and skips disconnected/decommissioned drives. Status is persisted into `settings.CrossDriveBackup` (method `rsync`), with the customer-preference fields (`UserDisabled`, `PreferredTarget`) preserved across runner writes by `withTier2Prefs` (`tier2.go:290`). The config-panel view is `Tier2Info` (`tier2.go:245`, read-only).

## 6. Concurrency, scheduling, and the quiesce loop

**Single-flight / running mutex.** `Manager` guards a `running` flag with a mutex; `acquireRunning`/`releaseRunning` (`backup.go:373`) reject a second backup or restore with "already in progress". `RestoreApp` and `RestoreFromRecoveryUnit` take the same flag (`restore.go:31`, `restore_unit.go:79`).

**Scheduling** (wired in `cmd/controller/main.go`, Europe/Budapest):

- `db-dump` daily at `cfg.Backup.DBDumpSchedule` (default `02:30`, `config.go:271`) → `RunDBDumps` (which also refreshes recovery units) (`main.go:328`).
- `tier2-backup` daily at `03:30` → `RunAllTier2` (`main.go:360`).
- `RefreshCache` runs on the 5-minute status refresh, re-scanning dump files and (idempotently) re-capturing recovery units (`backup.go:477`).

Manual triggers exist via the API router (`RunDBDumps`, `RunAllTier2` launched as goroutines, `router.go:764`/`781`).

**The quiesce loop** (`internal/quiesce/quiesce.go`) drives the whole-guest backup app-consistently. The agent's vzdump is crash-consistent only (an LXC has no fsfreeze), so the controller stops the app stacks first. `Loop.Run` polls `GET /backup/due`; when due, `quiesceAndPoll` (`quiesce.go:205`):

1. Writes a persisted marker (atomic, `0600`) listing the stacks it is about to stop — **before** stopping anything (`quiesce.go:207`).
2. Stops the running app stacks.
3. `POST /backup`, records the job id, polls `GET /backup/status`.
4. Resumes early at the `snapshotted` phase (8B.2 downtime optimization — the storage snapshot has captured the stopped state, so the app may come back up; the loop keeps polling to `done`/`failed`), or at `done` in stop/downgraded mode (`quiesce.go:256-279`).

Unquiesce is **guaranteed**: a deferred closure restarts exactly the stopped stacks and clears the marker on every exit path — backup error, status-poll error, the `MaxQuiesce` bound (default 30 min, restarts the app while the backup continues on the agent), or context cancellation (`quiesce.go:213-225`, `quiesce.go:243-249`). On startup, `Recover` (`quiesce.go:113`) restarts any stacks left stopped by a mid-quiesce crash, then clears the marker. Single-flight is enforced both within the process (a `TryLock` mutex shared by the scheduled loop and the manual `TriggerNow`, `quiesce.go:149`/`quiesce.go:181`) and across restarts (the active marker — a cycle refuses to start on top of one, `quiesce.go:157`). `TriggerNow` runs the same flow asynchronously for the manual "Mentés most" action, returning `ErrBackupInProgress` if a cycle is already running.

**Restic is gone from the controller.** All disk-tier backup (restic snapshots, cross-drive-to-other-disks, drive recovery, infra backup) is the agent's; the controller's only remaining off-drive copy is the Tier-2 rsync mirror in §5.