felhom-controller/REPORT.md

# REPORT — NAS Part B: off-box backup target (restic-over-SFTP), v0.93.0

**Date:** 2026-06-30
**Task:** `TASK.md` — NAS Part B (off-box restic-SFTP backup). Controller-only. Closes the NAS arc.

## Baseline
| | value |
|---|---|
| felhom-controller | v0.92.0 → **v0.93.0** |
| Commits | `2a7dead` (impl) · `02820d6` (restic sftp.command 0.14 fix) · `5e06254` (injection guard) |
| restic in image | **re-added** (`restic` + `openssh-client`); live `restic 0.14.0` in the container |
| Live image | `felhom-controller:0.93.0` on guest 9201 (`Up (healthy)`) |

## What was built

The customer's NAS as an **off-box backup destination**: an encrypted restic repo over SFTP holding the
app-data tier (each off-box app's recovery unit + DB dumps + volume tars) — the "1 off-site" leg of 3-2-1,
distinct from the local cross-drive rsync copy and the agent's PBS whole-CT DR. No kernel mount; restic
talks SFTP to the NAS directly; the NAS sees only ciphertext.

### Files changed
- `Dockerfile` — re-added `restic` + `openssh-client` (dropped when cross-drive migrated restic→rsync).
- `internal/backup/offbox.go` (NEW) — restic-SFTP backend + orchestration:
  - **Fail-fast (spike Q8):** every restic call carries `-o sftp.command="ssh … -oConnectTimeout=10 … -s
    sftp"` so a dead NAS errors in ~10 s, never a multi-minute hang; `-oStrictHostKeyChecking=yes
    -oUserKnownHostsFile=<pinned>` (no blind TOFU) + `-oBatchMode=yes`.
  - init-if-absent (idempotent), per-app `restic backup --tag`, `forget --keep-daily 7 --keep-weekly 4
    --keep-monthly 6 --prune`, single-flight + migration-guard, restic's **own exit code** checked, restore
    via `restic restore latest --tag <app> --target <scratch>` (non-destructive).
  - **Secrets** (SSH key + auto-gen repo password) are **0600 files** in the data dir — never logged, never
    in a non-0600/committed file; the repo is encrypted. They ride DR via the PBS whole-CT snapshot.
  - **Injection guard** (`ValidateOffboxTarget`): host/user/repo are charset-checked and must not start
    with `-` (ssh option-injection, e.g. `-oProxyCommand`); `OffboxConfigured` fails closed on an invalid
    target so a hostile value can never reach the ssh exec.
- `internal/settings/settings.go` — `OffboxTarget` + per-app `AppBackupPrefs.Offbox` + helpers (no secrets).
- `cmd/controller/main.go` — daily `offbox-backup` (04:15), failure → `backup_failed` operator alert.
- `internal/web/offbox_handlers.go` + `backups.html` — configure target (host/port/user/repo + out-of-band
  SSH key + known_hosts), per-app toggles, run-now, restore, status.

## Tests — `go build ./... && go vet ./... && go test ./...` GREEN
- **ConnectTimeout fail-fast companion (red-proof):** a fake SSH transport hangs to the ctx deadline
  WITHOUT the arg, fails fast WITH it — proving the arg is load-bearing.
- dead-NAS run fails fast + alerts + records status=error; restore round-trip byte-identical (SFTP-shaped
  seam); single-flight skip; repo init idempotency; secrets are 0600.
- **Security companion:** `ValidateOffboxTarget` rejects ssh-option-injection (`-oProxyCommand` host/user),
  metacharacters, traversal, spaces, non-absolute paths; accepts a clean target.

## Live e2e validation (isolated sim NAS, SFTP)

**Method:** isolated throwaway sim = a Debian LXC on felhom-pve running sshd + a `felhom` user + `/srv/repo`
(its own throwaway ed25519 key; never touched production storage). Drove the **real controller endpoints**
(demo box has no password → CSRF skipped — the full server pipeline runs) via curl from inside the
controller container; the off-box SSH key + known_hosts were injected through the config endpoint exactly as
the UI form does. Sim + key destroyed afterward.

- **A — configure + first backup:** config saved (secrets written **0600**); off-box run **initialized the
  restic repo on the NAS** and backed up `rallly` (1 snapshot, 5 s). The NAS holds a proper restic repo
  (config/data/index/keys/snapshots); the config blob is **ciphertext** and a grep for SQL/dump/`rallly`
  markers in the repo found **nothing** — the NAS sees only ciphertext.
- **B — dead NAS fails fast + alerts:** black-holed the sim (blackhole route in guest 9201, since the
  controller's egress is guest-side — the host iptables path used in A1/A2 doesn't catch guest→guest
  bridged traffic) → the run **failed in ~20 s** (= 2×ConnectTimeout for the cat-config + init probes) with
  restic's `connect to host … port 22: Connection timed out`; the **`backup_failed` operator alert fired**;
  the controller + guest stayed healthy (no hang). On route removal → a run succeeded again.
- **C — restore byte-identical:** restored `rallly` from the off-box repo to a scratch dir →
  `diff -r` against the local source = **BYTE_IDENTICAL** (non-destructive; live data untouched).
- **D — DR reachability:** the SSH key + repo password are **0600 files** in the controller data dir →
  captured by the agent's PBS whole-CT snapshot of the rootfs → a rebuilt box restores them and can
  list/restore the off-box repo. (The recovery-unit/dr-recipe stay secret-free by design; controller
  secrets ride DR via PBS — the existing escrow path.)
- **Caught live (version skew):** restic 0.14.0 rejects `-o sftp.args` (the spike's form) — fixed to the
  portable `-o sftp.command="ssh … -s sftp"` (ConnectTimeout preserved), rebuilt, re-validated.
- **Security finding (background commit review):** the injection guard above was added in response — then
  rebuilt + a regression backup confirmed a valid target still runs.

## Observations / follow-ups
- **NAS arc complete** (A1 agent mount ✅ · A2 controller media registry ✅ · B off-box backup ✅).
- Next queued: per-storage worst-fill alerting (hub host_disk follow-on) + a provisioning-side vzdump
  retention default so a box can't refill its own root.
- DR-escrow note: off-box secrets ride the PBS whole-CT path (rootfs data dir), NOT the secret-free
  recovery-unit/dr-recipe — documented, no new escrow code.
- The demo box's off-box target was disabled + its secrets removed at teardown (the sim is gone).

No secrets in any committed file. SSH key + repo password were throwaway and out-of-band only.