felhom-controller/REPORT.md

REPORT — NAS Part B: off-box backup target (restic-over-SFTP), v0.93.0

Date: 2026-06-30 Task: TASK.md — NAS Part B (off-box restic-SFTP backup). Controller-only. Closes the NAS arc.

Baseline

	value
felhom-controller	v0.92.0 → v0.93.0
Commits	2a7dead (impl) · 02820d6 (restic sftp.command 0.14 fix) · 5e06254 (injection guard)
restic in image	re-added (restic + openssh-client); live restic 0.14.0 in the container
Live image	felhom-controller:0.93.0 on guest 9201 (Up (healthy))

What was built

The customer's NAS as an off-box backup destination: an encrypted restic repo over SFTP holding the app-data tier (each off-box app's recovery unit + DB dumps + volume tars) — the "1 off-site" leg of 3-2-1, distinct from the local cross-drive rsync copy and the agent's PBS whole-CT DR. No kernel mount; restic talks SFTP to the NAS directly; the NAS sees only ciphertext.

Files changed

• Dockerfile — re-added restic + openssh-client (dropped when cross-drive migrated restic→rsync).
• internal/backup/offbox.go (NEW) — restic-SFTP backend + orchestration:
  • Fail-fast (spike Q8): every restic call carries -o sftp.command="ssh … -oConnectTimeout=10 … -s sftp" so a dead NAS errors in ~10 s, never a multi-minute hang; -oStrictHostKeyChecking=yes -oUserKnownHostsFile=<pinned> (no blind TOFU) + -oBatchMode=yes.
  • init-if-absent (idempotent), per-app restic backup --tag, forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune, single-flight + migration-guard, restic's own exit code checked, restore via restic restore latest --tag <app> --target <scratch> (non-destructive).
  • Secrets (SSH key + auto-gen repo password) are 0600 files in the data dir — never logged, never in a non-0600/committed file; the repo is encrypted. They ride DR via the PBS whole-CT snapshot.
  • Injection guard (ValidateOffboxTarget): host/user/repo are charset-checked and must not start with - (ssh option-injection, e.g. -oProxyCommand); OffboxConfigured fails closed on an invalid target so a hostile value can never reach the ssh exec.
• internal/settings/settings.go — OffboxTarget + per-app AppBackupPrefs.Offbox + helpers (no secrets).
• cmd/controller/main.go — daily offbox-backup (04:15), failure → backup_failed operator alert.
• internal/web/offbox_handlers.go + backups.html — configure target (host/port/user/repo + out-of-band SSH key + known_hosts), per-app toggles, run-now, restore, status.

Tests — go build ./... && go vet ./... && go test ./... GREEN

• ConnectTimeout fail-fast companion (red-proof): a fake SSH transport hangs to the ctx deadline WITHOUT the arg, fails fast WITH it — proving the arg is load-bearing.
• dead-NAS run fails fast + alerts + records status=error; restore round-trip byte-identical (SFTP-shaped seam); single-flight skip; repo init idempotency; secrets are 0600.
• Security companion: ValidateOffboxTarget rejects ssh-option-injection (-oProxyCommand host/user), metacharacters, traversal, spaces, non-absolute paths; accepts a clean target.

Live e2e validation (isolated sim NAS, SFTP)

Method: isolated throwaway sim = a Debian LXC on felhom-pve running sshd + a felhom user + /srv/repo (its own throwaway ed25519 key; never touched production storage). Drove the real controller endpoints (demo box has no password → CSRF skipped — the full server pipeline runs) via curl from inside the controller container; the off-box SSH key + known_hosts were injected through the config endpoint exactly as the UI form does. Sim + key destroyed afterward.

• A — configure + first backup: config saved (secrets written 0600); off-box run initialized the restic repo on the NAS and backed up rallly (1 snapshot, 5 s). The NAS holds a proper restic repo (config/data/index/keys/snapshots); the config blob is ciphertext and a grep for SQL/dump/rallly markers in the repo found nothing — the NAS sees only ciphertext.
• B — dead NAS fails fast + alerts: black-holed the sim (blackhole route in guest 9201, since the controller's egress is guest-side — the host iptables path used in A1/A2 doesn't catch guest→guest bridged traffic) → the run failed in ~20 s (= 2×ConnectTimeout for the cat-config + init probes) with restic's connect to host … port 22: Connection timed out; the backup_failed operator alert fired; the controller + guest stayed healthy (no hang). On route removal → a run succeeded again.
• C — restore byte-identical: restored rallly from the off-box repo to a scratch dir → diff -r against the local source = BYTE_IDENTICAL (non-destructive; live data untouched).
• D — DR reachability: the SSH key + repo password are 0600 files in the controller data dir → captured by the agent's PBS whole-CT snapshot of the rootfs → a rebuilt box restores them and can list/restore the off-box repo. (The recovery-unit/dr-recipe stay secret-free by design; controller secrets ride DR via PBS — the existing escrow path.)
• Caught live (version skew): restic 0.14.0 rejects -o sftp.args (the spike's form) — fixed to the portable -o sftp.command="ssh … -s sftp" (ConnectTimeout preserved), rebuilt, re-validated.
• Security finding (background commit review): the injection guard above was added in response — then rebuilt + a regression backup confirmed a valid target still runs.

Observations / follow-ups

• NAS arc complete (A1 agent mount ✅ · A2 controller media registry ✅ · B off-box backup ✅).
• Next queued: per-storage worst-fill alerting (hub host_disk follow-on) + a provisioning-side vzdump retention default so a box can't refill its own root.
• DR-escrow note: off-box secrets ride the PBS whole-CT path (rootfs data dir), NOT the secret-free recovery-unit/dr-recipe — documented, no new escrow code.
• The demo box's off-box target was disabled + its secrets removed at teardown (the sim is gone).

No secrets in any committed file. SSH key + repo password were throwaway and out-of-band only.