docs: REPORT + README for v0.93.0 (NAS Part B off-box backup, live-validated A-D)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs

REPORT.md

@@ -1,98 +1,84 @@
-# REPORT — NAS network storage Part A2 (controller registry/UI/health), v0.92.0
+# REPORT — NAS Part B: off-box backup target (restic-over-SFTP), v0.93.0
 
 **Date:** 2026-06-30
-**Task:** `TASK.md` — NAS Part A2 (+ hub dispatcher critical-severity fix, "Part 0"). Implementation class.
+**Task:** `TASK.md` — NAS Part B (off-box restic-SFTP backup). Controller-only. Closes the NAS arc.
 
-## Baselines
-
-| Repo | before → after |
-|------|----------------|
-| felhom-controller | v0.91.0 → **v0.92.0** (commit `364dc50`) |
-| felhom.eu (hub, Part 0) | v0.23.0 → **v0.24.0** (code `0ff1d3c`, manifest `87f0794`) |
-| felhom-agent | v0.50.0 (A1, unchanged) |
+## Baseline
+| | value |
+|---|---|
+| felhom-controller | v0.92.0 → **v0.93.0** |
+| Commits | `2a7dead` (impl) · `02820d6` (restic sftp.command 0.14 fix) · `5e06254` (injection guard) |
+| restic in image | **re-added** (`restic` + `openssh-client`); live `restic 0.14.0` in the container |
+| Live image | `felhom-controller:0.93.0` on guest 9201 (`Up (healthy)`) |
 
 ## What was built
 
-The controller side of NAS network storage: an operator adds a customer's NAS share and points a media app
-at it, all via the UI — proxying to the validated agent foundation (A1 `/netstorage/*`). A NAS is a
-**distinct "network" storage kind**, NOT a drive: no enroll/eject/decommission/migrate/wipe/SMART.
+The customer's NAS as an **off-box backup destination**: an encrypted restic repo over SFTP holding the
+app-data tier (each off-box app's recovery unit + DB dumps + volume tars) — the "1 off-site" leg of 3-2-1,
+distinct from the local cross-drive rsync copy and the agent's PBS whole-CT DR. No kernel mount; restic
+talks SFTP to the NAS directly; the NAS sees only ciphertext.
 
-### Part 0 — hub dispatcher (felhom.eu, v0.24.0, commit `0ff1d3c`)
-- `internal/notify/dispatcher.go` `ProcessEvent`: routed only `warning`/`error` → a `critical`-severity
-  event was **silently dropped**. Now routes `warning`/`error`/`critical` (`severityNotifies`); an
-  unrecognized severity is **logged**, not dropped; `info` stays a deliberate non-notify.
-- `host_disk_critical` now emits its **natural `critical` severity** (`FormatOperatorEmail` styles it 🔴).
-- **Latent crash guard:** `processCustomer` dereferenced `GetNotificationPrefs`, which returns `(nil,nil)`
-  for a customer with no notification row — an event for such a customer would have **panicked/crashed the
-  hub**. Now guards `prefs == nil`.
-- Verified safe first (§9 rule 5): **no** controller event emits `critical` (all info/warning/error), so no
-  surprise alert volume. Tests: `severityNotifies` + companion (pre-fix predicate drops `critical`),
-  ProcessEvent routes `critical`, unknown logged, info silent.
+### Files changed
+- `Dockerfile` — re-added `restic` + `openssh-client` (dropped when cross-drive migrated restic→rsync).
+- `internal/backup/offbox.go` (NEW) — restic-SFTP backend + orchestration:
+  - **Fail-fast (spike Q8):** every restic call carries `-o sftp.command="ssh … -oConnectTimeout=10 … -s
+    sftp"` so a dead NAS errors in ~10 s, never a multi-minute hang; `-oStrictHostKeyChecking=yes
+    -oUserKnownHostsFile=<pinned>` (no blind TOFU) + `-oBatchMode=yes`.
+  - init-if-absent (idempotent), per-app `restic backup --tag`, `forget --keep-daily 7 --keep-weekly 4
+    --keep-monthly 6 --prune`, single-flight + migration-guard, restic's **own exit code** checked, restore
+    via `restic restore latest --tag <app> --target <scratch>` (non-destructive).
+  - **Secrets** (SSH key + auto-gen repo password) are **0600 files** in the data dir — never logged, never
+    in a non-0600/committed file; the repo is encrypted. They ride DR via the PBS whole-CT snapshot.
+  - **Injection guard** (`ValidateOffboxTarget`): host/user/repo are charset-checked and must not start
+    with `-` (ssh option-injection, e.g. `-oProxyCommand`); `OffboxConfigured` fails closed on an invalid
+    target so a hostile value can never reach the ssh exec.
+- `internal/settings/settings.go` — `OffboxTarget` + per-app `AppBackupPrefs.Offbox` + helpers (no secrets).
+- `cmd/controller/main.go` — daily `offbox-backup` (04:15), failure → `backup_failed` operator alert.
+- `internal/web/offbox_handlers.go` + `backups.html` — configure target (host/port/user/repo + out-of-band
+  SSH key + known_hosts), per-app toggles, run-now, restore, status.
 
-### A2 — controller (v0.92.0, commit `364dc50`)
-- `internal/agentapi/client.go`: `AddNetStorage`/`ListNetStorage`/`RemoveNetStorage` + `NetworkMountStatus`
-  mirror. **SMB credential passes straight through to the agent (0600 file); never persisted by the
-  controller.**
-- `internal/settings/settings.go`: `StoragePath.Kind` discriminator + network descriptors (no password);
-  `IsNetwork()`/`IsNetworkStoragePath()`; `NetworkMountRoot`.
-- `internal/web/netstorage_handlers.go` (NEW): `POST /api/storage/netstorage/{add,remove}` + `GET`; registers
-  a Kind=network `StoragePath`; merges the agent's live per-share health.
-- **Kind-gating (safety centerpiece):** `refuseNetworkLifecycle` blocks the drive ops (eject/decommission/
-  migrate/wipe) on a network path; the drive-absent gate (`planDriveGates`) and the missing-storage surface
-  **skip network paths** → an `unreachable` NAS is a recoverable warning, never the drive stop-cascade.
-  `networkStorageWarnings` drives a distinct app-card badge.
-- UI (`settings.html`): a "Hálózati tárhely (NAS)" section (add form NFS/SMB, health badges, remove); shares
-  auto-selectable as a media app's `HDD_PATH`.
+## Tests — `go build ./... && go vet ./... && go test ./...` GREEN
+- **ConnectTimeout fail-fast companion (red-proof):** a fake SSH transport hangs to the ctx deadline
+  WITHOUT the arg, fails fast WITH it — proving the arg is load-bearing.
+- dead-NAS run fails fast + alerts + records status=error; restore round-trip byte-identical (SFTP-shaped
+  seam); single-flight skip; repo init idempotency; secrets are 0600.
+- **Security companion:** `ValidateOffboxTarget` rejects ssh-option-injection (`-oProxyCommand` host/user),
+  metacharacters, traversal, spaces, non-absolute paths; accepts a clean target.
 
-## Tests — both repos `go build ./... && go vet ./... && go test ./...` GREEN
-- agentapi: round-trip (creds forwarded, mapping/protocol forwarded, health states; `idle`/`ok` not
-  degraded, `unreachable` degraded).
-- registry **Kind-gate companion**: the drive-lifecycle handlers refuse a network path (400 before any
-  agent/stackMgr use); a DRIVE path is NOT gate-refused (proves the gate keys on Kind, not the path).
-- **`unreachable`≠`missing` companion**: a network path is never "missing"; the drive gate produces NO Stop
-  action for an absent network path, while it DOES for an absent drive path under the same parent.
-- Part 0 dispatcher tests (above).
+## Live e2e validation (isolated sim NAS, SFTP)
 
-## Live e2e validation (against the isolated sim NAS)
+**Method:** isolated throwaway sim = a Debian LXC on felhom-pve running sshd + a `felhom` user + `/srv/repo`
+(its own throwaway ed25519 key; never touched production storage). Drove the **real controller endpoints**
+(demo box has no password → CSRF skipped — the full server pipeline runs) via curl from inside the
+controller container; the off-box SSH key + known_hosts were injected through the config endpoint exactly as
+the UI form does. Sim + key destroyed afterward.
 
-**Method:** the demo controller has no password set, so CSRF is skipped — I drove the **exact UI endpoints**
-(`/api/storage/netstorage/*`) via curl from inside guest 9201's controller container (the full controller
-server pipeline runs; an acceptable proxy per CLAUDE.md). The storage-page badge was verified from the
-rendered HTML. **Sim NAS** = a throwaway privileged LXC (NFS, `anonuid/anongid=101000`, own config) on
-felhom-pve — never touched production storage; destroyed after.
+- **A — configure + first backup:** config saved (secrets written **0600**); off-box run **initialized the
+  restic repo on the NAS** and backed up `rallly` (1 snapshot, 5 s). The NAS holds a proper restic repo
+  (config/data/index/keys/snapshots); the config blob is **ciphertext** and a grep for SQL/dump/`rallly`
+  markers in the repo found **nothing** — the NAS sees only ciphertext.
+- **B — dead NAS fails fast + alerts:** black-holed the sim (blackhole route in guest 9201, since the
+  controller's egress is guest-side — the host iptables path used in A1/A2 doesn't catch guest→guest
+  bridged traffic) → the run **failed in ~20 s** (= 2×ConnectTimeout for the cat-config + init probes) with
+  restic's `connect to host … port 22: Connection timed out`; the **`backup_failed` operator alert fired**;
+  the controller + guest stayed healthy (no hang). On route removal → a run succeeded again.
+- **C — restore byte-identical:** restored `rallly` from the off-box repo to a scratch dir →
+  `diff -r` against the local source = **BYTE_IDENTICAL** (non-destructive; live data untouched).
+- **D — DR reachability:** the SSH key + repo password are **0600 files** in the controller data dir →
+  captured by the agent's PBS whole-CT snapshot of the rootfs → a rebuilt box restores them and can
+  list/restore the off-box repo. (The recovery-unit/dr-recipe stay secret-free by design; controller
+  secrets ride DR via PBS — the existing escrow path.)
+- **Caught live (version skew):** restic 0.14.0 rejects `-o sftp.args` (the spike's form) — fixed to the
+  portable `-o sftp.command="ssh … -s sftp"` (ConnectTimeout preserved), rebuilt, re-validated.
+- **Security finding (background commit review):** the injection guard above was added in response — then
+  rebuilt + a regression backup confirmed a valid target still runs.
 
-- **A — add via controller:** `POST /api/storage/netstorage/add` (nfs) → `registered:true`, registered as
-  `/mnt/felhom-drives/media kind=network sched=True proto=nfs`; the agent mounted it; `GET` showed health
-  `idle` then `ok` after first access. The share is absent from `/api/disks` (distinct class).
-- **B — media app reads from NAS:** jellyfin (uid 1000, healthy) bound to the controller-registered NAS path
-  **read the media library** (`Movies/SampleMovie (2024).mkv` owned `1000:1000` via the +100000 idmap, plus
-  `readme.txt`). (jellyfin isn't in the app catalog — run as a raw uid-1000 container to exercise the real
-  A1 chain; see Observations.)
-- **C — NAS disappears (no cascade):** black-holed the sim → `GET /api/storage/netstorage` health
-  `unreachable` in ~2s; the settings page rendered the "Nem elérhető" badge in the "Hálózati tárhely (NAS)"
-  section; **guest + controller + jellyfin stayed healthy**; over a drive-gate tick the network path was
-  **NOT** marked disconnected and **no app was stopped** (cascade prevented). On NAS return → health `ok`,
-  no remount.
-- **D — remove via controller:** `POST /api/storage/netstorage/remove` → `removed:true`, list empty, host
-  units gone + unmounted.
-- **E — Part 0:** unit-validated (critical routes; companion proves the pre-fix drop); hub v0.24.0 deployed
-  via ArgoCD (Synced/Healthy), `host_disk` now emits natural `critical`. (Not re-fired live — the demo host
-  root is at 24% after the prior task's prune, below threshold.)
+## Observations / follow-ups
+- **NAS arc complete** (A1 agent mount ✅ · A2 controller media registry ✅ · B off-box backup ✅).
+- Next queued: per-storage worst-fill alerting (hub host_disk follow-on) + a provisioning-side vzdump
+  retention default so a box can't refill its own root.
+- DR-escrow note: off-box secrets ride the PBS whole-CT path (rootfs data dir), NOT the secret-free
+  recovery-unit/dr-recipe — documented, no new escrow code.
+- The demo box's off-box target was disabled + its secrets removed at teardown (the sim is gone).
 
-Deployed: hub v0.24.0 (ArgoCD sync, Synced/Healthy); controller v0.92.0 on guest 9201
-(`docker ps … :0.92.0 Up (healthy)`, clean logs). Sim NAS + test jellyfin torn down; `/mnt/felhom-drives`
-back to only the real drives.
-
-## Observations / limitations
-- **Part B (restic-over-SFTP NAS backup target)** still pending — separate task.
-- **No media app in the catalog** (jellyfin/immich/…): A2's HDD_PATH selectability is wired (the NAS path is
-  Schedulable and appears in the dropdown), but the full "deploy a catalog media app onto the NAS via the
-  deploy flow" couldn't run; jellyfin was a raw uid-1000 container for the read proof. Adding jellyfin to
-  `app-catalog-felhom.eu` is a small follow-up that would make the app-card badge live-exercisable too.
-- **Mixed-uid share sharing** is a documented v1 limit (a share's +100000 uid mapping is fixed at add-time →
-  one app or same-uid apps).
-- **Write apps on a soft NFS NAS:** a mid-disappearance in-flight file can truncate (spike) → prefer
-  atomic-write apps or SSD-staging; v1 test used a READ app (jellyfin).
-- Per-storage worst-fill alerting (hub host_disk follow-on) still pending.
-
-No secrets in any committed file. SMB/NFS test credentials were throwaway and out-of-band only.
+No secrets in any committed file. SSH key + repo password were throwaway and out-of-band only.

controller/README.md

@@ -635,6 +635,25 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an
 
 ### 4. Storage Management
 
+> **Off-box (NAS) backup — restic-over-SFTP (v0.93.0, Part B).** The "1 off-site" leg of 3-2-1 for the
+> app-data tier: each off-box-toggled app's recovery unit + DB dumps + volume tars are backed up to the
+> customer's NAS as an **encrypted restic repo over SFTP** (`internal/backup/offbox.go`). No kernel mount —
+> restic talks SFTP directly; the NAS sees only ciphertext. Distinct from the local cross-drive rsync copy
+> and the agent's PBS whole-CT DR.
+> - **Fail-fast (load-bearing):** every restic call uses `-o sftp.command="ssh … -oConnectTimeout=10 … -s
+>   sftp"` so a dead NAS errors in ~10 s, never hangs the backup runner; a failure raises the `backup_failed`
+>   operator alert. `-oStrictHostKeyChecking=yes` + a pinned `known_hosts` (no blind TOFU).
+> - init-if-absent (idempotent), `forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune`,
+>   single-flight (shares `m.running`) + migration-guard, restic's own exit code checked, restore to a
+>   scratch dir (non-destructive).
+> - **Secrets** (SSH key + auto-gen repo password) are **0600 files in the data dir** — never logged/committed;
+>   they ride DR via the PBS whole-CT snapshot of the rootfs (the recovery-unit/dr-recipe stay secret-free).
+> - **Injection guard** (`ValidateOffboxTarget`): host/user/repo must not start with `-` (ssh
+>   option-injection) or carry metacharacters/traversal; `OffboxConfigured` fails closed on an invalid
+>   target. Image: `restic` + `openssh-client` (re-added; restic's sftp backend shells out to `ssh`).
+> - UI: the "Külső (NAS) mentés" section on the backups page (configure target, per-app toggles, run-now,
+>   restore, status). Config: `settings.OffboxTarget` + per-app `AppBackupPrefs.Offbox`. Daily at 04:15.
+
 > **NAS network storage (v0.92.0, Part A2; pairs with agent v0.50.0).** A customer NAS share (NFS or SMB)
 > is a **distinct storage KIND** from a physical drive (`StoragePath.Kind == "network"`), for **bulk media**.
 > The controller is a thin proxy over the agent's `/netstorage/*` (A1) + the local registry — it holds **no