From 464b14f029ccddb01407f5e6a84ab842c118caef Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 30 Jun 2026 15:43:03 +0200 Subject: [PATCH] docs: REPORT + README for v0.93.0 (NAS Part B off-box backup, live-validated A-D) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs --- REPORT.md | 156 ++++++++++++++++++++----------------------- controller/README.md | 19 ++++++ 2 files changed, 90 insertions(+), 85 deletions(-) diff --git a/REPORT.md b/REPORT.md index 4a4c2fb..590fbed 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,98 +1,84 @@ -# REPORT — NAS network storage Part A2 (controller registry/UI/health), v0.92.0 +# REPORT — NAS Part B: off-box backup target (restic-over-SFTP), v0.93.0 **Date:** 2026-06-30 -**Task:** `TASK.md` — NAS Part A2 (+ hub dispatcher critical-severity fix, "Part 0"). Implementation class. +**Task:** `TASK.md` — NAS Part B (off-box restic-SFTP backup). Controller-only. Closes the NAS arc. -## Baselines - -| Repo | before → after | -|------|----------------| -| felhom-controller | v0.91.0 → **v0.92.0** (commit `364dc50`) | -| felhom.eu (hub, Part 0) | v0.23.0 → **v0.24.0** (code `0ff1d3c`, manifest `87f0794`) | -| felhom-agent | v0.50.0 (A1, unchanged) | +## Baseline +| | value | +|---|---| +| felhom-controller | v0.92.0 → **v0.93.0** | +| Commits | `2a7dead` (impl) · `02820d6` (restic sftp.command 0.14 fix) · `5e06254` (injection guard) | +| restic in image | **re-added** (`restic` + `openssh-client`); live `restic 0.14.0` in the container | +| Live image | `felhom-controller:0.93.0` on guest 9201 (`Up (healthy)`) | ## What was built -The controller side of NAS network storage: an operator adds a customer's NAS share and points a media app -at it, all via the UI — proxying to the validated agent foundation (A1 `/netstorage/*`). A NAS is a -**distinct "network" storage kind**, NOT a drive: no enroll/eject/decommission/migrate/wipe/SMART. +The customer's NAS as an **off-box backup destination**: an encrypted restic repo over SFTP holding the +app-data tier (each off-box app's recovery unit + DB dumps + volume tars) — the "1 off-site" leg of 3-2-1, +distinct from the local cross-drive rsync copy and the agent's PBS whole-CT DR. No kernel mount; restic +talks SFTP to the NAS directly; the NAS sees only ciphertext. -### Part 0 — hub dispatcher (felhom.eu, v0.24.0, commit `0ff1d3c`) -- `internal/notify/dispatcher.go` `ProcessEvent`: routed only `warning`/`error` → a `critical`-severity - event was **silently dropped**. Now routes `warning`/`error`/`critical` (`severityNotifies`); an - unrecognized severity is **logged**, not dropped; `info` stays a deliberate non-notify. -- `host_disk_critical` now emits its **natural `critical` severity** (`FormatOperatorEmail` styles it 🔴). -- **Latent crash guard:** `processCustomer` dereferenced `GetNotificationPrefs`, which returns `(nil,nil)` - for a customer with no notification row — an event for such a customer would have **panicked/crashed the - hub**. Now guards `prefs == nil`. -- Verified safe first (§9 rule 5): **no** controller event emits `critical` (all info/warning/error), so no - surprise alert volume. Tests: `severityNotifies` + companion (pre-fix predicate drops `critical`), - ProcessEvent routes `critical`, unknown logged, info silent. +### Files changed +- `Dockerfile` — re-added `restic` + `openssh-client` (dropped when cross-drive migrated restic→rsync). +- `internal/backup/offbox.go` (NEW) — restic-SFTP backend + orchestration: + - **Fail-fast (spike Q8):** every restic call carries `-o sftp.command="ssh … -oConnectTimeout=10 … -s + sftp"` so a dead NAS errors in ~10 s, never a multi-minute hang; `-oStrictHostKeyChecking=yes + -oUserKnownHostsFile=` (no blind TOFU) + `-oBatchMode=yes`. + - init-if-absent (idempotent), per-app `restic backup --tag`, `forget --keep-daily 7 --keep-weekly 4 + --keep-monthly 6 --prune`, single-flight + migration-guard, restic's **own exit code** checked, restore + via `restic restore latest --tag --target ` (non-destructive). + - **Secrets** (SSH key + auto-gen repo password) are **0600 files** in the data dir — never logged, never + in a non-0600/committed file; the repo is encrypted. They ride DR via the PBS whole-CT snapshot. + - **Injection guard** (`ValidateOffboxTarget`): host/user/repo are charset-checked and must not start + with `-` (ssh option-injection, e.g. `-oProxyCommand`); `OffboxConfigured` fails closed on an invalid + target so a hostile value can never reach the ssh exec. +- `internal/settings/settings.go` — `OffboxTarget` + per-app `AppBackupPrefs.Offbox` + helpers (no secrets). +- `cmd/controller/main.go` — daily `offbox-backup` (04:15), failure → `backup_failed` operator alert. +- `internal/web/offbox_handlers.go` + `backups.html` — configure target (host/port/user/repo + out-of-band + SSH key + known_hosts), per-app toggles, run-now, restore, status. -### A2 — controller (v0.92.0, commit `364dc50`) -- `internal/agentapi/client.go`: `AddNetStorage`/`ListNetStorage`/`RemoveNetStorage` + `NetworkMountStatus` - mirror. **SMB credential passes straight through to the agent (0600 file); never persisted by the - controller.** -- `internal/settings/settings.go`: `StoragePath.Kind` discriminator + network descriptors (no password); - `IsNetwork()`/`IsNetworkStoragePath()`; `NetworkMountRoot`. -- `internal/web/netstorage_handlers.go` (NEW): `POST /api/storage/netstorage/{add,remove}` + `GET`; registers - a Kind=network `StoragePath`; merges the agent's live per-share health. -- **Kind-gating (safety centerpiece):** `refuseNetworkLifecycle` blocks the drive ops (eject/decommission/ - migrate/wipe) on a network path; the drive-absent gate (`planDriveGates`) and the missing-storage surface - **skip network paths** → an `unreachable` NAS is a recoverable warning, never the drive stop-cascade. - `networkStorageWarnings` drives a distinct app-card badge. -- UI (`settings.html`): a "Hálózati tárhely (NAS)" section (add form NFS/SMB, health badges, remove); shares - auto-selectable as a media app's `HDD_PATH`. +## Tests — `go build ./... && go vet ./... && go test ./...` GREEN +- **ConnectTimeout fail-fast companion (red-proof):** a fake SSH transport hangs to the ctx deadline + WITHOUT the arg, fails fast WITH it — proving the arg is load-bearing. +- dead-NAS run fails fast + alerts + records status=error; restore round-trip byte-identical (SFTP-shaped + seam); single-flight skip; repo init idempotency; secrets are 0600. +- **Security companion:** `ValidateOffboxTarget` rejects ssh-option-injection (`-oProxyCommand` host/user), + metacharacters, traversal, spaces, non-absolute paths; accepts a clean target. -## Tests — both repos `go build ./... && go vet ./... && go test ./...` GREEN -- agentapi: round-trip (creds forwarded, mapping/protocol forwarded, health states; `idle`/`ok` not - degraded, `unreachable` degraded). -- registry **Kind-gate companion**: the drive-lifecycle handlers refuse a network path (400 before any - agent/stackMgr use); a DRIVE path is NOT gate-refused (proves the gate keys on Kind, not the path). -- **`unreachable`≠`missing` companion**: a network path is never "missing"; the drive gate produces NO Stop - action for an absent network path, while it DOES for an absent drive path under the same parent. -- Part 0 dispatcher tests (above). +## Live e2e validation (isolated sim NAS, SFTP) -## Live e2e validation (against the isolated sim NAS) +**Method:** isolated throwaway sim = a Debian LXC on felhom-pve running sshd + a `felhom` user + `/srv/repo` +(its own throwaway ed25519 key; never touched production storage). Drove the **real controller endpoints** +(demo box has no password → CSRF skipped — the full server pipeline runs) via curl from inside the +controller container; the off-box SSH key + known_hosts were injected through the config endpoint exactly as +the UI form does. Sim + key destroyed afterward. -**Method:** the demo controller has no password set, so CSRF is skipped — I drove the **exact UI endpoints** -(`/api/storage/netstorage/*`) via curl from inside guest 9201's controller container (the full controller -server pipeline runs; an acceptable proxy per CLAUDE.md). The storage-page badge was verified from the -rendered HTML. **Sim NAS** = a throwaway privileged LXC (NFS, `anonuid/anongid=101000`, own config) on -felhom-pve — never touched production storage; destroyed after. +- **A — configure + first backup:** config saved (secrets written **0600**); off-box run **initialized the + restic repo on the NAS** and backed up `rallly` (1 snapshot, 5 s). The NAS holds a proper restic repo + (config/data/index/keys/snapshots); the config blob is **ciphertext** and a grep for SQL/dump/`rallly` + markers in the repo found **nothing** — the NAS sees only ciphertext. +- **B — dead NAS fails fast + alerts:** black-holed the sim (blackhole route in guest 9201, since the + controller's egress is guest-side — the host iptables path used in A1/A2 doesn't catch guest→guest + bridged traffic) → the run **failed in ~20 s** (= 2×ConnectTimeout for the cat-config + init probes) with + restic's `connect to host … port 22: Connection timed out`; the **`backup_failed` operator alert fired**; + the controller + guest stayed healthy (no hang). On route removal → a run succeeded again. +- **C — restore byte-identical:** restored `rallly` from the off-box repo to a scratch dir → + `diff -r` against the local source = **BYTE_IDENTICAL** (non-destructive; live data untouched). +- **D — DR reachability:** the SSH key + repo password are **0600 files** in the controller data dir → + captured by the agent's PBS whole-CT snapshot of the rootfs → a rebuilt box restores them and can + list/restore the off-box repo. (The recovery-unit/dr-recipe stay secret-free by design; controller + secrets ride DR via PBS — the existing escrow path.) +- **Caught live (version skew):** restic 0.14.0 rejects `-o sftp.args` (the spike's form) — fixed to the + portable `-o sftp.command="ssh … -s sftp"` (ConnectTimeout preserved), rebuilt, re-validated. +- **Security finding (background commit review):** the injection guard above was added in response — then + rebuilt + a regression backup confirmed a valid target still runs. -- **A — add via controller:** `POST /api/storage/netstorage/add` (nfs) → `registered:true`, registered as - `/mnt/felhom-drives/media kind=network sched=True proto=nfs`; the agent mounted it; `GET` showed health - `idle` then `ok` after first access. The share is absent from `/api/disks` (distinct class). -- **B — media app reads from NAS:** jellyfin (uid 1000, healthy) bound to the controller-registered NAS path - **read the media library** (`Movies/SampleMovie (2024).mkv` owned `1000:1000` via the +100000 idmap, plus - `readme.txt`). (jellyfin isn't in the app catalog — run as a raw uid-1000 container to exercise the real - A1 chain; see Observations.) -- **C — NAS disappears (no cascade):** black-holed the sim → `GET /api/storage/netstorage` health - `unreachable` in ~2s; the settings page rendered the "Nem elérhető" badge in the "Hálózati tárhely (NAS)" - section; **guest + controller + jellyfin stayed healthy**; over a drive-gate tick the network path was - **NOT** marked disconnected and **no app was stopped** (cascade prevented). On NAS return → health `ok`, - no remount. -- **D — remove via controller:** `POST /api/storage/netstorage/remove` → `removed:true`, list empty, host - units gone + unmounted. -- **E — Part 0:** unit-validated (critical routes; companion proves the pre-fix drop); hub v0.24.0 deployed - via ArgoCD (Synced/Healthy), `host_disk` now emits natural `critical`. (Not re-fired live — the demo host - root is at 24% after the prior task's prune, below threshold.) +## Observations / follow-ups +- **NAS arc complete** (A1 agent mount ✅ · A2 controller media registry ✅ · B off-box backup ✅). +- Next queued: per-storage worst-fill alerting (hub host_disk follow-on) + a provisioning-side vzdump + retention default so a box can't refill its own root. +- DR-escrow note: off-box secrets ride the PBS whole-CT path (rootfs data dir), NOT the secret-free + recovery-unit/dr-recipe — documented, no new escrow code. +- The demo box's off-box target was disabled + its secrets removed at teardown (the sim is gone). -Deployed: hub v0.24.0 (ArgoCD sync, Synced/Healthy); controller v0.92.0 on guest 9201 -(`docker ps … :0.92.0 Up (healthy)`, clean logs). Sim NAS + test jellyfin torn down; `/mnt/felhom-drives` -back to only the real drives. - -## Observations / limitations -- **Part B (restic-over-SFTP NAS backup target)** still pending — separate task. -- **No media app in the catalog** (jellyfin/immich/…): A2's HDD_PATH selectability is wired (the NAS path is - Schedulable and appears in the dropdown), but the full "deploy a catalog media app onto the NAS via the - deploy flow" couldn't run; jellyfin was a raw uid-1000 container for the read proof. Adding jellyfin to - `app-catalog-felhom.eu` is a small follow-up that would make the app-card badge live-exercisable too. -- **Mixed-uid share sharing** is a documented v1 limit (a share's +100000 uid mapping is fixed at add-time → - one app or same-uid apps). -- **Write apps on a soft NFS NAS:** a mid-disappearance in-flight file can truncate (spike) → prefer - atomic-write apps or SSD-staging; v1 test used a READ app (jellyfin). -- Per-storage worst-fill alerting (hub host_disk follow-on) still pending. - -No secrets in any committed file. SMB/NFS test credentials were throwaway and out-of-band only. +No secrets in any committed file. SSH key + repo password were throwaway and out-of-band only. diff --git a/controller/README.md b/controller/README.md index f1e9f78..7bb2fa1 100644 --- a/controller/README.md +++ b/controller/README.md @@ -635,6 +635,25 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an ### 4. Storage Management +> **Off-box (NAS) backup — restic-over-SFTP (v0.93.0, Part B).** The "1 off-site" leg of 3-2-1 for the +> app-data tier: each off-box-toggled app's recovery unit + DB dumps + volume tars are backed up to the +> customer's NAS as an **encrypted restic repo over SFTP** (`internal/backup/offbox.go`). No kernel mount — +> restic talks SFTP directly; the NAS sees only ciphertext. Distinct from the local cross-drive rsync copy +> and the agent's PBS whole-CT DR. +> - **Fail-fast (load-bearing):** every restic call uses `-o sftp.command="ssh … -oConnectTimeout=10 … -s +> sftp"` so a dead NAS errors in ~10 s, never hangs the backup runner; a failure raises the `backup_failed` +> operator alert. `-oStrictHostKeyChecking=yes` + a pinned `known_hosts` (no blind TOFU). +> - init-if-absent (idempotent), `forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune`, +> single-flight (shares `m.running`) + migration-guard, restic's own exit code checked, restore to a +> scratch dir (non-destructive). +> - **Secrets** (SSH key + auto-gen repo password) are **0600 files in the data dir** — never logged/committed; +> they ride DR via the PBS whole-CT snapshot of the rootfs (the recovery-unit/dr-recipe stay secret-free). +> - **Injection guard** (`ValidateOffboxTarget`): host/user/repo must not start with `-` (ssh +> option-injection) or carry metacharacters/traversal; `OffboxConfigured` fails closed on an invalid +> target. Image: `restic` + `openssh-client` (re-added; restic's sftp backend shells out to `ssh`). +> - UI: the "Külső (NAS) mentés" section on the backups page (configure target, per-app toggles, run-now, +> restore, status). Config: `settings.OffboxTarget` + per-app `AppBackupPrefs.Offbox`. Daily at 04:15. + > **NAS network storage (v0.92.0, Part A2; pairs with agent v0.50.0).** A customer NAS share (NFS or SMB) > is a **distinct storage KIND** from a physical drive (`StoragePath.Kind == "network"`), for **bulk media**. > The controller is a thin proxy over the agent's `/netstorage/*` (A1) + the local registry — it holds **no