diff --git a/documentation/audits/SPIKE-nas-storage-2026-06-29.md b/documentation/audits/SPIKE-nas-storage-2026-06-29.md new file mode 100644 index 0000000..a16527b --- /dev/null +++ b/documentation/audits/SPIKE-nas-storage-2026-06-29.md @@ -0,0 +1,232 @@ +# SPIKE — NAS network storage (media automount + backup target), simulated on DooPlex + +**Date:** 2026-06-29 +**Class:** Spike (empirical validation; output = this doc only). Throwaway probes; nothing merged into +agent/controller production packages. +**Goal:** validate the **bulk-media network-storage** mechanism (storage class 3) — host-side mount of a NAS +share → bind into guest 9201 → bind into media container, for both **SMB** and **NFS** — before writing the +production TASK. Secondary: restic-over-SFTP as a backup target (class 2). + +**Simulated NAS:** real Samba (SMB) + nfs-kernel-server (NFS) already running on **DooPlex 192.168.0.180**; +added an **isolated scratch export** `/srv/nas-sim/media` (sample media, ~513 MB incl. a 512 MB file), separate +from the live Longhorn-PVC NFS exports and PBS/k3s. Mount probes on the Proxmox host **felhom-pve** +(192.168.0.162) → guest **9201** (live demo guest) → throwaway alpine containers. + +> **Sim-vs-real-NAS caveat (stated up front):** this validates the Linux *mechanism* (kernel mount, propagation, +> idmap, failure semantics) against a Debian Samba/NFS server. It does **not** reproduce Synology/QNAP-specific +> quirks: their SMB dialect ceilings, NFS `squash`/`mapall` UIs, per-share ACL models, SMB multichannel, or +> appliance reboot timing. Those need a confirmation pass on the real appliance before GA. + +--- + +## Verdict: **READY** to write the production "network storage" TASK (with named caveats) + +NFS and SMB both work end-to-end through the host-mount→bind chain for **read and write**, with a precise +UID-mapping recipe; NFS `soft` mounts achieve clean failure isolation (fail-soft + auto-recover, no guest/df +wedge). SMB failure isolation is the one area needing the documented option set (below). Restic-SFTP backup is +trivial and failure-isolated. No blockers; see "Caveats / for the production TASK." + +--- + +## Per-Spike-Question findings + +### Q1 — Mount placement: **HOST-side, bind into guest (in-guest mount is kernel-blocked)** + +Guest 9201 is an **unprivileged** LXC (`unprivileged: 1`, `features: nesting,keyctl`). Mounting NFS or CIFS +**inside** the guest fails: +``` +# pct exec 9201 -- mount -t nfs ... → mount: permission denied (exit 32) +# pct exec 9201 -- mount -t cifs ... → mount: permission denied (exit 32) +# CapEff: 000001ffffffffff (full caps *within the userns* — but mounting these fs types needs the INIT userns) +``` +The kernel forbids NFS/CIFS mounts in a user namespace regardless of the in-userns capability set. **Therefore +the NAS mount must live on the Proxmox host (agent-owned)** and be bind-propagated into the guest — exactly +mirroring the existing drive-bind architecture. + +**Propagation works for free** via the existing `mp8: /mnt/felhom-drives` bind: the host path is `shared` +propagation, the guest sees it `shared,slave`, and existing drive mounts (felhom-flash/felhom-usb) already +propagate through. A **new host mount under `/mnt/felhom-drives/` appears live in the guest with no +restart** — confirmed. + +**Recommended chain:** `host: mount NAS → /mnt/felhom-drives/` → (auto-propagates) `guest: +/mnt/felhom-drives/` → `docker -v /mnt/felhom-drives/:/media`. + +### Q2 — SMB and NFS both work through the full chain — **YES (read+write)** + +**NFS** (`mount -t nfs -o vers=4.1,soft,timeo=50,retrans=2,_netdev,noatime`): host mount OK → propagated to +guest → container read the 512 MB file (md5 verified) and **wrote** a new file. Effective options included +`soft,timeo=50,retrans=2`. + +**SMB/CIFS** (`mount -t cifs -o credentials=…,uid=101000,gid=101000,forceuid,forcegid,file_mode=0664,dir_mode=0775,vers=3.0,_netdev`): +host mount OK → propagated → container read and **wrote** over SMB. + +Minimal viable option sets: +- **NFS:** `vers=4.1,soft,timeo=50,retrans=2,_netdev,noatime` + the export's `all_squash,anonuid=…,anongid=…` + (see Q5). `vers=4.1` avoids the rpcbind/111 + lock-manager surface of v3. +- **SMB:** `vers=3.0,credentials=,uid=…,gid=…,forceuid,forcegid,file_mode=0664,dir_mode=0775,_netdev` + (creds file 0600 on the host; never inline). `dir_mode`/`file_mode` MUST be plain octal (`0775`, not `2775`). + +### Q5 — UID / permission mapping: **the LXC +100000 offset is the whole story** (read+write proven) + +The established felhom convention (verified against the live `felhom-usb` userdata): a dir owned **host +`100000:101000`** appears in the guest as **`root:1000`**, mode **2775** (setgid, group-writable); media apps +write via **gid 1000**. So **guest/container uid/gid N = host uid/gid N+100000.** + +A naïve `anonuid=1000` export is WRONG — it lands as host-1000, which is *outside* the guest's 100000–165535 +idmap and shows as `nobody:nogroup` (readable because world-readable, but **not writable**). The fix is to +present the share as the **host-mapped** id: + +- **NFS:** export `…,all_squash,anonuid=101000,anongid=101000` and own the share dir `101000:101000`. Then the + guest sees `1000:1000`, and a uid/gid-1000 container **reads existing files and writes new ones**; the write + lands on the NAS as `101000:101000`. (Verified: `WRITE_OK`, server file `-rw-r--r-- 101000 101000`.) +- **SMB:** mount with `uid=101000,gid=101000,forceuid,forcegid` (client forces the guest-visible owner to + 1000:1000 regardless of server ownership) **and** the share's `force user`/`force group` must resolve to a + **uid/gid 101000** identity so server-side writes are actually permitted on the 101000-owned dir. (Verified: + with `force user=nasguest` (uid 101000) the container `SMB_WRITE_OK`; with `force user`=a uid-1000 identity + the write was denied — the mismatch is exactly the trap to document.) + +**Mapping recipe for the production class:** the NAS share for a customer's media must be owned/squashed to +`container_uid + 100000` (e.g. media app uid 1000 → NAS uses **101000**). On a real Synology/QNAP this means an +export `mapall`/`anonuid` of 101000 (NFS) or an SMB share whose force-user resolves to 101000. + +### Q3 — Failure isolation (THE risk) + +Method: with a media container looping reads, **black-holed** the NAS from felhom-pve via an **iptables DROP on +180 scoped to 192.168.0.162** (DROP, not REJECT → simulates a dead NAS / network black hole; leaves the NFS +service up for k3s/Longhorn — the safe way to "disappear" it). + +**NFS (`soft,timeo=50,retrans=2`) — clean fail-soft + auto-recover:** +- Uncached `stat` returned **`Input/output error` after ~16 s** (≈ retrans×timeo). It **errors, does not hang.** +- `df` on the mount did **not** hang (returned rc 0); no reader process in **D** (uninterruptible) state. +- The looping reader kept serving from page-cache briefly, then flipped to **FAIL** once the attribute cache + expired and revalidation hit the black-holed server — i.e. **fails soft, no wedge**. +- **Other apps unaffected:** gitea stayed **HTTP 200** throughout; the guest stayed healthy (`uptime`/commands + responsive). +- **Recovery on NAS return:** after removing the DROP, a fresh read was `RECOVERED_READ_OK`, and **the looping + reader auto-recovered on its own** (`FAIL → PASS`) with **no remount/intervention**. `stat` worked again. + +**SMB/CIFS (default options) — hangs, but blast-radius contained:** +- With the NAS black-holed, `df` on the mount **hung the full 40 s timeout** (rc 124); an uncached `stat` + **hung the full 60 s**; the reader's `dd` went **D (uninterruptible)** state. CIFS has **no `soft`/`hard` + knob** like NFS — a dead server blocks operations until the SMB session/TCP timeout. +- BUT the blast radius was still **contained**: gitea stayed **HTTP 200**, the guest stayed healthy + (`uptime` responsive) — only the process touching the SMB mount wedged. +- **Recovery:** on NAS return the wedged `dd` **unblocked on its own** (`df` instant, no D-state, reader + `PASS` again) — no kill/remount needed. + +**Q3 verdict:** **prefer NFS `soft` for the media class** — it fails-soft (clean EIO) and never wedges +`df`/the process. **SMB hangs the accessing process** during the outage (contained, self-healing) and is the +fallback only for NAS that speak SMB-only; pair it with **automount idle-unmount** (below) to minimise the +stale-mount window. The WRONG outcome (a hard mount wedging the *guest* or *df* box-wide) was **not** observed +in either case — the unprivileged-LXC bind boundary kept the blast radius to the accessing container. + +### Q4 — Write reliability — **integrity perfect; an interrupted in-flight write leaves a partial file** + +- **Clean write integrity:** a container (uid 1000) wrote a 100 MB file through the chain; md5 on the + host-mount side == md5 read **directly on the NAS server** (`2541a840…`) — byte-intact end to end. +- **Disappear mid-write** (write-loop of 1 MB fsync'd files, NAS black-holed at file ~221): every **completed** + file before the cut was byte-intact (exactly 1 MB); the **single in-flight file (`w221.bin`) was left + partial/truncated** on the NAS; the writer's `dd` went **D-state** (writes-in-flight soft-fail slower than + reads — soft eventually returns EIO, but the in-flight file is already partially persisted). +- **Safe-option note:** `soft` is correct for read-mostly media, but for **write apps a `soft`-interrupted + write can leave a partial file**. Mitigations (recommend, in order): (1) use apps with **atomic write** + semantics (temp file + `rename`) — a partial temp never gets the final name (immich, *arr, qBittorrent all + do this); (2) for write-critical non-atomic paths, **stage on local SSD then move**; (3) `hard` mounts avoid + the partial (the write blocks until the NAS returns then completes) at the cost of the wedge — **not** + recommended as the default. + +### Q6 — Performance — **~128 MB/s sequential through the chain — ample for streaming** + +512 MB sequential read through container→bind→guest→host→NFS measured **128.2 MB/s** (≈ saturating the +gigabit LAN). A 4K stream needs ≈3–6 MB/s, so this comfortably serves many concurrent HD/4K streams. (Host-side +raw read reported 12 GB/s = page-cache served, not representative.) **Recommendation: transcode scratch dirs and +thumbnail/preview caches stay on the local SSD**, not the NAS — they are latency-sensitive random I/O and +high-churn; only the bulk media library lives on the NAS. + +### Q8 (secondary) — restic-over-SFTP backup target — **works; failed-job-not-hang on NAS loss** + +`restic init` / `backup` / `restore` against `sftp:user@192.168.0.180:/srv/nas-sim/backup/repo` (no kernel +mount) succeeded: `snapshot … saved`, and a restored 20 MB file matched the source md5 (`c26d0b59…`) exactly. +**Failure isolation:** with SSH/22 black-holed, the backup **failed as a userspace job** (errored / had to be +timed out — set `-o sftp.args='-o ConnectTimeout=10'` to fail fast instead of retrying the TCP connect). The +key property: it is an ordinary **killable process with no kernel mount** — **no `df`/D-state/box wedge**, just +a failed backup. This is exactly the desired class-2 behaviour and de-risks that feature. + +### Q7 — Storage-class fit — **model it as a distinct "network storage" class, NOT a drive** + +A NAS share must be a **separate registry entry kind** from the physical-drive model, because it is network + +read-mostly + credentialed and has **no device lifecycle**: +- It **bypasses** the drive `enroll → format → eject → decommission → migrate` machinery entirely (there is no + block device to wipe, no durable-id, no SMART, no data-bearing-wipe gate). +- It carries instead: `kind: network`, `protocol: nfs|smb`, `server`, `export/share`, `credentials_ref` + (SMB; out-of-band), `mount_opts`, `mapped_uid/gid` (the +100000 recipe), `automount: true`, `idle_unmount`. +- It is **selectable as a media app's data path** (the `HDD_PATH`/userdata slot) but must be **disqualified as + a target for**: the DB/app-config class (class 1, SSD-only), and the live-write-critical fast paths. The + controller already sees storage as *paths*; a network-storage path slots in as a new path *kind* with + role-gating "bulk userdata / media only". +- Health is **per-share liveness** (is the automount currently mounted + last-access OK), surfaced as + "degraded: NAS unreachable" affecting only apps bound to it — never the box's overall health. + +--- + +## Recommended production architecture + +**The mount chain (mirror the existing drive-bind model):** +``` +NAS share ──(host automount, agent-owned)──▶ /mnt/felhom-drives/ on felhom-pve + ──(existing shared→slave propagation via mp8)──▶ same path inside guest 9201 + ──(docker -v /mnt/felhom-drives/:/media)──▶ media container +``` +The mount lives **host-side** (Q1: unprivileged LXC cannot mount NFS/CIFS); it propagates into the guest for +free through the existing `/mnt/felhom-drives` shared bind — **no new guest mountpoint, no guest restart**. + +**Mount via systemd `.automount` (on-demand + idle-unmount)** so an idle NAS reboot is a non-event and the +stale-mount window is minimised. autofs is an equivalent alternative; systemd automount needs **no extra +package** on the Proxmox host. **Validated:** before access the path is an `autofs` trigger (no real mount); a +single `ls` mounts it on demand (`nfs4`); after `TimeoutIdleSec` (tested 20 s) it **auto-unmounts**; a +re-access remounts. So when no app is reading, there is **no NAS mount at all** — a NAS reboot while idle +cannot affect the box, and an in-use outage is governed by the `soft` semantics above. + +**Protocol + option sets:** +- **NFS (preferred):** `vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev`; export `all_squash,anonuid=, + anongid=` where `/ = container_uid/gid + 100000`. Clean failure isolation. +- **SMB (SMB-only NAS):** `vers=3.0,credentials=<0600 file>,uid=,gid=,forceuid,forcegid, + file_mode=0664,dir_mode=0775,_netdev` (octal modes!), share `force user/group` = the uid/gid-`/` + identity. Hangs-but-contained on NAS loss → always pair with automount idle-unmount. + +**UID/permission mapping (the +100000 rule):** a media app at container uid/gid 1000 = host 101000. The NAS must +present/own/squash files as **101000** so the guest sees **1000** and the container reads+writes natively. NFS: +`anonuid/anongid=101000`. SMB: `force user/group` → uid/gid 101000 + client `uid/gid=101000,forceuid,forcegid`. + +**Failure / health model:** NFS `soft` → a down NAS yields EIO to the accessing app only; `df`/guest/other apps +stay healthy; auto-recovers on return. SMB → accessing app blocks (D-state) but contained + self-healing. Model +per-share health as "degraded, NAS unreachable"; never fold it into box health. **Writes:** prefer atomic-write +apps or local-staging (a `soft`-interrupted write leaves a partial in-flight file). + +**Local-SSD-stays-local:** DB + app config/state (class 1) and **transcode scratch + thumbnail/preview caches** +never go on the NAS — SSD only. The NAS holds bulk media library + (class 2) restic-SFTP backup repos. + +--- + +## Caveats / for the production TASK +- **Sim-vs-real-NAS** (restated): Synology/QNAP SMB dialect/ACL/`mapall` specifics, SMB multichannel, and + appliance reboot timing are unproven here — confirm on the real appliance before GA. +- **SMB credentials** live out-of-band (a 0600 creds file the agent writes); never in git, never in the + registry plaintext. The mapping uses a host uid/gid-101000 identity for `force user/group`. +- **restic-SFTP** needs an explicit `ConnectTimeout` to fail fast; otherwise a down NAS slow-fails the backup + job (still no kernel wedge). +- **Atomic-write requirement** for write-apps on a `soft` NFS NAS — document per-app, or stage writes on SSD. + +## Probe teardown — DONE +Fully torn down and verified: +- **180:** `/etc/exports` + `/etc/samba/smb.conf` restored from the pre-spike `.bak-nasspike` copies (then the + baks removed); the `[nas-sim]` share and `/srv/nas-sim` export are gone; `nassim`/`nasguest` users removed; + scratch `/srv/nas-sim` deleted; the throwaway pubkey removed from `authorized_keys`; SMB-pw files deleted. + **Verified intact:** both Longhorn-PVC NFS exports still present (count 2); all 6 original samba shares + (`kisfenyo_home`/`media`/`orsi_home`/…) preserved; PBS/k3s untouched. +- **felhom-pve:** all NAS mounts unmounted (0 remaining), the systemd automount units removed, the throwaway + restic binary + SSH keypair removed, `/mnt/felhom-drives` still holds only the real drives + (`felhom-flash`, `felhom-usb`). All iptables DROP rules removed throughout. +- **Live guest 9201 unharmed:** `gitea`, `rallly`, `felhom-controller` all `healthy` after the spike. + +No secrets in this doc. SMB/restic test credentials were throwaway and stored out-of-band only.