felhom.eu/documentation/audits/SPIKE-nas-storage-2026-06-29.md

SPIKE — NAS network storage (media automount + backup target), simulated on DooPlex

Date: 2026-06-29 Class: Spike (empirical validation; output = this doc only). Throwaway probes; nothing merged into agent/controller production packages. Goal: validate the bulk-media network-storage mechanism (storage class 3) — host-side mount of a NAS share → bind into guest 9201 → bind into media container, for both SMB and NFS — before writing the production TASK. Secondary: restic-over-SFTP as a backup target (class 2).

Simulated NAS: real Samba (SMB) + nfs-kernel-server (NFS) already running on DooPlex 192.168.0.180; added an isolated scratch export /srv/nas-sim/media (sample media, ~513 MB incl. a 512 MB file), separate from the live Longhorn-PVC NFS exports and PBS/k3s. Mount probes on the Proxmox host felhom-pve (192.168.0.162) → guest 9201 (live demo guest) → throwaway alpine containers.

Sim-vs-real-NAS caveat (stated up front): this validates the Linux mechanism (kernel mount, propagation, idmap, failure semantics) against a Debian Samba/NFS server. It does not reproduce Synology/QNAP-specific quirks: their SMB dialect ceilings, NFS squash/mapall UIs, per-share ACL models, SMB multichannel, or appliance reboot timing. Those need a confirmation pass on the real appliance before GA.

---

Verdict: READY to write the production "network storage" TASK (with named caveats)

NFS and SMB both work end-to-end through the host-mount→bind chain for read and write, with a precise UID-mapping recipe; NFS soft mounts achieve clean failure isolation (fail-soft + auto-recover, no guest/df wedge). SMB failure isolation is the one area needing the documented option set (below). Restic-SFTP backup is trivial and failure-isolated. No blockers; see "Caveats / for the production TASK."

---

Per-Spike-Question findings

Q1 — Mount placement: HOST-side, bind into guest (in-guest mount is kernel-blocked)

Guest 9201 is an unprivileged LXC (unprivileged: 1, features: nesting,keyctl). Mounting NFS or CIFS inside the guest fails:

# pct exec 9201 -- mount -t nfs  ... → mount: permission denied (exit 32)
# pct exec 9201 -- mount -t cifs ... → mount: permission denied (exit 32)
# CapEff: 000001ffffffffff  (full caps *within the userns* — but mounting these fs types needs the INIT userns)

The kernel forbids NFS/CIFS mounts in a user namespace regardless of the in-userns capability set. Therefore the NAS mount must live on the Proxmox host (agent-owned) and be bind-propagated into the guest — exactly mirroring the existing drive-bind architecture.

Propagation works for free via the existing mp8: /mnt/felhom-drives bind: the host path is shared propagation, the guest sees it shared,slave, and existing drive mounts (felhom-flash/felhom-usb) already propagate through. A new host mount under /mnt/felhom-drives/<name> appears live in the guest with no restart — confirmed.

Recommended chain: host: mount NAS → /mnt/felhom-drives/<share> → (auto-propagates) guest: /mnt/felhom-drives/<share> → docker -v /mnt/felhom-drives/<share>:/media.

Q2 — SMB and NFS both work through the full chain — YES (read+write)

NFS (mount -t nfs -o vers=4.1,soft,timeo=50,retrans=2,_netdev,noatime): host mount OK → propagated to guest → container read the 512 MB file (md5 verified) and wrote a new file. Effective options included soft,timeo=50,retrans=2.

SMB/CIFS (mount -t cifs -o credentials=…,uid=101000,gid=101000,forceuid,forcegid,file_mode=0664,dir_mode=0775,vers=3.0,_netdev): host mount OK → propagated → container read and wrote over SMB.

Minimal viable option sets:

• NFS: vers=4.1,soft,timeo=50,retrans=2,_netdev,noatime + the export's all_squash,anonuid=…,anongid=… (see Q5). vers=4.1 avoids the rpcbind/111 + lock-manager surface of v3.
• SMB: vers=3.0,credentials=<file>,uid=…,gid=…,forceuid,forcegid,file_mode=0664,dir_mode=0775,_netdev (creds file 0600 on the host; never inline). dir_mode/file_mode MUST be plain octal (0775, not 2775).

Q5 — UID / permission mapping: the LXC +100000 offset is the whole story (read+write proven)

The established felhom convention (verified against the live felhom-usb userdata): a dir owned host 100000:101000 appears in the guest as root:1000, mode 2775 (setgid, group-writable); media apps write via gid 1000. So guest/container uid/gid N = host uid/gid N+100000.

A naïve anonuid=1000 export is WRONG — it lands as host-1000, which is outside the guest's 100000–165535 idmap and shows as nobody:nogroup (readable because world-readable, but not writable). The fix is to present the share as the host-mapped id:

• NFS: export …,all_squash,anonuid=101000,anongid=101000 and own the share dir 101000:101000. Then the guest sees 1000:1000, and a uid/gid-1000 container reads existing files and writes new ones; the write lands on the NAS as 101000:101000. (Verified: WRITE_OK, server file -rw-r--r-- 101000 101000.)
• SMB: mount with uid=101000,gid=101000,forceuid,forcegid (client forces the guest-visible owner to 1000:1000 regardless of server ownership) and the share's force user/force group must resolve to a uid/gid 101000 identity so server-side writes are actually permitted on the 101000-owned dir. (Verified: with force user=nasguest (uid 101000) the container SMB_WRITE_OK; with force user=a uid-1000 identity the write was denied — the mismatch is exactly the trap to document.)

Mapping recipe for the production class: the NAS share for a customer's media must be owned/squashed to container_uid + 100000 (e.g. media app uid 1000 → NAS uses 101000). On a real Synology/QNAP this means an export mapall/anonuid of 101000 (NFS) or an SMB share whose force-user resolves to 101000.

Q3 — Failure isolation (THE risk)

Method: with a media container looping reads, black-holed the NAS from felhom-pve via an iptables DROP on 180 scoped to 192.168.0.162 (DROP, not REJECT → simulates a dead NAS / network black hole; leaves the NFS service up for k3s/Longhorn — the safe way to "disappear" it).

NFS (soft,timeo=50,retrans=2) — clean fail-soft + auto-recover:

• Uncached stat returned Input/output error after ~16 s (≈ retrans×timeo). It errors, does not hang.
• df on the mount did not hang (returned rc 0); no reader process in D (uninterruptible) state.
• The looping reader kept serving from page-cache briefly, then flipped to FAIL once the attribute cache expired and revalidation hit the black-holed server — i.e. fails soft, no wedge.
• Other apps unaffected: gitea stayed HTTP 200 throughout; the guest stayed healthy (uptime/commands responsive).
• Recovery on NAS return: after removing the DROP, a fresh read was RECOVERED_READ_OK, and the looping reader auto-recovered on its own (FAIL → PASS) with no remount/intervention. stat worked again.

SMB/CIFS (default options) — hangs, but blast-radius contained:

• With the NAS black-holed, df on the mount hung the full 40 s timeout (rc 124); an uncached stat hung the full 60 s; the reader's dd went D (uninterruptible) state. CIFS has no soft/hard knob like NFS — a dead server blocks operations until the SMB session/TCP timeout.
• BUT the blast radius was still contained: gitea stayed HTTP 200, the guest stayed healthy (uptime responsive) — only the process touching the SMB mount wedged.
• Recovery: on NAS return the wedged dd unblocked on its own (df instant, no D-state, reader PASS again) — no kill/remount needed.

Q3 verdict: prefer NFS soft for the media class — it fails-soft (clean EIO) and never wedges df/the process. SMB hangs the accessing process during the outage (contained, self-healing) and is the fallback only for NAS that speak SMB-only; pair it with automount idle-unmount (below) to minimise the stale-mount window. The WRONG outcome (a hard mount wedging the guest or df box-wide) was not observed in either case — the unprivileged-LXC bind boundary kept the blast radius to the accessing container.

Q4 — Write reliability — integrity perfect; an interrupted in-flight write leaves a partial file

• Clean write integrity: a container (uid 1000) wrote a 100 MB file through the chain; md5 on the host-mount side == md5 read directly on the NAS server (2541a840…) — byte-intact end to end.
• Disappear mid-write (write-loop of 1 MB fsync'd files, NAS black-holed at file ~221): every completed file before the cut was byte-intact (exactly 1 MB); the single in-flight file (w221.bin) was left partial/truncated on the NAS; the writer's dd went D-state (writes-in-flight soft-fail slower than reads — soft eventually returns EIO, but the in-flight file is already partially persisted).
• Safe-option note: soft is correct for read-mostly media, but for write apps a soft-interrupted write can leave a partial file. Mitigations (recommend, in order): (1) use apps with atomic write semantics (temp file + rename) — a partial temp never gets the final name (immich, *arr, qBittorrent all do this); (2) for write-critical non-atomic paths, stage on local SSD then move; (3) hard mounts avoid the partial (the write blocks until the NAS returns then completes) at the cost of the wedge — not recommended as the default.

Q6 — Performance — ~128 MB/s sequential through the chain — ample for streaming

512 MB sequential read through container→bind→guest→host→NFS measured 128.2 MB/s (≈ saturating the gigabit LAN). A 4K stream needs ≈3–6 MB/s, so this comfortably serves many concurrent HD/4K streams. (Host-side raw read reported 12 GB/s = page-cache served, not representative.) Recommendation: transcode scratch dirs and thumbnail/preview caches stay on the local SSD, not the NAS — they are latency-sensitive random I/O and high-churn; only the bulk media library lives on the NAS.

Q8 (secondary) — restic-over-SFTP backup target — works; failed-job-not-hang on NAS loss

restic init / backup / restore against sftp:user@192.168.0.180:/srv/nas-sim/backup/repo (no kernel mount) succeeded: snapshot … saved, and a restored 20 MB file matched the source md5 (c26d0b59…) exactly. Failure isolation: with SSH/22 black-holed, the backup failed as a userspace job (errored / had to be timed out — set -o sftp.args='-o ConnectTimeout=10' to fail fast instead of retrying the TCP connect). The key property: it is an ordinary killable process with no kernel mount — no df/D-state/box wedge, just a failed backup. This is exactly the desired class-2 behaviour and de-risks that feature.

Q7 — Storage-class fit — model it as a distinct "network storage" class, NOT a drive

A NAS share must be a separate registry entry kind from the physical-drive model, because it is network + read-mostly + credentialed and has no device lifecycle:

• It bypasses the drive enroll → format → eject → decommission → migrate machinery entirely (there is no block device to wipe, no durable-id, no SMART, no data-bearing-wipe gate).
• It carries instead: kind: network, protocol: nfs|smb, server, export/share, credentials_ref (SMB; out-of-band), mount_opts, mapped_uid/gid (the +100000 recipe), automount: true, idle_unmount.
• It is selectable as a media app's data path (the HDD_PATH/userdata slot) but must be disqualified as a target for: the DB/app-config class (class 1, SSD-only), and the live-write-critical fast paths. The controller already sees storage as paths; a network-storage path slots in as a new path kind with role-gating "bulk userdata / media only".
• Health is per-share liveness (is the automount currently mounted + last-access OK), surfaced as "degraded: NAS unreachable" affecting only apps bound to it — never the box's overall health.

---

Recommended production architecture

The mount chain (mirror the existing drive-bind model):

NAS share ──(host automount, agent-owned)──▶ /mnt/felhom-drives/<share> on felhom-pve
          ──(existing shared→slave propagation via mp8)──▶ same path inside guest 9201
          ──(docker -v /mnt/felhom-drives/<share>:/media)──▶ media container

The mount lives host-side (Q1: unprivileged LXC cannot mount NFS/CIFS); it propagates into the guest for free through the existing /mnt/felhom-drives shared bind — no new guest mountpoint, no guest restart.

Mount via systemd .automount (on-demand + idle-unmount) so an idle NAS reboot is a non-event and the stale-mount window is minimised. autofs is an equivalent alternative; systemd automount needs no extra package on the Proxmox host. Validated: before access the path is an autofs trigger (no real mount); a single ls mounts it on demand (nfs4); after TimeoutIdleSec (tested 20 s) it auto-unmounts; a re-access remounts. So when no app is reading, there is no NAS mount at all — a NAS reboot while idle cannot affect the box, and an in-use outage is governed by the soft semantics above.

Protocol + option sets:

• NFS (preferred): vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev; export all_squash,anonuid=<U>, anongid=<G> where <U>/<G> = container_uid/gid + 100000. Clean failure isolation.
• SMB (SMB-only NAS): vers=3.0,credentials=<0600 file>,uid=<U>,gid=<G>,forceuid,forcegid, file_mode=0664,dir_mode=0775,_netdev (octal modes!), share force user/group = the uid/gid-<U>/<G> identity. Hangs-but-contained on NAS loss → always pair with automount idle-unmount.

UID/permission mapping (the +100000 rule): a media app at container uid/gid 1000 = host 101000. The NAS must present/own/squash files as 101000 so the guest sees 1000 and the container reads+writes natively. NFS: anonuid/anongid=101000. SMB: force user/group → uid/gid 101000 + client uid/gid=101000,forceuid,forcegid.

Failure / health model: NFS soft → a down NAS yields EIO to the accessing app only; df/guest/other apps stay healthy; auto-recovers on return. SMB → accessing app blocks (D-state) but contained + self-healing. Model per-share health as "degraded, NAS unreachable"; never fold it into box health. Writes: prefer atomic-write apps or local-staging (a soft-interrupted write leaves a partial in-flight file).

Local-SSD-stays-local: DB + app config/state (class 1) and transcode scratch + thumbnail/preview caches never go on the NAS — SSD only. The NAS holds bulk media library + (class 2) restic-SFTP backup repos.

---

Caveats / for the production TASK

• Sim-vs-real-NAS (restated): Synology/QNAP SMB dialect/ACL/mapall specifics, SMB multichannel, and appliance reboot timing are unproven here — confirm on the real appliance before GA.
• SMB credentials live out-of-band (a 0600 creds file the agent writes); never in git, never in the registry plaintext. The mapping uses a host uid/gid-101000 identity for force user/group.
• restic-SFTP needs an explicit ConnectTimeout to fail fast; otherwise a down NAS slow-fails the backup job (still no kernel wedge).
• Atomic-write requirement for write-apps on a soft NFS NAS — document per-app, or stage writes on SSD.

Probe teardown — DONE

Fully torn down and verified:

• 180: /etc/exports + /etc/samba/smb.conf restored from the pre-spike .bak-nasspike copies (then the baks removed); the [nas-sim] share and /srv/nas-sim export are gone; nassim/nasguest users removed; scratch /srv/nas-sim deleted; the throwaway pubkey removed from authorized_keys; SMB-pw files deleted. Verified intact: both Longhorn-PVC NFS exports still present (count 2); all 6 original samba shares (kisfenyo_home/media/orsi_home/…) preserved; PBS/k3s untouched.
• felhom-pve: all NAS mounts unmounted (0 remaining), the systemd automount units removed, the throwaway restic binary + SSH keypair removed, /mnt/felhom-drives still holds only the real drives (felhom-flash, felhom-usb). All iptables DROP rules removed throughout.
• Live guest 9201 unharmed: gitea, rallly, felhom-controller all healthy after the spike.

No secrets in this doc. SMB/restic test credentials were throwaway and stored out-of-band only.