felhom.eu/documentation/controller/network-storage-nas.md

Network storage (NAS) — bulk-media shares with verify-before-commit

Status: authoritative feature doc, code-verified against felhom-agent v0.81.0 + felhom-controller v0.113.0 (2026-07-11). Evidence base: documentation/audits/SPIKE-nas-storage-2026-06-29.md (the locked mount recipe) and documentation/audits/SPIKE-nas-verify-2026-07-11.md (the verify mechanisms, squash matrix, error taxonomy — commit b57f6c1). Code: agent internal/storage/netmount.go / internal/storage/netverify.go / internal/localapi/netverifyjob.go; controller internal/web/netstorage_job.go / internal/web/netprobe*.go.

What it is

A customer's NAS serves bulk media (film, photo, music) to media apps. The share is mounted host-side by the agent under /mnt/felhom-drives/<name> via a systemd .automount + .mount pair (on-demand + idle-unmount), propagates into the guest through the existing shared mp8 bind, and is selectable as a media app's data path. A NAS is a distinct storage class from a drive: no durable-id, no SMART, no enroll/eject/decommission/wipe/migrate — remove is the only lifecycle action (refuseNetworkLifecycle blocks the drive verbs).

The locked mount recipe

• NFS (nfs4): vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev,retry=0. soft = failure isolation (clean EIO, never a wedge). retry=0 (SPIKE-nas-verify Q4-vi): without it a dead-NAS on-demand access wedges the accessing app until systemd's 90 s start cap (measured 91 s); with it the access fails clean in ~3.8 s (ENODEV) and each autofs re-access is a fresh attempt. retry only governs retrying a FAILED first attempt — the happy path is untouched. The uid mapping is the EXPORT's job (squash), never the client mount.
• SMB (cifs): vers=3.0,credentials=<0600 file>,uid=<uid+100000>,gid=<gid+100000>,forceuid, forcegid,file_mode=0664,dir_mode=0775,_netdev. No retry= (a mount.nfs option; mount.cifs rejects it). Credentials live in an agent-written 0600 file, never in the registry, never controller-persisted.
• The +100000 rule: container uid/gid N = host N+100000 (unprivileged-LXC idmap).

Verify-before-commit (the add pipeline)

POST /api/storage/netstorage/add never registers blind. The controller orchestrates (detached job, single-flight, ~150 s budget, status on GET /api/storage/netstorage/add/status):

1. agent add (POST /netstorage/add): role-gate → sync fast-fail (spec validation + 2 s TCP pre-probe; an unreachable server is refused with NOTHING installed) → stage SMB creds → install unit pair + enable the automount → start the agent's detached verify job.
2. agent verify (polled on GET /netstorage/verify-status every 2 s): a directory read through the automount triggers a REAL mount (spike Q1: an in-guest or host-side access wakes it, ~1 s LAN); success is judged from /proc/mounts only (truth table below); a failure is classified from the mount unit's journal and auto-rolled-back agent-side (units + creds).
3. in-guest uid-1000 write probe: the controller re-execs itself (felhom-controller --netprobe <dir>) with SysProcAttr.Credential{Uid:1000,Gid:1000} — create a .felhom-proba-<random> dot-file, write a nonce, read back, compare, delete. This catches the squash trap: an export that mounts fine but denies every uid-1000 write (spike Q3 row c). A failed delete after a good write is a WARN, not a failure.
4. register (AddStoragePath, Kind=network, Schedulable) — the LAST step. The worst crash outcome is an agent-side orphan (surfaced in the list as a remove-only "Árva megosztás" row), never a registered-but-broken path.

Any failure = full rollback: nothing registered, no units left installed, no creds file. If the agent restarts mid-verify, its in-memory verify slot reports phase none; the controller treats that as verify-lost and rolls back (the orphan row is the belt-and-braces surface).

Mount-success truth table (SPIKE-nas-verify §8)

Trigger ReadDir(where)	/proc/mounts shows nfs4/cifs at where	Verdict
ok	yes	mount OK → the uid-1000 probe decides writability
EACCES / EPERM	yes	mount OK (the agent user just can't read it — a 0700 export is fine)
any result	no	mount FAILED → classify from the journal
still blocked at 95 s	—	timeout (black-holed-but-routed; systemd's 90 s cap resolves it)

Readability is NEVER the truth source — /proc/mounts is.

Error taxonomy (live-measured; classification is string-based BY DESIGN — every failure is rc=32)

Category	Journal substring (verbatim)	Meaning
unreachable	No route to host / Connection refused / Connection timed out (or the 2 s sync pre-probe)	no NAS behind the address
nfs_export	reason given by server: No such file or directory	export missing OR not permitted — MERGED: NFSv4 returns the identical string for both (Q4 ii≡iii); the customer message names both possibilities
smb_auth	mount error(13)	wrong SMB username/password
smb_share	mount error(2)	SMB share name not found
timeout	Mounting timed out. Terminating	routed but not answering (systemd 90 s cap)
mount_failed	(none matched / journal unavailable)	generic; detail carries the raw journal
not_writable	(probe exit 2)	mounts, but uid-1000 cannot write — the squash trap
probe_io	(probe exit 3)	write readback failed/differed

The journal is read unprivileged (journalctl -u <unit> -n 20 -o cat): the agent user is in the systemd-journal group (host-install ≥ v1.13.0 adds it; existing hosts: usermod -aG systemd-journal felhom-agent && systemctl restart felhom-agent). No sudoers grant — journal access is group-based by rule. Journal unavailable degrades to mount_failed with a hint; the rollback still runs.

NAS-side recipes (protocol-honest guidance — what the UI tells the customer)

• SMB (Synology, QNAP — the consumer default, listed FIRST in the UI): a plain user account with rw on the share is sufficient — zero server-side uid configuration (spike Q5). Files land on the NAS owned by the connecting account; the client mount forces the guest view to uid/gid 1000. Hardlinks worked on Debian Samba/ext4 — re-verify per appliance.
• NFS, simple recipe (Route A — spike Q3 row b, ACCEPTED): enable "map all users / all squash" on the export, rw, to ANY local user. The guest-uid-1000 app gets full read/write/rename/hardlink/delete. Caveats:
  • guest-visible ownership is nobody:nogroup (65534) — cosmetic for apps that just read/write;
  • chown fails with an immediate clean EPERM (no hang);
  • chmod SUCCEEDS and persists server-side (the squashed identity owns every file) — modes are app-controlled; operators should know a mode change on the NAS side is real.
• NFS, full-fidelity recipe (TrueNAS / Linux server): rw,all_squash,anonuid=<uid+100000>,anongid=<uid+100000> (e.g. 101000 for a uid-1000 app) — ownership information is exact end-to-end.
• The pinned WRONG case (what not_writable usually means): anonuid=<x> without all_squash is a no-op for non-root traffic — the app's wire uid (101000) hits the export unmapped and fails on others-perms. The consumer recipe REQUIRES the map-ALL-users mode; an anonymous-uid field alone does nothing. The UI's not_writable message and the guidance block tell this same story.
• Naming caveat (R-66, controller v0.159.0): Windows network names („FELHOM") generally do NOT resolve here. The Szerver field takes an IP or a DNS name; NetBIOS/WSD flat-name resolution is Windows machinery the guest deliberately does not run (the R-6 LAN-discovery spike: the docker bridge is deaf, and nothing NetBIOS-resolves on the mount path). The add form's helper text says so under the field, and an unreachable-class failure for a single-label non-IP server appends a purely lexical hint („Tipp: a(z) »FELHOM« Windows-hálózati névnek tűnik — használja az eszköz IP-címét") — no NetBIOS/mDNS resolution is ever attempted. Where the customer FINDS the peer Felhom box's address: the serving box's Megosztás page („közvetlen cím" row) or its Beállítások → Rendszer → „Hálózat" card (Helyi cím) — both live-computed per render, never stored (S-5), so they are current by construction.

Browsing (FileBrowser) — R-67, controller v0.160.0

A registered share is browsable in FileBrowser: the sync binds the share ROOT (/mnt/felhom-drives/<name>:/srv/<name>:rslave) and adds a sidebar source named by the share's display label. The :rslave propagation is load-bearing — host-side automount wake and idle-unmount events reach the running container, and an in-container access through the bind WAKES an idle trigger (Phase-0-probed live on demo-hp, 2026-07-22). Differences from drives, both deliberate: there is no userdata scoping and no skeleton (Felhom convention dirs are never written onto a customer's own NAS; the sync is read-only toward the share), and the gate is not the drive-absent check (idle is healthy) but the stub classifier: when this namespace sees a local stub dir instead of the NAS, the share is excluded from FileBrowser until propagation recovers — an exposed stub would accept uploads that the real mount later shadows, silently hiding the customer's files. A dead NAS surfaces as a fast clean error in FileBrowser (the soft/retry=0 mount recipe), and recovery after the NAS returns needs no intervention.

Health model

Per-share liveness (ok | idle | unreachable | stub): idle (automount idle-unmounted) is the benign steady state; unreachable degrades the affected share's apps ONLY — never box health, never the drive missing→stop cascade (kind-gated). Timing/budget note for integrators: the verify worst case is systemd's 90 s (black-holed server) — verify traffic therefore rides the detached job + status poll, never a single long HTTP call (the agentapi client keeps its global 15 s timeout).

One classification, two surfaces (F8, controller v0.119.0, CAMPAIGN-3)

The agent's per-share health derives from a server-level TCP dial (server:2049/445). That dial stays GREEN when a single export is exportfs -u'd — the server still answers on 2049 for its other exports — so the agent reports benign idle+reachable:true for a share whose consuming namespace is actually a stub. The campaign found the resulting contradictory UI: the share row showed "Készenlét" while the stacks/dashboard cards showed the stub badge (which reads the consuming-namespace classifier shipped in v0.117.0).

The fix (controller-only — the §3 fork's recommended option B): the share row (networkStorageItems → fuseNetHealth) reads the SAME system.ClassifyPathFS the stacks stub badge reads and fuses it over the agent view. Precedence: a whole-server unreachable (agent dial failed) is the most actionable and WINS; otherwise a stub classification at Where (namespace sees local disk, not the NAS) overrides a benign idle/ok — the row badge becomes "Hibás — az alkalmazások nem a NAS-t látják"; an autofs-healthy trigger, a real network fs, or an inconclusive unknown read leave the agent-derived health untouched (never force-mount an idle trigger — that is healthy by June's idle-unmount design). The share row and the stacks badge now derive from ONE classification and can never contradict. Live-proven 2026-07-12: an exportfs -u while idle flipped the row to stub in lockstep with the stacks badge; re-export cleared both to idle/ok, with reachable:true throughout (the agent's server-level view unchanged — the fusion is what carries the export-level truth).

Input validation — mapped_uid range (F4, v0.119.0)

The add handler range-checks the container uid/gid (1..65533) at the door, after the <=0 default. The guest maps <uid> to <uid>+100000 on the host, so 65534 is nobody and a host-side mapped value (e.g. 101000 = 1000+100000) must never be entered as the app uid. Out of range → a friendly Hungarian 400, nothing installed — the campaign's mapped_uid:101000 previously slipped past the controller and failed only at the agent with a raw agent_error.

Synology (DSM) — validated recipe (SPIKE-nas-dsm-2026-07-11, real DSM 7.2)

The consumer recipes were validated end-to-end against a real DSM 7.2 (virtual-dsm) through the live add pipeline. Exact steps for the customer/operator:

• Enable NFS first (off by default, and defaults to NFSv3 which our mount rejects): Control Panel → File Services → NFS → "Enable NFS service" → Maximum NFS protocol: NFSv4.1 → Apply.
• NFS rule: Control Panel → Shared Folder → → Edit → NFS Permissions → Create → the Felhom host IP, Read/Write, Squash: "Map all users to admin" → Save. The export path shown on that tab (/volume1/<share>) is what goes in the add form.
• SMB: Control Panel → User & Group → create a user with Read/Write on the share — nothing else (no force-user). Hardlinks work on DSM's SMB stack (the Q5 caveat is closed for Synology).
• Route A on DSM behaves exactly as on Debian: guest sees nobody, chown clean-EPERM, chmod persists. The SMB error taxonomy (smb_auth/smb_share) classifies identically to Debian.

QNAP remains a stated caveat — no emulator exists; not validated.

Reboot survival (Q1c) — automatic since agent v0.84.0, live-proven 2026-07-11

The Q1c failure (a fresh guest namespace inherits real mounts but NOT an idle autofs trigger → silent local stub after any guest reboot; RCA documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md) is FIXED by the pair agent v0.84.0 + controller v0.117.0:

• Agent ReassertNetworkMounts (storage ReassertNetworkAutomounts): per share — real mount → skip (inherited); idle autofs trigger → systemctl stop + enable --now the .automount (existing sudoers verbs; the fresh trigger-mount event propagates into every running guest's slave bind), then best-effort per-guest GuestSeesMount verify. Runs (a) from the guest-hook post-start phase (root, direct systemctl, non-fatal by contract) and (b) once at agent startup (guests that autostarted before the agent). Never from periodic health paths — an idle trigger is healthy.
• Controller consuming-namespace verification (the detection net beneath it): statfs f_type classifier; --netprobe refuses a non-network dir (not_network_fs, full rollback); deploys onto a stubbed network HDD_PATH are refused (409); a distinct dashboard stub badge ("Hálózati tárhely hibás — az alkalmazás nem a NAS-t látja") wins over the recoverable unreachable badge.

Live proof (2026-07-11 21:20, demo pct reboot 9201): the start-task transcript itself carried felhom-agent guest-hook: vmid 9201 post-start — network share nas-media visible in guest (rearmed); post-boot with NO manual action the guest had trigger + nfs4, calibre-web auto-started healthy on the NAS library, uid-1000 write-through OK, health status=ok on the first cycle, no storage badge. (The backlog spec documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md is implemented by this.)

Boot ordering — the F12 fix (agent v0.85.0, CAMPAIGN-3)

The bug (CRITICAL): the .mount and .automount templates ordered After=/Wants=network-online.target. An automount is implicitly Before=local-fs.target, and networking.service runs after local-fs, so the units closed the cycle networking → local-fs → automount → network-online → networking. systemd breaks such a cycle by deleting an arbitrary job in it — on one host boot it deleted networking.service (host up, no network, for 7 h), on the next it deleted the automount (network up, NAS dead). Every boot of a host with an enrolled network share was a coin flip.

The fix: neither rendered unit carries any network-online ordering. The .mount keeps _netdev in Options — that is the correct + sufficient network ordering for the REAL mount (systemd classes a _netdev mount under remote-fs.target and orders it after the network). The .automount gets NO network relation at all — a trigger needs none, and it must stay orderable before local-fs without dragging the network into the transaction.

Installed-unit migration: SudoHostOps.MigrateNetworkUnits is a general template-drift reconcile (SHA-256 content compare of each marker-owned unit against a fresh render of its reconstructed spec; rewrite + one batched daemon-reload; idempotent). It runs at agent startup (before the reassert sweep) and at the head of EnsureNetworkMount, so pre-0.85 units carrying the cycle are repaired, not just future adds. Live-proven 2026-07-12: felhom-pve rebooted ×2 after the 0.85 deploy — journalctl -b | grep -c "ordering cycle" == 0 on both boots, networking up, automount armed, guest + apps healthy. The coin flip is dead.

Reassert hardening (same release, matrix-corrected)

• F11 — read the right unit: the re-arm decision is driven ONLY by the host /proc/mounts fstype at the mountpoint; the .automount unit's own state is never consulted (an armed trigger always reports "active"). Matrix-corrected live finding (2026-07-12): a freshly-rebooted guest inherits neither an idle trigger NOR an active host mount — only a FRESH trigger event propagates in. So skip-active is proactively safe (don't churn shares guests already see) but the caller re-arms reactively when a specific running guest is found blind (RearmNetworkAutomount after a failed GuestSeesMount/GuestSeesPath).
• F10 — re-arm for real: a .mount/.automount left failed/start-limit-hit (unexport → idle-timeout → repeated access) is reset-failed FIRST (new sudoers verb) — without it the enable --now is refused by the start limit and the share stays dead across every boot.
• F9 — say what you did: the sweep enumerates by marker-owned unit files and logs an INFO verdict per share (reasserted / reset-failed+rearmed / skip-active / skip-foreign / error).
• The hook can never take a guest down: every guest-hook phase runs recover-wrapped under a hard timeout and returns cleanly; the installed wrapper snippet no longer execs (… || true; exit 0).

Known limitation — F13 (active NFS under the mp8 bind at guest start)

An actively-mounted nfs4 at /mnt/felhom-drives/<name> at the moment a guest starts can make PVE's rbind of mp8 fail with rc255 (an unprivileged-container idmap incompatibility), independent of the felhom hook (which returns 0). The realistic steady state is IDLE (the share idle-unmounts after TimeoutIdleSec=60), where the guest starts clean and the post-start reassert propagates it — but a share in active use at the reboot moment is a hazard. Deferred to a follow-up (candidate fixes: a pre-start conditional idle-unmount for single-guest hosts, or an idmapped nfs mount). Related: apps whose HDD_PATH is on a NAS that is down at guest boot land in docker Created state (the deferred fix-3, boot-time app-start-failure alerting).

Node self-heal (appliance) — F12-class defense in depth (agent v0.85.0)

internal/selfheal is a minimal check/remedy registry gated on deployment_mode. One heal ships: host networking recovery. Healthy ⇔ networking.service active AND a default route exists; the remedy — systemctl start networking.service (new FELHOM_SELFHEAL sudoers verb, ≤3 attempts, 10/30/60 s backoff, terminal give-up logged) — runs ONLY on deployment_mode:"appliance". A byo host runs the check + WARNs but the remedy is structurally unreachable (the Manager gates before any exec; unit-tested: byo + unhealthy → zero privileged invocations). Absent/unknown mode → byo (fail-safe). The F12 template fix is the CURE for the ordering instance; this watchdog is the belt for the CLASS (any boot that leaves networking down, for any reason). Live-proven 2026-07-12: systemctl stop networking.service on the appliance → the watchdog WARNed (networking.service active=false), fired the privileged start, and recovered on attempt 1. deployment_mode is set by host-install --mode appliance|byo (default byo; the byo config assert refuses deployment_mode=appliance).

Open items

• QNAP appliance fidelity pass (no emulator) before GA — Synology is now validated.
• The demo's nas-media share predates retry=0; re-adding re-creates the unit with the current option string.