# Network storage (NAS) — bulk-media shares with verify-before-commit **Status:** authoritative feature doc, code-verified against felhom-agent v0.81.0 + felhom-controller v0.113.0 (2026-07-11). **Evidence base:** `documentation/audits/SPIKE-nas-storage-2026-06-29.md` (the locked mount recipe) and `documentation/audits/SPIKE-nas-verify-2026-07-11.md` (the verify mechanisms, squash matrix, error taxonomy — commit b57f6c1). Code: agent `internal/storage/netmount.go` / `internal/storage/netverify.go` / `internal/localapi/netverifyjob.go`; controller `internal/web/netstorage_job.go` / `internal/web/netprobe*.go`. ## What it is A customer's NAS serves **bulk media** (film, photo, music) to media apps. The share is mounted **host-side** by the agent under `/mnt/felhom-drives/` via a systemd `.automount` + `.mount` pair (on-demand + idle-unmount), propagates into the guest through the existing shared `mp8` bind, and is selectable as a media app's data path. A NAS is a **distinct storage class from a drive**: no durable-id, no SMART, no enroll/eject/decommission/wipe/migrate — remove is the only lifecycle action (`refuseNetworkLifecycle` blocks the drive verbs). ## The locked mount recipe - **NFS (nfs4):** `vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev,retry=0`. `soft` = failure isolation (clean EIO, never a wedge). **`retry=0`** (SPIKE-nas-verify Q4-vi): without it a dead-NAS on-demand access wedges the accessing app until systemd's 90 s start cap (measured 91 s); with it the access fails clean in ~3.8 s (ENODEV) and each autofs re-access is a fresh attempt. `retry` only governs retrying a FAILED first attempt — the happy path is untouched. The uid mapping is the EXPORT's job (squash), never the client mount. - **SMB (cifs):** `vers=3.0,credentials=<0600 file>,uid=,gid=,forceuid, forcegid,file_mode=0664,dir_mode=0775,_netdev`. No `retry=` (a mount.nfs option; mount.cifs rejects it). Credentials live in an agent-written 0600 file, never in the registry, never controller-persisted. - **The +100000 rule:** container uid/gid N = host N+100000 (unprivileged-LXC idmap). ## Verify-before-commit (the add pipeline) `POST /api/storage/netstorage/add` never registers blind. The controller orchestrates (detached job, single-flight, ~150 s budget, status on `GET /api/storage/netstorage/add/status`): 1. **agent add** (`POST /netstorage/add`): role-gate → **sync fast-fail** (spec validation + 2 s TCP pre-probe; an unreachable server is refused with NOTHING installed) → stage SMB creds → install unit pair + enable the automount → start the agent's detached verify job. 2. **agent verify** (polled on `GET /netstorage/verify-status` every 2 s): a directory read through the automount triggers a REAL mount (spike Q1: an in-guest or host-side access wakes it, ~1 s LAN); success is judged from **/proc/mounts only** (truth table below); a failure is classified from the mount unit's journal and **auto-rolled-back** agent-side (units + creds). 3. **in-guest uid-1000 write probe:** the controller re-execs itself (`felhom-controller --netprobe `) with `SysProcAttr.Credential{Uid:1000,Gid:1000}` — create a `.felhom-proba-` dot-file, write a nonce, read back, compare, delete. This catches the **squash trap**: an export that mounts fine but denies every uid-1000 write (spike Q3 row c). A failed delete after a good write is a WARN, not a failure. 4. **register** (`AddStoragePath`, Kind=network, Schedulable) — **the LAST step**. The worst crash outcome is an agent-side orphan (surfaced in the list as a remove-only "Árva megosztás" row), never a registered-but-broken path. Any failure = **full rollback**: nothing registered, no units left installed, no creds file. If the agent restarts mid-verify, its in-memory verify slot reports phase `none`; the controller treats that as verify-lost and rolls back (the orphan row is the belt-and-braces surface). ### Mount-success truth table (SPIKE-nas-verify §8) | Trigger `ReadDir(where)` | /proc/mounts shows nfs4/cifs at where | Verdict | |---|---|---| | ok | yes | mount OK → the uid-1000 probe decides writability | | EACCES / EPERM | yes | mount OK (the agent user just can't read it — a 0700 export is fine) | | any result | no | mount FAILED → classify from the journal | | still blocked at 95 s | — | `timeout` (black-holed-but-routed; systemd's 90 s cap resolves it) | Readability is NEVER the truth source — /proc/mounts is. ### Error taxonomy (live-measured; classification is string-based BY DESIGN — every failure is rc=32) | Category | Journal substring (verbatim) | Meaning | |---|---|---| | `unreachable` | `No route to host` / `Connection refused` / `Connection timed out` (or the 2 s sync pre-probe) | no NAS behind the address | | `nfs_export` | `reason given by server: No such file or directory` | export missing **OR not permitted — MERGED**: NFSv4 returns the identical string for both (Q4 ii≡iii); the customer message names both possibilities | | `smb_auth` | `mount error(13)` | wrong SMB username/password | | `smb_share` | `mount error(2)` | SMB share name not found | | `timeout` | `Mounting timed out. Terminating` | routed but not answering (systemd 90 s cap) | | `mount_failed` | (none matched / journal unavailable) | generic; detail carries the raw journal | | `not_writable` | (probe exit 2) | mounts, but uid-1000 cannot write — the squash trap | | `probe_io` | (probe exit 3) | write readback failed/differed | The journal is read **unprivileged** (`journalctl -u -n 20 -o cat`): the agent user is in the `systemd-journal` group (host-install ≥ v1.13.0 adds it; existing hosts: `usermod -aG systemd-journal felhom-agent && systemctl restart felhom-agent`). **No sudoers grant** — journal access is group-based by rule. Journal unavailable degrades to `mount_failed` with a hint; the rollback still runs. ## NAS-side recipes (protocol-honest guidance — what the UI tells the customer) - **SMB (Synology, QNAP — the consumer default, listed FIRST in the UI):** a plain user account with rw on the share is sufficient — **zero server-side uid configuration** (spike Q5). Files land on the NAS owned by the connecting account; the client mount forces the guest view to uid/gid 1000. Hardlinks worked on Debian Samba/ext4 — re-verify per appliance. - **NFS, simple recipe (Route A — spike Q3 row b, ACCEPTED):** enable "map all users / all squash" on the export, rw, to ANY local user. The guest-uid-1000 app gets full read/write/rename/**hardlink**/delete. Caveats: - guest-visible ownership is `nobody:nogroup` (65534) — cosmetic for apps that just read/write; - `chown` fails with an immediate clean EPERM (no hang); - **`chmod` SUCCEEDS and persists server-side** (the squashed identity owns every file) — modes are app-controlled; operators should know a mode change on the NAS side is real. - **NFS, full-fidelity recipe (TrueNAS / Linux server):** `rw,all_squash,anonuid=,anongid=` (e.g. 101000 for a uid-1000 app) — ownership information is exact end-to-end. - **The pinned WRONG case (what `not_writable` usually means):** `anonuid=` **without** `all_squash` is a no-op for non-root traffic — the app's wire uid (101000) hits the export unmapped and fails on others-perms. The consumer recipe REQUIRES the map-ALL-users mode; an anonymous-uid field alone does nothing. The UI's `not_writable` message and the guidance block tell this same story. - **Naming caveat (R-66, controller v0.159.0): Windows network names („FELHOM") generally do NOT resolve here.** The Szerver field takes an IP or a DNS name; NetBIOS/WSD flat-name resolution is Windows machinery the guest deliberately does not run (the R-6 LAN-discovery spike: the docker bridge is deaf, and nothing NetBIOS-resolves on the mount path). The add form's helper text says so under the field, and an `unreachable`-class failure for a single-label non-IP server appends a purely lexical hint („Tipp: a(z) »FELHOM« Windows-hálózati névnek tűnik — használja az eszköz IP-címét") — no NetBIOS/mDNS resolution is ever attempted. Where the customer FINDS the peer Felhom box's address: the serving box's **Megosztás** page („közvetlen cím" row) or its **Beállítások → Rendszer → „Hálózat"** card (Helyi cím) — both live-computed per render, never stored (S-5), so they are current by construction. ## Browsing (FileBrowser) — R-67, controller v0.160.0 A registered share is browsable in FileBrowser: the sync binds the share ROOT (`/mnt/felhom-drives/:/srv/:rslave`) and adds a sidebar source named by the share's display label. The `:rslave` propagation is load-bearing — host-side automount wake and idle-unmount events reach the running container, and an in-container access through the bind WAKES an idle trigger (Phase-0-probed live on demo-hp, 2026-07-22). Differences from drives, both deliberate: there is **no userdata scoping and no skeleton** (Felhom convention dirs are never written onto a customer's own NAS; the sync is read-only toward the share), and the gate is not the drive-absent check (idle is healthy) but the **stub classifier**: when this namespace sees a local stub dir instead of the NAS, the share is **excluded from FileBrowser until propagation recovers** — an exposed stub would accept uploads that the real mount later shadows, silently hiding the customer's files. A dead NAS surfaces as a fast clean error in FileBrowser (the soft/retry=0 mount recipe), and recovery after the NAS returns needs no intervention. ## Health model Per-share liveness (`ok | idle | unreachable | stub`): `idle` (automount idle-unmounted) is the benign steady state; `unreachable` degrades the affected share's apps ONLY — never box health, never the drive missing→stop cascade (kind-gated). Timing/budget note for integrators: the verify worst case is systemd's 90 s (black-holed server) — verify traffic therefore rides the detached job + status poll, never a single long HTTP call (the agentapi client keeps its global 15 s timeout). ### One classification, two surfaces (F8, controller v0.119.0, CAMPAIGN-3) The agent's per-share `health` derives from a **server-level TCP dial** (`server:2049/445`). That dial stays GREEN when a *single* export is `exportfs -u`'d — the server still answers on 2049 for its other exports — so the agent reports benign `idle`+`reachable:true` for a share whose consuming namespace is actually a stub. The campaign found the resulting **contradictory UI**: the share row showed "Készenlét" while the stacks/dashboard cards showed the stub badge (which reads the consuming-namespace classifier shipped in v0.117.0). The fix (controller-only — the §3 fork's recommended option B): the share row (`networkStorageItems` → `fuseNetHealth`) reads the SAME `system.ClassifyPathFS` the stacks stub badge reads and fuses it over the agent view. Precedence: a whole-server `unreachable` (agent dial failed) is the most actionable and WINS; otherwise a **`stub`** classification at `Where` (namespace sees local disk, not the NAS) overrides a benign idle/ok — the row badge becomes "Hibás — az alkalmazások nem a NAS-t látják"; an autofs-healthy trigger, a real network fs, or an inconclusive `unknown` read leave the agent-derived health untouched (**never force-mount an idle trigger** — that is healthy by June's idle-unmount design). The share row and the stacks badge now derive from ONE classification and can never contradict. **Live-proven 2026-07-12:** an `exportfs -u` while idle flipped the row to `stub` in lockstep with the stacks badge; re-export cleared both to `idle`/`ok`, with `reachable:true` throughout (the agent's server-level view unchanged — the fusion is what carries the export-level truth). ### Input validation — mapped_uid range (F4, v0.119.0) The add handler range-checks the container uid/gid (**1..65533**) at the door, after the `<=0` default. The guest maps `` to `+100000` on the host, so 65534 is `nobody` and a host-side mapped value (e.g. 101000 = 1000+100000) must never be entered as the app uid. Out of range → a friendly Hungarian 400, nothing installed — the campaign's `mapped_uid:101000` previously slipped past the controller and failed only at the agent with a raw `agent_error`. ## Synology (DSM) — validated recipe (SPIKE-nas-dsm-2026-07-11, real DSM 7.2) The consumer recipes were validated end-to-end against a real DSM 7.2 (virtual-dsm) through the live add pipeline. Exact steps for the customer/operator: - **Enable NFS first (off by default, and defaults to NFSv3 which our mount rejects):** Control Panel → File Services → NFS → "Enable NFS service" → **Maximum NFS protocol: NFSv4.1** → Apply. - **NFS rule:** Control Panel → Shared Folder → → Edit → NFS Permissions → Create → the Felhom host IP, Read/Write, **Squash: "Map all users to admin"** → Save. The export path shown on that tab (`/volume1/`) is what goes in the add form. - **SMB:** Control Panel → User & Group → create a user with Read/Write on the share — nothing else (no force-user). **Hardlinks work on DSM's SMB stack** (the Q5 caveat is closed for Synology). - Route A on DSM behaves exactly as on Debian: guest sees `nobody`, chown clean-EPERM, chmod persists. The SMB error taxonomy (`smb_auth`/`smb_share`) classifies identically to Debian. **QNAP remains a stated caveat** — no emulator exists; not validated. ## Reboot survival (Q1c) — automatic since agent v0.84.0, live-proven 2026-07-11 The Q1c failure (a fresh guest namespace inherits real mounts but NOT an idle autofs trigger → silent local stub after any guest reboot; RCA `documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md`) is FIXED by the pair **agent v0.84.0 + controller v0.117.0**: - **Agent `ReassertNetworkMounts`** (storage `ReassertNetworkAutomounts`): per share — real mount → skip (inherited); idle autofs trigger → `systemctl stop` + `enable --now` the `.automount` (existing sudoers verbs; the fresh trigger-mount event propagates into every running guest's slave bind), then best-effort per-guest `GuestSeesMount` verify. Runs (a) from the **guest-hook post-start phase** (root, direct systemctl, non-fatal by contract) and (b) once at **agent startup** (guests that autostarted before the agent). Never from periodic health paths — an idle trigger is healthy. - **Controller consuming-namespace verification** (the detection net beneath it): statfs f_type classifier; `--netprobe` refuses a non-network dir (`not_network_fs`, full rollback); deploys onto a stubbed network `HDD_PATH` are refused (409); a distinct dashboard **stub badge** ("Hálózati tárhely hibás — az alkalmazás nem a NAS-t látja") wins over the recoverable unreachable badge. **Live proof (2026-07-11 21:20, demo `pct reboot 9201`):** the start-task transcript itself carried `felhom-agent guest-hook: vmid 9201 post-start — network share nas-media visible in guest (rearmed)`; post-boot with NO manual action the guest had trigger + nfs4, calibre-web auto-started healthy on the NAS library, uid-1000 write-through OK, health `status=ok` on the first cycle, no storage badge. (The backlog spec `documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md` is implemented by this.) ## Boot ordering — the F12 fix (agent v0.85.0, CAMPAIGN-3) **The bug (CRITICAL):** the `.mount` and `.automount` templates ordered `After=`/`Wants=network-online.target`. An automount is implicitly `Before=local-fs.target`, and `networking.service` runs after `local-fs`, so the units closed the cycle `networking → local-fs → automount → network-online → networking`. systemd breaks such a cycle by deleting an **arbitrary** job in it — on one host boot it deleted `networking.service` (host up, **no network, for 7 h**), on the next it deleted the automount (network up, NAS dead). **Every boot of a host with an enrolled network share was a coin flip.** **The fix:** neither rendered unit carries any network-online ordering. The `.mount` keeps `_netdev` in `Options` — that is the correct + sufficient network ordering for the REAL mount (systemd classes a `_netdev` mount under remote-fs.target and orders it after the network). The `.automount` gets NO network relation at all — a trigger needs none, and it must stay orderable before local-fs without dragging the network into the transaction. **Installed-unit migration:** `SudoHostOps.MigrateNetworkUnits` is a general template-drift reconcile (SHA-256 content compare of each marker-owned unit against a fresh render of its reconstructed spec; rewrite + one batched `daemon-reload`; idempotent). It runs at agent startup (before the reassert sweep) and at the head of `EnsureNetworkMount`, so pre-0.85 units carrying the cycle are repaired, not just future adds. **Live-proven 2026-07-12:** `felhom-pve` rebooted ×2 after the 0.85 deploy — `journalctl -b | grep -c "ordering cycle"` == **0 on both boots**, networking up, automount armed, guest + apps healthy. The coin flip is dead. ### Reassert hardening (same release, matrix-corrected) - **F11 — read the right unit:** the re-arm decision is driven ONLY by the host `/proc/mounts` fstype at the mountpoint; the `.automount` unit's own state is never consulted (an armed trigger always reports "active"). **Matrix-corrected live finding (2026-07-12):** a freshly-rebooted guest inherits neither an idle trigger NOR an *active* host mount — only a FRESH trigger event propagates in. So skip-active is proactively safe (don't churn shares guests already see) but the caller re-arms *reactively* when a specific running guest is found blind (`RearmNetworkAutomount` after a failed `GuestSeesMount`/`GuestSeesPath`). - **F10 — re-arm for real:** a `.mount`/`.automount` left `failed`/start-limit-hit (unexport → idle-timeout → repeated access) is `reset-failed` FIRST (new sudoers verb) — without it the `enable --now` is refused by the start limit and the share stays dead across every boot. - **F9 — say what you did:** the sweep enumerates by marker-owned unit files and logs an INFO verdict per share (reasserted / reset-failed+rearmed / skip-active / skip-foreign / error). - **The hook can never take a guest down:** every guest-hook phase runs recover-wrapped under a hard timeout and returns cleanly; the installed wrapper snippet no longer `exec`s (`… || true; exit 0`). ### Known limitation — F13 (active NFS under the mp8 bind at guest start) An **actively-mounted** nfs4 at `/mnt/felhom-drives/` at the moment a guest starts can make PVE's rbind of `mp8` fail with `rc255` (an unprivileged-container idmap incompatibility), independent of the felhom hook (which returns 0). The realistic steady state is IDLE (the share idle-unmounts after `TimeoutIdleSec=60`), where the guest starts clean and the post-start reassert propagates it — but a share in active use at the reboot moment is a hazard. Deferred to a follow-up (candidate fixes: a pre-start conditional idle-unmount for single-guest hosts, or an idmapped nfs mount). Related: apps whose HDD_PATH is on a NAS that is down at guest boot land in docker `Created` state (the deferred fix-3, boot-time app-start-failure alerting). ## Node self-heal (appliance) — F12-class defense in depth (agent v0.85.0) `internal/selfheal` is a minimal check/remedy registry gated on `deployment_mode`. One heal ships: **host networking recovery.** Healthy ⇔ `networking.service` active AND a default route exists; the remedy — `systemctl start networking.service` (new `FELHOM_SELFHEAL` sudoers verb, ≤3 attempts, 10/30/60 s backoff, terminal give-up logged) — runs ONLY on `deployment_mode:"appliance"`. A **byo** host runs the check + WARNs but the remedy is structurally unreachable (the Manager gates before any exec; unit-tested: byo + unhealthy → zero privileged invocations). Absent/unknown mode → byo (fail-safe). The F12 template fix is the CURE for the ordering instance; this watchdog is the belt for the CLASS (any boot that leaves networking down, for any reason). **Live-proven 2026-07-12:** `systemctl stop networking.service` on the appliance → the watchdog WARNed (`networking.service active=false`), fired the privileged start, and recovered on attempt 1. `deployment_mode` is set by host-install `--mode appliance|byo` (default byo; the byo config assert refuses `deployment_mode=appliance`). ## Open items - QNAP appliance fidelity pass (no emulator) before GA — Synology is now validated. - The demo's `nas-media` share predates `retry=0`; re-adding re-creates the unit with the current option string.