felhom.eu/documentation/controller/network-storage-nas.md

# Network storage (NAS) — bulk-media shares with verify-before-commit

**Status:** authoritative feature doc, code-verified against felhom-agent v0.81.0 +
felhom-controller v0.113.0 (2026-07-11).
**Evidence base:** `documentation/audits/SPIKE-nas-storage-2026-06-29.md` (the locked mount recipe)
and `documentation/audits/SPIKE-nas-verify-2026-07-11.md` (the verify mechanisms, squash matrix,
error taxonomy — commit b57f6c1). Code: agent `internal/storage/netmount.go` /
`internal/storage/netverify.go` / `internal/localapi/netverifyjob.go`; controller
`internal/web/netstorage_job.go` / `internal/web/netprobe*.go`.

## What it is

A customer's NAS serves **bulk media** (film, photo, music) to media apps. The share is mounted
**host-side** by the agent under `/mnt/felhom-drives/<name>` via a systemd `.automount` + `.mount`
pair (on-demand + idle-unmount), propagates into the guest through the existing shared `mp8` bind,
and is selectable as a media app's data path. A NAS is a **distinct storage class from a drive**:
no durable-id, no SMART, no enroll/eject/decommission/wipe/migrate — remove is the only lifecycle
action (`refuseNetworkLifecycle` blocks the drive verbs).

## The locked mount recipe

- **NFS (nfs4):** `vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev,retry=0`. `soft` = failure
  isolation (clean EIO, never a wedge). **`retry=0`** (SPIKE-nas-verify Q4-vi): without it a
  dead-NAS on-demand access wedges the accessing app until systemd's 90 s start cap (measured
  91 s); with it the access fails clean in ~3.8 s (ENODEV) and each autofs re-access is a fresh
  attempt. `retry` only governs retrying a FAILED first attempt — the happy path is untouched.
  The uid mapping is the EXPORT's job (squash), never the client mount.
- **SMB (cifs):** `vers=3.0,credentials=<0600 file>,uid=<uid+100000>,gid=<gid+100000>,forceuid,
  forcegid,file_mode=0664,dir_mode=0775,_netdev`. No `retry=` (a mount.nfs option; mount.cifs
  rejects it). Credentials live in an agent-written 0600 file, never in the registry, never
  controller-persisted.
- **The +100000 rule:** container uid/gid N = host N+100000 (unprivileged-LXC idmap).

## Verify-before-commit (the add pipeline)

`POST /api/storage/netstorage/add` never registers blind. The controller orchestrates
(detached job, single-flight, ~150 s budget, status on `GET /api/storage/netstorage/add/status`):

1. **agent add** (`POST /netstorage/add`): role-gate → **sync fast-fail** (spec validation + 2 s
   TCP pre-probe; an unreachable server is refused with NOTHING installed) → stage SMB creds →
   install unit pair + enable the automount → start the agent's detached verify job.
2. **agent verify** (polled on `GET /netstorage/verify-status` every 2 s): a directory read
   through the automount triggers a REAL mount (spike Q1: an in-guest or host-side access wakes
   it, ~1 s LAN); success is judged from **/proc/mounts only** (truth table below); a failure is
   classified from the mount unit's journal and **auto-rolled-back** agent-side (units + creds).
3. **in-guest uid-1000 write probe:** the controller re-execs itself
   (`felhom-controller --netprobe <dir>`) with `SysProcAttr.Credential{Uid:1000,Gid:1000}` —
   create a `.felhom-proba-<random>` dot-file, write a nonce, read back, compare, delete. This
   catches the **squash trap**: an export that mounts fine but denies every uid-1000 write
   (spike Q3 row c). A failed delete after a good write is a WARN, not a failure.
4. **register** (`AddStoragePath`, Kind=network, Schedulable) — **the LAST step**. The worst
   crash outcome is an agent-side orphan (surfaced in the list as a remove-only "Árva megosztás"
   row), never a registered-but-broken path.

Any failure = **full rollback**: nothing registered, no units left installed, no creds file.
If the agent restarts mid-verify, its in-memory verify slot reports phase `none`; the controller
treats that as verify-lost and rolls back (the orphan row is the belt-and-braces surface).

### Mount-success truth table (SPIKE-nas-verify §8)

| Trigger `ReadDir(where)` | /proc/mounts shows nfs4/cifs at where | Verdict |
|---|---|---|
| ok | yes | mount OK → the uid-1000 probe decides writability |
| EACCES / EPERM | yes | mount OK (the agent user just can't read it — a 0700 export is fine) |
| any result | no | mount FAILED → classify from the journal |
| still blocked at 95 s | — | `timeout` (black-holed-but-routed; systemd's 90 s cap resolves it) |

Readability is NEVER the truth source — /proc/mounts is.

### Error taxonomy (live-measured; classification is string-based BY DESIGN — every failure is rc=32)

| Category | Journal substring (verbatim) | Meaning |
|---|---|---|
| `unreachable` | `No route to host` / `Connection refused` / `Connection timed out` (or the 2 s sync pre-probe) | no NAS behind the address |
| `nfs_export` | `reason given by server: No such file or directory` | export missing **OR not permitted — MERGED**: NFSv4 returns the identical string for both (Q4 ii≡iii); the customer message names both possibilities |
| `smb_auth` | `mount error(13)` | wrong SMB username/password |
| `smb_share` | `mount error(2)` | SMB share name not found |
| `timeout` | `Mounting timed out. Terminating` | routed but not answering (systemd 90 s cap) |
| `mount_failed` | (none matched / journal unavailable) | generic; detail carries the raw journal |
| `not_writable` | (probe exit 2) | mounts, but uid-1000 cannot write — the squash trap |
| `probe_io` | (probe exit 3) | write readback failed/differed |

The journal is read **unprivileged** (`journalctl -u <unit> -n 20 -o cat`): the agent user is in
the `systemd-journal` group (host-install ≥ v1.13.0 adds it; existing hosts:
`usermod -aG systemd-journal felhom-agent && systemctl restart felhom-agent`). **No sudoers
grant** — journal access is group-based by rule. Journal unavailable degrades to `mount_failed`
with a hint; the rollback still runs.

## NAS-side recipes (protocol-honest guidance — what the UI tells the customer)

- **SMB (Synology, QNAP — the consumer default, listed FIRST in the UI):** a plain user account
  with rw on the share is sufficient — **zero server-side uid configuration** (spike Q5). Files
  land on the NAS owned by the connecting account; the client mount forces the guest view to
  uid/gid 1000. Hardlinks worked on Debian Samba/ext4 — re-verify per appliance.
- **NFS, simple recipe (Route A — spike Q3 row b, ACCEPTED):** enable "map all users / all
  squash" on the export, rw, to ANY local user. The guest-uid-1000 app gets full
  read/write/rename/**hardlink**/delete. Caveats:
  - guest-visible ownership is `nobody:nogroup` (65534) — cosmetic for apps that just read/write;
  - `chown` fails with an immediate clean EPERM (no hang);
  - **`chmod` SUCCEEDS and persists server-side** (the squashed identity owns every file) — modes
    are app-controlled; operators should know a mode change on the NAS side is real.
- **NFS, full-fidelity recipe (TrueNAS / Linux server):**
  `rw,all_squash,anonuid=<uid+100000>,anongid=<uid+100000>` (e.g. 101000 for a uid-1000 app) —
  ownership information is exact end-to-end.
- **The pinned WRONG case (what `not_writable` usually means):** `anonuid=<x>` **without**
  `all_squash` is a no-op for non-root traffic — the app's wire uid (101000) hits the export
  unmapped and fails on others-perms. The consumer recipe REQUIRES the map-ALL-users mode; an
  anonymous-uid field alone does nothing. The UI's `not_writable` message and the guidance block
  tell this same story.
- **Naming caveat (R-66, controller v0.159.0): Windows network names („FELHOM") generally do NOT
  resolve here.** The Szerver field takes an IP or a DNS name; NetBIOS/WSD flat-name resolution is
  Windows machinery the guest deliberately does not run (the R-6 LAN-discovery spike: the docker
  bridge is deaf, and nothing NetBIOS-resolves on the mount path). The add form's helper text says
  so under the field, and an `unreachable`-class failure for a single-label non-IP server appends a
  purely lexical hint („Tipp: a(z) »FELHOM« Windows-hálózati névnek tűnik — használja az eszköz
  IP-címét") — no NetBIOS/mDNS resolution is ever attempted. Where the customer FINDS the peer
  Felhom box's address: the serving box's **Megosztás** page („közvetlen cím" row) or its
  **Beállítások → Rendszer → „Hálózat"** card (Helyi cím) — both live-computed per render, never
  stored (S-5), so they are current by construction.

## Browsing (FileBrowser) — R-67, controller v0.160.0

A registered share is browsable in FileBrowser: the sync binds the share ROOT
(`/mnt/felhom-drives/<name>:/srv/<name>:rslave`) and adds a sidebar source named by the share's
display label. The `:rslave` propagation is load-bearing — host-side automount wake and
idle-unmount events reach the running container, and an in-container access through the bind
WAKES an idle trigger (Phase-0-probed live on demo-hp, 2026-07-22). Differences from drives, both
deliberate: there is **no userdata scoping and no skeleton** (Felhom convention dirs are never
written onto a customer's own NAS; the sync is read-only toward the share), and the gate is not
the drive-absent check (idle is healthy) but the **stub classifier**: when this namespace sees a
local stub dir instead of the NAS, the share is **excluded from FileBrowser until propagation
recovers** — an exposed stub would accept uploads that the real mount later shadows, silently
hiding the customer's files. A dead NAS surfaces as a fast clean error in FileBrowser (the
soft/retry=0 mount recipe), and recovery after the NAS returns needs no intervention.

## Health model

Per-share liveness (`ok | idle | unreachable | stub`): `idle` (automount idle-unmounted) is the
benign steady state; `unreachable` degrades the affected share's apps ONLY — never box health,
never the drive missing→stop cascade (kind-gated). Timing/budget note for integrators: the verify
worst case is systemd's 90 s (black-holed server) — verify traffic therefore rides the detached
job + status poll, never a single long HTTP call (the agentapi client keeps its global 15 s
timeout).

### One classification, two surfaces (F8, controller v0.119.0, CAMPAIGN-3)

The agent's per-share `health` derives from a **server-level TCP dial** (`server:2049/445`). That dial
stays GREEN when a *single* export is `exportfs -u`'d — the server still answers on 2049 for its other
exports — so the agent reports benign `idle`+`reachable:true` for a share whose consuming namespace is
actually a stub. The campaign found the resulting **contradictory UI**: the share row showed
"Készenlét" while the stacks/dashboard cards showed the stub badge (which reads the consuming-namespace
classifier shipped in v0.117.0).

The fix (controller-only — the §3 fork's recommended option B): the share row (`networkStorageItems` →
`fuseNetHealth`) reads the SAME `system.ClassifyPathFS` the stacks stub badge reads and fuses it over
the agent view. Precedence: a whole-server `unreachable` (agent dial failed) is the most actionable and
WINS; otherwise a **`stub`** classification at `Where` (namespace sees local disk, not the NAS)
overrides a benign idle/ok — the row badge becomes "Hibás — az alkalmazások nem a NAS-t látják"; an
autofs-healthy trigger, a real network fs, or an inconclusive `unknown` read leave the agent-derived
health untouched (**never force-mount an idle trigger** — that is healthy by June's idle-unmount
design). The share row and the stacks badge now derive from ONE classification and can never
contradict. **Live-proven 2026-07-12:** an `exportfs -u` while idle flipped the row to `stub` in lockstep
with the stacks badge; re-export cleared both to `idle`/`ok`, with `reachable:true` throughout (the
agent's server-level view unchanged — the fusion is what carries the export-level truth).

### Input validation — mapped_uid range (F4, v0.119.0)

The add handler range-checks the container uid/gid (**1..65533**) at the door, after the `<=0` default.
The guest maps `<uid>` to `<uid>+100000` on the host, so 65534 is `nobody` and a host-side mapped value
(e.g. 101000 = 1000+100000) must never be entered as the app uid. Out of range → a friendly Hungarian
400, nothing installed — the campaign's `mapped_uid:101000` previously slipped past the controller and
failed only at the agent with a raw `agent_error`.

## Synology (DSM) — validated recipe (SPIKE-nas-dsm-2026-07-11, real DSM 7.2)

The consumer recipes were validated end-to-end against a real DSM 7.2 (virtual-dsm) through the live
add pipeline. Exact steps for the customer/operator:
- **Enable NFS first (off by default, and defaults to NFSv3 which our mount rejects):** Control
  Panel → File Services → NFS → "Enable NFS service" → **Maximum NFS protocol: NFSv4.1** → Apply.
- **NFS rule:** Control Panel → Shared Folder → <share> → Edit → NFS Permissions → Create → the
  Felhom host IP, Read/Write, **Squash: "Map all users to admin"** → Save. The export path shown on
  that tab (`/volume1/<share>`) is what goes in the add form.
- **SMB:** Control Panel → User & Group → create a user with Read/Write on the share — nothing else
  (no force-user). **Hardlinks work on DSM's SMB stack** (the Q5 caveat is closed for Synology).
- Route A on DSM behaves exactly as on Debian: guest sees `nobody`, chown clean-EPERM, chmod
  persists. The SMB error taxonomy (`smb_auth`/`smb_share`) classifies identically to Debian.

**QNAP remains a stated caveat** — no emulator exists; not validated.

## Reboot survival (Q1c) — automatic since agent v0.84.0, live-proven 2026-07-11

The Q1c failure (a fresh guest namespace inherits real mounts but NOT an idle autofs trigger →
silent local stub after any guest reboot; RCA `documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md`)
is FIXED by the pair **agent v0.84.0 + controller v0.117.0**:

- **Agent `ReassertNetworkMounts`** (storage `ReassertNetworkAutomounts`): per share — real mount →
  skip (inherited); idle autofs trigger → `systemctl stop` + `enable --now` the `.automount`
  (existing sudoers verbs; the fresh trigger-mount event propagates into every running guest's slave
  bind), then best-effort per-guest `GuestSeesMount` verify. Runs (a) from the **guest-hook
  post-start phase** (root, direct systemctl, non-fatal by contract) and (b) once at **agent
  startup** (guests that autostarted before the agent). Never from periodic health paths — an idle
  trigger is healthy.
- **Controller consuming-namespace verification** (the detection net beneath it): statfs f_type
  classifier; `--netprobe` refuses a non-network dir (`not_network_fs`, full rollback); deploys onto
  a stubbed network `HDD_PATH` are refused (409); a distinct dashboard **stub badge** ("Hálózati
  tárhely hibás — az alkalmazás nem a NAS-t látja") wins over the recoverable unreachable badge.

**Live proof (2026-07-11 21:20, demo `pct reboot 9201`):** the start-task transcript itself carried
`felhom-agent guest-hook: vmid 9201 post-start — network share nas-media visible in guest (rearmed)`;
post-boot with NO manual action the guest had trigger + nfs4, calibre-web auto-started healthy on
the NAS library, uid-1000 write-through OK, health `status=ok` on the first cycle, no storage badge.
(The backlog spec `documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md` is
implemented by this.)

## Boot ordering — the F12 fix (agent v0.85.0, CAMPAIGN-3)

**The bug (CRITICAL):** the `.mount` and `.automount` templates ordered `After=`/`Wants=network-online.target`.
An automount is implicitly `Before=local-fs.target`, and `networking.service` runs after `local-fs`,
so the units closed the cycle `networking → local-fs → automount → network-online → networking`.
systemd breaks such a cycle by deleting an **arbitrary** job in it — on one host boot it deleted
`networking.service` (host up, **no network, for 7 h**), on the next it deleted the automount (network
up, NAS dead). **Every boot of a host with an enrolled network share was a coin flip.**

**The fix:** neither rendered unit carries any network-online ordering. The `.mount` keeps `_netdev`
in `Options` — that is the correct + sufficient network ordering for the REAL mount (systemd classes
a `_netdev` mount under remote-fs.target and orders it after the network). The `.automount` gets NO
network relation at all — a trigger needs none, and it must stay orderable before local-fs without
dragging the network into the transaction.

**Installed-unit migration:** `SudoHostOps.MigrateNetworkUnits` is a general template-drift reconcile
(SHA-256 content compare of each marker-owned unit against a fresh render of its reconstructed spec;
rewrite + one batched `daemon-reload`; idempotent). It runs at agent startup (before the reassert
sweep) and at the head of `EnsureNetworkMount`, so pre-0.85 units carrying the cycle are repaired,
not just future adds. **Live-proven 2026-07-12:** `felhom-pve` rebooted ×2 after the 0.85 deploy —
`journalctl -b | grep -c "ordering cycle"` == **0 on both boots**, networking up, automount armed,
guest + apps healthy. The coin flip is dead.

### Reassert hardening (same release, matrix-corrected)

- **F11 — read the right unit:** the re-arm decision is driven ONLY by the host `/proc/mounts` fstype
  at the mountpoint; the `.automount` unit's own state is never consulted (an armed trigger always
  reports "active"). **Matrix-corrected live finding (2026-07-12):** a freshly-rebooted guest inherits
  neither an idle trigger NOR an *active* host mount — only a FRESH trigger event propagates in. So
  skip-active is proactively safe (don't churn shares guests already see) but the caller re-arms
  *reactively* when a specific running guest is found blind (`RearmNetworkAutomount` after a failed
  `GuestSeesMount`/`GuestSeesPath`).
- **F10 — re-arm for real:** a `.mount`/`.automount` left `failed`/start-limit-hit (unexport →
  idle-timeout → repeated access) is `reset-failed` FIRST (new sudoers verb) — without it the
  `enable --now` is refused by the start limit and the share stays dead across every boot.
- **F9 — say what you did:** the sweep enumerates by marker-owned unit files and logs an INFO verdict
  per share (reasserted / reset-failed+rearmed / skip-active / skip-foreign / error).
- **The hook can never take a guest down:** every guest-hook phase runs recover-wrapped under a hard
  timeout and returns cleanly; the installed wrapper snippet no longer `exec`s (`… || true; exit 0`).

### Known limitation — F13 (active NFS under the mp8 bind at guest start)

An **actively-mounted** nfs4 at `/mnt/felhom-drives/<name>` at the moment a guest starts can make
PVE's rbind of `mp8` fail with `rc255` (an unprivileged-container idmap incompatibility), independent
of the felhom hook (which returns 0). The realistic steady state is IDLE (the share idle-unmounts
after `TimeoutIdleSec=60`), where the guest starts clean and the post-start reassert propagates it —
but a share in active use at the reboot moment is a hazard. Deferred to a follow-up (candidate fixes:
a pre-start conditional idle-unmount for single-guest hosts, or an idmapped nfs mount). Related: apps
whose HDD_PATH is on a NAS that is down at guest boot land in docker `Created` state (the deferred
fix-3, boot-time app-start-failure alerting).

## Node self-heal (appliance) — F12-class defense in depth (agent v0.85.0)

`internal/selfheal` is a minimal check/remedy registry gated on `deployment_mode`. One heal ships:
**host networking recovery.** Healthy ⇔ `networking.service` active AND a default route exists; the
remedy — `systemctl start networking.service` (new `FELHOM_SELFHEAL` sudoers verb, ≤3 attempts,
10/30/60 s backoff, terminal give-up logged) — runs ONLY on `deployment_mode:"appliance"`. A **byo**
host runs the check + WARNs but the remedy is structurally unreachable (the Manager gates before any
exec; unit-tested: byo + unhealthy → zero privileged invocations). Absent/unknown mode → byo
(fail-safe). The F12 template fix is the CURE for the ordering instance; this watchdog is the belt for
the CLASS (any boot that leaves networking down, for any reason). **Live-proven 2026-07-12:**
`systemctl stop networking.service` on the appliance → the watchdog WARNed
(`networking.service active=false`), fired the privileged start, and recovered on attempt 1.
`deployment_mode` is set by host-install `--mode appliance|byo` (default byo; the byo config assert
refuses `deployment_mode=appliance`).

## Open items
- QNAP appliance fidelity pass (no emulator) before GA — Synology is now validated.
- The demo's `nas-media` share predates `retry=0`; re-adding re-creates the unit with the current
  option string.