felhom.eu/documentation/audits/SPIKE-nas-verify-2026-07-11.md

# SPIKE — NAS verify mechanisms + NFS squash matrix, simulated on DooPlex

**Date:** 2026-07-11
**Class:** Spike (empirical validation; output = this doc only). Throwaway probes; no production code
written or modified.
**Goal:** validate the four unproven mechanisms the upcoming "NAS verify-before-commit + UI redesign"
production TASK depends on, and pin down the June spike's `anonuid=1000` WRONG case. Q1 in-guest
automount trigger; Q2 controller uid-1000 probe; Q3 NFS squash matrix (Route A decision); Q4 verify
error taxonomy + timing; Q5 plain-user SMB.

**Simulated NAS:** real nfs-kernel-server + Samba on **DooPlex 192.168.0.180**, isolated scratch root
`/srv/nas-spike2/` (all exports/shares spike-named; live Longhorn exports + the six original Samba
sections untouched, verified by count at teardown). Mount side: **felhom-pve** (192.168.0.162), all
mounts under `/mnt/felhom-drives/spike2-*` via hand-written throwaway units carrying EXACTLY
`netmount.go`'s option strings. Probe side: live demo guest **9201**, probes as guest uid 1000 via
`docker exec -u 1000:1000 felhom-controller` (the production probe's exact vantage point).

> **Sim-vs-real-NAS caveat (restated from June):** this validates the Linux *mechanism* against a
> Debian NFS/Samba server. Synology/QNAP specifics — their `squash`/"Map all users" UIs, SMB dialect
> ceilings, per-share ACL models, whether their SMB stack permits hardlinks — remain unvalidated until
> the virtual-dsm confirmation pass.

**Baselines (read-only anchors, verified live 2026-07-11):**

| Repo | `main` @ commit | Version | Role |
|------|-----------------|---------|------|
| felhom-agent | `300f06722b` | v0.80.0 | `internal/storage/netmount.go` = the locked recipe under test |
| felhom-controller | `3db9126121` | v0.112.0 | the container the Q2/Q3/Q5 probes ran in |
| felhom.eu | `8ca1272db6` | — | this doc's home |

**This doc's commit:** `b57f6c1` (hash inserted by the follow-up commit).

---

## Verdict: **ALL FIVE QUESTIONS ANSWERED — no blocker for the production TASK**

- **Q1 (make-or-break) PASS** — an in-guest access (bare `pct exec ls` AND a docker-container
  bind-mount access) wakes the **host** systemd automount; content appears correctly, never an empty
  dir or ENOENT.
- **Q2 PASS** — the controller container (root, in-guest) can drop to uid/gid 1000 (`setpriv`, the
  same setresuid/setresgid syscalls `syscall.Credential` uses) and write through the share path;
  SETUID/SETGID capability bits confirmed present.
- **Q3 — Route A ACCEPTED-with-caveats** — `all_squash` to an arbitrary server-local uid gives the
  guest-uid-1000 app full read+write+rename+**hardlink**+delete. Caveats: guest-visible ownership is
  `nobody:nogroup` (65534), `chown` fails (immediate clean EPERM, no hang), `chmod` **succeeds** (and
  persists server-side). The June WRONG case is pinned: `anonuid` **without** `all_squash` is a no-op
  for non-root traffic — the wire uid 101000 hits the export unmapped and fails on others-perms.
- **Q4 taxonomy captured** — table below. Two headline facts: NFSv4 makes "export doesn't exist" and
  "export not permitted for this client" **indistinguishable** (identical error string), and the real
  ceiling on a dead-server mount is **systemd's 90 s start timeout**, not `mount.nfs`'s 2-min retry.
  `retry=0` turns the dead-server case into a **3.1 s clean "No route to host"** and an on-demand
  guest access from 91 s ENODEV into **3.8 s** — recommended for the PRODUCTION unit (evidence in
  Q4-vi).
- **Q5 PASS** — a plain Samba user + rw share (zero server-side `force user`/uid config) works through
  the client-forced-uid chain, **including hardlinks** (on this Debian/ext4 Samba).

---

## Q1 — In-guest automount trigger: **PASS (both variants)**

Setup: export `/srv/nas-spike2/q1` (`rw,all_squash,anonuid=101000,anongid=101000,no_subtree_check`,
dir `101000:101000`), throwaway `.mount`+`.automount` pair at `/mnt/felhom-drives/spike2-q1` with the
locked NFS option string (`vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev`), `TimeoutIdleSec=30`.
Idle state confirmed by grep of `/proc/mounts` only (autofs, no nfs4) — the path was never
stat'ed host-side.

**Q1a — first access from inside the running guest:**
```
# host /proc/mounts before: autofs trigger ONLY
$ time pct exec 9201 -- ls -la /mnt/felhom-drives/spike2-q1
-rw-r--r-- 1   1000    1000      30 ... MARKER.txt        ← marker listed, owned 1000:1000
-rw-r--r-- 1   1000    1000 5242880 ... sample.bin
real  0m1.053s   rc=0
# host /proc/mounts after:  192.168.0.180:/srv/nas-spike2/q1 ... nfs4 ...soft,timeo=50,retrans=2...
# guest /proc/mounts after: the SAME nfs4 entry (propagated) — plus the autofs trigger itself
```
The in-guest access woke the host automount in ~1 s (including `pct exec` overhead). None of the
WRONG outcomes occurred (no hang, no empty dir, no host-only mount).

**Q1b — idle-unmount + container re-trigger:** after 45 s untouched, host was back to autofs-only
(idle-unmount works). Then a container bind-mount access —
`docker run --rm -v /mnt/felhom-drives/spike2-q1:/m alpine ls -la /m` — listed the marker and
remounted the host nfs4 (3.2 s total, dominated by the one-time alpine image pull). So the
docker `-v` bind of an idle automount path also triggers correctly — the exact media-container
pattern.

**Q1c — ordering observation (read-only, no restart performed):** `mp8: /mnt/felhom-drives` is a
plain bind; host side is propagation `shared`, guest side `shared,slave`. The **autofs trigger mount
itself is visible in the guest's `/proc/mounts`** (it propagated in live when the automount was
enabled, guest running). What this spike could NOT prove without a guest restart: whether a trigger
that exists BEFORE guest start is carried into the guest by the mp8 bind at boot (the June premise
covered real mounts — felhom-flash/felhom-usb — not autofs triggers). **Open item: needs a
restart-window test.** Until then the production task should treat "automount installed while guest
runs" as the proven path (it is also the actual add-storage flow).

---

## Q2 — Controller drops to uid 1000 and writes: **PASS (both proofs)**

**Q2a — chain proof (docker-level):**
```
$ docker exec -u 1000:1000 felhom-controller /bin/sh -c 'id && echo probe > .../q2a && cat .../q2a && rm .../q2a'
uid=1000 gid=1000 groups=1000
probe
Q2A_WRITE_READ_DELETE_OK
# second write left in place; on 180: stat → 101000:101000 644  ← the +100000 recipe end-to-end
```

**Q2b — mechanism proof (the `syscall.Credential` path):**
```
CapEff: 00000000a80425fb    CapBnd: 00000000a80425fb   ← bits 6 (SETGID) + 7 (SETUID) present
$ command -v setpriv → /usr/bin/setpriv               ← in the image (util-linux, debian bookworm-slim)
$ docker exec felhom-controller setpriv --reuid 1000 --regid 1000 --clear-groups /bin/sh -c 'id && echo x > .../q2b && ...'
uid=1000 gid=1000 groups=1000
Q2B_SETPRIV_OK
```
`setpriv` performs the same setresuid/setresgid syscalls `os/exec` + `SysProcAttr.Credential` will
use — no EPERM anywhere, so the production probe shape (root controller spawning a uid-1000
subprocess) is viable as designed. Bonus: `docker exec -u 1000:1000` (Q2a) also works, so a
future probe could even avoid setuid entirely by exec-ing into itself via the docker socket —
not needed, just noted.

---

## Q3 — NFS squash matrix: **Route A ACCEPTED-with-caveats; June WRONG case pinned**

Three exports, each probed from the controller container as guest uid 1000
(`docker exec -u 1000:1000 felhom-controller`), 8-step probe (read / create / mkdir+rename /
**hardlink** / delete / stat / chown+chmod / server-side stat):

| # | Export | Dir owner | S1 read | S2 create | S3 mkdir+mv | S4 hardlink | S5 delete | S6 stat (guest view) | S7 chown / chmod | server-side new file |
|---|--------|-----------|---------|-----------|-------------|-------------|-----------|----------------------|------------------|----------------------|
| a | `rw,all_squash,anonuid=101000,anongid=101000` | 101000:101000 | PASS | PASS | PASS | PASS (links=2) | PASS | `1000:1000` | OK / OK | `101000:101000 644` |
| b | `rw,all_squash,anonuid=1050,anongid=1050` (throwaway user `spike2a`) | 1050:1050 | PASS | PASS | PASS | **PASS (links=2)** | PASS | `65534:65534` (nobody) | **EPERM (clean, ~7 ms total)** / **SUCCEEDS** | `1050:1050 644` |
| c | `rw,anonuid=1000,anongid=1000` (NO all_squash) | 1000:1000 (755) | PASS (world-readable) | **FAIL** `Permission denied` | **FAIL** | — | — | `65534:65534` | EPERM / EPERM | none created |

**Route A verdict (one-line decision input): ACCEPTED-with-caveats — `all_squash` to ANY
server-local uid gives the guest-uid-1000 app full read+write+rename+hardlink+delete; no 101000
needed anywhere on the NAS.** Exact caveat list:
1. **Guest-visible ownership is `nobody:nogroup` (65534:65534)** — cosmetic for apps that just
   read/write, but apps that *check* ownership may complain.
2. **`chown` fails** — immediate clean `Operation not permitted`, no hang (measured ~7 ms for the
   chown+chmod pair).
3. **`chmod` SUCCEEDS** (refinement over the expected caveat): with `all_squash` every request runs
   as the anon identity, which owns every file on the share — so mode changes are permitted and
   **persist server-side** (the q3b marker really became `600` on the NAS). Apps that chmod won't
   error; operators should know modes are app-controlled.
4. **Hardlink works** — the *arr atomic-move pattern is safe on Route A.

**The June WRONG case, pinned (q3c):** `anonuid=1000` without `all_squash` squashes **root only**;
the guest-uid-1000 app arrives on the wire as its host-mapped uid **101000, unmapped**, and is
evaluated against the `1000:1000 755` dir's **others**-perms → read OK, every write
`Permission denied`. So the June finding ("anonuid=1000 → not writable") was real but its cause is
"anonuid without all_squash is a no-op for non-root traffic", NOT "uid 1000 is inherently wrong".
With `all_squash` added, even `anonuid=1000` would behave like row (b). The production guidance can
state this truthfully: **the consumer recipe requires the "map/squash ALL users" mode, not just an
anonymous-uid field.**

---

## Q4 — Verify error taxonomy + timing (production verify inputs)

Method: throwaway `.mount` unit per case, `time systemctl start <unit>` (the production verify
mechanism), journal captured, unit stopped/reset between cases. NFS cases use the locked NFS option
string; SMB cases the locked SMB string (creds file 0600 on the host).

| # | Case | Duration | start rc | Unit result | Distinguishing journal/dmesg string (verbatim) |
|---|------|----------|----------|-------------|-----------------------------------------------|
| i | server unreachable (192.168.0.199, verified unused) | **90.1 s** | 1 | `timeout` | `Mounting timed out. Terminating.` + `Mount process exited, code=killed, status=15/TERM` (NO mount.nfs line — systemd kills it first) |
| i′ | same, options + `retry=0` | **3.1 s** | 1 | `exit-code` (32) | `mount.nfs4: No route to host for 192.168.0.199:/srv/nope on /mnt/...` |
| ii | reachable, nonexistent export | 0.081 s | 1 | `exit-code` (32) | `mount.nfs4: mounting 192.168.0.180:/srv/nas-spike2/nope failed, reason given by server: No such file or directory` |
| iii | export exists, NOT permitted for this client (scoped to 192.168.0.1) | 0.080 s | 1 | `exit-code` (32) | **IDENTICAL to (ii)** — `...failed, reason given by server: No such file or directory` |
| iv | SMB wrong password | 0.175 s | 1 | `exit-code` (32) | `mount error(13): Permission denied`; dmesg: `Status code returned 0xc000006d STATUS_LOGON_FAILURE` |
| v | SMB wrong share name (valid creds) | 0.458 s | 1 | `exit-code` (32) | `mount error(2): No such file or directory`; dmesg: `CIFS: VFS: BAD_NETWORK_NAME: \\192.168.0.180\nope2` |

Happy-path references measured the same day: NFS automount trigger <0.1 s (mount side), SMB plain
mount 0.085 s.

**Taxonomy consequences for the production verify's Hungarian error map:**
- Cases (ii) and (iii) MUST be one merged category over NFSv4 ("a megosztás nem található, vagy a
  szerver nem engedélyezi ennek a gépnek") — the pseudo-fs hides non-permitted exports; do not promise
  the user a distinction the protocol cannot give.
- The unreachable case is cleanly classifiable only with `retry=0` (`No route to host` / connection
  errors); with the default retry the verify sees a generic systemd `timeout` with no mount.nfs
  diagnostic at all.
- SMB splits cleanly: `error(13)` = credentials, `error(2)` = share name, plus dmesg NT-status lines
  if ever needed.
- All failures share `exit status 32` — classify on the journal string, not the exit code.

**Q4-vi — the `retry=0` stance: put it in the PRODUCTION unit** (not only a verify-time transient
unit). Evidence — a guest process accessing the **idle automount path while the NAS is down**:
- without `retry=0`: blocks **91 s**, then `ls: cannot open directory ...: No such device` (ENODEV;
  systemd's 90 s start timeout is what ends it);
- with `retry=0`: same clean ENODEV in **3.8 s**; an immediate second access re-attempts and fails
  in the same ~3.8 s (autofs retries fresh per access — no stuck state).
`retry` only governs retrying a FAILED first attempt, so the happy path is untouched, and a briefly
down NAS costs an app ~4 s per access instead of a 90 s wedge. There is no scenario in this model
where the 2-min foreground retry helps: the automount re-triggers a fresh attempt on every access
anyway. Caveat, stated honestly: 3.1–3.8 s is the same-LAN ARP-failure case; a black-holed-but-routed
server (SYN drops) would sit in TCP connect until systemd's 90 s cap — `retry=0` narrows, but does
not eliminate, the worst case.

**Q4-vii — verify timeout budget recommendation:** with `retry=0` in the unit, the measured cases
bound a LAN verify at **≤ ~5 s** (mount trigger + probe write), but the true worst case remains
**systemd's 90 s** (black-holed server). The agentapi client's global 15 s `http.Client` Timeout
(`client.go:87`) therefore **cannot carry the verify call**: a synchronous verify inside
`POST /api/storage/netstorage/add` would time out client-side exactly in the cases verify exists to
catch. Recommendation: the production task gives AddNetStorage its own longer-lived path — either a
dedicated client/context with a **~120 s** end-to-end budget, or (better, matching the existing
detached-format pattern) an async start + status-poll. Agent-side, bound the `systemctl start` step
at ~95 s (just above systemd's cap) so the agent always returns a classified result rather than its
own timeout.

---

## Q5 — SMB without server-side force user: **PASS (hardlink included)**

Server: Samba share `[spike2-plain]`, `read only = no`, `valid users = spike2smb` (a plain throwaway
user, uid 1051), **no `force user`/`force group`**, dir owned `spike2smb`. Client: the locked SMB
option string verbatim (`vers=3.0,credentials=…,uid=101000,gid=101000,forceuid,forcegid,
file_mode=0664,dir_mode=0775,_netdev`). Mounted in 0.085 s; probe as guest uid 1000:

| Step | Result |
|------|--------|
| S1 read | PASS — marker listed `1000:1000` (client-forced), content read |
| S2 create + read-back | PASS |
| S3 mkdir + rename | PASS |
| S4 **hardlink** | **PASS** (`link-count=2`) — on THIS Samba/ext4; a NAS SMB stack may still refuse it, re-verify on the appliance |
| S5 delete | PASS |
| S6 stat | `1000:1000, mode 664` (uid/gid + file_mode client overrides, as designed) |
| S7 chown/chmod | both "succeed" (client accepts them; guest view stays 1000:1000/0664; server mode of a touched file ended `644`/`744` — i.e. cosmetic client-side, not faithfully persisted) |
| S8 server-side stat | `spike2smb:spike2smb (1051:1051)` — **writes land as the authenticated user, zero server-side uid config** |

Consumer-SMB consequence: a stock NAS account with rw on the share is sufficient; ownership fidelity
on the NAS is "everything belongs to the connecting account" — same shape as Route A on NFS.

---

## Teardown verification — DONE (all counts restored)

- **180:** `/etc/exports` + `/etc/samba/smb.conf` restored from the `.bak-nasspike2` copies, baks
  deleted. Verified: **2** non-comment export lines (pre-spike count 2), **6** smb.conf `[...]`
  sections (pre-spike 6), `exportfs -v` shows **0** spike2 entries, users `spike2a`/`spike2smb`
  removed (`id` → no such user), `/srv/nas-spike2` gone. Longhorn exports, PBS, k3s, Gitea, the six
  Samba sections untouched.
- **felhom-pve:** all spike2 units stopped (automounts first, then mounts), unit files removed,
  `daemon-reload` + `reset-failed`; `grep -c spike2 /proc/mounts` → **0**; `systemctl list-units
  '*spike2*' --all` → empty; `/mnt/felhom-drives` holds only `felhom-usb`; both SMB creds files
  removed; `/mnt/felhom-drives/spike2-*` mountpoint dirs removed.
- **Guest 9201:** `grep -c spike2 /proc/mounts` → **0**; `felhom-controller:0.112.0` Up (healthy);
  the alpine image pulled during Q1b removed.
- SMB passwords were throwaway, never committed; the creds files lived 0600 on felhom-pve for the
  spike's duration only.

---

## Observations (out of scope — documented, NOT acted on)

1. **Guest `/proc/mounts` duplicates propagated entries** (each spike mount appeared twice in the
   guest) — an artifact of the mp8 bind topology; harmless, but a naive "grep -c" health check would
   double-count.
2. The live NFS4 mount line shows **`fatal_neterrors=none`** — a newer kernel NFS option present on
   the PVE host's kernel; not in the locked recipe string, purely informational.
3. `docker exec -u 1000:1000` into the production controller container works today (Q2a) — an
   alternative probe transport via the docker socket if the Credential path ever grew a problem.
4. `useradd --system --uid 1050` warns (`uid greater than SYS_UID_MAX`) but works — irrelevant to
   findings.
5. The Q4 error strings were produced by `mount.nfs4`/`mount.cifs` on PVE (util-linux/nfs-utils of
   Proxmox 8) — string-matching in the production verify should tolerate minor version drift
   (match on the stable substrings: `No such file or directory`, `No route to host`,
   `error(13)`, `error(2)`, `timed out`).
6. Samba on 180 reloaded config via `smbcontrol all reload-config` without disturbing the six live
   shares — useful for the future virtual-dsm pass too.

No secrets in this doc. All spike credentials were throwaway and are now deleted.