felhom.eu/documentation/audits/SPIKE-nas-verify-2026-07-11.md

SPIKE — NAS verify mechanisms + NFS squash matrix, simulated on DooPlex

Date: 2026-07-11 Class: Spike (empirical validation; output = this doc only). Throwaway probes; no production code written or modified. Goal: validate the four unproven mechanisms the upcoming "NAS verify-before-commit + UI redesign" production TASK depends on, and pin down the June spike's anonuid=1000 WRONG case. Q1 in-guest automount trigger; Q2 controller uid-1000 probe; Q3 NFS squash matrix (Route A decision); Q4 verify error taxonomy + timing; Q5 plain-user SMB.

Simulated NAS: real nfs-kernel-server + Samba on DooPlex 192.168.0.180, isolated scratch root /srv/nas-spike2/ (all exports/shares spike-named; live Longhorn exports + the six original Samba sections untouched, verified by count at teardown). Mount side: felhom-pve (192.168.0.162), all mounts under /mnt/felhom-drives/spike2-* via hand-written throwaway units carrying EXACTLY netmount.go's option strings. Probe side: live demo guest 9201, probes as guest uid 1000 via docker exec -u 1000:1000 felhom-controller (the production probe's exact vantage point).

Sim-vs-real-NAS caveat (restated from June): this validates the Linux mechanism against a Debian NFS/Samba server. Synology/QNAP specifics — their squash/"Map all users" UIs, SMB dialect ceilings, per-share ACL models, whether their SMB stack permits hardlinks — remain unvalidated until the virtual-dsm confirmation pass.

Baselines (read-only anchors, verified live 2026-07-11):

Repo	main @ commit	Version	Role
felhom-agent	300f06722b	v0.80.0	internal/storage/netmount.go = the locked recipe under test
felhom-controller	3db9126121	v0.112.0	the container the Q2/Q3/Q5 probes ran in
felhom.eu	8ca1272db6	—	this doc's home

This doc's commit: b57f6c1 (hash inserted by the follow-up commit).

---

Verdict: ALL FIVE QUESTIONS ANSWERED — no blocker for the production TASK

• Q1 (make-or-break) PASS — an in-guest access (bare pct exec ls AND a docker-container bind-mount access) wakes the host systemd automount; content appears correctly, never an empty dir or ENOENT.
• Q2 PASS — the controller container (root, in-guest) can drop to uid/gid 1000 (setpriv, the same setresuid/setresgid syscalls syscall.Credential uses) and write through the share path; SETUID/SETGID capability bits confirmed present.
• Q3 — Route A ACCEPTED-with-caveats — all_squash to an arbitrary server-local uid gives the guest-uid-1000 app full read+write+rename+hardlink+delete. Caveats: guest-visible ownership is nobody:nogroup (65534), chown fails (immediate clean EPERM, no hang), chmod succeeds (and persists server-side). The June WRONG case is pinned: anonuid without all_squash is a no-op for non-root traffic — the wire uid 101000 hits the export unmapped and fails on others-perms.
• Q4 taxonomy captured — table below. Two headline facts: NFSv4 makes "export doesn't exist" and "export not permitted for this client" indistinguishable (identical error string), and the real ceiling on a dead-server mount is systemd's 90 s start timeout, not mount.nfs's 2-min retry. retry=0 turns the dead-server case into a 3.1 s clean "No route to host" and an on-demand guest access from 91 s ENODEV into 3.8 s — recommended for the PRODUCTION unit (evidence in Q4-vi).
• Q5 PASS — a plain Samba user + rw share (zero server-side force user/uid config) works through the client-forced-uid chain, including hardlinks (on this Debian/ext4 Samba).

---

Q1 — In-guest automount trigger: PASS (both variants)

Setup: export /srv/nas-spike2/q1 (rw,all_squash,anonuid=101000,anongid=101000,no_subtree_check, dir 101000:101000), throwaway .mount+.automount pair at /mnt/felhom-drives/spike2-q1 with the locked NFS option string (vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev), TimeoutIdleSec=30. Idle state confirmed by grep of /proc/mounts only (autofs, no nfs4) — the path was never stat'ed host-side.

Q1a — first access from inside the running guest:

# host /proc/mounts before: autofs trigger ONLY
$ time pct exec 9201 -- ls -la /mnt/felhom-drives/spike2-q1
-rw-r--r-- 1   1000    1000      30 ... MARKER.txt        ← marker listed, owned 1000:1000
-rw-r--r-- 1   1000    1000 5242880 ... sample.bin
real  0m1.053s   rc=0
# host /proc/mounts after:  192.168.0.180:/srv/nas-spike2/q1 ... nfs4 ...soft,timeo=50,retrans=2...
# guest /proc/mounts after: the SAME nfs4 entry (propagated) — plus the autofs trigger itself

The in-guest access woke the host automount in ~1 s (including pct exec overhead). None of the WRONG outcomes occurred (no hang, no empty dir, no host-only mount).

Q1b — idle-unmount + container re-trigger: after 45 s untouched, host was back to autofs-only (idle-unmount works). Then a container bind-mount access — docker run --rm -v /mnt/felhom-drives/spike2-q1:/m alpine ls -la /m — listed the marker and remounted the host nfs4 (3.2 s total, dominated by the one-time alpine image pull). So the docker -v bind of an idle automount path also triggers correctly — the exact media-container pattern.

Q1c — ordering observation (read-only, no restart performed): mp8: /mnt/felhom-drives is a plain bind; host side is propagation shared, guest side shared,slave. The autofs trigger mount itself is visible in the guest's /proc/mounts (it propagated in live when the automount was enabled, guest running). What this spike could NOT prove without a guest restart: whether a trigger that exists BEFORE guest start is carried into the guest by the mp8 bind at boot (the June premise covered real mounts — felhom-flash/felhom-usb — not autofs triggers). Open item: needs a restart-window test. Until then the production task should treat "automount installed while guest runs" as the proven path (it is also the actual add-storage flow).

---

Q2 — Controller drops to uid 1000 and writes: PASS (both proofs)

Q2a — chain proof (docker-level):

$ docker exec -u 1000:1000 felhom-controller /bin/sh -c 'id && echo probe > .../q2a && cat .../q2a && rm .../q2a'
uid=1000 gid=1000 groups=1000
probe
Q2A_WRITE_READ_DELETE_OK
# second write left in place; on 180: stat → 101000:101000 644  ← the +100000 recipe end-to-end

Q2b — mechanism proof (the syscall.Credential path):

CapEff: 00000000a80425fb    CapBnd: 00000000a80425fb   ← bits 6 (SETGID) + 7 (SETUID) present
$ command -v setpriv → /usr/bin/setpriv               ← in the image (util-linux, debian bookworm-slim)
$ docker exec felhom-controller setpriv --reuid 1000 --regid 1000 --clear-groups /bin/sh -c 'id && echo x > .../q2b && ...'
uid=1000 gid=1000 groups=1000
Q2B_SETPRIV_OK

setpriv performs the same setresuid/setresgid syscalls os/exec + SysProcAttr.Credential will use — no EPERM anywhere, so the production probe shape (root controller spawning a uid-1000 subprocess) is viable as designed. Bonus: docker exec -u 1000:1000 (Q2a) also works, so a future probe could even avoid setuid entirely by exec-ing into itself via the docker socket — not needed, just noted.

---

Q3 — NFS squash matrix: Route A ACCEPTED-with-caveats; June WRONG case pinned

Three exports, each probed from the controller container as guest uid 1000 (docker exec -u 1000:1000 felhom-controller), 8-step probe (read / create / mkdir+rename / hardlink / delete / stat / chown+chmod / server-side stat):

#	Export	Dir owner	S1 read	S2 create	S3 mkdir+mv	S4 hardlink	S5 delete	S6 stat (guest view)	S7 chown / chmod	server-side new file
a	rw,all_squash,anonuid=101000,anongid=101000	101000:101000	PASS	PASS	PASS	PASS (links=2)	PASS	1000:1000	OK / OK	101000:101000 644
b	rw,all_squash,anonuid=1050,anongid=1050 (throwaway user spike2a)	1050:1050	PASS	PASS	PASS	PASS (links=2)	PASS	65534:65534 (nobody)	EPERM (clean, ~7 ms total) / SUCCEEDS	1050:1050 644
c	rw,anonuid=1000,anongid=1000 (NO all_squash)	1000:1000 (755)	PASS (world-readable)	FAIL Permission denied	FAIL	—	—	65534:65534	EPERM / EPERM	none created

Route A verdict (one-line decision input): ACCEPTED-with-caveats — all_squash to ANY server-local uid gives the guest-uid-1000 app full read+write+rename+hardlink+delete; no 101000 needed anywhere on the NAS. Exact caveat list:

1. Guest-visible ownership is nobody:nogroup (65534:65534) — cosmetic for apps that just read/write, but apps that check ownership may complain.
2. chown fails — immediate clean Operation not permitted, no hang (measured ~7 ms for the chown+chmod pair).
3. chmod SUCCEEDS (refinement over the expected caveat): with all_squash every request runs as the anon identity, which owns every file on the share — so mode changes are permitted and persist server-side (the q3b marker really became 600 on the NAS). Apps that chmod won't error; operators should know modes are app-controlled.
4. Hardlink works — the *arr atomic-move pattern is safe on Route A.

The June WRONG case, pinned (q3c): anonuid=1000 without all_squash squashes root only; the guest-uid-1000 app arrives on the wire as its host-mapped uid 101000, unmapped, and is evaluated against the 1000:1000 755 dir's others-perms → read OK, every write Permission denied. So the June finding ("anonuid=1000 → not writable") was real but its cause is "anonuid without all_squash is a no-op for non-root traffic", NOT "uid 1000 is inherently wrong". With all_squash added, even anonuid=1000 would behave like row (b). The production guidance can state this truthfully: the consumer recipe requires the "map/squash ALL users" mode, not just an anonymous-uid field.

---

Q4 — Verify error taxonomy + timing (production verify inputs)

Method: throwaway .mount unit per case, time systemctl start <unit> (the production verify mechanism), journal captured, unit stopped/reset between cases. NFS cases use the locked NFS option string; SMB cases the locked SMB string (creds file 0600 on the host).

#	Case	Duration	start rc	Unit result	Distinguishing journal/dmesg string (verbatim)
i	server unreachable (192.168.0.199, verified unused)	90.1 s	1	timeout	Mounting timed out. Terminating. + Mount process exited, code=killed, status=15/TERM (NO mount.nfs line — systemd kills it first)
i′	same, options + retry=0	3.1 s	1	exit-code (32)	mount.nfs4: No route to host for 192.168.0.199:/srv/nope on /mnt/...
ii	reachable, nonexistent export	0.081 s	1	exit-code (32)	mount.nfs4: mounting 192.168.0.180:/srv/nas-spike2/nope failed, reason given by server: No such file or directory
iii	export exists, NOT permitted for this client (scoped to 192.168.0.1)	0.080 s	1	exit-code (32)	IDENTICAL to (ii) — ...failed, reason given by server: No such file or directory
iv	SMB wrong password	0.175 s	1	exit-code (32)	mount error(13): Permission denied; dmesg: Status code returned 0xc000006d STATUS_LOGON_FAILURE
v	SMB wrong share name (valid creds)	0.458 s	1	exit-code (32)	mount error(2): No such file or directory; dmesg: CIFS: VFS: BAD_NETWORK_NAME: \\192.168.0.180\nope2

Happy-path references measured the same day: NFS automount trigger <0.1 s (mount side), SMB plain mount 0.085 s.

Taxonomy consequences for the production verify's Hungarian error map:

• Cases (ii) and (iii) MUST be one merged category over NFSv4 ("a megosztás nem található, vagy a szerver nem engedélyezi ennek a gépnek") — the pseudo-fs hides non-permitted exports; do not promise the user a distinction the protocol cannot give.
• The unreachable case is cleanly classifiable only with retry=0 (No route to host / connection errors); with the default retry the verify sees a generic systemd timeout with no mount.nfs diagnostic at all.
• SMB splits cleanly: error(13) = credentials, error(2) = share name, plus dmesg NT-status lines if ever needed.
• All failures share exit status 32 — classify on the journal string, not the exit code.

Q4-vi — the retry=0 stance: put it in the PRODUCTION unit (not only a verify-time transient unit). Evidence — a guest process accessing the idle automount path while the NAS is down:

• without retry=0: blocks 91 s, then ls: cannot open directory ...: No such device (ENODEV; systemd's 90 s start timeout is what ends it);
• with retry=0: same clean ENODEV in 3.8 s; an immediate second access re-attempts and fails in the same ~3.8 s (autofs retries fresh per access — no stuck state). retry only governs retrying a FAILED first attempt, so the happy path is untouched, and a briefly down NAS costs an app ~4 s per access instead of a 90 s wedge. There is no scenario in this model where the 2-min foreground retry helps: the automount re-triggers a fresh attempt on every access anyway. Caveat, stated honestly: 3.1–3.8 s is the same-LAN ARP-failure case; a black-holed-but-routed server (SYN drops) would sit in TCP connect until systemd's 90 s cap — retry=0 narrows, but does not eliminate, the worst case.

Q4-vii — verify timeout budget recommendation: with retry=0 in the unit, the measured cases bound a LAN verify at ≤ ~5 s (mount trigger + probe write), but the true worst case remains systemd's 90 s (black-holed server). The agentapi client's global 15 s http.Client Timeout (client.go:87) therefore cannot carry the verify call: a synchronous verify inside POST /api/storage/netstorage/add would time out client-side exactly in the cases verify exists to catch. Recommendation: the production task gives AddNetStorage its own longer-lived path — either a dedicated client/context with a ~120 s end-to-end budget, or (better, matching the existing detached-format pattern) an async start + status-poll. Agent-side, bound the systemctl start step at ~95 s (just above systemd's cap) so the agent always returns a classified result rather than its own timeout.

---

Q5 — SMB without server-side force user: PASS (hardlink included)

Server: Samba share [spike2-plain], read only = no, valid users = spike2smb (a plain throwaway user, uid 1051), no force user/force group, dir owned spike2smb. Client: the locked SMB option string verbatim (vers=3.0,credentials=…,uid=101000,gid=101000,forceuid,forcegid, file_mode=0664,dir_mode=0775,_netdev). Mounted in 0.085 s; probe as guest uid 1000:

Step	Result
S1 read	PASS — marker listed 1000:1000 (client-forced), content read
S2 create + read-back	PASS
S3 mkdir + rename	PASS
S4 hardlink	PASS (link-count=2) — on THIS Samba/ext4; a NAS SMB stack may still refuse it, re-verify on the appliance
S5 delete	PASS
S6 stat	1000:1000, mode 664 (uid/gid + file_mode client overrides, as designed)
S7 chown/chmod	both "succeed" (client accepts them; guest view stays 1000:1000/0664; server mode of a touched file ended 644/744 — i.e. cosmetic client-side, not faithfully persisted)
S8 server-side stat	spike2smb:spike2smb (1051:1051) — writes land as the authenticated user, zero server-side uid config

Consumer-SMB consequence: a stock NAS account with rw on the share is sufficient; ownership fidelity on the NAS is "everything belongs to the connecting account" — same shape as Route A on NFS.

---

Teardown verification — DONE (all counts restored)

• 180: /etc/exports + /etc/samba/smb.conf restored from the .bak-nasspike2 copies, baks deleted. Verified: 2 non-comment export lines (pre-spike count 2), 6 smb.conf [...] sections (pre-spike 6), exportfs -v shows 0 spike2 entries, users spike2a/spike2smb removed (id → no such user), /srv/nas-spike2 gone. Longhorn exports, PBS, k3s, Gitea, the six Samba sections untouched.
• felhom-pve: all spike2 units stopped (automounts first, then mounts), unit files removed, daemon-reload + reset-failed; grep -c spike2 /proc/mounts → 0; systemctl list-units '*spike2*' --all → empty; /mnt/felhom-drives holds only felhom-usb; both SMB creds files removed; /mnt/felhom-drives/spike2-* mountpoint dirs removed.
• Guest 9201: grep -c spike2 /proc/mounts → 0; felhom-controller:0.112.0 Up (healthy); the alpine image pulled during Q1b removed.
• SMB passwords were throwaway, never committed; the creds files lived 0600 on felhom-pve for the spike's duration only.

---

Observations (out of scope — documented, NOT acted on)

1. Guest /proc/mounts duplicates propagated entries (each spike mount appeared twice in the guest) — an artifact of the mp8 bind topology; harmless, but a naive "grep -c" health check would double-count.
2. The live NFS4 mount line shows fatal_neterrors=none — a newer kernel NFS option present on the PVE host's kernel; not in the locked recipe string, purely informational.
3. docker exec -u 1000:1000 into the production controller container works today (Q2a) — an alternative probe transport via the docker socket if the Credential path ever grew a problem.
4. useradd --system --uid 1050 warns (uid greater than SYS_UID_MAX) but works — irrelevant to findings.
5. The Q4 error strings were produced by mount.nfs4/mount.cifs on PVE (util-linux/nfs-utils of Proxmox 8) — string-matching in the production verify should tolerate minor version drift (match on the stable substrings: No such file or directory, No route to host, error(13), error(2), timed out).
6. Samba on 180 reloaded config via smbcontrol all reload-config without disturbing the six live shares — useful for the future virtual-dsm pass too.

No secrets in this doc. All spike credentials were throwaway and are now deleted.