From b57f6c15090124b5bab521518fde03afa1607625 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 11 Jul 2026 08:20:00 +0200 Subject: [PATCH] =?UTF-8?q?docs(audit):=20SPIKE=20NAS=20verify=20mechanism?= =?UTF-8?q?s=20+=20squash=20matrix=20=E2=80=94=20Q1-Q5=20answered,=20Route?= =?UTF-8?q?=20A=20accepted-with-caveats?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CONTEXT.md | 12 + REPORT.md | 93 +++--- .../audits/SPIKE-nas-verify-2026-07-11.md | 275 ++++++++++++++++++ 3 files changed, 328 insertions(+), 52 deletions(-) create mode 100644 documentation/audits/SPIKE-nas-verify-2026-07-11.md diff --git a/CONTEXT.md b/CONTEXT.md index 6a612c3..2d470f2 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,18 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-11 — SPIKE: NAS verify mechanisms + NFS squash matrix — ALL FIVE QUESTIONS ANSWERED, no + blocker.** Record: `documentation/audits/SPIKE-nas-verify-2026-07-11.md`. Q1 in-guest access DOES + wake the host automount (make-or-break PASS; container bind-mount too); Q2 controller drops to uid + 1000 and writes (setpriv proof; SETUID/SETGID caps present); Q3 **Route A ACCEPTED-with-caveats** + (`all_squash` to ANY server uid = full rw+hardlink for the guest app; guest sees nobody; chown + EPERM-clean, chmod persists; June's `anonuid=1000` WRONG case pinned = anonuid without all_squash + is a no-op for non-root); Q4 error taxonomy captured (NFSv4 makes no-export vs not-permitted + INDISTINGUISHABLE; systemd 90 s start-timeout is the real ceiling; **`retry=0` → production unit**: + dead-NAS access 91 s→3.8 s; verify must NOT ride the 15 s agentapi timeout → own path/status-poll, + ~120 s budget); Q5 plain-user SMB PASS incl. hardlink. Teardown verified to pre-counts. Production + TASK is the operator side's next move. + - **2026-07-11 — PBS DR tier SLICE 2 SHIPPED (agent v0.80.0): the apply-bridge — demo ADOPTION-PROVEN live + PUBLISHED; the supervised Peti phase PENDING.** Full chain fired on the demo: hub enable (storage-id `felhom-offsite`) → tenantsync provisioned ns `demo-felhom` on ep0 diff --git a/REPORT.md b/REPORT.md index dd13aa6..ffa753f 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,59 +2,48 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## TASK — PBS DR tier SLICE 1: ep0 tenantsync surface + hub provisioning (hub v0.44.0) — 2026-07-10 +## SPIKE — NAS verify mechanisms + NFS squash matrix — 2026-07-11 -**LIVE on k3s (Synced/Healthy, image 0.44.0, startup logs `WG peer-sync enabled` + `PBS DR tenantsync -enabled`). NOT live-provisioned — no real customer was provisioned from this task; slice 2 (the agent -apply-bridge) + the supervised Peti enable are pending.** Builds on SPIKE-pbs-tier-provisioning -(00afadc). Commits: ce6a566 (code) + 32ca1f6 (live fix + image bump). +**Class:** spike (findings doc is the only artifact; no production code touched). +**Record:** `documentation/audits/SPIKE-nas-verify-2026-07-11.md`. +**Baselines (read-only):** felhom-agent `300f06722b` (v0.80.0, `netmount.go` = recipe under test), +felhom-controller `3db9126121` (v0.112.0, the probe container), felhom.eu `8ca1272db6`. -### What shipped -- **ep0 surface (🛑 Viktor-approved, installed + smoked live):** `scripts/felhom-tenantsync.sh` v1.0.0 - as a SECOND forced-command key for `felhom-peersync` (own sudoers drop-in; **peersync script + - sudoers sha256 asserted byte-identical before/after**). Ops: `provision` (existing token → typed - `token_exists`; own-ns self-check with one regen retry then rollback), `reissue` (delete→recreate→ - re-grant — delete purges token ACLs, the spike fact), `fingerprint`. Secret hygiene: the token - secret rides stdout ONLY (all tool stdout → stderr; hub error paths never embed stdout). - Smoke (throwaway `spike-smoke`, torn down): all ops green; **live fix caught by the smoke** — - `proxmox-backup-client --output-format json` wraps as `{"data":[…]}` → ns-exists check fixed to - `(.data // .)` and reinstalled before the hub ever used the surface. - **Orphan cleanup (spike flag #1):** `root@pam!spike` token + `/datastore/scratch` DatastoreAdmin - ACL removed; ep0 now holds exactly the real `demo-felhom-01` tenancy. -- **Hub `internal/tenantsync`:** wgsync-twin SSH client (pinned host key, constrained - HostKeyAlgorithms, per-op JSON exec, typed `ErrTokenExists`). Contract test - `TestErrors_NeverEmbedStdout` pins the no-stdout-in-errors rule. -- **Provisioning flow (`web/pbsdr.go`):** config-form section "PBS DR tier (ep0)" → on enable+save - (detached-ctx atom): WG peer existence check (the agent self-registers its key; absence - fail-closes with "has not reported a WG key yet") → tenantsync provision → secret stored - consume-once HOST-scoped (`host_pbs_secrets`, the `one_time_secrets` twin) → non-secret `pbs_dr` - descriptor (`enabled, storage_id, pbs_tunnel_ip, datastore, namespace, token_id, fingerprint`) - merged into the HOST `desired_json` + `SetHostDesired` generation bump. Already-provisioned - re-save = success-no-op; disable = `enabled:false` descriptor, tenancy kept; **Re-issue PBS - credentials** button = the F4 precedent (fresh consume-once secret + bump). -- **Consume endpoint:** `POST /api/v1/hosts/{id}/pbs/consume-token` — per-host key self-scoped - (global = operator recovery); 200 exactly once → 404; a foreign key's 403 does NOT burn the - secret. *(Deviation: task spec wrote `/host/{id}/…`; implemented under `/hosts/` matching every - other agent-facing route — slice 2 must use the `/hosts/` form.)* -- **Deploy:** `TENANTSYNC_SSH_KEY_FILE` env + optional `Secret/tenantsync` mount (key generated on - 180, moved into the Secret, local copy shredded); same endpoint addr + pinned host key as peersync. +### Verdicts (all five questions answered live, evidence in the record) -### Tests + red-proofs (all run, mutations reverted, gate green) -- Happy path: descriptor lands in desired_json + gen bump exactly 1; secret consumable once and - absent from ConfigJSON/desired-state/captured logs (grep-asserts). -- **Red-proof consume-once:** dropping the `consumed_at` UPDATE → store test re-served `"secret-1"`, - API test returned 200 with the secret again → both FAIL as designed. -- **Red-proof fail-closed:** swallowing the applyPBSDR error in handleConfigUpdate → all four - fail-closed subtests FAIL with 303 (half-enabled save). -- **Red-proof idempotency:** dropping the already-provisioned short-circuit → re-save re-keyed the - token (2 provision calls), staged a fresh secret, bumped spuriously → FAIL. -- Also covered: foreign-host 403 non-burning, reissue rotation (old secret dead), disable no-op on - second save, storage-id edit updates descriptor without re-provision, tenantsync client - round-trip against an in-process SSH server incl. wrong-host-key refusal. -- UI leg: render tests (`TestPBSDR_FormRendersState` + suite) — the hub UI is password-gated, so - render tests are the verification method per repo convention. +- **Q1 — in-guest automount trigger: PASS (make-or-break).** A `pct exec` ls AND a docker + bind-mount access from inside guest 9201 both wake the host systemd automount (~1 s); content + correct, nfs4 propagates host→guest; idle-unmount + re-trigger proven. Q1c open item: whether a + trigger installed BEFORE guest start carries through at boot needs a restart-window test (no + restart permitted this spike). +- **Q2 — controller uid-1000 probe: PASS.** `docker exec -u 1000:1000` chain proof + `setpriv + --reuid 1000` mechanism proof (same syscalls as `syscall.Credential`); CapEff `a80425fb` has + SETUID/SETGID; server-side files land `101000:101000`. +- **Q3 — Route A: ACCEPTED-with-caveats.** `all_squash` to an arbitrary server uid (1050) gives the + guest-uid-1000 app full read/create/rename/**hardlink**/delete. Caveats: guest sees + `nobody:nogroup`; `chown` fails immediate-clean EPERM; `chmod` SUCCEEDS and persists server-side. + q3c pinned the June WRONG case: `anonuid` without `all_squash` never applies to non-root traffic — + wire uid 101000 fails on others-perms. Consumer guidance must require the "map/squash ALL users" + mode, not just an anon-uid field. +- **Q4 — error taxonomy + timing:** full table in the record. Headlines: NFSv4 returns the IDENTICAL + `No such file or directory` for nonexistent vs not-permitted exports (merge those error + categories); dead server = systemd `timeout` at **90 s** (systemd kills mount.nfs before its 2-min + retry — no diagnostic string); **`retry=0` recommended for the PRODUCTION unit** (dead-server + verify 90 s→3.1 s with clean `No route to host`; guest on-demand access 91 s→3.8 s ENODEV; happy + path untouched; autofs retries fresh per access). SMB splits cleanly: `error(13)`=credentials, + `error(2)`=share name. **Budget (Q4-vii):** verify cannot ride the agentapi global 15 s timeout + (`client.go:87`) — worst case is systemd's 90 s; give AddNetStorage its own longer-lived path or + async status-poll, ~120 s end-to-end, agent-side `systemctl start` bounded at ~95 s. +- **Q5 — plain-user SMB: PASS.** Stock user + rw share, zero server-side uid config, all 8 probe + steps pass INCLUDING hardlink (on Debian Samba/ext4 — re-verify on a real appliance); server-side + ownership = the authenticated account; guest sees forced `1000:1000/0664`. -### Operator follow-ups -- Slice 2 (agent apply-bridge) then the supervised Peti enable (floor + agent update first). -- The demo host's missing §4b step-3 storage ACL grants (spike flag #2) — re-assert before slice 2 - codifies the self-grant. +### Safety rails held +Isolated scratch root `/srv/nas-spike2/` on 180 (backups `.bak-nasspike2`); no iptables/network +manipulation (unreachable case = verified-unused 192.168.0.199); no guest restart; spike mounts only +under `/mnt/felhom-drives/spike2-*`. **Teardown verified:** exports back to pre-count 2, smb.conf +sections 6, spike users/scratch/units/creds/mountpoints all gone, 0 spike2 mounts on host and in +guest, controller 0.112.0 healthy, spike-pulled alpine image removed. + +**Next:** the production "NAS verify-before-commit + UI redesign" TASK is written by the operator +side on these findings (STOP honored — no production work started). diff --git a/documentation/audits/SPIKE-nas-verify-2026-07-11.md b/documentation/audits/SPIKE-nas-verify-2026-07-11.md new file mode 100644 index 0000000..f4d6129 --- /dev/null +++ b/documentation/audits/SPIKE-nas-verify-2026-07-11.md @@ -0,0 +1,275 @@ +# SPIKE — NAS verify mechanisms + NFS squash matrix, simulated on DooPlex + +**Date:** 2026-07-11 +**Class:** Spike (empirical validation; output = this doc only). Throwaway probes; no production code +written or modified. +**Goal:** validate the four unproven mechanisms the upcoming "NAS verify-before-commit + UI redesign" +production TASK depends on, and pin down the June spike's `anonuid=1000` WRONG case. Q1 in-guest +automount trigger; Q2 controller uid-1000 probe; Q3 NFS squash matrix (Route A decision); Q4 verify +error taxonomy + timing; Q5 plain-user SMB. + +**Simulated NAS:** real nfs-kernel-server + Samba on **DooPlex 192.168.0.180**, isolated scratch root +`/srv/nas-spike2/` (all exports/shares spike-named; live Longhorn exports + the six original Samba +sections untouched, verified by count at teardown). Mount side: **felhom-pve** (192.168.0.162), all +mounts under `/mnt/felhom-drives/spike2-*` via hand-written throwaway units carrying EXACTLY +`netmount.go`'s option strings. Probe side: live demo guest **9201**, probes as guest uid 1000 via +`docker exec -u 1000:1000 felhom-controller` (the production probe's exact vantage point). + +> **Sim-vs-real-NAS caveat (restated from June):** this validates the Linux *mechanism* against a +> Debian NFS/Samba server. Synology/QNAP specifics — their `squash`/"Map all users" UIs, SMB dialect +> ceilings, per-share ACL models, whether their SMB stack permits hardlinks — remain unvalidated until +> the virtual-dsm confirmation pass. + +**Baselines (read-only anchors, verified live 2026-07-11):** + +| Repo | `main` @ commit | Version | Role | +|------|-----------------|---------|------| +| felhom-agent | `300f06722b` | v0.80.0 | `internal/storage/netmount.go` = the locked recipe under test | +| felhom-controller | `3db9126121` | v0.112.0 | the container the Q2/Q3/Q5 probes ran in | +| felhom.eu | `8ca1272db6` | — | this doc's home | + +**This doc's commit:** `PENDING` (inserted by the follow-up CHANGELOG commit). + +--- + +## Verdict: **ALL FIVE QUESTIONS ANSWERED — no blocker for the production TASK** + +- **Q1 (make-or-break) PASS** — an in-guest access (bare `pct exec ls` AND a docker-container + bind-mount access) wakes the **host** systemd automount; content appears correctly, never an empty + dir or ENOENT. +- **Q2 PASS** — the controller container (root, in-guest) can drop to uid/gid 1000 (`setpriv`, the + same setresuid/setresgid syscalls `syscall.Credential` uses) and write through the share path; + SETUID/SETGID capability bits confirmed present. +- **Q3 — Route A ACCEPTED-with-caveats** — `all_squash` to an arbitrary server-local uid gives the + guest-uid-1000 app full read+write+rename+**hardlink**+delete. Caveats: guest-visible ownership is + `nobody:nogroup` (65534), `chown` fails (immediate clean EPERM, no hang), `chmod` **succeeds** (and + persists server-side). The June WRONG case is pinned: `anonuid` **without** `all_squash` is a no-op + for non-root traffic — the wire uid 101000 hits the export unmapped and fails on others-perms. +- **Q4 taxonomy captured** — table below. Two headline facts: NFSv4 makes "export doesn't exist" and + "export not permitted for this client" **indistinguishable** (identical error string), and the real + ceiling on a dead-server mount is **systemd's 90 s start timeout**, not `mount.nfs`'s 2-min retry. + `retry=0` turns the dead-server case into a **3.1 s clean "No route to host"** and an on-demand + guest access from 91 s ENODEV into **3.8 s** — recommended for the PRODUCTION unit (evidence in + Q4-vi). +- **Q5 PASS** — a plain Samba user + rw share (zero server-side `force user`/uid config) works through + the client-forced-uid chain, **including hardlinks** (on this Debian/ext4 Samba). + +--- + +## Q1 — In-guest automount trigger: **PASS (both variants)** + +Setup: export `/srv/nas-spike2/q1` (`rw,all_squash,anonuid=101000,anongid=101000,no_subtree_check`, +dir `101000:101000`), throwaway `.mount`+`.automount` pair at `/mnt/felhom-drives/spike2-q1` with the +locked NFS option string (`vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev`), `TimeoutIdleSec=30`. +Idle state confirmed by grep of `/proc/mounts` only (autofs, no nfs4) — the path was never +stat'ed host-side. + +**Q1a — first access from inside the running guest:** +``` +# host /proc/mounts before: autofs trigger ONLY +$ time pct exec 9201 -- ls -la /mnt/felhom-drives/spike2-q1 +-rw-r--r-- 1 1000 1000 30 ... MARKER.txt ← marker listed, owned 1000:1000 +-rw-r--r-- 1 1000 1000 5242880 ... sample.bin +real 0m1.053s rc=0 +# host /proc/mounts after: 192.168.0.180:/srv/nas-spike2/q1 ... nfs4 ...soft,timeo=50,retrans=2... +# guest /proc/mounts after: the SAME nfs4 entry (propagated) — plus the autofs trigger itself +``` +The in-guest access woke the host automount in ~1 s (including `pct exec` overhead). None of the +WRONG outcomes occurred (no hang, no empty dir, no host-only mount). + +**Q1b — idle-unmount + container re-trigger:** after 45 s untouched, host was back to autofs-only +(idle-unmount works). Then a container bind-mount access — +`docker run --rm -v /mnt/felhom-drives/spike2-q1:/m alpine ls -la /m` — listed the marker and +remounted the host nfs4 (3.2 s total, dominated by the one-time alpine image pull). So the +docker `-v` bind of an idle automount path also triggers correctly — the exact media-container +pattern. + +**Q1c — ordering observation (read-only, no restart performed):** `mp8: /mnt/felhom-drives` is a +plain bind; host side is propagation `shared`, guest side `shared,slave`. The **autofs trigger mount +itself is visible in the guest's `/proc/mounts`** (it propagated in live when the automount was +enabled, guest running). What this spike could NOT prove without a guest restart: whether a trigger +that exists BEFORE guest start is carried into the guest by the mp8 bind at boot (the June premise +covered real mounts — felhom-flash/felhom-usb — not autofs triggers). **Open item: needs a +restart-window test.** Until then the production task should treat "automount installed while guest +runs" as the proven path (it is also the actual add-storage flow). + +--- + +## Q2 — Controller drops to uid 1000 and writes: **PASS (both proofs)** + +**Q2a — chain proof (docker-level):** +``` +$ docker exec -u 1000:1000 felhom-controller /bin/sh -c 'id && echo probe > .../q2a && cat .../q2a && rm .../q2a' +uid=1000 gid=1000 groups=1000 +probe +Q2A_WRITE_READ_DELETE_OK +# second write left in place; on 180: stat → 101000:101000 644 ← the +100000 recipe end-to-end +``` + +**Q2b — mechanism proof (the `syscall.Credential` path):** +``` +CapEff: 00000000a80425fb CapBnd: 00000000a80425fb ← bits 6 (SETGID) + 7 (SETUID) present +$ command -v setpriv → /usr/bin/setpriv ← in the image (util-linux, debian bookworm-slim) +$ docker exec felhom-controller setpriv --reuid 1000 --regid 1000 --clear-groups /bin/sh -c 'id && echo x > .../q2b && ...' +uid=1000 gid=1000 groups=1000 +Q2B_SETPRIV_OK +``` +`setpriv` performs the same setresuid/setresgid syscalls `os/exec` + `SysProcAttr.Credential` will +use — no EPERM anywhere, so the production probe shape (root controller spawning a uid-1000 +subprocess) is viable as designed. Bonus: `docker exec -u 1000:1000` (Q2a) also works, so a +future probe could even avoid setuid entirely by exec-ing into itself via the docker socket — +not needed, just noted. + +--- + +## Q3 — NFS squash matrix: **Route A ACCEPTED-with-caveats; June WRONG case pinned** + +Three exports, each probed from the controller container as guest uid 1000 +(`docker exec -u 1000:1000 felhom-controller`), 8-step probe (read / create / mkdir+rename / +**hardlink** / delete / stat / chown+chmod / server-side stat): + +| # | Export | Dir owner | S1 read | S2 create | S3 mkdir+mv | S4 hardlink | S5 delete | S6 stat (guest view) | S7 chown / chmod | server-side new file | +|---|--------|-----------|---------|-----------|-------------|-------------|-----------|----------------------|------------------|----------------------| +| a | `rw,all_squash,anonuid=101000,anongid=101000` | 101000:101000 | PASS | PASS | PASS | PASS (links=2) | PASS | `1000:1000` | OK / OK | `101000:101000 644` | +| b | `rw,all_squash,anonuid=1050,anongid=1050` (throwaway user `spike2a`) | 1050:1050 | PASS | PASS | PASS | **PASS (links=2)** | PASS | `65534:65534` (nobody) | **EPERM (clean, ~7 ms total)** / **SUCCEEDS** | `1050:1050 644` | +| c | `rw,anonuid=1000,anongid=1000` (NO all_squash) | 1000:1000 (755) | PASS (world-readable) | **FAIL** `Permission denied` | **FAIL** | — | — | `65534:65534` | EPERM / EPERM | none created | + +**Route A verdict (one-line decision input): ACCEPTED-with-caveats — `all_squash` to ANY +server-local uid gives the guest-uid-1000 app full read+write+rename+hardlink+delete; no 101000 +needed anywhere on the NAS.** Exact caveat list: +1. **Guest-visible ownership is `nobody:nogroup` (65534:65534)** — cosmetic for apps that just + read/write, but apps that *check* ownership may complain. +2. **`chown` fails** — immediate clean `Operation not permitted`, no hang (measured ~7 ms for the + chown+chmod pair). +3. **`chmod` SUCCEEDS** (refinement over the expected caveat): with `all_squash` every request runs + as the anon identity, which owns every file on the share — so mode changes are permitted and + **persist server-side** (the q3b marker really became `600` on the NAS). Apps that chmod won't + error; operators should know modes are app-controlled. +4. **Hardlink works** — the *arr atomic-move pattern is safe on Route A. + +**The June WRONG case, pinned (q3c):** `anonuid=1000` without `all_squash` squashes **root only**; +the guest-uid-1000 app arrives on the wire as its host-mapped uid **101000, unmapped**, and is +evaluated against the `1000:1000 755` dir's **others**-perms → read OK, every write +`Permission denied`. So the June finding ("anonuid=1000 → not writable") was real but its cause is +"anonuid without all_squash is a no-op for non-root traffic", NOT "uid 1000 is inherently wrong". +With `all_squash` added, even `anonuid=1000` would behave like row (b). The production guidance can +state this truthfully: **the consumer recipe requires the "map/squash ALL users" mode, not just an +anonymous-uid field.** + +--- + +## Q4 — Verify error taxonomy + timing (production verify inputs) + +Method: throwaway `.mount` unit per case, `time systemctl start ` (the production verify +mechanism), journal captured, unit stopped/reset between cases. NFS cases use the locked NFS option +string; SMB cases the locked SMB string (creds file 0600 on the host). + +| # | Case | Duration | start rc | Unit result | Distinguishing journal/dmesg string (verbatim) | +|---|------|----------|----------|-------------|-----------------------------------------------| +| i | server unreachable (192.168.0.199, verified unused) | **90.1 s** | 1 | `timeout` | `Mounting timed out. Terminating.` + `Mount process exited, code=killed, status=15/TERM` (NO mount.nfs line — systemd kills it first) | +| i′ | same, options + `retry=0` | **3.1 s** | 1 | `exit-code` (32) | `mount.nfs4: No route to host for 192.168.0.199:/srv/nope on /mnt/...` | +| ii | reachable, nonexistent export | 0.081 s | 1 | `exit-code` (32) | `mount.nfs4: mounting 192.168.0.180:/srv/nas-spike2/nope failed, reason given by server: No such file or directory` | +| iii | export exists, NOT permitted for this client (scoped to 192.168.0.1) | 0.080 s | 1 | `exit-code` (32) | **IDENTICAL to (ii)** — `...failed, reason given by server: No such file or directory` | +| iv | SMB wrong password | 0.175 s | 1 | `exit-code` (32) | `mount error(13): Permission denied`; dmesg: `Status code returned 0xc000006d STATUS_LOGON_FAILURE` | +| v | SMB wrong share name (valid creds) | 0.458 s | 1 | `exit-code` (32) | `mount error(2): No such file or directory`; dmesg: `CIFS: VFS: BAD_NETWORK_NAME: \\192.168.0.180\nope2` | + +Happy-path references measured the same day: NFS automount trigger <0.1 s (mount side), SMB plain +mount 0.085 s. + +**Taxonomy consequences for the production verify's Hungarian error map:** +- Cases (ii) and (iii) MUST be one merged category over NFSv4 ("a megosztás nem található, vagy a + szerver nem engedélyezi ennek a gépnek") — the pseudo-fs hides non-permitted exports; do not promise + the user a distinction the protocol cannot give. +- The unreachable case is cleanly classifiable only with `retry=0` (`No route to host` / connection + errors); with the default retry the verify sees a generic systemd `timeout` with no mount.nfs + diagnostic at all. +- SMB splits cleanly: `error(13)` = credentials, `error(2)` = share name, plus dmesg NT-status lines + if ever needed. +- All failures share `exit status 32` — classify on the journal string, not the exit code. + +**Q4-vi — the `retry=0` stance: put it in the PRODUCTION unit** (not only a verify-time transient +unit). Evidence — a guest process accessing the **idle automount path while the NAS is down**: +- without `retry=0`: blocks **91 s**, then `ls: cannot open directory ...: No such device` (ENODEV; + systemd's 90 s start timeout is what ends it); +- with `retry=0`: same clean ENODEV in **3.8 s**; an immediate second access re-attempts and fails + in the same ~3.8 s (autofs retries fresh per access — no stuck state). +`retry` only governs retrying a FAILED first attempt, so the happy path is untouched, and a briefly +down NAS costs an app ~4 s per access instead of a 90 s wedge. There is no scenario in this model +where the 2-min foreground retry helps: the automount re-triggers a fresh attempt on every access +anyway. Caveat, stated honestly: 3.1–3.8 s is the same-LAN ARP-failure case; a black-holed-but-routed +server (SYN drops) would sit in TCP connect until systemd's 90 s cap — `retry=0` narrows, but does +not eliminate, the worst case. + +**Q4-vii — verify timeout budget recommendation:** with `retry=0` in the unit, the measured cases +bound a LAN verify at **≤ ~5 s** (mount trigger + probe write), but the true worst case remains +**systemd's 90 s** (black-holed server). The agentapi client's global 15 s `http.Client` Timeout +(`client.go:87`) therefore **cannot carry the verify call**: a synchronous verify inside +`POST /api/storage/netstorage/add` would time out client-side exactly in the cases verify exists to +catch. Recommendation: the production task gives AddNetStorage its own longer-lived path — either a +dedicated client/context with a **~120 s** end-to-end budget, or (better, matching the existing +detached-format pattern) an async start + status-poll. Agent-side, bound the `systemctl start` step +at ~95 s (just above systemd's cap) so the agent always returns a classified result rather than its +own timeout. + +--- + +## Q5 — SMB without server-side force user: **PASS (hardlink included)** + +Server: Samba share `[spike2-plain]`, `read only = no`, `valid users = spike2smb` (a plain throwaway +user, uid 1051), **no `force user`/`force group`**, dir owned `spike2smb`. Client: the locked SMB +option string verbatim (`vers=3.0,credentials=…,uid=101000,gid=101000,forceuid,forcegid, +file_mode=0664,dir_mode=0775,_netdev`). Mounted in 0.085 s; probe as guest uid 1000: + +| Step | Result | +|------|--------| +| S1 read | PASS — marker listed `1000:1000` (client-forced), content read | +| S2 create + read-back | PASS | +| S3 mkdir + rename | PASS | +| S4 **hardlink** | **PASS** (`link-count=2`) — on THIS Samba/ext4; a NAS SMB stack may still refuse it, re-verify on the appliance | +| S5 delete | PASS | +| S6 stat | `1000:1000, mode 664` (uid/gid + file_mode client overrides, as designed) | +| S7 chown/chmod | both "succeed" (client accepts them; guest view stays 1000:1000/0664; server mode of a touched file ended `644`/`744` — i.e. cosmetic client-side, not faithfully persisted) | +| S8 server-side stat | `spike2smb:spike2smb (1051:1051)` — **writes land as the authenticated user, zero server-side uid config** | + +Consumer-SMB consequence: a stock NAS account with rw on the share is sufficient; ownership fidelity +on the NAS is "everything belongs to the connecting account" — same shape as Route A on NFS. + +--- + +## Teardown verification — DONE (all counts restored) + +- **180:** `/etc/exports` + `/etc/samba/smb.conf` restored from the `.bak-nasspike2` copies, baks + deleted. Verified: **2** non-comment export lines (pre-spike count 2), **6** smb.conf `[...]` + sections (pre-spike 6), `exportfs -v` shows **0** spike2 entries, users `spike2a`/`spike2smb` + removed (`id` → no such user), `/srv/nas-spike2` gone. Longhorn exports, PBS, k3s, Gitea, the six + Samba sections untouched. +- **felhom-pve:** all spike2 units stopped (automounts first, then mounts), unit files removed, + `daemon-reload` + `reset-failed`; `grep -c spike2 /proc/mounts` → **0**; `systemctl list-units + '*spike2*' --all` → empty; `/mnt/felhom-drives` holds only `felhom-usb`; both SMB creds files + removed; `/mnt/felhom-drives/spike2-*` mountpoint dirs removed. +- **Guest 9201:** `grep -c spike2 /proc/mounts` → **0**; `felhom-controller:0.112.0` Up (healthy); + the alpine image pulled during Q1b removed. +- SMB passwords were throwaway, never committed; the creds files lived 0600 on felhom-pve for the + spike's duration only. + +--- + +## Observations (out of scope — documented, NOT acted on) + +1. **Guest `/proc/mounts` duplicates propagated entries** (each spike mount appeared twice in the + guest) — an artifact of the mp8 bind topology; harmless, but a naive "grep -c" health check would + double-count. +2. The live NFS4 mount line shows **`fatal_neterrors=none`** — a newer kernel NFS option present on + the PVE host's kernel; not in the locked recipe string, purely informational. +3. `docker exec -u 1000:1000` into the production controller container works today (Q2a) — an + alternative probe transport via the docker socket if the Credential path ever grew a problem. +4. `useradd --system --uid 1050` warns (`uid greater than SYS_UID_MAX`) but works — irrelevant to + findings. +5. The Q4 error strings were produced by `mount.nfs4`/`mount.cifs` on PVE (util-linux/nfs-utils of + Proxmox 8) — string-matching in the production verify should tolerate minor version drift + (match on the stable substrings: `No such file or directory`, `No route to host`, + `error(13)`, `error(2)`, `timed out`). +6. Samba on 180 reloaded config via `smbcontrol all reload-config` without disturbing the six live + shares — useful for the future virtual-dsm pass too. + +No secrets in this doc. All spike credentials were throwaway and are now deleted.